General – Why Are Certification Exams Not Higher Quality?

I was reading Ivan’s blog as I often do when I came across this post about why certifications suck.

The author Robert Graham had a sample question from the GIAC Penetration Tester (GPEN) exam. The question looked like this:

By default, which protocol do Linux systems use to transmit packets for tracing a network path?

a) UDP
b) TCP
c) ICMP
d) TTL
e) ECHO

Obviously being a networking expert I have my networking glasses on but I have to respectfully disagree with these gentlemen that I don’t think this is such a bad question at all. Trust me, I’ve seen much worse.

So traceroute works differently on different operating systems. If you work with penetration testing I would argue that you need to have a good understanding of different operating systems. You should know how they behave, what their characteristics are and how you can fingerprint them. The correct answer here is UDP. Linux systems and Cisco devices normally use UDP to send packets for a traceroute while Windows systems use ICMP when doing a traceroute. TCP is not the right answer since it’s not used by default but as my edit below shows it’s possible to use it. ICMP is the only really other viable answer so if you pick between UDP and ICMP you have a 50/50 shot. The TTL is a field in the IP header and not a protocol although Robert argues about the semantics what a protocol is. I don’t agree with his conclusion there. Echo is probably in there because people recognize echo since we use it when pinging but it should say echo request or echo reply in that case.

Edit: As Carlos pointed out in the comments it’s definitely possible to do traceroute with TCP packets. As long as we have an IP header the payload that we carry isn’t that important. By using TCP it is possible to do traceroute even for devices that filter ICMP. We don’t actually need to complete a three way handshake. We can just wait for SYN/ACK to come back and ignore it or wait for RST to come back. There are also other useful tools like MTR.

I don’t consider this knowledge to be trivia. In fact when I interview a person asking them about traceroute is a very good gauge of their overall networking knowledge. When asking that question I’m not looking for a binary answer. I’m checking to see if they know that traceroute can use both UDP and ICMP depending on the operating system. Are the familiar with the TTL? Do they know that the TTL is incremented for each hop? What kind of packet is sent back by the destination IP we are tracing? Do they know that ICMP packets could be filtered? Do they know that packets coming back may take another path than packets going there(asymmetry)? Asking a simple question like this can really open up to a much bigger discussion.

I would therefore argue that this question is not irrelevant and a person that gets this question correct is likely to have a pretty good general networking knowledge in my opinion.

I’ve taken my fair share of tests over the years and for example the CCIE RS written has been less than stellar lately. So what is the problem with certifications and why aren’t they better?

Scale

An exam like the CCNA is taken by likely thousands of people every day. While most of us agree that an interview process is more accurate and likely to stop cheaters this just doesn’t scale to the number of people taking the test. Not to mention that the vendor would need a massive force of proctors all around the world. At the scale of an exam like the CCNA there’s pretty much only one way of doing a test unfortunately and that is the format we use today.

Large pool of questions

Every exam gets dumped. It’s a fact of life. There are always people willing to take shortcuts. It would be much easier to have a pool of 200 really good quality questions for a test consisting of 60-70 questions. In reality the tests must have pools consisting of probably thousands of questions which dilutes the quality. It’s really difficult writing this amount of questions and still keeping the quality high.

Writing content is hard

Have you ever written a test for someone else to take? How was it? Was it easy? Now try writing 200 questions instead of 20. Not so easy any longer? Writing quality content is very very difficult, especially at scale. You want to ask a question that is relevant and then offer some options. The difficult part is writing the options. One answer should be correct obviously but the other ones should be feasible so that it isn’t obvious which one is the correct one without having the actual knowledge. It’s often easy to come up with a 2nd option which is almost correct but coming up with 1 or 2 more is a lot more difficult.

Bias

We all have a bias. The test takers have a bias. The content development team have a bias. By this I mean that we all work on different technologies in our jobs. Depending on what we work with this becomes our reality. Anyone else having another reality is seen as having the wrong reality. “Why are they asking me on technology X?!” “Noone uses technology X!”. Trust me. I had the same feeling when I took the CCIE lab but ended up working with some of those technologies later. We have to accept that the test takers come from a large geographical area where everyone has a different reality.

Refresh cycles

Most exams are refreshed every 2-3 years. This was fine earlier when technology was not moving at such a rapid pace. A technology from 3 years ago may be almost obsolete today. Which is why people react when they see what they consider to be old or outdated technologies. I know that this is being acknowledged within vendors and we will start to see more rapid refresh cycles soon where minor updates are coming out every few months instead of a major refresh every 2-3 years.

We can make them better

A lot of people openly complain about exams but how many of us are willing to get involved in making the exam better? There are often subject matter expert (SME) programs where people that are certified can get involved in creating exam content. There are also advisory boards where you can help form the future of the certification. What technologies should be added? What technologies should be removed? When people have complained about an exam in the past I have asked why they don’t join a SME program. The response has usually been something like “I don’t work for free for profit organizations”. While I can fully understand why someone would feel like this if we all took that stance then exams would never get better.

Certification exams aren’t great but writing a good test is very difficult. It takes a lot of time and effort. Always give feedback about a test and if you are really serious about making a test better, join a SME program. Good luck in your studies!