F5 Certifications Mega Meta Series: The cool side of the exam development process

In this series of articles, I will go into the details of how the exams are developed (...as far as I know at least), and how I think you can improve your chances of passing them. In this article (part four of eight), I will share some insights in how they (F5) come up with the exam questions and how to use this to your advantage.

Guest Author: Alex Tijhuis

An evangelist for anything software designed and security, and a self-described massive network geek, Alex is an F5 trainer and consultant at ABCT.net. While certified and highly skilled and interested in all things F5, he's just as happy pulling cables in a data center and designing scalable systems as he is messing around with the latest cool kids toys our fine industry has to offer.

Over the years, I’ve been lucky enough to have been able to participate in the development of some of the F5 exams and get an insight in the workings of them. My top tip; if you ever get the chance to help with the exam development process – I would highly recommend it! One of the strong points of the F5 exams is that they are developed by a bunch of industry experts, rather than “some idiot with a book”. I’ve seen many exam questions myself over the years that are asking for such obscure knowledge that the only way to answer it correctly is to have read the exact book that the question comes from. What a waste… You have just proven that someone can read a book! (and to be honest, in real life you’ve got Google to tell you that stuff anyway) Having a bunch of people from around the world work together and come up with questions makes for a much more varied and scenario-based experience. Think about developers, consultants, support engineers and all those. They are speaking from their own experiences when writing questions, and if you ever find a question that thinly-veiled asks how “an F5 engineer has misconfigured something, what may have caused this” – you bet that that person was speaking from their own experience…

Once someone has come up with a good question, it gets shown to the rest of the group; who of you can answer this question? There are normally about 10-15 people in a room, the facilitator would expect that at least half to three-quarters of all people should be able to spot the right answer. If everybody can quickly guess the right answer, it’s clearly too easy, if nobody can spot the right answer, it’s clearly too hard. It’s a first test to check a question’s validity, but a pretty good one in my eyes! Not that this is the whole process – not by a long shot! (and I’m sure I will get beaten up for making it sound so simple…) After this, questions first go into a beta exam program, all kinds of funky data analysis will be done on it and any comments from real test-takers assessed, before it may actually make it into the real exam. So if a question makes it to the real exam, you can be pretty certain that it makes sense and they are not trying to catch you out!

Saying that, if you don’t agree with questions during an exam, feel free to leave comments. Every question has a comments field, so feel free to use it and they do read them. Unfortunately, your exam timer does not stop when writing comments, so be quick. I believe in Beta exams you get an extra ten minutes or so afterwards to fill in comments.

This brings me to the Practice Exam. This is a facility that is available for you when you sign up to the exam certification process. Something like 25 dollars for a go, and a nice facility to have available. Personally, I’m not a big fan of it, but it does have its place; if you want to get a feel for whether you are ready for the real thing, and want to get an idea of what the exam actually looks like, feel free to give it a go. As you will likely get the same set of questions (none of which are on the real exam), there is also no point in doing it more than once really. You just start to remember the questions, instead of the topics.

There are also some no-no’s when it comes to writing the question, the correct answer and the “distractors” – a cool term for the wrong answers. Such as:

  • Questions are not meant to catch you out. If an answer looks like the right one, it probably is. If you have two or more answers that look pretty good to you (and you only need one), then you probably will have missed something, rather than the exam trying to play a trick on you.
  • Someone who just walks in with no knowledge of the exam, should NOT be able to guess the right answer. For example; “From what direction does the wind blow?” A: North, B: South, C: West, D: It depends on the weather of that moment.” In other words, is the question idiot proof?
  • The distractors can NOT be correct in certain cases. It needs to be clear from the question that only one answer can be correct.
  • Don’t make the question overly complex and don’t use double negatives. The exam is to test your F5 knowledge, not to test if someone can read. In fact, with the exam being taken worldwide, the simpler the language used, the better. And remember, simple to read questions doesn’t mean a simple question to answer - “How many stars are there in the Universe?”
  • F5 works with a measurement called a Minimally Qualified Candidate (MQC), which means a person who is JUST competent enough to not be fired. They should be able to pass the exam. The exam is not checking if you are the best in your field. As such, questions will not be checking if you know about this obscure knowledge that you only learned on page 275 of the book, or which you must have seen once while working in the dark pits of a datacentre during a blackout.

F5 has something called the Wheel of Validity (it almost sounds like a game show!) which they use during the exam development process, and always hold it against what they want to really test. Starting with the subject of the exam this will turn into the blueprint, an assessment of how often these topics are actually popping up in real life, and followed by the questions per category. Even the number of questions per category being dependent on the importance of a category. See it as a risk assessment. If you normally do a certain F5 task only once a year, and if it goes wrong, nobody cares, it’s clearly not THAT important to have it right. Whereas if you have a job to do every week that MUST be done right, otherwise the place burns down, you better know what you are doing! The latter subject will therefore result into more questions. Many moons ago, when I had a quick chat with them, I mentioned that I was surprised to get quite a number of questions in my 201 exam about raising a support ticket. As it turned out – at least back then, that it is something that many people need to do “regularly” and it often goes wrong! Apparently raising a priority 1 call in the middle of the night to sort out a minor lab issue is not the nice thing to do. …but I digress.

If you want to know more about what’s going in to the exam development process, there are a few really good videos / interviews with the brains behind the exam process on the DevCentral YouTube channel, such as the one embedded below – it shows you how much work goes into delivering a few lines of text on your screen while you are sitting there and sweating over them.



Dr. Ken Says...

There’s a lot in this post; and, while, yes from my perspective it is extremely simplified, it does capture the gist of how we put every effort into making the exams do one thing: determine if someone is minimally qualified or not. That is the only goal.

I did want to clarify two things: the minimally qualified candidate (MQC), and the value of practice exams.

MQC: This is not an F5 concept, but a standard concept in psychometric assessment development; and, while it sounds a bit incongruent or backwards, it is extremely important. Defining the MQC means detailing, exactly, how much knowledge we require before we are confident to allow the individual to do something. It is drawing a line in the sand saying, “if you know, at least this much, then we will trust you to do the job.” There are thousands, or millions, of minimally qualified airline pilots, brain surgeons, teachers, etc. practicing their professions today. It is really about simply defining the line between feeling confident you can do the job or not. This is also why our exams are pass/fail: you either make the mark or you don’t. We don’t actually care if you know more than the MQC, we just need to know whether you know enough to be one.

Practice Exams: I know practice exams are somewhat of a contentious topic, but I believe that is because most people only know them in the way they have been used for many years: a list of questions that people study and review as a “learning” tool. Our practice exams are not designed for that. Our practice exams are built using the same process as our production (i.e. “real”) exams; in fact, we could easily use the practice exam for the real exam as it is equated (i.e., “identical” to the real exams in terms of number of questions, construction, and difficulty) to the real exams. The only difference is that it does not have the same questions. The value of this is that candidates taking the practice exam can get the same information (passing/failing, which sections they need to work on, same time constraints, understanding the style of questions, etc.) all without having the cost of taking the real exam and without it affecting your retake status. This is all extremely valuable, and I’ve known many a candidate who has just taken the real exam (from many vendors) just to get this information. The one caveat, as pointed out in the article, is the value of the practice exam diminishes each time you take it as there is only one set of practice exam questions. The more times you take the practice exam, the less the results will reflect your real exam performance. For that reason, we suggest using the practice exam only two (at most three) times; after that, the results will simply tell you how well you know the questions, rather than the material underneath.

Published Jun 17, 2021
Version 1.0

Was this article helpful?

No CommentsBe the first to comment