F5 Certifications Mega Meta Series: The cool side of the exam development process

In this series of articles, I will go into the details of how the exams are developed (...as far as I know at least), and how I think you can improve your chances of passing them. In this article (part four of eight), I will share some insights in how they (F5) come up with the exam questions and how to use this to your advantage.

Over the years, I’ve been lucky enough to have been able to participate in the development of some of the F5 exams and get an insight in the workings of them. My top tip; if you ever get the chance to help with the exam development process – I would highly recommend it! One of the strong points of the F5 exams is that they are developed by a bunch of industry experts, rather than “some idiot with a book”. I’ve seen many exam questions myself over the years that are asking for such obscure knowledge that the only way to answer it correctly is to have read the exact book that the question comes from. What a waste… You have just proven that someone can read a book! (and to be honest, in real life you’ve got Google to tell you that stuff anyway) Having a bunch of people from around the world work together and come up with questions makes for a much more varied and scenario-based experience. Think about developers, consultants, support engineers and all those. They are speaking from their own experiences when writing questions, and if you ever find a question that thinly-veiled asks how “an F5 engineer has misconfigured something, what may have caused this” – you bet that that person was speaking from their own experience…

Once someone has come up with a good question, it gets shown to the rest of the group; who of you can answer this question? There are normally about 10-15 people in a room, the facilitator would expect that at least half to three-quarters of all people should be able to spot the right answer. If everybody can quickly guess the right answer, it’s clearly too easy, if nobody can spot the right answer, it’s clearly too hard. It’s a first test to check a question’s validity, but a pretty good one in my eyes! Not that this is the whole process – not by a long shot! (and I’m sure I will get beaten up for making it sound so simple…) After this, questions first go into a beta exam program, all kinds of funky data analysis will be done on it and any comments from real test-takers assessed, before it may actually make it into the real exam. So if a question makes it to the real exam, you can be pretty certain that it makes sense and they are not trying to catch you out!

Saying that, if you don’t agree with questions during an exam, feel free to leave comments. Every question has a comments field, so feel free to use it and they do read them. Unfortunately, your exam timer does not stop when writing comments, so be quick. I believe in Beta exams you get an extra ten minutes or so afterwards to fill in comments.

This brings me to the Practice Exam. This is a facility that is available for you when you sign up to the exam certification process. Something like 25 dollars for a go, and a nice facility to have available. Personally, I’m not a big fan of it, but it does have its place; if you want to get a feel for whether you are ready for the real thing, and want to get an idea of what the exam actually looks like, feel free to give it a go. As you will likely get the same set of questions (none of which are on the real exam), there is also no point in doing it more than once really. You just start to remember the questions, instead of the topics.

There are also some no-no’s when it comes to writing the question, the correct answer and the “distractors” – a cool term for the wrong answers. Such as:

• Questions are not meant to catch you out. If an answer looks like the right one, it probably is. If you have two or more answers that look pretty good to you (and you only need one), then you probably will have missed something, rather than the exam trying to play a trick on you.
• Someone who just walks in with no knowledge of the exam, should NOT be able to guess the right answer. For example; “From what direction does the wind blow?” A: North, B: South, C: West, D: It depends on the weather of that moment.” In other words, is the question idiot proof?
• The distractors can NOT be correct in certain cases. It needs to be clear from the question that only one answer can be correct.
• Don’t make the question overly complex and don’t use double negatives. The exam is to test your F5 knowledge, not to test if someone can read. In fact, with the exam being taken worldwide, the simpler the language used, the better. And remember, simple to read questions doesn’t mean a simple question to answer - “How many stars are there in the Universe?”
• F5 works with a measurement called a Minimally Qualified Candidate (MQC), which means a person who is JUST competent enough to not be fired. They should be able to pass the exam. The exam is not checking if you are the best in your field. As such, questions will not be checking if you know about this obscure knowledge that you only learned on page 275 of the book, or which you must have seen once while working in the dark pits of a datacentre during a blackout.

F5 has something called the Wheel of Validity (it almost sounds like a game show!) which they use during the exam development process, and always hold it against what they want to really test. Starting with the subject of the exam this will turn into the blueprint, an assessment of how often these topics are actually popping up in real life, and followed by the questions per category. Even the number of questions per category being dependent on the importance of a category. See it as a risk assessment. If you normally do a certain F5 task only once a year, and if it goes wrong, nobody cares, it’s clearly not THAT important to have it right. Whereas if you have a job to do every week that MUST be done right, otherwise the place burns down, you better know what you are doing! The latter subject will therefore result into more questions. Many moons ago, when I had a quick chat with them, I mentioned that I was surprised to get quite a number of questions in my 201 exam about raising a support ticket. As it turned out – at least back then, that it is something that many people need to do “regularly” and it often goes wrong! Apparently raising a priority 1 call in the middle of the night to sort out a minor lab issue is not the nice thing to do. …but I digress.

If you want to know more about what’s going in to the exam development process, there are a few really good videos / interviews with the brains behind the exam process on the DevCentral YouTube channel, such as the one embedded below – it shows you how much work goes into delivering a few lines of text on your screen while you are sitting there and sweating over them.