Writing better MCQs and fight against the ‘test takers’:
We have always come across students (and peers) who have amazing test scores and we wonder what is unique about them. We also come across questions in tests we take where we make a ‘guess’ but why we pick one option over the other when we really don’t know the answer. Why we ‘feel’ one options seems to be truer than others. There is actually reason for that and Test Wise-ness will be defined as that ability to pick up on patterns or errors in design of a question to find out the right option (answer) without actually knowing anything about the question.
This is a summary of basic problems with multiple choice questions writing and why certain people can score higher on tests as they have an ability to pick up the right option choices from the limited choices of 4 or 5 in multiple choice questions. As they can pick up the patterns that suggest the right answer even when not familiar with the information.
This test wise-ness is a problem because the whole idea of testing is to understand where the strengths and weaknesses are for a particular participant. That might be a formative assessment where the information will then be used to design the lectures better in future or use to give feedback to the test taker. Or it could even be a summative assessment where we’re trying to understand if they have worked hard enough to pick up the relevant information and its application and are able to use it in problem solving for given clinical scenarios and is currently the backbone of summative assessment for most of the areas in medical field and thus needs to be comprehensive and reliable enough that it can provide a true picture of the participants abilities.
Common Pitfalls in MCQs and how to pick up the right options:
Following are the 8 main issues or errors that are picked up by wise test takers who can then make a very good intelligent guess of what might be the right option even when they’re not familiar with the content or information and actually thus do not know the answer.
1. Word repetition: also called clanging and clang clue. The test takers could see repetition of the words used in the stem of the question that matches with one of the choice thus making it more likely to be the right answer or the relevant answer.
2. Grammatical clues: Sometimes the multiple choice questions are written in a way that when read, in relation to or context of the stem of the multiple choice questions that only one of the option is grammatically correct or is appropriate wording for the answer. Other options just do not make any grammatical or wording sense to go with the question.
3. Syntax: The answers or options may be selected in a way that they are just not appropriate for the syntax of these stem of multiple choice question except for the one right answer.
4. Use of absolute words: The use of absolute words in a multiple choice question answers typically means they were the wrong choices because things in world things in life are not absolute especially in medical sciences. So giving options with words like all or none or never is typically not a correct answer. Options which don’t have these absolute words are likely to be the more correct options.
5. Qualifying the right choice: This is a problem that is seen where the right choice is given with such details so as to be absolutely correct choice and is overqualified (additional information added into the option). Other options which are incorrect do not have that kind of detail and more detailed information is given the correct option or choice within the multiple choice question answers and thus can be picked up by wise test taker.
6. Convergence bias or partially correct answers: One of the problems with test writers is when they are writing different options they usually start with the right option and then they have to make up the wrong options that sound believable or sound close to the right answer so what they do is that they take different parts of the right option and add partially wrong information. So what happens is that all different options have partially correct answers and one of them has the fully correct answer. That option has the words that are overlapping with all the other options and is the most overlapping or most common information from all options and can be picked up.
7. Convergence of numerical answer: Another problem seen is that the options given in mathematical and numerical choices. Sometimes the test writers write false options which are numerically above and below the right answers if the right answer is 260 then the option may be given 26 and 2600 which is above and below 260 and usually the middle answer is the right answer as one option is given above and below.
8. Question memory: This is a very common problem especially when the questions are from a small block or data bank where the related questions are written and the answer or detail or stem of one question give information which is relevant to another question few questions down and thus if the test taker has a good memory then he would likely do recall that question has previously been asked or explained in a stem of another question and be able to quickly answer that question.
Writing better MCQs and fighting the test takers:
So what to do to avoid pitfalls or problems in writing a multiple choice question to avoid test wise-ness or answering to the test without knowing the information.
1. Avoid absolute terms: Do not use all none or never in your option choices as life is never absolute especially in medical sciences.
2. Write in positive tone: Do not use “which of the following options are true except” or “which of the following is not true”. These are not appropriate questions.
3. Writes stem concisely: Although always longer than the options themselves but still not too long and not excessive information which may be repeated later on in another question. And avoid putting words in your options so try not to have a grammatical flavor to your options so that grammatical incorrectness can help pick the right answer.
4. Avoid teaching points and questions: Again these can be relevant to the next questions later on and may be easily remembered and recorded later on.
5. Avoid week frequency words such as seldom or frequently in options as it becomes hard for a person to pick the right option.
6. Option should not overlap so this should not be convergence or partially wrong answers which carries the words on the right answer.
7. There is another problem of stem writing that should be avoided which is that the questions is answerable by reading the last line of the question or looking at the option without reading the whole stem. When writing using ‘cover the option rule’ then the stem itself should be able to give the answer without looking at the options and options are just there to be picked one up for convenience because we don’t want to grade a written short answer by hand. The stem are not there as teaching points or context they have to have information necessary to enter the options below.
A word on level of MCQs:
Level 1 multiple choice questions are designed to test the information or knowledge of the person taking those questions.
Level 2 of MCQs will be application of knowledge where a problem is given that can be solved only after knowing certain facts. Those are even harder questions to write where you have to synthesize it in a way that a recall of knowledge is being used to apply it in a practical sense.
Level 3 is the highest level of MCQs with problem solving where the straightforward application is not being asked but applications in various contexts are being asked to solve the problem. For example, the problem solving questions where a scenario is described and then the question is asked what will be the next best step in management. This is superior to asking the most likely diagnosis which is an application question and although that is better than a knowledge of recall question which is what is the most common type of infection seen in this situations.
