Sometimes the Correct Answer is the Best Answer

Sometimes the Correct Answer is the Best Answer

Liberty Munson (Microsoft)

Within my organization (certification development), our focus this year is innovation in item types. Although we are exploring new item types (more on that in a future blog), we are also exploring how we can leverage our current item types in innovative ways. Take the old standby--the multiple choice (MC) item. This item type has been unfairly criticized--after all, it's used on the vast majority of exams for a reason, but I understand the perceptions that are driving this criticism. Complaints include: it's too easy to guess the correct answer; MC questions are not rigorous measures of skills; and they don't reflect real world experiences. So, the question is "how can we make MC questions more difficult to guess, increase their rigor, and make them a better reflection of real world experience?" One solution is to change the focus from selecting the "correct" answer to selecting the "best" answer. What's the difference? In a "correct" answer item, only one answer is correct; in a "best" answer item, more than one answer might solve the problem but there is one best solution.

Because we believe that truly qualified candidates, especially at the professional level, should know what the best solution is in situations where multiple good solutions exist or are possible, we have decided to incorporte "best answer" items into some of our exams. After all, how often do you encounter a situation where there is one and only one correct solution to the problem? Problems rarely have clear cut solutions. As a result, these items should be more real world than a correct answer question because they require the candidate to carefully consider the situation presented in the question, just as they would carefully consider their organizational environment and constraints when making decisions in their jobs, and determine the best solution given those circumstances. These questions require more critical thinking than correct answer items, which makes them more rigorous measures of skills and more difficult to guess.

So, let's cut to this chase. What does this mean to you?

We will be slowly adding best answer items to our exams over time, and I want to let you know what to expect:

  • Best answer items will only appear on PRO level exams where being able to identify the best solution given a set of specific environmental characteristics or constraints is critical to planning, designing, and managing Microsoft technologies in organizations.
  • The best answer will be the solution that takes the least effort, requires the fewest steps, and/or is least costly AND solves the problem. Remember with a best answer choice item, more than one answer will likely solve the problem (others might not), but one answer will be the best solution from a resource (budget, time, people) perspective in that particular situation.
  • The best answer will be the best answer GIVEN THE SITUATION DESCRIBED IN THE QUESTION. It may not be how you would do it in your organization, but it's how it should be solved given the situation.
  • At this time, only a small proportion of items on any exam will be best answer choice items; although this proportion is likely to increase over time, we are not to the point where every question on a particular exam will be a "best answer" question. Not all PRO exams have best answer questions yet, but we're working on it.
  • You will know these items when you see them. The question statement will include words such as "best" or "most" (depending what is being asked) that make it clear that the question requires a deeper evaluation to determine the best answer. In addition, the following statement appears following the question: "(More than one answer choice may achieve the goal. Select the BEST answer.)"
  • Most important, the best answer has been vetted by many SMEs who agreed on what the best answer is. The best answer isn't what Microsoft thinks is the best answer; the best answer is what many SMEs have agreed is the best answer.

I have to admit best answer questions are not innovative in and of themselves; you can find them on most certification and licensure exams, but they are new to Microsoft Learning. I know some of you won't like this change, but I believe that best answer questions have a place in our certification program and will ultimately increase the value of your certifications because of the higher level thinking required to answer these items correctly.

Comments
  • rellufgerg
    |

    sometimes the correct answer is the answer with lowest ammout of wrong steps and ideas ... choose the best even if all are wrong

  • Pete Jones
    |

    Sorry, but won't change much except for the legitimate test takers.

    Multiple choice questions, even with "best" answers will have a right answer: one that earns the most points. And braindumps will show this one as the right answer. So all that is needed is to memorise the question and answer pair, without understanding the underlying reasons.

    Legit test takers will simply have more to memorise. So this does little except make it less likely for regular candidates to pass, and possibly even take the exam.

    I can see what you are trying to do, and I would say that having the best answers worth say 10 points, with other correct answers on a sliding scale (for example, a 10, two 7's, a 5 and a 0 as possible answers) does mitigate the feeling of "trick" questions somewhat. But if the idea is just to make questions harder, you are doing it to the wrong people.

  • rellufgerg
    |

    I thought for sure that I had seen these types of questions on the exams in the MCSE 2003 track, along with many other question types other than multiple choice.  In fact, I never saw a straight multiple choice exam from Microsoft until I went to upgrade my MCSE to MCITP:Enterprise Administrator.  Most of the people that I talked to around that time thought that the new generation of exams were far easier than the older exams specifically because of this, so I am glad to see this changing.

  • rellufgerg
    |

    @Pete:

    Unfortunately the "BEST" way to put a hurt on the braindumpers is to have lab-based exams.  We've already seen that can be a difficult proposition...

  • rellufgerg
    |

    I've never liked "Best Answer" type questions - primarily because there is occasionally conflicting advice in documents posted on TechNet and MSDN. It would make sense to go with this if TechNet and product team advice was always consistent, but it is not. Go to any of the teams responsible for TechNet documentation and ask them if they suspect that their documentation contains conflicting advice.

    Two separate answers that achieve the same goal where both are right is fine, but a "best answer" based on TechNet documents where those documents and solution accelerators offer different advice? MSL is relying upon a fantasy about the coherence and innerrancy of TechNet that anyone who uses it on a regular basis knows is impossible. You should only start playing these games once the documentation is consistent and that, simply, because of the evolving nature of the documentation, will. never. happen.

  • Pete Jones
    |

    @OtherKevin I know, but what I don't know is why. When it is possible for them to do blogs.technet.com/.../hands-on-labs-at-teched-virtualization-at-work.aspx things like this, why can't they do virtualised exams? Where is the failure occuring?

    It would be interesting to see from the technical view, as well as reassuring from a candidate view that they are working on it.

    1.5 days to create and destroy 3,750 VMs doesn't sound so dissimilar from what might occur when hosting virtualised exams.

  • Umapathy
    |

    I do think simulations are best to keep away from crammers which devalue the certification which also make who passed the exam in a legitimate way in trouble.

    Regarding the best answer it's quite difficult to guess what is the best in real life. What is best for me may not be best for Microsoft. For example may be for real life virus problem from my experience booting the system via live Linux (knoppix or ubuntu) and copy the files then do a clean installation probably save time and and also system will come to initial speed. While probably I guess Microsoft will suggest to start in Last known good configuration if not safe mode if not to repair Widows and so on.  I also been watching Microsoft sysinternals tools they are really good tools to trouble shoot. Almost 2-3 months back I had a problem in the office which I managed to sort it out by sysinternals where I found the particular application is writing a file in a particular place, that seems to be problem behind it. I am in an opinion at least Microsoft should promote it's own best tools like sysinternals at least in the client examinations. In real life  atleast for me life mostly we have to do what management asked to do so that I can survive, anyway that's a bitter truth. From my experience their best solution for Management is the shortest downtime. I mean which ever the way makes their solutions faster, that is the thing they prefer rather than going for a perfect solution that takes time.  

    Another problem other than making innovative is that not all exams are available in the whole world. I am from Sri Lanka and I can't find any Microsoft Exam centers for Office product where Certiport is responsible for conducting the exams. I think first what Microsoft should do is that to make sure that people can take the exam irrespective of the developing nations or from the developed nations. It's a shame that I can take open office exams but not the Microsoft Office exams in Sri Lanka.

  • Chris Rhodes (Regional Lead - UK)

    I for one am looking forward to seeing how this works out.

    It should make the exam taker think more about an answer, and therefore their poroduct knowldege should be better tested.

    It could help differentiate the candidates more than the existing format by itself.

  • Chris Barker (Regional Lead -- New Zealand)

    While I agree with the sentiment and applaude it (along with the idea of continually improving quality in exams) I am concerned about the objectivity of "best". My idea of best and MSL's idea of best can be *very* different--subjective. This will ony work if there is absolutely no ambiguity, cultural bias and a totally objective concept of "best".

  • Umapathy
    |

    I am working for GIS (Geographic Information system) and I remembered some one suggested to put  Mankulam as a center for Northern Sri Lanka while it sounds good as in a computer as it is almost the middle of the place of the northern province. This result came from a GIS analysis but the reality is that in that place hard to find good drinking water. However the drinking water data hardly available in GIS so one cannot analysis.

    Let me come back to the best answer issue I also think that should be more practical one. I may be wrong. I am not against the innovative idea of selecting the best answer but care should be taken and Microsoft should provide at least some sort of example questions with answers preferably on windows 7 environment to say why the answer is the best one and why others are not so best based on what criteria. Otherwise the exam takers will be in nightmare.

  • rellufgerg
    |

    @Pete:  Spinning up 3750 VMs for labs at TechNet is easier than effectively spinning up virtual labs for exam takers.  For starters, the team prepping for TechNet controlled the entire network being used to access the VMs.  With the virtual lab exams they did not.  Before rolling out lab-based exams they have to make sure that the exam providers can accommodate the connectivity requirements of the remote labs.  Systems need to be configured to support the remote technology, appropriate bandwidth has to be available and of sufficiently low latency, etc.  I've done some virtual lab-based exams before (not just for Microsoft), and I have yet to see one where connecting to an off-site datacenter to access the lab DIDN'T cause issues with latency or connectivity.

  • Mike Corkery
    |

    In theory I like this idea of best answer questions. My concerns are similar to what Chris Barker mentioned. The definition of "best" can be very subjective. It will be important to ensure that the correct answer represents the best real world answer given the question and scenario given. To determine the best answer it might be necessary to give more detailed scenarios. It will be important to know how different cultures would act given the same question.  Would someone in another country take the same actions given a scenario?

  • rellufgerg
    |

    I agree with the concerns raised about the "Best Answers" style questions. Unless you were able to score partial credit for a right answer. It  is just to tricky and to easy for a candidate to make implicit assumptions about context given their  experience. even with the case studies in the PRO exams it seems dubious to have to pick out the one clue that makes an answer better than the other. I would be in favor of adding some small amount of hand graded open ended short answer type questions to allow the candidate to respond to a given case scenario. This probably isn't feasible; but would raise the bar a little for the the pro exams

  • rellufgerg
    |

    Speaking of "best answer", something similar came to mind.  "Least administrative effort".  I've seen this on questions often.  In the real world, yes that's important for the support person, but I think "least impact to the user" is more important,  and then "least admin effort" comes in second.  Just a thought!

  • Umapathy
    |

    I agree with twist. I also think it's better do what is best for the customer than what is easy for us.

    One of the problem with giving a feed back at the end of the exam is that, we are not in a position to see what Microsoft thinks as the right answer. We never know what is Microsoft's answer, until we see the marks sheet by that time it's too late as we left the feedback session. May be it's better to give the score card then allow to make the comment that way we know (well it may not always practical but chances are there) what went wrong. Is there any disputes we should be able to challenge it. In my humble opinion, in that way not only the exam takers but also Microsoft will benefit from it. Currently for the Microsoft exam we accept what ever the decision made Microsoft as the correct answer. Over the years hardware has been changed. I hardly come across a questions about Multicore processor. Which may a question asked often in the office as well as from the colleagues in order to a make a purchasing decision.  What makes Microsoft to think licenses differently if there are two physical processors are in the mother board or dualcore chip in a mother board? Although I started hating java I have no other option to run it as it is required for my office. There are applications that do not report to Microsoft Management Console. If they do then I know how to deal with it. real life is not so easy.

    Just to reemphasize what I told before, it may be good to time as a factor on making decision. May be a question can be made like which will give fastest solution so that the customers can be  get the solution quickly if required. Also make the question pool as big as  possible. Make a statistics of the questions and answers. May be it's better idea to take out the questions that 99% (or 95% or what ever you decide) of them answers correctly as this may be an easy question or may be a leaked one, same way need to look at the questions that or 5% (1% or what ever you decide) answered correctly that may be giving a wrong perception (may be for the people like me where English is not a native language) or may be you guys by mistakenly put wrong one as the correct one.

View All