A few days ago we invited y’all to submit questions about certification development. Steve Maier asked “How much do you look at the comments that people put into the exam?” This is a great question and one that gives me the opportunity to tell you lots of good stuff. First I have to explain the difference between beta exam comments and live exam comments.
Most exams that we create go through a phase that we call beta. The beta exam is available for a limited time on an invitation-only basis, which allows us to collect data about each item and determine which items distinguish a qualified candidate from an unqualified candidate. After the beta phase is complete, we review the data about each item and decide which items we are going to keep for the live exam (more details on this process will be provided in a future blog). Scores for the beta candidates are then determined based on the items that we decide to keep.
Just like during a live exam, the candidates who take the beta exam can submit comments about each item. We read every comment submitted during the beta phase of exam development. These comments are one of the data points that we consider when we are deciding whether to keep or kill an item during this phase. If candidate comments indicate a technical flaw, multiple correct answers, no correct answer, or lack of clarity in an item, we will review that item and its associated comments with subject matter experts. These beta comments are SUPER IMPORTANT to help us understand what’s going on with an item, and we depend on you submitting them. You should also be aware that we have no way of knowing when you mark an item for review but don’t have time to provide feedback on it. Because we have to limit the time allotted for comments for security reasons, be sure to comment on the items that you believe have the most critical issues before you comment on items that might have minor issues.
[Tangential side note: Folks who are invited to take the beta version get to take the exam for free. Yes, that’s right, I said free. Click here to find out more about how to participate.]
Unfortunately, we cannot review all the comments that you submit on live exams. There are simply too many. We do, however, read a sampling of candidate comments from each exam as part of our annual review process.
At this point I’ll bet you’re saying, “How am I supposed to submit feedback about a live exam if you don’t read my comments?” That’s easy. If you have feedback about an exam, send an e-mail to email@example.com. If you are concerned that a specific item is technically inaccurate, you can submit an Item Challenge, and we’ll look into it.
Well, that about covers it. Funny how such a short question can lead to such a long answer!
If you have other questions about exam development, add them here as comments, and we’ll work the answers into our posts over the next few months.
Posted by Krista
Friday, February 20, 2009 1:46 PM by Edward Laverick
The link for the item challenge form is wrong, it looks like its got your webmail URL tagged to the start of it.
Friday, February 20, 2009 3:40 PM by Krista
Thanks for pointing this out, I fixed it. It was only a matter of time before the world figured out that I am hopelessly link-challenged.
Friday, February 20, 2009 7:08 PM by rhagman
The "click here" link to the beta exam information has the same issue
Saturday, February 21, 2009 9:28 AM by Bart
I ve been taking Beta exams since 1996. Last summer I took the great 70-113 but was kicked out during the comment period of one of the tests. I m pretty sure my comment was not saved and regret it thoroughly. Can work with Prometric to enhance their (beta) test engine to ensure that (beta) comments are saved, even when you re kicked out because comment-time times out? Maybe even add a sentence to the dialog box that says that comment-time is over, like "your current comment has been saved". Would make me feel much better. Thanks, Bart
Saturday, February 21, 2009 12:54 PM by Christopher Kusek
I take great pride in providing comments for the beta exams, though it saddens me that even after the comments were taken into account (which I know they re read in the Beta s) that very simple things like questions being completely wrong, are not corrected - not to mention a series of grammatical mistakes which violate the integrity of the question, or a series of inconsistencies within the questions.
I m not saying that is the case with every and all exams, but in general when the comments are read, and perhaps just ignored (when you know they re read) it doesn t give me a whole lot of faith that the beta period was anything more than simple a trialing of the content which either will or will not be updated to reflect the reality of how it should be delivered.
Monday, February 23, 2009 2:34 AM by Kjetil
I miss a "tag question for comment" during the exam. When I m finished with the exam I don t want to browse through all questions to locate them.
Love the beta tests! :-)
Monday, February 23, 2009 12:09 PM by libertymunson
I m back. I have become a blog junkie...
Let me take a minute to provide a little more detail about how comments are used to evaluate items after beta, which should address some of the concerns raised in Christopher s comment. First, please, please, please, comment if you find typos, misspellings, or grammatical errors. Although we go through a rigorous item development, review, and edit process, it is very easy to overlook mistakes like this in large item pools given the schedules that we are working toward. We will fix them (with confirmation from SMEs or our planners) before we publish the exam, but we can t correct them if we don t know about them.
Second, it is not uncommon for one candidate to comment that a question is “bad” while another comments that it s "great." In these cases, it’s difficult for us to know which candidate is “right.” Because we don’t have the time to review every comment that we receive (we literally receive hundreds of comments on a typical beta) with SMEs or our planners, we have to make a judgment on which comments will be reviewed further. To do this, we consider whether the person making the comment answered the question correctly, how that candidate performed on the beta, how many candidates provided similar comments, how the item is performing psychometrically, and how critical the comment is in light of the other comments that we receive. As a result, some items may fall through the cracks of our review process.
So, as a tip when you’re providing beta comments, the best way to catch our attention is to tell us when the item is technically inaccurate, if there is no correct answer, if there are multiple correct answers, or if it’s unclear. Explaining ‘why’ is helpful when you have time to do so, but seriously, the magic words for us to review an item more closely are: this question is technically inaccurate; this question doesn’t have a correct answer; or this question has multiple correct answers.
By the way, the purpose of the beta exam is to “test” the content in an environment that’s similar to the live exam. We are trying to determine what works and what doesn’t and to identify errors that we may have missed and opportunities to correct the content before publication. This is why beta candidates don’t receive a score immediately after the exam. We only want to score people on the content that “works.”
I hope that clarifies the process a bit more.
Tuesday, February 24, 2009 5:38 PM by Greg Low
The biggest problem I see with this process is that insufficient time is provided for comments.
In one recent exam I did, I had a four hour timeslot allocated. I did the exam in 40 minutes, found probably 50 questions that needed comment and was happy to spend the rest of my allocated time commenting on them.
After a few minutes, the system said "you have one minute left in the comment period".
What exactly is the point of that? I was up to about the fourth question that I wanted to comment on. Most of the questions with real issues were later in the deck and I didn t even get close to commenting on them.
Tuesday, February 24, 2009 5:44 PM by Greg Low
The second problem with this process is how SMEs are located.
I m regularly seeing situations where the process of locating the SMEs is outsourced. The requests that come from the outsourcers have so little pre-planning that they effectively say "can you be in Seattle next Monday?".
If the SME pool is limited to those who don t know what they re doing for work next week, there is a significant problem with the process and it s hard to imagine you getting the outcome you re after.
Tuesday, February 24, 2009 6:12 PM by libertymunson
I get this type of...let s just call it "feedback"...a lot about the amount of time allotted for comments. Unfortunately, for exam security reasons, we have to limit the amount of time allowed for comments to something that would give candidates a reasonable amount of time to comment while not giving those that would use this time for nefarious reasons additional opportunities to take advantage of our process. It s a true balancing act between getting the feedback that we need and the security of the exam (so that the certification remains valued in the industry).
For beta exams, we double the amount of time allowed for comments to 30 minutes because this feedback is so important to our development process, and to be perfectly honest, I strongly believe that this is the most that we can give candidates to maintain the security of the exams. By the way, my team has an Anti-Piracy Program Manager (who you’ll meet later), and I bet she wishes we would reduce it even more.
My recommendation is that you try to prioritize the feedback that you provide and start with the most important problems/concerns/issues that you have. I realize that this is difficult, but it is the best solution that I can provide given that possibility of providing more time for comments is remote at best.
To your question about SME recruitment, we have this planned for a future blog, so I’ll forward your comment to the person writing that blog so they can address your issue.
Thanks for the great feedback on our process!
Friday, February 27, 2009 6:20 AM by Greg Low
Are you sure that 30 minutes is allowed? I can t think I had more than about 8 or 9 minutes before it said "one minute left". There s no chance I had 30 minutes or I wouldn t have got to only question 4 or 5.
Is that supposed to apply to all beta exams?
Friday, February 27, 2009 4:27 PM by libertymunson
For most of our beta exams, you will get 30 minutes for comments. I can only think of one exception in the 2+ years that I ve worked on this program. In that situation, we could only give candidates 20 minutes for beta comments because the number and type of items on each instance of the beta exam was sufficiently large that we couldn t provide 30 minutes for comments and still remain in our 4 hour seat time window. This is an extremely rare event though and something we try to avoid at all costs. (In fact, I received enough... again, let s just call it "feedback"… about the decision to reduce the comment period to 20 minutes that it s unlikely to happen again on my watch).
If you don’t think you are being given the full 30 minutes for comments on a beta exam, contact your Regional Service Center (http://www.microsoft.com/learning/support/worldsites.mspx). We will investigate the issue and ensure that Prometric is delivering the exam according to our exam specifications.
By the way, we allow a maximum of 15 minutes for comments on live exams. Although most live exams allow for a 15 minute comment period, some do not. None of our exams can exceed a 4 hour seat time which includes launching the exam, reading through all the instructions, signing the NDA, taking the exam, and receiving the score report upon completion. Really, we only have about 3.5 hours for the actual exam (note that amount of time you actually receive to complete the exam is based on the time estimates we receive from beta participants—our goal is to ensure that 95% of candidates can complete the exam in the time allotted). Because some items take candidates longer to complete, requiring a longer exam time, I sometimes have to “steal” time form the comment period on live exams to ensure a valid and reliable exam is administered. This doesn’t happen often, but it does happen on a few live exams.
Tuesday, March 10, 2009 3:11 PM by Coleen
I m taking a psych-assessments course this semester, so thinking a lot about psychometrics--which has put me in mind of my Microsoft testing experiences. So, it s interesting to see this come up.
I have commented a few times on exam questions, but found one thing that kept me from doing so: as soon as I was a click or two away from finding out whether I d passed, the suspense was too overpowering--I had to see how I d done, and to heck with the comments. If I could have the opportunity to comment _after_ finding out, I would likely have taken the time more often.
Maybe it s test anxiety, I don t know...is there any reason one couldn t be allowed to comment _after_ finding out the score (keeping in mind that this step wouldn t have to show whether a taker had gotten a particular answer right or wrong)?
Thursday, March 12, 2009 2:00 PM by libertymunson
What a great observation! I hadn t given much thought to why we don t get as many comments for live exams as we do for beta exams, but I suspect that the excitement/fear/anxiety of getting your final score is likely a big reason why people don t spend the time commenting on live exams.
The reason that we don t allow candidates to provide comments after they know their score is because many will spend the time trying to identify the items that they missed rather than provide us feedback on the items. Because exams include a limited number of questions, by definition, they can only cover a subset of all possible knowledge/skills/abilities that could be measured. Being certified means that you are at least minimally qualified on how to use the technology or perform the job role (as defined by the content domain (i.e., objective domain)) regardless of the actual questions included on the exam (let’s ignore piracy (a.k.a. cheating) for now). This is one reason why we don’t tell you which items you missed. Rather, the score reports indicate the content areas where you should focus your efforts because doing so will likely give you more knowledge and skills in that area than you would gain if we told you which items you missed. The bottom line is that certified people should be competent in the content domain regardless of the specific questions asked. Focusing on the specific items missed might help you pass the exam in the future, but this won’t give you broader/deeper/stronger competence in the content domain that is truly needed to be minimally competent.
I’m guessing someone will see this and comment about braindumps and other cheating behaviors. As I’ve mentioned in previous comments to other threads, we take cheating seriously (http://www.microsoft.com/learning/mcpexams/policies/default.mspx). In fact, we have someone on our team who is responsible for defining and enforcing policies to address these issues. She will post a blog soon sharing what we can about our anti-piracy program.
Your friendly psycho(metrician)
Friday, March 13, 2009 3:15 AM by Alice
Exam security is certainly important, but I am wondering what difference it would make to allow early finishers to use some of their left-over exam time to get a longer comment period.
Let s say someone takes an exam with 120 minutes to complete the exam and 15 minutes to comment. They finish taking the exam in 90 minutes. Now instead of only having 15 minutes to comment, the left-over 30 minutes from the exam-taking time could be added to the comment period, letting them comment for as much as 45 minutes if they want to. They d still only have a total of 135 minutes to potentially look at the exam questions, the same as if they had used the whole time as originally allocated.
Friday, March 13, 2009 1:10 PM by libertymunson
My guess is that you re not going to like this answer. Don t worry, you re not alone. I have to stand firm on this...I have to be the bad gal...because, unfortunately, it only takes one person to abuse the comment process to create security issues. The amount of time provided has nothing to do with seat time--it has everything to do with balancing the security of the exam with our desire to get additional feedback on the quality of the items.
Although I understand why people would legitimately want more time to provide comments, I cannot compromise on this issue; I ve weighed the risks of increasing (and decreasing) comment time, and I will not be increasing the comment time for live or beta exams.
As much as I d like to be Microsoft s psychometrician forever, odds are, some day in the very distant future (I hope), someone else will have this job. That person may disagree with my position and change the comment time, but unless something changes in ways that I can t imagine, it seems unlikely.
Friday, March 13, 2009 2:37 PM by Krista
And Alice, following on what Liberty was saying, I d like to reiterate that comments on a live exam are actually NOT the best way to get us feedback about an exam...we can t guarantee that we ll actually read them, because we get so many. You can always submit feedback about the exam by simply sending an e-mail to firstname.lastname@example.org. Try to include as many details as possible about the item or issue you encountered and which exam.
I took the beta exams 71-321 and 71-323 and dont have a clue on the exam results. As per the website, prometric is supposed to mail the score report however they claim it is sent by MS. So who send the result and where can I check if I passed these exams?