A Look Back at 2013: A Year in Numbers from an Exam Perspective

A Look Back at 2013: A Year in Numbers from an Exam Perspective

Liberty Munson (Microsoft)

As many of you know, I love "year in review" retrospectives, so my holiday gift to all of you is my annual summary of what happened in the world of exams from a numbers perspective. Without further ado, let’s take a look back at 2013:

    • Published more than 600 total MCSA, MCSE, Dynamics, TS, and PRO exams, including new publications and republications related to updating exam content and maintaining the validity and reliability of the exams over their lifecycle.
    • Of these, ~200 were ENU, ~400 were in localized languages with the most popular languages being French and German followed by Japanese and simplified Chinese.
    • Published ~50 new exams
    • Of these, ~30 Dynamics exams, 10 MCSE/MCSA exams, 8 Specialist exams, and few others mixed
    • Published 6 beta exams taken by more than 1500 candidates with nearly 2000 comments—all of which I read, of course! By the way, in 2012 with the release of Windows Server 2012, SQL Server 2012, and Windows 8, etc., we had more than 30 beta exams! So, this is a bit on the low side… which might explain why you may not have been invited to participate in a beta in 2013!

So, those are the highlights from an exam publication perspective, but this year, I thought it might be more fun to talk about what I did in 2013. If you know me, you know I love to talk about what I do…

      • Psychometrically analyzed more than 100 exams, some more than once.
      • Psychometrically analyzed more than 10,000 items. Not only is this information used to make decisions about the effectiveness of our questions in terms of differentiating qualified and unqualified candidates, I also use this data to inform our question development process as well as make decisions about which item types are more psychometrically sound for our various audiences.
      • Pilot tested eight new question types that are more interactive than traditional multiple choice questions; if you haven’t seen them on our exams yet, you will soon!
      • Read more than 10,000 comments.
      • Pilot tested “polytomous” scoring of items, meaning that some questions may be worth more than one point. Candidates can earn “partial” credit for answering parts of questions correctly rather than the all or none point process (known as “dichotomous scoring”) that we’ve used to date. I’m still evaluating the effectiveness of this scoring strategy, but if a question is worth more than 1 point, it will be stated in the instructions for that question.
      • Tracked candidate perceptions of the quality of our exam questions on more than 100 exams weekly. This information comes from candidates who take the time to complete the Exam Satisfaction Survey (sent by ComScore, a third party who manages the distribution of the survey for us). I really do monitor these results weekly and plan our sustainment efforts accordingly!!
      • Attended MCT summits, participated in MCT Quarterly updates, launched ACE Chronicles, blogged, and tweeted, etc. in attempt to shed some light on our exam development and certification processes. In this case, I want you to pay attention to the (wo)man behind the curtain and these are the droids you’re seeking.

Wishing you an awesome 2014! And, just for fun... a picture of me on my favorite hike from 2013 with the best dog ever!

Comments
  • KevinM
    |

    Liberty,

    Should we be getting ComScore surveys after every exam?  I've taken approximately 25 Microsoft exams over the past 6+ years, and I have received exactly 6 ComScore survey invitation emails over that time.

  • Mike Corkery
    |

    Hi Liberty, My comment is similar to Kevins. I have taken over 80 exams, and I am not sure if I have seen one of the ComScore surveys.

  • Liberty Munson (Microsoft)
    |

    The trick to receiving the invite is that first you have to opt in to letting Microsoft contact you; if you have opted out at any point, you won't receive this email. If you have opted in, then getting these surveys depends on how often and when you've been contacted by Microsoft. We have to follow Microsoft's contact rules. You can only be contacted by MS once every 90 days. Finally, we only send the survey to people who took the exam in ENU because we don't have the resources to set up and administer loc versions of this survey.

  • Liberty Munson (Microsoft)
    |

    Oh, and we have to have the right email to contact you! :)

  • KevinM
    |

    That explains it.  I tend to take most of my exams in clusters.

  • Davin Mickelson
    |

    Hi, Liberty.

    Do you also read all the comments entered by test takers upon completing their exams? For exams I have had to retake in the last sixteen years, I have never seen any exam "repairs" performed in respect to glaring typos or other obvious errors I have pointed out in the exams I have taken.

  • Bibble
    |

    2013 - the year MS retired MCM / MCSM.

    The day my further career aspirations died :o

  • ImmoRhys
    |

    Davin, I sat 70-413 back in March 2013, commented every question with problems, submitted an Item Review form for the exam to Microsoft listing all the problems. An MVP sat the same exam months later, still has the same problems.

    I've put it to MS a few times: Exams never change, and are not updated until the technology changes. If the questions changed, brain dumps wouldn't work.

    There's some weird lack of interest in stopping brain dumps. MS needs brain dumps working, to get partners certified to silver/gold status?

    There's some shocking mistakes, while apparently these are written by subject matter experts, there were questions that were so far removed from any real world scenario it was laughable.

  • Davin Mickelson
    |

    Well, I just passed 70-483 (C# exam) on January 6th. I passed the exam with what I consider a good score (875) but I also notice that one of the questions was impossible to answer. Like always, I submitted feedback at the end of the exam. When I got home that day, I decided to take it to a new level so I filled in and submitted the exam item challenge form for the bad exam question. It felt weird trying to write down an entire exam question/answers from memory, like I was creating a brain dump, but I was the submitting it to Microsoft to help the team fix the problem. That was my only goal.

    **I am starting to regret it.

    Even though I got an 875 (700 needed to pass, like most other Microsoft IT exams), they have withheld adding the passed exam to my transcript. They also told me that they usually do not change the exam content anyway - especially if the exam will be retired or modified within six months. They also told me that it will take between four (4) to six (6) weeks for them to get back to me because of a needed SME review.

    Thus, I created my question (above) to Liberty asking if she (or someone else) reads those comments and possibly considers examining test questions that seem to get above average comments so I don't have to go through this pain.

    I only want to help MSL.

    Brain dumps? Unfortunately, since Microsoft is changing the exam content so frequently as of late, the only people making money on current exam study materials are those evil companies selling illegal brain dumps. The MCPTPs (STS, Transcender, MU) are struggling to keep up with creating legal study materials. They can only create materials for the most popular exams.

  • KevinM
    |

    Wow, I've been taking exams for years and never knew that there was even an "item challenge" process.  I had always assumed that bad questions were identified and eventually filtered out based on exam comments.  I've seen loads of poorly written or just plain incorrect questions over the years, presumably because I often take exams in beta or shortly after they are released.  If there is a separate process to report bad questions, what is the purpose of letting us comment on specific questions?

  • Liberty Munson (Microsoft)
    |

    Hi Davin,

    I'm looking into the escalation that you submitted, but regardless, we shouldn't be "withholding the exam from your transcript." Please contact the regional service center to escalate this issue. The fact that it's not showing up on your transcript is not related to the escalation that you submitted. You can find your RSC here:

    www.microsoft.com/.../help.aspx

    As for reading exam comments, we actually do this, and over the last 18 months have become more strategic about the process for reading, analyzing, and updating exam content accordingly. Because we can't read all comments on all exams (there are simply too many), we focus on the exams and items with the most comments and go from there.

    Although it may not seem like we have fixed issues identified in comments, we do take them seriously; however, we have to prioritize comments and fixes and, of course, it does take some time to find SMEs to review the comments, make the fixes, and then republish exams with the corrections in place. Because of the churn this sometimes creates, I try to bundle these changes with any other changes that I'm making to exams related to psychometric issues into one publication.

    If you want to formally challenge an item, follow the process described here:

    www.microsoft.com/.../certification-exam-policies.aspx in the Challenging Microsoft Certification exam questions section.

    Thanks for your feedback!

  • Liberty Munson (Microsoft)
    |

    Hi KevinM,

    If you want to formally challenge an item, follow the process described here:

    www.microsoft.com/.../certification-exam-policies.aspx in the Challenging Microsoft Certification exam questions section.

    Please take the time to do this, because we really don't know what we don't know. Although exams go through a review process before they are published, the shear volume of publications makes it difficult to catch everything... and there's no such thing as a perfect exam (or item for that matter), so bad items occasionally slip through (eventually, we find them through the comments and psychometric performance, but you can help us find them earlier). Please don't assume we know there's an issue with an exam. We only know if you tell us.

    Thanks!

  • Davin Mickelson
    |

    Hi, Liberty.

    Thank you for taking the time to respond! Your post was very helpful. I have wondered for years about the comments.

    I will get my passed exam taken care of. I suspect it didn't make it to my transcript because I have been testing at all the test centers in the Saint Paul, MN area, trying to find the best one.

    I would have waited another few weeks (4 to 6) before contacting Microsoft about my transcript. Not any more, thanks your response.

    Thank you very much and have a great weekend.

  • mohammad haroon1
    |

    Liberty

    i just wanted to ask that can i give the exam 70-640 Windows Server 2008 Active Directory,  mcsa exam

    or it has been replaced by some new exam  please help me out and kindly give a brief and clear answer for that

  • Davin Mickelson
    |

    Mohammad,

    That exam is still available for people who wish to become certified in Windows Server 2008. You may also want to consider the new 70-410 exam for Windows Serve 2012. Check out all the MCSA certification titles here: www.microsoft.com/.../mcsa-certification.aspx

View All