Awhile back, in the comments of one of my blog posts, there was a bit of chatter about what happens when someone escalates an issue or concern that they have with a question on an exam. How does this work? What really happens?
Let's start at the beginning...
So, you're taking an exam and find a question that you feel is technically inaccurate, doesn't have a correct answer, or has some other flaw that prevents you from answering it correctly. What do you do?
You have to let us know!!! Sometimes, there are issues with a question or even the exam delivery that we aren't aware of... unless you tell us. Don't assume that we know! We are not trying to trick you. Really!
To escalate an issue with the content of an exam, you need to complete the exam item challenge form found here (expand the section "Challenging a Microsoft Certification exam item"). Complete the form within three days of taking the exam, and submit it following the provided instructions. Once we receive the form, we start our investigation into the issue or concern raised.
Using the information provided, we will identify the question being challenged. This means that you need to provide as much information as possible about the question and your concerns so that we can identify the correct question. Some tips to help us identify the right question: What was the context of the question? What was particularly memorable about it? Did it mention a company or server or code? What type of question was it (multiple choice, build list, hot area, etc.)? Currently, we are unable to see the order in which you saw the items, so telling us that it was question #5 doesn’t help. We need as many details as you can recall!
Once we have done so, we work with subject matter experts to determine whether the question is flawed. Based on their feedback, we will provide you with a response. To protect the integrity of the exam content, we can't provide specific details about the feedback that the experts provided, but we will provide a general summary of the results of the investigation. It can take up to six weeks to receive a response from us, although we do our best to provide one sooner.
If we determine that there is no issue, nothing changes in the exam. It is not uncommon for someone to misremember the content of a question, leading them to believe it to be flawed when it is not. There is some interesting psychology at play here... Humans have notoriously poor memories, which are shaped by our expectations and self-fulfilling prophesies. If something doesn't align to our expectations or if we don't know something, we have an amazing ability to recreate our memories to fit our expectations and our existing knowledge. And, sometimes the person raising the issue is simply incorrect.
However, if the feedback has merit, we will fix or remove the flawed question. It can take a little time to make these types of fixes because of the psychometric implications of fixing questions on or removing questions from an exam. That means that if you retake the exam soon after providing feedback about an issue, you might see the flawed question again. What happens to your score in these cases? I check to see whether removing the question changes your score as well as the score needed to pass, but it is unusual for the removal of a question to change either. Why? Well, sometimes you answered the question correctly even though it was flawed, so removing the question actually "hurts" your score--you now have one less point. If you didn't answer the question correctly, then your score doesn't change. What might change is the cut score, but even this is surprisingly rare because we always round up to the nearest whole number when we set the cut score. Rounding up ensures that you have demonstrated at least minimal competence in the content domain. If the cut score does change, you and everyone who saw that version of the question will be rescored and contacted by VUE with an explanation of what happened. And, sometimes, you are more than one point away from passing although you may not realize it.
The most common question I get at this point is: Why not just give me the point? Because I can’t assume that you would have answered the question correctly without the flaw. I have to ensure you are at least minimally competent. I cannot make any assumptions about your performance. I have to use the answers you provided to our questions to make this decision. I can remove questions from this consideration if they are flawed, but I cannot assume you would have known something if the question had not been flawed.
This is an overview of the exam content escalation process. We take your feedback very seriously. I wish I could provide specific examples of changes that we've made not only based on your escalations but also on the comments that you provide during the exam, but I cannot for the same reasons why we don't provide detailed explanations about the results of our investigations into the issues that you raise - providing too many details might undermine the integrity and security of the exam content.
Know that your comments and feedback help us identify content that is flawed or outdated. You also help us find typos. Yes, typos shouldn't ever make it on to our exams, but there's no such thing as a perfect question and sometimes, they do. So, I really want to thank you for being passionate enough about our exams to take the time to provide this type of feedback. I believe our exams get better every day because you do.
Now, what questions do you have?
Do you know Office 365? Would you like to help us evaluate a new item type: text entry/short answer? Of course, you do! Microsoft Learning is evaluating the possibility of including short answer (text-based) questions on our certification exams and in our online courses. By completing this short survey, you can help us evaluate the efficiency and accuracy of various automated scoring technologies, as well as if and how we can best leverage short answer (text-based) questions in our learning products.
The example questions in the survey are based on 0365 skills and represent different variations of short answer questions that we are exploring. These variations are also designed to allow us to evaluate several automated scoring systems. As a result, we are asking questions that require complex responses and/or to provide multiple pieces of information. By doing this, we can understand the limitations of these scoring systems.
This survey will take less than 5 minutes to complete and will help us ensure the validity, reliability, and relevance of our certification exams and program.
You can access the survey at this link: http://aka.ms/shortqa
Thank you for taking the time to provide your input!
Continuing in the series of ACE NewsBytes covering the questions posed by the viewers of my webinar interview with Greg Shields from Pluralsight, this one covers those related to value of certification. This is a hot topic, and I expect even more questions from our loyal followers to help us better understand what you're thinking on this topic and what we can provide to help you build a business case with your manager, team, organization, etc. to get certified!
Here are the questions that I answer in this edition:
Watch this ACE NewsByte as I shed some light on the real value of certification!
As a reminder, in our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In the second installment, we discussed question formats/item types, in the third installment, we discussed questions about scoring, and in the latest installment, we discussed exam development.
UPDATE 3 (April 21)
The Microsoft Virtual Academy course "Querying with Transact-SQL" is now available. This course completes the path needed to recertify your MCSE: Business Intelligence certification, and is also part of the MCSE: Data Platform recertification path. Find out more here: http://www.microsoftvirtualacademy.com/training-courses/querying-with-transact-sql
UPDATE 2 (April 19):
Thanks for the continued feedback on this thread.
Stay tuned this coming week for another blog, where Ken Rosen and I sit down and talk through the questions, comments, and concerns raised by the BTL community.
Please note that this option only applies to recertification. Individuals seeking MCSE certification must still pass a series of robust, proctored exams.
Please also see step 3 for clarification - Microsoft will be verifying the completion of your Microsoft Virtual Academy activities for recertification.
We know you have a lot of demands on your time, so we're rolling out a new option to help you keep your certifications current. Now, you can recertify by completing a prescribed path of Microsoft Virtual Academy courses.
Although you can still choose to recertify by passing a recertification exam, recertifying with Microsoft Virtual Academy provides a great way to both maintain your current skills and catch up on advances in your area of expertise, all on your own schedule.
Recertification can now be achieved simply by completing these three steps:
If your MCSE: Data Platform or MCSE: Business Intelligence credential is due for recertification, you can take advantage of Recertification Through Microsoft Virtual Academy immediately. We’ll add recertification paths for other MCSE and MCSD certifications over the next year.
To learn more about this recertification option, click here.
Continuing in the series of ACE NewsBytes covering the questions posed by the viewers of my webinar interview with Greg Shields from Pluralsight, this one covers those related to exam development:
Want the answers to these questions and more? Watch this ACE NewsByte!
As a reminder, in our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In the second installment, we discussed question formats/item types, and in the third installment, we discussed questions about scoring.
Microsoft is supercharging its AppToCert program for Spring 2015, broadening appeal to a wider segment of developers. Here are the updated details:
We know that you have real world experience in building apps, and hard decisions on how to spend your limited time and money for training and certification. So, why not simplify your life and get credit for those apps you’re building? Elevate yourself by getting certified as a Microsoft Certified Solutions Developer through a special streamlined path, courtesy of Microsoft Learning and Microsoft Developer Support Experts.
With the AppToCert Program, HTML5/JS or C#/XAML developers can save hundreds of dollars and get credit for up to two exams by showing the app development work you are already doing.
Along the way, the app review will provide you with tips on app design and implementation, and address common pitfalls and blockers to getting apps into the marketplace.
We have a number of great updates to share with you in this ACE NewsByte!
1. The Bing Rewards Campaign has been extended through April 30! Take a MCP certification exam until April 30, and regardless of if you pass or fail, you will receive 525 Bing Rewards Credits.
2. MCPs receive a 40% discount on print books and 50% on eBooks.
3. Rock it at Ignite at the MCP party and by taking an exam for half price! MCTs plan to be there a day early for Day Zero fun (and it really is fun!).
4. Larry Kaye (he envisions, designs, and implements our developer certification paths, requirements, and strategy), won the FIRST ever ITCC Innovation Award for his AppToCert program. The Innovation Award celebrates professionals in the IT industry responsible for creating a product, service, or initiative that positively impacts customers, companies, and/or the candidate experience. This is an AMAZING honor for him!! I really cannot stress how innovative his idea is in the certification industry. To be recognized for this program is quite an honor.
Learn about these great updates and more by watching this ACE NewsByte!
As you may recall, I participated in a webinar interview with Greg Shields from Pluralsight in December, where I answered many of your questions about Microsoft exams. I had a great time during it and because I wanted to continue answering your questions, we began a short series right here on Born to Learn where we do just that. In our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In the second installment, we discussed question formats/item types.
Today, we're back with a brand new installment of our mini-series where we discuss exam scoring. Specifically, we answer the following questions:
Are all questions objectively scored? In other words, are they all scored automatically or are some human graded?
If you don’t answer a question because you either run out of time or skip it, is it wrong?
You always have lots of questions about scoring... What else do you want to know? Let us know in the comments section below.
As you may recall, I participated in a webinar interview with Greg Shields from Pluralsight in December. You had a lot of questions that I didn't have time to answer, so I'm answering them in a series of ACE NewsBytes because, you know, why not?
In our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In this installment, we will be discussing question types and answering these questions:
Have you seen either of these two new question types on an exam yet? What did you think? Let me know in the comments section below.
What else do you want to know? Your questions will be answered!
Hit "play" to watch this ACE NewsByte:
If you're looking for more expert insights and tips for preparing for a Microsoft exam, please be sure to check out our first question and answer session here.
You may have seen me mention this in a few other blogs or in recent ACE NewsBytes that I was interviewed by Greg Shields for Pluralsight at the beginning of December. It just occurred to me that while I have mentioned this in several different places, I have never actually blogged about the interview itself. Silly me!
This was a spontaneous interview--in other words, Greg didn't give me a heads up on the specific questions that he would be asking, and many of them grew organically from our conversation and the responses that I provided to his first questions. In our preparation, he gave me the basic outline for what he'd like to discuss and then asked if anything was off limits. My response "Nope. If I can't talk about it, I'll tell you what I can and why I can't share any other details." Mostly what I can't talk about are topics related to privacy issues and information Microsoft considers proprietary (e.g., passing rates, volumes, number of people with a particular certification), but I can talk about nearly everything else in the space of exam development, program design, item writing, statistics, how we manage the rapid pace with which technology is changing, etc. All of these topics and more came up during this interview, and I have a feeling that the people who tuned in learned a few things that they didn't know about how Microsoft designs and develops our exams and certification requirements to stay relevant to our candidates, how we pilot test new item formats, the difference between a certificate and certification program and what that means in terms of the alignment between exam and learning content.
Are you curious? Check out the webinar:
I love doing this stuff. I love answering your questions. I had a lot of fun and hope to do something like this again. If you have questions, please ask them. If you want me to speak to your students or your colleagues or ???, I'm totally onboard. Let me know...
Did you hear the news? In this ACE NewsByte, Briana and I discuss our new MCSE certification for those who are interested in Enterprise Devices and Apps! To earn this certification, you need to take and pass:
Interested? Register at: https://www.microsoft.com/learning/en-us/exam-70-695.aspx and https://www.microsoft.com/learning/en-us/exam-70-696.aspx.
We are also pleased to announce a straightforward path to upgrade to your MCSA Windows 8. If you hold an XP certification, you can migrate to MCSA Windows 8 certification by passing exam 692: Upgrading Your Windows XP Skills to MCSA: Windows 8. Isn't about time that you did?
In December, I participated in a webinar interview with Greg Shields from Pluralsight. We had great participation and a lot of questions from the audience. Although I was able to answer some of those questions, I wasn’t able to answer all of them in the time that we had. I asked the faciliator of the webinar to send me your questions so I could answer them over a series of NewsBytes. In this series, Briana will "interview" me by asking those questions that were submitted by participants.
In the first installment, we will be discussing exam preparation. In this video, I will be answering these questions:
So, I admit... the video is a bit long because we have a lot of great exam preparation tips, but you really want to watch to the end... Why, you ask? Because I will let you in on a little secret...I explain why there can be a disconnect between our learning options and exam content and what are we doing to ensure better alignment between them going forward.
Got more questions? You know I love to answer them so leave them below!
A few months ago, I posted a blog post where I had the audacity to say that exams could be "fun." Based on some of your reactions, "fun" was clearly not the best word to use. I meant that exams should be challenging in a good way--in my own experience, a good challenge is something fun, but this is not true for everyone. More specifically, what I was trying to say was that when you complete an exam (whether you pass or fail), you should feel a sense of accomplishment. You should feel that you were tested on the right skills in ways that fairly and effectively evaluated whether or not you have those skills. Even more, we want to create an “engaging” experience by using question types that require you to think in ways similar to how you would have thought had you faced that problem in the real world (even if you personally would not or have not experienced that particular problem).
But, your reaction to the word “fun” creates an interesting question for me. Across the certification industry, the notion of “gamification” is taking hold. It is becoming more common in training, and it’s only a matter of time before it finds its way into certification. Because it requires immersive, interactive technologies, gamification enables new forms of engagement, personalization, and assessment within training and, possibly, within exams. Games have become a medium for giving students engaging experiences in areas where it might be otherwise impossible in terms of training… why couldn’t it be used to change the face of certification exams? Imagine an exam that simulates your experiences as an ITPro or Developer in an organization as a game like experience. As you make decisions on how to solve problems, the parameters of the game change to reflect those decisions. As you complete easier tasks, the tasks become more challenging.
Don’t get me started on the psychometric challenges of implementing the level of personalization for exams because this will be extremely complicated…but imagine a world where this is not an issue, what do you think of this idea? Do you see an opportunity for gamification within our certification exams? How do you think it can/will play a role in a certification context?
In this ACE NewsByte, Briana and I highlight some really awesome program updates that YOU need to know about!
First, we have updated the Certification and Training Roadmap to reflect changes we've recently made to the program (e.g., electives, new Azure exams, new certifications, etc.)... this includes updating the associated app, which you can get from the Windows Store today (updates will be complete shortly).
Second, we want let you know of several really cool MCP benefits that were recently announced in case you missed them:
Finally, have you heard of the Know It. Prove It. campaign? Throughout February, challenge yourself to learn a new skill in one of eight areas, including cloud development, game development, Office365, and SharePoint.
We hit the highlights in this video! Watch and get excited about training, certification, challenging yourself to do more, and being (or becoming) a MCP!
[Update]: A new version of the certification roadmap has been uploaded on March 5, 2015!
We’re excited to announce we've updated the Skills and Certification Roadmap, which has been revised to reflect the latest skills development and certification information. This popular resource provides a one-stop-shop for certification pathways. The updates include the new Devices MCSE, Azure exams and exam electives. You can download the roadmap as PDF here.
We’re also updating the Training and Certification Guide app to reflect the updated information. Watch for an announcement on that soon!
Download the FREE certification roadmap: