On August 4, I had the distinct pleasure of giving the keynote address at TechMentor. Greg Shields and I took a different approach to the keynote—it wasn’t a presentation but a fireside chat--complete with a roaring fire (video-based, of course) where he and the audience asked me questions about Microsoft’s certification and learning program. It was an awesome experience that I hope the attendees enjoyed as much as I did!
Missed it? Here’s a high level view of the our conversation which focused on Microsoft's efforts to modernize skills validation and make it more relevant and valuable to technical audiences and employers. As our industry is changing so are the ways technical professionals want to learn and prove their skills.
We started the conversation with an overview of the exam development process and how IT professionals and developers could become involved in that process (hint: add your profile to our SME database--http://aka.ms/MSLSME and don't forget to update it as Microsoft releases new products!). From there, we opened up the conversation to questions from the attendees. They were particularly interested in badging because it has the potential to highlight skills in a different way than certification, and badges could be easier to attain, paving a smoother road to certification.
We also discussed why people were interested in certification and why they were not. A common complaint is that candidates want to demonstrate competence with hands on/performance-based methodologies, not with multiple choice questions. This led to how we might certify "work products." Is there a way to grant a certification for something that you're already doing? In the developer space, this is relatively easy--see the CertToApp program for an example. With the right resources and support, we could expand this to other developer "products." The IT Pro space is a tougher nut to crack, so I asked the audience for ideas on how we might take the idea of what we're doing with CertToApp and apply it to the ITPro audience. They offered some great suggestions, such as having someone submit their design specs for building a system/process to pre-specified requirements or even submitting the design spec for a system they have created or implemented (e.g., building a server and adding the features and rolls specified by Microsoft or needed for your organization's implementation). By the way, if you have ideas, I would love to hear them.
My favorite part of the conversation, though, was reminding attendees that we value their feedback. Leave comments when you take exams, escalate issues or concerns that you have about exam questions--see https://www.microsoft.com/learning/en-us/certification-exam-policies.aspx for more details, and complete the Exam Satisfaction survey when you receive the invite from ComScore, our survey vendor. We carefully review all of this information to make improvements that (I hope) result in the highest quality exam and experience. There's no such thing as a perfect exam; we don't always know when there is an issue, and we need your help to tell us if something doesn't seem right. If all else fails, email me...seriously.
We also spent some time talking about the crazy ideas that I have to shake up the world of certification. Some are on Microsoft Learning's radar, but others are dreams that I have on how we might revolutionize certification as we know it today... More on that in a future blog if you're interested. Let me know in your comments.
I really enjoyed this conversation and am always looking for more opportunities to answer your questions. Ask away!!
Photo credit: Douglas DeCamp
Have you been paying attention to what's been happening in the world of Microsoft Certification? June brought two big announcements that are highlighted in this ACE NewsByte:
1) Online proctoring expands to most European countries with plans to continue the worldwide expansion over the next few months, and
2) MCP Profiles were launched, allowing you to create a public profile that showcases your skills not only to others in our community, but to potential and current clients, customers, and hiring managers!
Learn more by watching this ACE NewsByte!
And, by checking out these links:
Because many of you were so kind to respond to our text entry survey (see this blog post: https://borntolearn.mslearn.net/b/weblog/archive/2015/05/11/help-us-evaluate-a-text-entry-scoring-tool), I wanted to follow up with some details on what we learned.
What did we learn about the scoring tool that we were testing?
What do people think of including text entry questions on our exams? Respondents indicated that:
Overall, survey responses provided support for including these types of questions on our exams. So, never fear...although this scoring tool didn't work as well as we'd hope, we are not giving up. We continue to explore other options to creating an innovative solution that allows you "code" or write commands/syntax in such a way that 1) gets us closer to the real world experience than multiple choice questions, 2) takes into account the multiple different ways a solution could be implemented, and 3) is scored correctly on the exam!
Thanks to everyone who took time to respond to our survey and provide this valuable feedback!
Calling all Windows developers! Microsoft Learning is in the process of defining the next set of developer exams on the new Universal Windows Platform, and we would like your input on the importance, frequency, and rigor for each functional group and objective for these exams.
If you wish to participate in the online survey, please click on the following link, and submit your responses by July 6th:
What are you doing on August 4 at 8:30am PT? If you're looking for something to do, I have an idea. Come to my keynote address at TechMentor 2015.
I cannot tell you how honored I am to be invited to do this! I am SO excited about this amazing opportunity. Of course, it won't be your typical keynote address filled with a PowerPoint presentation because I've never been one to do the normal. Instead, it will be more like a fireside chat with TechMentor co-chair Greg Shields. I am planning to discuss research showing the benefits IT Pros receive from becoming certified and explain how Microsoft designs and develops its MCSE certification program to ensure ongoing relevance and value.
I am going to reveal the "secret” sauce for how we develop valid and reliable certification exams and how we ensure that our exams AND your skills stay relevant and up to date. I will also be sharing some changes that are coming, not only to certification exams but to our certification program, and other great ideas we have in the pipeline. As you know, we continue to look for ways to improve our certification process, innovate, and increase our certification's value not only to you but to hiring managers and organizations... I hoping to be able to share some really cool stuff that I'm working on that should help with this!
Want to know more about my session (or see what I look like)?
Well, what are you waiting for? Register today! I would love to see you there!
Oh, I have your attention now, don't I?! Here's the deal. To help you get more from your investment in earning Microsoft credentials, Microsoft Learning is exploring several options. Some popular ideas are: free retakes, free practice tests, free online training, free exam prep materials, etc. We would like your feedback. How much to do you value each of these options? Do you have other ideas we should be considering? Which ideas do you prefer?
Please take a few minutes to complete this brief survey: http://microsoftlearning.co1.qualtrics.com/SE/?SID=SV_3UjGMTYmmc9pNQN
Your input will be used to help us assess the value of each option. During the survey, you'll also get some insight into some ideas we’re considering for enhancing your exam and certification experience. So, what are you waiting for?! Complete the survey today!
The fine print: While I always welcome comments on my posts, please be sure to respond to the survey so your input will be included in the results. Thanks!
Got your attention, didn't I!?!
In this NewsByte, Briana and I share some great improvements and program enhancements that Microsoft Learning has introduced to make skilling up easier:
1) Our amazing partnership with edX.org,
2) Free eBooks if you pass a MCSA, MCSE, MCSD, or Specialist exam before June 30 (while supplies last),
3) Improvements to lab linking in Digital MOC and interactive assessments, and
4) The MCP Insider series--getting answers and insight from industry leading experts
Watch the video to learn more!
Awhile back, in the comments of one of my blog posts, there was a bit of chatter about what happens when someone escalates an issue or concern that they have with a question on an exam. How does this work? What really happens?
Let's start at the beginning...
So, you're taking an exam and find a question that you feel is technically inaccurate, doesn't have a correct answer, or has some other flaw that prevents you from answering it correctly. What do you do?
You have to let us know!!! Sometimes, there are issues with a question or even the exam delivery that we aren't aware of... unless you tell us. Don't assume that we know! We are not trying to trick you. Really!
To escalate an issue with the content of an exam, you need to complete the exam item challenge form found here (expand the section "Challenging a Microsoft Certification exam item"). Complete the form within three days of taking the exam, and submit it following the provided instructions. Once we receive the form, we start our investigation into the issue or concern raised.
Using the information provided, we will identify the question being challenged. This means that you need to provide as much information as possible about the question and your concerns so that we can identify the correct question. Some tips to help us identify the right question: What was the context of the question? What was particularly memorable about it? Did it mention a company or server or code? What type of question was it (multiple choice, build list, hot area, etc.)? Currently, we are unable to see the order in which you saw the items, so telling us that it was question #5 doesn’t help. We need as many details as you can recall!
Once we have done so, we work with subject matter experts to determine whether the question is flawed. Based on their feedback, we will provide you with a response. To protect the integrity of the exam content, we can't provide specific details about the feedback that the experts provided, but we will provide a general summary of the results of the investigation. It can take up to six weeks to receive a response from us, although we do our best to provide one sooner.
If we determine that there is no issue, nothing changes in the exam. It is not uncommon for someone to misremember the content of a question, leading them to believe it to be flawed when it is not. There is some interesting psychology at play here... Humans have notoriously poor memories, which are shaped by our expectations and self-fulfilling prophesies. If something doesn't align to our expectations or if we don't know something, we have an amazing ability to recreate our memories to fit our expectations and our existing knowledge. And, sometimes the person raising the issue is simply incorrect.
However, if the feedback has merit, we will fix or remove the flawed question. It can take a little time to make these types of fixes because of the psychometric implications of fixing questions on or removing questions from an exam. That means that if you retake the exam soon after providing feedback about an issue, you might see the flawed question again. What happens to your score in these cases? I check to see whether removing the question changes your score as well as the score needed to pass, but it is unusual for the removal of a question to change either. Why? Well, sometimes you answered the question correctly even though it was flawed, so removing the question actually "hurts" your score--you now have one less point. If you didn't answer the question correctly, then your score doesn't change. What might change is the cut score, but even this is surprisingly rare because we always round up to the nearest whole number when we set the cut score. Rounding up ensures that you have demonstrated at least minimal competence in the content domain. If the cut score does change, you and everyone who saw that version of the question will be rescored and contacted by VUE with an explanation of what happened. And, sometimes, you are more than one point away from passing although you may not realize it.
The most common question I get at this point is: Why not just give me the point? Because I can’t assume that you would have answered the question correctly without the flaw. I have to ensure you are at least minimally competent. I cannot make any assumptions about your performance. I have to use the answers you provided to our questions to make this decision. I can remove questions from this consideration if they are flawed, but I cannot assume you would have known something if the question had not been flawed.
This is an overview of the exam content escalation process. We take your feedback very seriously. I wish I could provide specific examples of changes that we've made not only based on your escalations but also on the comments that you provide during the exam, but I cannot for the same reasons why we don't provide detailed explanations about the results of our investigations into the issues that you raise - providing too many details might undermine the integrity and security of the exam content.
Know that your comments and feedback help us identify content that is flawed or outdated. You also help us find typos. Yes, typos shouldn't ever make it on to our exams, but there's no such thing as a perfect question and sometimes, they do. So, I really want to thank you for being passionate enough about our exams to take the time to provide this type of feedback. I believe our exams get better every day because you do.
Now, what questions do you have?
I blinked, and it was over. It’s been just about a month since I took the stage at the Microsoft Build Conference (//build) in San Francisco, CA. It all happened so fast. My heartbeat was spiking, my smile spreading. Lights flash and the Twitterverse explodes. GeekWire reports “Microsoft adds Minecraft mod support in Visual Studio.” VentureBeat writes “Microsoft announces Minecraft modding add-in for Visual Studio.” Aidan Brady and I walk off stage and high-five. Beaming, we walk back to our green room and watch the aftermath unfold.
The previous 48 hours leading up to that moment were a blur with endless rehearsals. My eyes wide as I take in everything going on behind the scenes to pull of the show. In contrast to the thumping pulse of the conference, behind the stage is a soft quiet. Lights are muted, Microsoft executives sit in their green rooms and rehearse their lines—yes, and they even pace while doing so! There are more computers, services, screens, and gadgets than I’ve ever seen.
To say the experience was surreal is truly an understatement. Working in my office in Redmond, it’s sometimes hard to remember what drives me. When I can’t see the faces of the students I am trying to reach, when I briefly stray from understanding my audience. A highlight of my job is when I get outside my office and into a classroom, and I get to talk to students and teachers and find out what devices they use and what technology is important in their daily lives. Multiply any one of those experiences a hundredfold and that’s how //build felt. I was honored to be a part of it.
Satya Nadella has been completely transparent about the fact that Microsoft must focus on education. Microsoft needs to win the hearts and minds of students. It has to become our lifeblood. Without students, there will be no enterprise in the future. We have to understand that the concept of learning has changed—it now happens everywhere. And students are technology natives. Yet, one of the biggest things I notice when talking to students is that they don’t see the power and opportunity behind the devices they hold. It starts with coding, but it’s about computational thinking and problem-solving, the creative process behind programming.
At //build, not only did we highlight an amazing example of a student whose story is an inspiration to others, we highlighted that we—Microsoft—have listened. We understand the audience we need to reach, and we understand that we have work to do. Minecraft is the rage right now for students of all ages, and Microsoft is committed to engaging these student developers. The Visual Studio extension I announced allows programming in Java—the programming language behind Minecraft—within Visual Studio. This is a huge step to winning the hearts and minds of students who will get to play and change the game they love, all within Visual Studio, building a positive relationship in association with Microsoft.
Here’s a glimpse, including some photos, into the milestone moments along the way on my journey to //build.
Do you know Office 365? Would you like to help us evaluate a new item type: text entry/short answer? Of course, you do! Microsoft Learning is evaluating the possibility of including short answer (text-based) questions on our certification exams and in our online courses. By completing this short survey, you can help us evaluate the efficiency and accuracy of various automated scoring technologies, as well as if and how we can best leverage short answer (text-based) questions in our learning products.
The example questions in the survey are based on 0365 skills and represent different variations of short answer questions that we are exploring. These variations are also designed to allow us to evaluate several automated scoring systems. As a result, we are asking questions that require complex responses and/or to provide multiple pieces of information. By doing this, we can understand the limitations of these scoring systems.
This survey will take less than 5 minutes to complete and will help us ensure the validity, reliability, and relevance of our certification exams and program.
You can access the survey at this link: http://aka.ms/shortqa
Thank you for taking the time to provide your input!
Continuing in the series of ACE NewsBytes covering the questions posed by the viewers of my webinar interview with Greg Shields from Pluralsight, this one covers those related to value of certification. This is a hot topic, and I expect even more questions from our loyal followers to help us better understand what you're thinking on this topic and what we can provide to help you build a business case with your manager, team, organization, etc. to get certified!
Here are the questions that I answer in this edition:
Watch this ACE NewsByte as I shed some light on the real value of certification!
As a reminder, in our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In the second installment, we discussed question formats/item types, in the third installment, we discussed questions about scoring, and in the latest installment, we discussed exam development.
UPDATE 3 (April 21)
The Microsoft Virtual Academy course "Querying with Transact-SQL" is now available. This course completes the path needed to recertify your MCSE: Business Intelligence certification, and is also part of the MCSE: Data Platform recertification path. Find out more here: http://www.microsoftvirtualacademy.com/training-courses/querying-with-transact-sql
UPDATE 2 (April 19):
Thanks for the continued feedback on this thread.
Stay tuned this coming week for another blog, where Ken Rosen and I sit down and talk through the questions, comments, and concerns raised by the BTL community.
Please note that this option only applies to recertification. Individuals seeking MCSE certification must still pass a series of robust, proctored exams.
Please also see step 3 for clarification - Microsoft will be verifying the completion of your Microsoft Virtual Academy activities for recertification.
We know you have a lot of demands on your time, so we're rolling out a new option to help you keep your certifications current. Now, you can recertify by completing a prescribed path of Microsoft Virtual Academy courses.
Although you can still choose to recertify by passing a recertification exam, recertifying with Microsoft Virtual Academy provides a great way to both maintain your current skills and catch up on advances in your area of expertise, all on your own schedule.
Recertification can now be achieved simply by completing these three steps:
If your MCSE: Data Platform or MCSE: Business Intelligence credential is due for recertification, you can take advantage of Recertification Through Microsoft Virtual Academy immediately. We’ll add recertification paths for other MCSE and MCSD certifications over the next year.
To learn more about this recertification option, click here.
Continuing in the series of ACE NewsBytes covering the questions posed by the viewers of my webinar interview with Greg Shields from Pluralsight, this one covers those related to exam development:
Want the answers to these questions and more? Watch this ACE NewsByte!
As a reminder, in our first installment, we answered your questions related to exam preparation and alignment of learning content to exam content. In the second installment, we discussed question formats/item types, and in the third installment, we discussed questions about scoring.
Microsoft is supercharging its AppToCert program for Spring 2015, broadening appeal to a wider segment of developers. Here are the updated details:
We know that you have real world experience in building apps, and hard decisions on how to spend your limited time and money for training and certification. So, why not simplify your life and get credit for those apps you’re building? Elevate yourself by getting certified as a Microsoft Certified Solutions Developer through a special streamlined path, courtesy of Microsoft Learning and Microsoft Developer Support Experts.
With the AppToCert Program, HTML5/JS or C#/XAML developers can save hundreds of dollars and get credit for up to two exams by showing the app development work you are already doing.
Along the way, the app review will provide you with tips on app design and implementation, and address common pitfalls and blockers to getting apps into the marketplace.
We have a number of great updates to share with you in this ACE NewsByte!
1. The Bing Rewards Campaign has been extended through April 30! Take a MCP certification exam until April 30, and regardless of if you pass or fail, you will receive 525 Bing Rewards Credits.
2. MCPs receive a 40% discount on print books and 50% on eBooks.
3. Rock it at Ignite at the MCP party and by taking an exam for half price! MCTs plan to be there a day early for Day Zero fun (and it really is fun!).
4. Larry Kaye (he envisions, designs, and implements our developer certification paths, requirements, and strategy), won the FIRST ever ITCC Innovation Award for his AppToCert program. The Innovation Award celebrates professionals in the IT industry responsible for creating a product, service, or initiative that positively impacts customers, companies, and/or the candidate experience. This is an AMAZING honor for him!! I really cannot stress how innovative his idea is in the certification industry. To be recognized for this program is quite an honor.
Learn about these great updates and more by watching this ACE NewsByte!