Survey Results Are In: Progressive Case Studies

Liberty Munson (Microsoft)

A few months ago, we proposed a new item type--progressive case studies--to the community and asked for your feedback. We created a video describing this new item type and asked you for feedback via a brief survey, and the results are in!

As a reminder, progressive case study is a variation of the case study item type in which the case builds progressively. More resources are introduced to candidates as they answer questions. The progressive case study scenarios will include information in a format that reflects how candidates might receive it on the job. For example, the case may include emails, graphics, exhibits, and documents that candidates need to review and synthesize in order to answer questions.

So, what did you think? Highlights include:

  • 63% felt this item type was real world and relevant
  • 83% thought it was a challenging way to test
  • 47% felt that it would be an effective method to validate skills and would increase the prestige of the associated certification
  • 62% thought that the ability to process information that is added as the case progresses was a valuable skill to test
  • Several respondents commented that this item time was a great innovation and tests one's ability to apply solutions rather than to memorize content

Several respondents also expressed some concerns that:

  • Test takers wouldn't be able to ask clarifying questions, but that's true of all exam items.
  • These items may require making inferences...again, true of all exam items--we simply cannot provide you with all of the information needed to cover every nuance that might be related to answering a question; as a minimally qualified candidate, we expect you to make appropriate inferences about the situation and solve the problem accordingly. At the expert level, where these items will appear, we expect you to demonstrate the ability to make the correct inferences and to make the best decision possible given the information that you have. Through our exam development process, we ensure that subject matter experts agree that a minimally qualified candidate has sufficient information to make appropriate inferences and that they agree on what the best answer is in that specific situation.
  • Candidates will not be able to return to previous questions. I agree that this will be challenging, but many certification and licensure programs adopt this practice--so, it's not new in this space. Most of the research related to the candidate impact of not being able to review questions has been done in relation to computer adaptive testing. While candidates may not like this (and it will certainly change your approach to the way that you take exams!), research suggests that scores may be better reflections of ability and skill because candidates can't use information obtained in latter items to change their answers on previous items. Don't worry about what this means for your score. This part of the exam experience will be accounted for in the way the passing score is set. In other words, if this makes the exam more difficult, the passing score is adjusted accordingly. The same level of skills needed to demonstrate minimal competence will be required, so while the exam may feel more difficult as the result of this change, this isn't reflected in the passing score.

Given the generally positive feedback on this item type, we expect to pilot test it sometime in 2014. Thanks to everyone who provided their feedback!

  • KevinM
    | |

    The one thing that I have to say about this (and I included it earlier in the survey) is that the more info that you put into an exam question, the more important it is that people review the question for clarity, especially clarity of language.  I have taken many beta exams over the years and have seen more than my fair share of poorly worded or unclear/imprecise questions.  I've even seen a few such questions on live exams, and I always make it a point to comment on them when they are encountered.

    I have often felt that during the process of writing exam questions that many of the question authors have read, re-read, and discussed the question so often that the intent may be crystal clear to the community of authors, but unclear to someone who is reading the question for only the first or second time.  I have noticed this problem particularly with Microsoft exams, and not so much with exams that I have taken from CompTIA or VMware.  I think that you would be well-served by having another wave of review, before beta, that focuses purely on the clarity\comprehensability of the questions.

  • Peter Kessler
    | |

    I think the telling number is 47 %. Less then half the People found it effective.

    And as long as clarity of writing is an issue in each and every exam I take, this is a bad idea.

  • b20810bc-900a-486a-bcfd-578a341b4415

    Do you know how the current exams are scored? If you miss a question in the case study, do you lose points for that question or the entire case study?