On the series finale of this Memorial Day Trilogy, I will reveal the final phases of our process. (Find a diagram of the entire process and details on first three phases here and details on phases 4-6 here.)
The results of the beta exam are analyzed to determine which items should be included in the live item pool. I had a four part series on how we make these decisions. To learn more, see:
Item Selection Part 1
Item Selection Part 2
Item Selection Part 2.5
Item Selection Part 3
Phase 8: Standard (Cut-Score) Setting
Once the final item pool has been established, the cut score is set using industry recognized methodologies. Regardless of the method used, the cut score is always based on SME input of some kind—cut scores for MCP exams are never set in an arbitrary manner.
After the item selection and standard setting meeting, the final item pool and cut score information are used to assemble forms, or instances, of the exam. If an exam has multiple forms, these forms are designed to be psychometrically equivalent in terms of difficulty, meaning that one form is not harder than another.
This phase is self explanatory. Prometric publishes the exam, and it becomes available for candidates to take.
Localization is handled by an external vendor. Our exams are typically translated into French, German, and Japanese; additional languages are added based on market research, reach, and/or other strategic initiatives.
This is technically not a phase of exam development. Rather, it’s a phase of the exam lifecycle that is critical to maintaining the quality of the exam, which is why I like to include it as part of our process.
Exams are psychometrically analyzed at least annually. We also review the results of our candidate satisfaction survey (the survey you are invited to complete via email after you take an exam—your feedback drives change, so take a minute to complete it the next time you take an exam) and other sources of candidate feedback to identify ways to improve our exams.
So, that’s the long and short of our exam development process. Each of these phases will be described in more detail in a future blog if they haven’t been already.
What else do you want to know? I’m partial to trilogy blogs… will work for blog ideas.