Although I’ve complained a fair bit about validity and reliability issues in CME assessment, I haven’t offered much on this blog to actually address these concerns. Well, the thought of thousands (and thousands and…) of dear and devoted readers facing each new day with the same, tired CME assessment questions has become too much to bear. That, and I was recently required to do a presentation on guidelines and common flaws in the creation of multiple-choice questions…so I thought I’d share it here.
I’d love to claim these pearls are all mine, but they’re just borrowed. Nevertheless, this slide deck may serve as a handy single-resource when constructing your next assessment (and it contains some cool facts about shark attacks).
Outcomes assessment is “summative”, which is fancy for measures whether desired results have been achieved. A “formative” assessment, however, addresses something while in development to be sure it’s on track. Moore et al (2009) make a strong case for formative assessment in CME, but leave the “how-to” details to our imagination (I guess when you’re covering every aspect of CME you need to leave a few bits out).
Here’s one recipe for formative assessment (for live CME activities):
- Have your course faculty develop knowledge and/or case vignette questions relative to their pending talks
- Turn these questions into a web-based survey (www.SurveyMonkey.com)
- At least two weeks prior to the activity date, email the survey to all activity registrants
- Share the registrants’ responses with your course faculty
- Adjust the pending talks accordingly
If you feel the need to incentivize respondents (which I never discourage), offer them a discount off registration for another activity. If you want more detail, check out this short JCEHP article.
I’ve used this approach a few times and it’s been generally successful (i.e., good response rate and faculty have used some of the data to modify their presentations). However, I don’t want to pretend this approach is “setting-the-bar” for formative assessment. If you’re not doing any such assessment, this is a good way to get started. Play with this for a while and you’ll discover ways to get more sophisticated – just remember to share what you’re doing with the rest us!