Category Archives: Summative assessment

Give me Moore

Ten years ago, the Journal of Continuing Education in the Health Professions published Achieving Desired Results and Improved Outcomes: Integrating Planning and Assessment Throughout Learning Activities, which quickly became known as the “Moore’s outcomes paper” (sorry, Green and Gallis). While applauded for several years, it has now become vogue to wave aside as an antiquated interpretation of physician learning outcomes.

Why a paper primarily focused on the design and implementation of educational interventions for clinicians was ever considered bedrock for outcome assessment, escapes me. Then again, maybe I shouldn’t be surprised that the learning objectives of a passive, print intervention would be so poorly translated to its target audience.

So what were Moore, Green and Gallis trying to communicate? Specifically…the central point of this article is that before outcomes can be measured, educational planning focused on the outcomes must occur so that these outcomes can be expected to happen (JCEHP 2009; 29: p. 5). To be fair, they do use the word “outcomes” a lot in that sentence, but the key terms are clearly “educational planning”. Overall, this was meant to be an instructional guide for planning continuing medical education (CME) – not an outcomes paper. Here are the key points:

  1. There may be five stages of physician learning
  2. If there are such stages, designing CME using the predisposing-enabling-reinforcement framework may be a good idea
  3. The seven level outcomes framework may help CME providers apply the predisposing-enabling-reinforcement framework
  4. Formative assessment is really important and can be incorporated in the predisposing-enabling-reinforcement framework

Noting a pattern here? Whole lotta chatter about the predisposing-enabling-reinforcement framework.  As far as outcomes, the “Moore’s model” is a simple amalgam of frameworks – I suspect that neither Moore nor Green nor Gallis really care which framework you use, as long as it also incorporates…let’s all say it together now…the predisposing-enabling-reinforcement framework!

While I recognize that many a fine point has been made in criticism of the Moore’s outcomes framework as an independent entity (ie, outside of the context of the article in which it was published), my concern is that we’re tossing the baby with the bathwater for those newly initiated into the field of CME. Everyone in the practice of CME should read this paper. The insights neatly tucked into 15 pages may not instantly transform a CME providers’ practice, but they will at least help tune their attention to the evidence-based barriers and facilitators to transferring clinical education to practice.

Advertisements

Leave a comment

Filed under CME, Formative assessment, JCEHP, Methodology, Outcomes, Predisposing-enabling-reinforcement, Summative assessment

Writing questions good

Although I’ve complained a fair bit about validity and reliability issues in CME assessment, I haven’t offered much on this blog to actually address these concerns. Well, the thought of thousands (and thousands and…) of dear and devoted readers facing each new day with the same, tired CME assessment questions has become too much to bear. That, and I was recently required to do a presentation on guidelines and common flaws in the creation of multiple-choice questions…so I thought I’d share it here.

I’d love to claim these pearls are all mine, but they’re just borrowed.  Nevertheless, this slide deck may serve as a handy single-resource when constructing your next assessment (and it contains some cool facts about shark attacks).

1 Comment

Filed under Best practices, CME, MCQs, multiple-choice questions, Reliability, Summative assessment, Survey, survey design, Validity

Formative Assessment

Outcomes assessment is “summative”, which is fancy for measures whether desired results have been achieved.  A “formative” assessment, however, addresses something while in development to be sure it’s on track.  Moore et al (2009) make a strong case for formative assessment in CME, but leave the “how-to” details to our imagination (I guess when you’re covering every aspect of CME you need to leave a few bits out).

Here’s one recipe for formative assessment (for live CME activities):

  1. Have your course faculty develop knowledge and/or case vignette questions relative to their pending talks
  2. Turn these questions into a web-based survey (www.SurveyMonkey.com)
  3. At least two weeks prior to the activity date, email the survey to all activity registrants
  4. Share the registrants’ responses with your course faculty
  5. Adjust the pending talks accordingly

If you feel the need to incentivize respondents (which I never discourage), offer them a discount off registration for another activity.  If you want more detail, check out this short JCEHP article.

I’ve used this approach a few times and it’s been generally successful (i.e., good response rate and faculty have used some of the data to modify their presentations).  However, I don’t want to pretend this approach is “setting-the-bar” for formative assessment.  If you’re not doing any such assessment, this is a good way to get started.  Play with this for a while and you’ll discover ways to get more sophisticated – just remember to share what you’re doing with the rest us!

1 Comment

Filed under Formative assessment, Methodology, Needs Assessment, Summative assessment