Ten years ago, the Journal of Continuing Education in the Health Professions published Achieving Desired Results and Improved Outcomes: Integrating Planning and Assessment Throughout Learning Activities, which quickly became known as the “Moore’s outcomes paper” (sorry, Green and Gallis). While applauded for several years, it has now become vogue to wave aside as an antiquated interpretation of physician learning outcomes.
Why a paper primarily focused on the design and implementation of educational interventions for clinicians was ever considered bedrock for outcome assessment, escapes me. Then again, maybe I shouldn’t be surprised that the learning objectives of a passive, print intervention would be so poorly translated to its target audience.
So what were Moore, Green and Gallis trying to communicate? Specifically…the central point of this article is that before outcomes can be measured, educational planning focused on the outcomes must occur so that these outcomes can be expected to happen (JCEHP 2009; 29: p. 5). To be fair, they do use the word “outcomes” a lot in that sentence, but the key terms are clearly “educational planning”. Overall, this was meant to be an instructional guide for planning continuing medical education (CME) – not an outcomes paper. Here are the key points:
- There may be five stages of physician learning
- If there are such stages, designing CME using the predisposing-enabling-reinforcement framework may be a good idea
- The seven level outcomes framework may help CME providers apply the predisposing-enabling-reinforcement framework
- Formative assessment is really important and can be incorporated in the predisposing-enabling-reinforcement framework
Noting a pattern here? Whole lotta chatter about the predisposing-enabling-reinforcement framework. As far as outcomes, the “Moore’s model” is a simple amalgam of frameworks – I suspect that neither Moore nor Green nor Gallis really care which framework you use, as long as it also incorporates…let’s all say it together now…the predisposing-enabling-reinforcement framework!
While I recognize that many a fine point has been made in criticism of the Moore’s outcomes framework as an independent entity (ie, outside of the context of the article in which it was published), my concern is that we’re tossing the baby with the bathwater for those newly initiated into the field of CME. Everyone in the practice of CME should read this paper. The insights neatly tucked into 15 pages may not instantly transform a CME providers’ practice, but they will at least help tune their attention to the evidence-based barriers and facilitators to transferring clinical education to practice.
What does it mean when your CME participants score worse on a post-test assessment (compared to pre-test)?
Here are some likely explanations:
- The data was not statistically significant. Significance testing determines whether we reject the null hypothesis (null hypothesis = pre- and post-test scores are equivalent). If the difference was not significant (ie, P > .05), we can’t reject this assumption. If the pre/post response was too low to warrant statistical testing, the direction of change is meaningless – you don’t have a representative sample.
- Measurement bias (specifically, “multiple comparisons”). This measurement bias results from multiple comparisons being conducted within a single sample (ie, asking dozens of pre/post questions within a single audience). The issue with multiple comparisons is that the more questions you ask, the more likely you are to find a significant difference where it shouldn’t exist (and wouldn’t if subject to more focused assessment). Yes, this is a bias to which many CME assessments are subject.
- Bad question design. Did you follow key question development guidelines? If not, the post-activity knowledge drop could be due to misinterpretation of the question. You can learn more about question design principles here.
I was very excited to have my CMEPalooza session (Secrets of CME Outcome Assessment) officially sanctioned by the League of Assessors (LoA). Accordingly, participants who passed the associated examination were awarded “CME Outcome Statistician, first grade” certifications. It’s a grueling test, but three candidates made it through and received their certifications today (names withheld due to exclusivity).
More good news…I petitioned the LoA to extend the qualifying exam for another six weeks (expiring May 29, 2015) and was officially approved! So you can still view the CMEPalooza session (here) and then take the qualifying exam (sorry, exam is now closed). Good luck!
Let’s officially retire this pre/post-activity question:
<pre-activity> How would you rate your knowledge of X? (or the common variant: How confident are you in your ability to do X?)
<post-activity> After having participated in this activity, how would you rate your knowledge of X? (or …how confident are you now in your ability to do X?)
First and foremost, it’s really lazy. Second, we’ve known for long enough that physician self-assessments are reliably unreliable (Davis et al, 2006). It’s better to ask no question, than a bad one.
Oh, I so want to say I measure patient outcomes. Everyone gets so excited. Imagine these two presentation titles: 1) “Reliability and Validity in Educational Outcome Assessment” and 2) “Measuring Patient Outcomes Associated with CME Participation”. Which one are you going to attend? Well…yes, to most folks those both sound pretty boring. But this is a CME blog. And in this part of town, it’d be like asking whether you’d rather hang out with some guy who runs a strip mall accounting firm or Will Ferrell.
But I’m not Will Ferrell. And instead of an accountant, I’d like to introduce you to Drs. Cook and West who present a very clear and thoughtful piece on
why Will Ferrell really isn’t that funny why patient outcomes may not be the best CME outcome target (click here for the article).
Read this article and be prepared. If you’re presenting on patient outcomes, I’m going to ask about things like “dilution” and “teaching-to-the-test”. Unless, of course, you are Will Ferrell. In which case, thank you for Elf.
How do you cook CME? Maybe simmer KOL in a venue sauce and add enduring material to taste? And how do you select your ingredients? Are you a student of food theory or do you just feel your way through?
Well, I’m supposed to be scientifically-minded, so my pantry is full of evidence-based options. Wait…did I say full? I meant I know these four things:
- Live activities are more savory than print
- You’ll make a better soup with multi-media
- Multiple tastes are preferred to just one
- Case-based discussions are the most important seasoning
According to Marinopolous SS, et al. that’s all we’ve got to work with. When you don’t know who’s coming to dinner, how hungry they are, or any of their possible dietary restrictions, you’ve got to make CME magic using only these four things. That’s pretty bleak.
Why don’t we know more? Too few studies with no standardization and very little reliability or validity data to support findings. Us outcome experts may all be wearing toques, but apparently only make french fries.
Commitment to change (CTC) questions are the caboose of every post-activity CME evaluation – stripped of all relevancy and sustained solely by nostalgia. Thirty years since its introduction, we can now all retire this method, confident that it has served us well, but that it’s now time for something more…app-ish. And off it goes, grumbling it’s final words toward obscurity: “…but, you never really knew me”.
Before you dismiss CTC, check out this article. People have been studying CTC for a long time. And there’s value to this approach – assuming you use it correctly. Should you use a follow-up survey? When? How? How should you word the questions? Include a rating scale? And how should you sort through and interpret the results? This stuff all matters. And you won’t find an easier to digest summary than this 2010 article in Evaluation & the Health Professions.
So, yes, if you’re simply maintaining a “what are you going to change in your practice” question at the end of every CME evaluation – definitely send that packing. Then read the aforementioned article. You’ll find that CTC has limitations, but when done in accordance with the latest evidence, there’s a lot of good data to be had.