I’ve talked a lot about effect size: what it is (here), how to calculate it (here, here and here), what to do with the result (here and here)…and then some about limitations (here). Overall, I’ve been trying to convince you that effect size is a sound (and simple) approach to quantifying the magnitude of CME effectiveness. Now it’s time to talk about how it may be total garbage.
All this effect size talk includes the supposition that the data from which it is calculated is both reliable and valid. In CME, the data source is overwhelming survey – and the questions within typically include self-efficacy scales, single-correct answer knowledge tests and / or case vignettes. But how do you know that your survey questions actually measure their intention (validity) and do so with consistency (reliability)? CME has been repeatedly dinged for not using validated measurement tools. And if your survey isn’t valid (or reliable), why would your data be worth anything? Effect size does not correct for bad questions. So maybe next time you’re touting a great effect size (or trying to bury a bad one), you should also consider (and be able to document) whether you’ve demonstrated the effectiveness of your CME or the ineffectiveness of your survey.
So what can be done? Well, you can hire a psychometrist and add complicated-sounding things like “factor analysis” and “Cronbach’s alpha” to your lexicon (yell those out during the next CME presentation you attend…and then quickly run of the room). Or (actually “and”), you can start with sound question-design principles. The key thing to note, no amount of complex statistics can make a bad question good – so you need to know the fundamentals of assessing knowledge and competence in medical education. Where do you get those? Here are some suggestions to get you started:
- Take the National Board of Medical Examiners (NBME) U course entitled: Assessment Principles, Methods, and Competency Framework. This is an awesome (daresay, the best) resource for anyone assessing knowledge and competence in medical education. Complete this course (there are 20 lessons, each under 30 minutes) and you’ll be as expert as anyone in CME. You can register here. And it’s free!
- Check out Dr. Wendy Turell’s session entitled Tips to Make You a Survey Measurement Rock Star during the next CMEpalooza (April 8th at 1:30 eastern). This is her wheelhouse – so steal every bit of her expertise you can. Once again, it’s free.
The ACCME just released an updated synthesis of published systematic reviews regarding the effectiveness of CME. You can find it here. In short, the authors offer the following conclusions (this is pulled verbatim from the report on p. 14):
- CME does improve physician performance and patient health outcomes;
- CME has a more reliably positive impact on physician performance than on patient health outcomes; and
- CME leads to greater improvement in physician performance and patient health if it is more interactive, uses more methods, involves multiple exposures, is longer, and is focused on outcomes that are considered important by physicians.
Yes, there are issues of validity, heterogeneity, standardization and good-ole-fashioned publication bias in CME research, but that aside, there’s enough evidence out there to comfortably assume CME can positively affect physician performance and patient health. While that’s good news, we can’t ignore the next question: Why is it effective?
To borrow another section from this report (p. 15):
The authors of the systematic reviews make clear that the research regarding mechanisms of action by which CME improves physician performance and patient health outcomes is in the early stages and needs greater theoretical and methodological sophistication. Several authors make the argument that future research must take account of the wider social, political, and organizational factors that play a role in physician performance and patient health outcomes.
The third bullet point above shines some light on these “mechanisms of action”, but the recipe for effective CME is still vague. For example….How do I make my activity more interactive? More importantly, what qualifies as interactive in the first place? If multiple exposures is better, how many, and at what intensity? How effective are these “mechanisms of action” across various physician audiences? Do oncologists and internists learn the same way? What internal and external (e.g., practice environment) factors are influential?
There’s several careers worth of research questions here. Anyone funding?
CMEPalooza will be on Thursday March 20 and Friday March 21. Like the annual professional meeting for CME (Alliance for Continuing Education in the Health Professions), CMEpalooza is a collection of “best practice” talks. Unlike the Alliance, the entire event will be online, archived and free. A big thank you to Derek Warnick (aka “the CME Guy“) for putting this all together.
Based on the agenda (of 21 presentations), there are many promising talks ranging from audience recruitment, adult learning theory, linking educational objectives with outcomes, qualitative analysis, and measuring patient outcomes (I’ll be representing Imedex with a presentation on statistical analysis in CME outcomes). Regardless of your scope of work, I suspect there will be at least one presentation in the agenda of interest.
If you can’t participate live, no worries, everything will be archived, so view at your convenience – but make sure to check it out.
If you’re worried about how your CME Journal Club and/or Case Conference jives with the 2006 Updated ACCME criteria…this article has some great recommendations (abstract).