The ACCME just released an updated synthesis of published systematic reviews regarding the effectiveness of CME. You can find it here. In short, the authors offer the following conclusions (this is pulled verbatim from the report on p. 14):
- CME does improve physician performance and patient health outcomes;
- CME has a more reliably positive impact on physician performance than on patient health outcomes; and
- CME leads to greater improvement in physician performance and patient health if it is more interactive, uses more methods, involves multiple exposures, is longer, and is focused on outcomes that are considered important by physicians.
Yes, there are issues of validity, heterogeneity, standardization and good-ole-fashioned publication bias in CME research, but that aside, there’s enough evidence out there to comfortably assume CME can positively affect physician performance and patient health. While that’s good news, we can’t ignore the next question: Why is it effective?
To borrow another section from this report (p. 15):
The authors of the systematic reviews make clear that the research regarding mechanisms of action by which CME improves physician performance and patient health outcomes is in the early stages and needs greater theoretical and methodological sophistication. Several authors make the argument that future research must take account of the wider social, political, and organizational factors that play a role in physician performance and patient health outcomes.
The third bullet point above shines some light on these “mechanisms of action”, but the recipe for effective CME is still vague. For example….How do I make my activity more interactive? More importantly, what qualifies as interactive in the first place? If multiple exposures is better, how many, and at what intensity? How effective are these “mechanisms of action” across various physician audiences? Do oncologists and internists learn the same way? What internal and external (e.g., practice environment) factors are influential?
There’s several careers worth of research questions here. Anyone funding?
How do you cook CME? Maybe simmer KOL in a venue sauce and add enduring material to taste? And how do you select your ingredients? Are you a student of food theory or do you just feel your way through?
Well, I’m supposed to be scientifically-minded, so my pantry is full of evidence-based options. Wait…did I say full? I meant I know these four things:
- Live activities are more savory than print
- You’ll make a better soup with multi-media
- Multiple tastes are preferred to just one
- Case-based discussions are the most important seasoning
According to Marinopolous SS, et al. that’s all we’ve got to work with. When you don’t know who’s coming to dinner, how hungry they are, or any of their possible dietary restrictions, you’ve got to make CME magic using only these four things. That’s pretty bleak.
Why don’t we know more? Too few studies with no standardization and very little reliability or validity data to support findings. Us outcome experts may all be wearing toques, but apparently only make french fries.
Imagine you’ve created a survey to assess changes in competence among participants in a CME activity (pre- vs. post-participation). Let’s say there’s a total of eight questions in your survey (each one asking participant to self-report their utilization of a specific clinical task related to the CME activity content on a 5-point scale). One week after the CME activity is completed, you’re staring at an Excel spreadsheet containing all of the pre- and post-activity responses. Now what? How do you use this data to determine whether your activity was effective? And how do you compare this data to the results of other outcome assessments?
If this was an infomercial, I’d now break into scene (colorfully sweatered) and introduce the latest advancement in outcome measurement: Cohen’s d. It slices, it dices…it reduces all of your outcomes data into a single metric that summarizes the overall effectiveness of your CME. But wait, there’s more…you can use Cohen’s d to compare the effectiveness of your CME across activities. That’s right. Want to know how effective this year’s conference was relative to last year? Cohen’s d. Want to know how effective your live CME activity was relative to its repurposed enduring material? Cohen’s d. Want to know the effectiveness of your overall CME program by topic? Or by format? That’s right, Cohen’s d.
Although this may sound like an innovation, Cohen’s d has been around for decades. It’s even been used in CME; however, this has largely been restricted to academic publications. But the claims made above are true. Cohen’s d can answer the following questions: 1) Was my CME effective? 2) How effective was it? And 3) how effective was my CME relative to other CME activities? Better still, it’s remarkably simple to calculate – if you can calculate a mean and a standard deviation (not by hand, of course, use Excel), then you can calculate a Cohen’s d.
I’m going to dedicate the next two blog posts to Cohen’s d. The first will provide step-by-step instructions of how to calculate it and interpret the results. The next will demonstrate how you can use Cohen’s d to assess the effectiveness of your CME beyond the individual activity (i.e., by topic, or format, or year). And I promise, you’ll be amazed at how easy this is to do.
Want to be the smartest CME person in the room? Better be able to answer these four questions:
- What is the relative effectiveness of each CME delivery method?
- How sustainable are changes in physician competence/practice or patient outcomes resulting from CME participation?
- What audience characteristics influence the effectiveness of CME?
- How good of a job have we been doing in assessing CME outcomes?
All the answers are here: http://www.ahrq.gov/downloads/pub/evidence/pdf/cme/cme.pdf