Category Archives: Outcomes

Cause and effect in CME

There is rumor of a sacred mountain in Tibet, the peak of which can be only ascended when Jupiter, Mercury and Venus are in triangular alignment. At the summit, there lives a man who will provide the truth for any one question a plucky adventurer may pose. One day, I hope to be that adventurer. My question…is CME an effective means for impacting clinician competence, performance and (daresay) patient health?

nature-sky-sunset-man.jpeg

Unfortunately, the next anticipated triangular alignment isn’t until 2021. In the interim, I have to: 1) learn how to climb mountains and 2) go about establishing cause and effect the old-fashioned way.

To that end…If I want to argue that a relationship exists between CME and some effect (eg, competence gain), I must establish three things:

  • Temporal precedence: the effect comes after the presumed cause. For example, CME participants score better on a case-based, post-activity assessment than pre-activity. Pretty straightforward, right? Who needs a mountain guru?
  • Covariation: the effect is systematically (ie, not randomly) related to the presumed cause. For example, a high level of competence would be more likely among CME participants than non-participants and/or more CME participation would equal more competence than less CME participation. Wait…this sounds like a control group study. Didn’t we (ie, me in this conversation with myself) say control groups in CME are bunk? Okay, I exaggerated a skosh. Simple post-test only nonequivalent control group design (ie, surveys to participants and nonparticipants after a CME activity) is pretty much at the bottom of the research credibility scale, but there are more robust methods to employ control groups. I’ll cover these in a subsequent post.
  • Plausible alternatives: once both temporal precedence and covariation are established, all other possible explanations for the effect (ie, confounders) must be explored. This addresses the internal validity of your assessment (ie, how well it avoids confounding). I’ll talk about some threats to internal validity in a subsequent post. Until then, note there is no perfect study: interval validity exists on a spectrum. The more internally valid (ie, the less confounded), the more confident you can be in your interpretation of cause and effect.

In absence of divine wisdom, every CME outcome assessment should speak to these three factors. I’d say we do a pretty good job establishing temporal precedence, but it’s a rare occasion to discuss covariation or confounders. Next time you find yourself creating or reviewing an outcome report, take that opportunity to push us all forward a bit on these critical factors to establishing the value of CME.

Advertisements

Leave a comment

Filed under Causality, CME, Covariation, Internal validity, Outcomes, Temporal precedence, Uncategorized

Losing Control

CME has been walking around with spinach in its teeth for more than 10 years.  And while my midwestern mindset defaults to “don’t make waves”, I think it’s officially time to offer a toothpick to progress and pluck that pesky control group from the front teeth of our standard outcomes methodology.

That’s right, CME control groups are bunk. Sure, they make sense at first glance: randomized controlled trials (RCTs) use control groups and they’re the empirical gold standard.  However, as we’ll see, the magic of RCTs is the randomization, not the control: without the “R” the “C” falls flat.  Moreover, efforts to demographically-match controls to CME participants on a few simple factors (eg, degree, specialty, practice type and self-report patient experience) fall well short of the vast assemblage of confounders that could account for differences between these groups. In the end, only you can prevent forest fires and only randomization can ensure balance between samples.

So let’s dig into this randomization thing.  Imagine you wanted to determine the efficacy of a new treatment for detrimental modesty (a condition in which individuals are unable to communicate mildly embarrassing facts).  A review of clinical history shows that individuals who suffer this condition represent a wide range of race, ethnicity and socioeconomic strata, as well as vary in health metrics such as age, BMI and comorbidities.  Accordingly, you recruit a sufficient sample* of patients with this diagnosis and randomly designate them into two categories: 1) those who will receive the new treatment and 2) those who will receive a placebo.  The purpose of this randomization is to balance the factors that could confound the relationship you wish to examine (ie, treatment to outcome).  Assume the outcome of interest is likelihood to tell a stranger he has spinach in his teeth.  Is there a limit to the number of factors you can imagine that might influence an individual’s ability for such candor?  And remember, clinical history indicated that patients with detrimental modesty are diverse in regard social and physical characteristics.  How can you know that age, gender, height, religious affiliation, ethnicity or odontophobia won’t enhance or reduce the effect of your treatment?  If these factors are not evenly distributed across the treatment and control groups, your conclusion about treatment efficacy will be confounded.

So…you could attempt to match the treatment and control groups on all potential confounders or you could take the considerably less burdensome route and simply randomize your subjects into either group.  While all of these potential confounders still exist, randomization ensures that both the treatment and control group are equally “not uniform” across all these factors and therefore comparable.  It’s very important to note that the “control” group is simply what you call the population who doesn’t receive treatment.  The only reason it works is because of randomization.  Accordingly, simply applying a control group to your CME outcome assessment without randomization is like giving a broke man a wallet – it’s so not the thing that matters.

Now let’s bring this understanding to CME.  There are approximately, 18,000 oncology physicians in the United States.  In only two scenarios will the participants in your oncology-focused CME represent an unbiased sample of this population: 1) all 18,000 physicians participate or 2) at least 377 participate (sounds much more likely) that have been randomly sampled (wait…what?).  For option #2, the CME provider would require access to the entire population of oncology physicians from which they would apply a randomization scheme to create a sample based on their empirically expected response rate to invitations in order to achieve the 377 participation target.  Probably not standard practice.  If neither scenario applies to your CME activity, then the participants are a biased representation of your target learners.  Of note, biased doesn’t mean bad.  It just means that there are likely factors that differentiate your CME participants from the overall population of target learners and, most importantly, these factors could influence your target outcomes.  How many potential factors? Some CME researchers suggest more than 30.

Now think about a control group. Are you pulling a random sample of your target physician population?  See scenario #2 above.  Also, are you having any difficulty attracting physicians to participate in control surveys?  What’s your typical response rate?  Maybe you use incentives to help?  Does it seem plausible that the physicians who choose to respond to your control group surveys would be distinct from the overall physician population you hope they represent?  Do you think matching this control group to participants based on just profession, specialty, practice location and type is sufficient to balance these groups?  Remember, it not the control group, it’s the randomization that matters.  RCTs would be a lot less cumbersome if they only had to match comparison groups on four factors.  Of course, our resulting pharmacy would be terrifying.

So, based on current methods, we’re comparing a biased sample of CME participants to a biased sample of non-participants (control) and attributing any measured differences to CME exposure.  This is a flawed model.  Without balancing the inherent differences between these two samples, it is impossible to associate any measured differences in response to survey questions to any specific exposure.  So why are you finding significant differences (ie, P < .05) between groups?  Because they are different.  The problem is we have no idea why.

By what complicated method can we pluck this pesky piece of spinach?  Simple pre- versus post-activity comparison.  Remember, we want to ensure that confounding factors are balanced between comparison groups.  While participants in your CME activity will always be a biased representation of your overall target learner population, those biases are balanced when participants are used as their own controls (as in the pre- vs. post-activity comparison).  That is, both comparison groups are equally “non-uniform” in that they are comprised of the same individuals. In the end, you won’t know how participants differ from non-participants, but you will be able to associate post-activity changes to your CME.

Leave a comment

Filed under Best practices, CME, Confounders, Control groups, Needs Assessment, Outcomes, Power calculation, Pre vs. Post

Where did the knowledge go?

What does it mean when your CME participants score worse on a post-test assessment (compared to pre-test)?

Here are some likely explanations:

  1. The data was not statistically significant.  Significance testing determines whether we reject the null hypothesis (null hypothesis = pre- and post-test scores are equivalent).  If the difference was not significant (ie, P > .05), we can’t reject this assumption.  If the pre/post response was too low to warrant statistical testing, the direction of change is meaningless – you don’t have a representative sample.
  2. Measurement bias (specifically, “multiple comparisons”).  This measurement bias results from multiple comparisons being conducted within a single sample (ie, asking dozens of pre/post questions within a single audience).  The issue with multiple comparisons is that the more questions you ask, the more likely you are to find a significant difference where it shouldn’t exist (and wouldn’t if subject to more focused assessment).  Yes, this is a bias to which many CME assessments are subject.
  3. Bad question design. Did you follow key question development guidelines?  If not, the post-activity knowledge drop could be due to misinterpretation of the question.  You can learn more about question design principles here.

Leave a comment

Filed under Outcomes, question design, Statistical tests of significance

CME Outcomes Statistician, first grade

I was very excited to have my CMEPalooza session (Secrets of CME Outcome Assessment) officially sanctioned by the League of Assessors (LoA).  Accordingly, participants who passed the associated examination were awarded “CME Outcome Statistician, first grade” certifications.  It’s a grueling test, but three candidates made it through and received their certifications today (names withheld due to exclusivity).

Picture2

More good news…I petitioned the LoA to extend the qualifying exam for another six weeks (expiring May 29, 2015) and was officially approved!  So you can still view the CMEPalooza session (here) and then take the qualifying exam (sorry, exam is now closed). Good luck!

Leave a comment

Filed under CME, CMEpalooza, League of Assessors, Outcomes

Physician self-assessment questions

Let’s officially retire this pre/post-activity question:

<pre-activity> How would you rate your knowledge of X? (or the common variant: How confident are you in your ability to do X?)

<post-activity> After having participated in this activity, how would you rate your knowledge of X?  (or …how confident are you now in your ability to do X?)

First and foremost, it’s really lazy.  Second, we’ve known for long enough that physician self-assessments are reliably unreliable (Davis et al, 2006).   It’s better to ask no question, than a bad one.

2 Comments

Filed under CME, Outcomes, Self-assessment

Be patient on those outcomes

Oh, I so want to say I measure patient outcomes.  Everyone gets so excited.  Imagine these two presentation titles: 1) “Reliability and Validity in Educational Outcome Assessment” and 2) “Measuring Patient Outcomes Associated with CME Participation”.  Which one are you going to attend?  Well…yes, to most folks those both sound pretty boring.  But this is a CME blog.  And in this part of town, it’d be like asking whether you’d rather hang out with some guy who runs a strip mall accounting firm or Will Ferrell.

But I’m not Will Ferrell.  And instead of an accountant, I’d like to introduce you to Drs. Cook and West who present a very clear and thoughtful piece on  why Will Ferrell really isn’t that funny why patient outcomes may not be the best  CME outcome target (click here for the article).

Read this article and be prepared.  If you’re presenting on patient outcomes, I’m going to ask about things like “dilution” and “teaching-to-the-test”.  Unless, of course, you are Will Ferrell.  In which case, thank you for Elf.

2 Comments

Filed under CME, Outcomes, Patient Health, Reliability, Validity

Recipe for CME

How do you cook CME?  Maybe simmer KOL in a venue sauce and add enduring material to taste?  And how do you select your ingredients?  Are you a student of food theory or do you just feel your way through?

Well, I’m supposed to be scientifically-minded, so my pantry is full of evidence-based options.  Wait…did I say full?  I meant I know these four things:

  1. Live activities are more savory than print
  2. You’ll make a better soup with multi-media
  3. Multiple tastes are preferred to just one
  4. Case-based discussions are the most important seasoning

According to Marinopolous SS, et al. that’s all we’ve got to work with.  When you don’t know who’s coming to dinner, how hungry they are, or any of their possible dietary restrictions, you’ve got to make CME magic using only these four things. That’s pretty bleak.

Why don’t we know more?  Too few studies with no standardization and very little reliability or validity data to support findings.  Us outcome experts may all be wearing toques, but apparently only make french fries.

2 Comments

Filed under CME, Effectiveness, Outcomes