Category Archives: Internal validity

Outcomes test drive

australia-162760_1920(broke car)

I bought my first car at 16. It was an awesome little blue 4×4 (Bronco II). The test drive was perfect. I got to blast the radio and drive off-road through a sub-division under construction. Bouncing over piles of debris, I can still remember the exhilaration. Both the seller and I laughed the whole time. Only problem…he was still laughing two weeks later, while I was on the side of the highway spitting steam and pouring oil mixed with engine coolant. That 4×4 rusted in my driveway for another year before a neighbor bought it for less than 20% of what I paid.

Yeah…I skipped the inspection part. It was just too much fun to think about that. And since it handled the test drive, what could really go wrong? I was going to be so freakin’ cool come fall in high school.

Tell me I’m the only one who’s ever dreamed of the stars and ended up on the bus.

Now that brings us to outcomes. Maybe you’ve been kicking the tires of a new CME program and hoping it will generate great outcomes? Don’t get distracted by the shiny bits…there are three key things to inspect for every outcomes project (in descending order of importance and ascending in order of coolness):

  1. Study design: the main concern here is “internal validity”, which refers to how well a study controls for the factors that could confound the relationship between the intervention and outcome (ie, how do we know something else isn’t accelerating or breaking our path toward the desired outcome?). There are many threats to internal validity and correspondingly, many distinct study designs to address them. One group pretest-posttest is a study design, so is posttest only with nonequivalent groups (ie, post-test administered to CME participants and a non-participant “control” group). There are about a dozen more options. You should understand why a particular study design was selected and what answers it can (and cannot) provide.


  1. Data collection: second to study design, is data collection. The big deal here is “construct validity” (ie, can the data collection tool measure what it claims?). Just because you want your survey or chart abstraction to measure a certain outcome, doesn’t mean it actually will. Can you speak to the data that supports the effectiveness of your tool in measuring its intention? If not, you should consider another option. Note: it is really fun to say “chart abstraction”, but it’s a data collection tool, not a study design. If your study design is flawed, you have to consider those challenges to internal validity plus any construct validity issues associated with your chart abstraction. The more issues you collect, the weaker your final argument regarding your desired outcome. An expensive study (eg, chart review) does not guarantee a result of any importance, but it does sound good.


  1. Analysis: this is the shiny bit, and just like your parents told you, the least important. Remember Mom’s advice: if your friends don’t think you’re cool, then they aren’t really your friends. Well, think about study design and data collection as the “beauty on the inside” and analysis as a really groovy jacket and great hair. Oh yeah, it matters, but rather less so if they keep getting you stuck on the highway. You may have heard statisticians are nerds, but they’re the NASCAR drivers of the research community – and I’m here to tell you the car and pit crew are more important. In short, if your outcomes are all about analysis, they probably aren’t worth much.


Filed under CME, Confounders, Construct validity, Internal validity, Methodology, Uncategorized

Cause and effect in CME

There is rumor of a sacred mountain in Tibet, the peak of which can be only ascended when Jupiter, Mercury and Venus are in triangular alignment. At the summit, there lives a man who will provide the truth for any one question a plucky adventurer may pose. One day, I hope to be that adventurer. My question…is CME an effective means for impacting clinician competence, performance and (daresay) patient health?


Unfortunately, the next anticipated triangular alignment isn’t until 2021. In the interim, I have to: 1) learn how to climb mountains and 2) go about establishing cause and effect the old-fashioned way.

To that end…If I want to argue that a relationship exists between CME and some effect (eg, competence gain), I must establish three things:

  • Temporal precedence: the effect comes after the presumed cause. For example, CME participants score better on a case-based, post-activity assessment than pre-activity. Pretty straightforward, right? Who needs a mountain guru?
  • Covariation: the effect is systematically (ie, not randomly) related to the presumed cause. For example, a high level of competence would be more likely among CME participants than non-participants and/or more CME participation would equal more competence than less CME participation. Wait…this sounds like a control group study. Didn’t we (ie, me in this conversation with myself) say control groups in CME are bunk? Okay, I exaggerated a skosh. Simple post-test only nonequivalent control group design (ie, surveys to participants and nonparticipants after a CME activity) is pretty much at the bottom of the research credibility scale, but there are more robust methods to employ control groups. I’ll cover these in a subsequent post.
  • Plausible alternatives: once both temporal precedence and covariation are established, all other possible explanations for the effect (ie, confounders) must be explored. This addresses the internal validity of your assessment (ie, how well it avoids confounding). I’ll talk about some threats to internal validity in a subsequent post. Until then, note there is no perfect study: interval validity exists on a spectrum. The more internally valid (ie, the less confounded), the more confident you can be in your interpretation of cause and effect.

In absence of divine wisdom, every CME outcome assessment should speak to these three factors. I’d say we do a pretty good job establishing temporal precedence, but it’s a rare occasion to discuss covariation or confounders. Next time you find yourself creating or reviewing an outcome report, take that opportunity to push us all forward a bit on these critical factors to establishing the value of CME.

Leave a comment

Filed under Causality, CME, Covariation, Internal validity, Outcomes, Temporal precedence, Uncategorized