Tag Archives: case vignettes

Effect size kryptonite

I’ve talked a lot about effect size: what it is (here), how to calculate it (here, here and here), what to do with the result (here and here)…and then some about limitations (here).  Overall, I’ve been trying to convince you that effect size is a sound (and simple) approach to quantifying the magnitude of CME effectiveness.  Now it’s time to talk about how it may be total garbage.

All this effect size talk includes the supposition that the data from which it is calculated is both reliable and valid.  In CME, the data source is overwhelming survey – and the questions within typically include self-efficacy scales, single-correct answer knowledge tests and / or case vignettes.  But how do you know that your survey questions actually measure their intention (validity) and do so with consistency (reliability)?  CME has been repeatedly dinged for not using validated measurement tools.  And if your survey isn’t valid (or reliable), why would your data be worth anything?  Effect size does not correct for bad questions.  So maybe next time you’re touting a great effect size (or trying to bury a bad one), you should also consider (and be able to document) whether you’ve demonstrated the effectiveness of your CME or the ineffectiveness of your survey.

So what can be done?  Well, you can hire a psychometrist and add complicated-sounding things like “factor analysis” and “Cronbach’s alpha” to your lexicon (yell those out during the next CME presentation you attend…and then quickly run of the room).  Or (actually “and”), you can start with sound question-design principles.  The key thing to note, no amount of complex statistics can make a bad question good – so you need to know the fundamentals of assessing knowledge and competence in medical education.  Where do you get those?  Here are some suggestions to get you started:

  • Take the National Board of Medical Examiners (NBME) U course entitled: Assessment Principles, Methods, and Competency Framework.  This is an awesome (daresay, the best) resource for anyone assessing knowledge and competence in medical education.  Complete this course (there are 20 lessons, each under 30 minutes) and you’ll be as expert as anyone in CME.  You can register here.  And it’s free!
  • Check out Dr. Wendy Turell’s session entitled Tips to Make You a Survey Measurement Rock Star during the next CMEpalooza (April 8th at 1:30 eastern).  This is her wheelhouse – so steal every bit of her expertise you can.  Once again, it’s free.

2 Comments

Filed under ACCME, CMEpalooza, Item writing, question design, Reliability, Validity

Script Concordance Tests: where have you been hiding?

When I saw the  JCEHP editorial title lead with “How Significant is Statistical Significance…” I knew I’d be blogging about it.  As I remember the progression through graduate school statistic courses, it began with learning how to select the appropriate significance test, progressed to application and then concluded with all the reasons why the results didn’t really mean much.  So I was ready to build a “cut-and-paste” blog post out of old class papers detailing an unhealthy dependence on the results of statistical tests (which I  expected to be the opinion of this editorial).  And that would have worked fine, but then I found a rabbit hole: script concordance test (SCTs).

Casually introduced by the authors via an educational scenario illustrating the limitations of statistical significance, SCT is a case-based assessment method designed to measure the clinical decision-making process (as opposed to simply identifying whether someone knows a correct diagnosis or treatment).  As educators, this could be quite helpful in clarifying educational gaps.  For evaluators, this approach has some encouraging validity data.  I’ve got a way to go before I can even claim familiarity with SCTs, but will be diving into the literature immediately (and assuming expert status by hopefully next week).  If anyone else is interested, here are some suggestions to learn more:

  1. Fournier JP, Demeester A, Charlin B. Script concordance tests: guidelines for construction. BMC Med Inform Decis Mak 2008;8:18. (click here for full article)
  2. Charlin B, Roy L, Brailovsky C, Goulet F, van der Vleuten C. The script concordance test: A tool to assess the reflective clinician. Teach Learn Med 2000; 12:189-195. (click here for abstract)
  3. Dory V, Gagnon R, Dominique V, Charlin B: How to construct and implement script concordance tests: insights from a systematic review. Med Educ 2012, 46:552–563. (click here for full article)
  4. Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten C: Script concordance method: a review of published validity evidence. Med Educ 2011, 45:329–338. (click here for full article)

FYI – it turns out SCTs were introduced in the late 1990s.  So I’m less than 20 years behind the curve, and perfectly in tune with the traditional adoption curve of evidence to clinical practice (which hovers around 17 years).

Leave a comment

Filed under Case vignettes, CME, Script concordance tests, Statistical tests of significance, Statistics