I was very excited to have my CMEPalooza session (Secrets of CME Outcome Assessment) officially sanctioned by the League of Assessors (LoA). Accordingly, participants who passed the associated examination were awarded “CME Outcome Statistician, first grade” certifications. It’s a grueling test, but three candidates made it through and received their certifications today (names withheld due to exclusivity).
More good news…I petitioned the LoA to extend the qualifying exam for another six weeks (expiring May 29, 2015) and was officially approved! So you can still view the CMEPalooza session (here) and then take the qualifying exam (sorry, exam is now closed). Good luck!
On Tuesday, Chicago will decide on either Rahm on Chuy. But Wednesday, it’s all about CMEPalooza. Thank you to our industry’s “Jane’s Addiction” for organizing the third installment of this CME free-for-all. I’ll be presenting on CME outcomes assessment (11 AM eastern). My session is designed for those that fall into the following categories:
- Regularly use surveys to measure learning and competence change
- No formal process for reviewing survey questions
- Unsure of how to utilize statistical tests
Oh, but there’s more…this session has been accredited by the apocryphal League of CME Assessors (sorry, can’t provide a link due to exclusivity). If, after completing the session, you wish to be considered for eligibility as “CME Outome Statistician, first grade”, click here (sorry, this test is now closed) to take their test. There’s even a certificate if you pass. Good luck!
I’ve talked a lot about effect size: what it is (here), how to calculate it (here, here and here), what to do with the result (here and here)…and then some about limitations (here). Overall, I’ve been trying to convince you that effect size is a sound (and simple) approach to quantifying the magnitude of CME effectiveness. Now it’s time to talk about how it may be total garbage.
All this effect size talk includes the supposition that the data from which it is calculated is both reliable and valid. In CME, the data source is overwhelming survey – and the questions within typically include self-efficacy scales, single-correct answer knowledge tests and / or case vignettes. But how do you know that your survey questions actually measure their intention (validity) and do so with consistency (reliability)? CME has been repeatedly dinged for not using validated measurement tools. And if your survey isn’t valid (or reliable), why would your data be worth anything? Effect size does not correct for bad questions. So maybe next time you’re touting a great effect size (or trying to bury a bad one), you should also consider (and be able to document) whether you’ve demonstrated the effectiveness of your CME or the ineffectiveness of your survey.
So what can be done? Well, you can hire a psychometrist and add complicated-sounding things like “factor analysis” and “Cronbach’s alpha” to your lexicon (yell those out during the next CME presentation you attend…and then quickly run of the room). Or (actually “and”), you can start with sound question-design principles. The key thing to note, no amount of complex statistics can make a bad question good – so you need to know the fundamentals of assessing knowledge and competence in medical education. Where do you get those? Here are some suggestions to get you started:
- Take the National Board of Medical Examiners (NBME) U course entitled: Assessment Principles, Methods, and Competency Framework. This is an awesome (daresay, the best) resource for anyone assessing knowledge and competence in medical education. Complete this course (there are 20 lessons, each under 30 minutes) and you’ll be as expert as anyone in CME. You can register here. And it’s free!
- Check out Dr. Wendy Turell’s session entitled Tips to Make You a Survey Measurement Rock Star during the next CMEpalooza (April 8th at 1:30 eastern). This is her wheelhouse – so steal every bit of her expertise you can. Once again, it’s free.
Don’t forget to check out CMEPalooza this Wednesday (Oct 15th) – it starts at 9 AM eastern. I’d like to catch all seven sessions, but I’m particularly interested in the 11 AM set: Death of the MECC – Fact or Fiction? If it’s fact, I guess I’m sleeping in on Thursday.
Filed under CME, CMEpalooza
Statistics can help answer important questions about your CME. For example, was there an educational effect and, if so, how big was it? The first question is typically answered with a P value and the second with an effect size.
If this were 10 years ago, you’d either be purchasing some expensive statistical software or hiring a consultant to answer these questions. Today (thank you Internet), it’s simple and basically free.
A step-by-step approach can be found here.
CMEPalooza will be on Thursday March 20 and Friday March 21. Like the annual professional meeting for CME (Alliance for Continuing Education in the Health Professions), CMEpalooza is a collection of “best practice” talks. Unlike the Alliance, the entire event will be online, archived and free. A big thank you to Derek Warnick (aka “the CME Guy“) for putting this all together.
Based on the agenda (of 21 presentations), there are many promising talks ranging from audience recruitment, adult learning theory, linking educational objectives with outcomes, qualitative analysis, and measuring patient outcomes (I’ll be representing Imedex with a presentation on statistical analysis in CME outcomes). Regardless of your scope of work, I suspect there will be at least one presentation in the agenda of interest.
If you can’t participate live, no worries, everything will be archived, so view at your convenience – but make sure to check it out.