ASCO 2013 Annual Meeting – bring your CME outcome questions

Once again, we’re exhibiting at the ASCO Annual Meeting in Chicago.  I could pretend that staffing a booth is awesome, but I suspect you know better.  So if you stop by and brighten my day with some complex outcome questions, I promise to let you in on some of Chicago’s secrets (yes, there is so much more than pizza, Michigan Ave and Wrigley Field).

The exhibit hall is open from Saturday, June 1 to Monday, June 3 (9-5).  See you there!

So what’s a good effect size, part II

In the previous post, I promised to provide effect size data for activities we’ve produced over the past two years.  So here you go (note, knowledge and competence effect sizes were calculated using pre/post questions – multiple choice “test” questions for knowledge outcome and case-based questions for competence outcome):

Activity Format	Knowledge Effect Size (# of activities)	Competence Effect Size (# of activities)
Live activities
Multi-day conference: Consists of proprietary meetings with two or more days of education.  Meetings typically include didactic presentations, case-based discussions, interactive debates, and question and answer sessions.  Audience response technology is utilized to enhance interactivity and capture educational outcomes.	.51 (16)	.34 (19)
One-day conference: Consists of proprietary meetings with one day of education. Meetings typically include didactic presentations, case-based discussions, interactive debates, and question and answer sessions.  Audience response technology is utilized to enhance interactivity and capture educational outcomes.	.71 (10)	.42 (14)
Symposium / Dinner Series: Meetings typically include didactic presentations, case-based discussions, and question and answer sessions.  Audience response technology is utilized to enhance interactivity and capture educational outcomes.	.67 (11)	.41 (20)
e-Learning
Interactive Case Series: Each case (in a typical three-part, weekly series) contains four multimedia components: case presentation, Q&A, real-time results (chart), and expert commentary (audio/slides). An audio podcast, slides, and transcript of each case are also made available for download.	1.11 (11)	.40 (10)
New Frontiers: Webcast video program that features a one-on-one interview between a nationally recognized thought leader and a community oncologist probing the key issues that affect and impact clinical practice in the community. A series of questions are posed to the thought leader, whose response includes PowerPoint slides illustrating the points to be made and the data described.	1.46 (7)	.59 (6)
DocTalk / Investigator Insights: DocTalk is an online streaming video talk show filmed in studio and composed of four segments: introduction, interview with therapy area expert, round table discussion/debate and case review.Investigator Insights is an online activity that is comprised of an interview with a nationally renowned expert. The interview is built around treatment decisions drawn from emerging practice trends and evidence-based medical literature.	.62 (7)	.32 (7)
OncoReader / MDPracticeGuideTM: OncoReader features evidence-based commentary and discussion by expert faculty interspersed with patient cases, wherever applicable, to reinforce learning. The video and audio are enhanced by medical animations that support key points of learning throughout the activity.MDPracticeGuide™ is a multi-format educational resource featuring:  an Executive Summary, a disease state animation, slide presentation, case studies, treatment and diagnostic algorithms, physician self-evaluations, patient education resources and a discussion forum.	1.38 (6)	.48 (5)
Webinars: Webinars are broadcast live on the web to an extensive audience of healthcare professionals. Webinars include slide presentations with audio narration by a faculty expert interspersed with polling questions to encourage reflection and reinforce learning, as well as interactive discussion and Q&A with participants via the internet. Webinars are subsequently archived and made available on the Imedex E-Learning Center at elc.imedex.com.	1.04 (4)	.57 (7)

So what’s a good effect size for CME?

You have been calculating an effect size for each of your CME activities, right?  And now you have a database full of activities with corresponding effect sizes for say, knowledge and competence outcomes.  Sound familiar?  Anyone…anyone…Bueller?  Okay, for the one straggler, here’s a refresher:

1. What is effect size? (link)
2. How to calculate effect size (link)
3. Reporting effect size (link)
4. Effect size – other methodologic/statistical considerations (link)

Now that we’re all on the same page, let’s move on to the next question…what exactly is a “good” effect size?  Well, you would first start with Cohen (Cohen. J. [1988]. Statistical power analysis for the behavioral sciences [2nd ed.]. Hillsdale, NJ: Lawrence Earlbaum Associates), who identified the following general benchmarks: 0.2 = small effect, 0.5 = medium effect, and 0.8 = large effect.  Although effect size is relatively new to CME, thankfully more specific effect size data is available.  Starting with recent literature (specifically, meta-analyses), the following effect sizes have been reported:

• Competence effect size (live activities) = .85 (Drexel et al, 2011)
• Knowledge effect size (live activities) = .6 (Mansouri & Lockyer, 2007)
• Knowledge effect size (eLearning) = .82 (Casebeer et al, 2011)

It’s important to note that these effect sizes are the result of mixed measurement methods (and that measurement approach influences effect size), but they are certainly more relevant than Cohen’s benchmarks (and we know that Cohen wouldn’t take offense, because refining effect sizes through repeated measurement in a given area is exactly what he recommended).

In regard to repeated measurement, we have been measuring knowledge- and competence-level effect sizes for a variety of CME activities over the past two years.  In the next post, I’ll be publishing our effect size results for a variety of live and enduring material formats.  I’d love to hear how these results jive with your findings.

Outcomes session at Alliance 2013

According to the Bay Citizen, taxi passengers registered 1,733 complaints with the city’s 311 complaint line from July 1, 2011 to June 30, 2012 (a 13 percent increase from the previous fiscal year and nearly double the goal set by the San Francisco Municipal Transportation Agency, which regulates taxis).  So if you’re going to the Alliance this year, maybe you should keep close to conference hotel.

And hey, if you’re stuck near the conference, might as well stop by the NAMEC-sponsored outcome session on Wednesday night (Jan 30th at 8 PM).  We’ve got a great panel with representatives from MECs, industry and an independent outcome assessment company.  You can find more detail here.

Formative Assessment

Outcomes assessment is “summative”, which is fancy for measures whether desired results have been achieved.  A “formative” assessment, however, addresses something while in development to be sure it’s on track.  Moore et al (2009) make a strong case for formative assessment in CME, but leave the “how-to” details to our imagination (I guess when you’re covering every aspect of CME you need to leave a few bits out).

Here’s one recipe for formative assessment (for live CME activities):

1. Have your course faculty develop knowledge and/or case vignette questions relative to their pending talks
2. Turn these questions into a web-based survey (www.SurveyMonkey.com)
3. At least two weeks prior to the activity date, email the survey to all activity registrants
4. Share the registrants’ responses with your course faculty
5. Adjust the pending talks accordingly

If you feel the need to incentivize respondents (which I never discourage), offer them a discount off registration for another activity.  If you want more detail, check out this short JCEHP article.

I’ve used this approach a few times and it’s been generally successful (i.e., good response rate and faculty have used some of the data to modify their presentations).  However, I don’t want to pretend this approach is “setting-the-bar” for formative assessment.  If you’re not doing any such assessment, this is a good way to get started.  Play with this for a while and you’ll discover ways to get more sophisticated – just remember to share what you’re doing with the rest us!

Perceived self-efficacy evaluations

CME outcome assessments typically involve at least one of the following three question types: commitment to change, case vignettes, or perceived self-efficacy.  Although a well established assessment in other fields, CME literature regarding perceived self-efficacy evaluation is lacking.  As much as I’d like to claim authority over this approach, I can’t describe it any better than Eric Peterson did here.

If you follow the methodology as described by Eric, you’re capturing competency (Level 4)  outcomes.  One word of caution, this evaluation approach is only suitable for CME designed to teach clinical skills (e.g., latest surgical approach to tonsillectomy).  CME designed to disseminate information (e.g., new diabetes guidelines) should consider a different assessment.

Evaluating regularly scheduled series (RSS)

I’m familiar with two types of RSS, one in which the topics and speakers change with each session (e.g., Grand Rounds) and another in which the topics and speakers remain relatively constant, but unique clinical cases are addressed in each session (e.g., M&M case conference).  For the former, I’ve used a paper-based, post-activity evaluation that looks like this.  For the case-based RSS activities, however, evaluation completions using this approach were very low (< 20%).

In effort to increase the response rate, I piloted three approaches and eventually decided on a quarterly, web-based survey of participants using an adaptation of a previously validated satisfaction instrument and the commitment-to-change evaluation (click here for an example).

Evaluation response rate using this approach averaged 50% across seven case-based RSS activities.  Click here for an example of outcomes reporting using this approach.