CS2day: Award-Winning, 9-Collaborator, Performance-Improvement CME With an Outcomes-Based Evaluation Model

I saved the best for the last entry in the Back to School Tweet Fest. The Cease Smoking Today (CS2day) initiative cannot be ignored in a series about effective educational interventions in changing practice and improving quality of health care. An entire 2011 supplement of the Journal of Continuing Education in the Health Professions (JCEHP) reports the complex CS2day educational program and its findings, with six research articles [1-6] and three forum articles [7-9] written by multi-institutional teams among the nine initiative partners. This study was awarded the Alliance for CME (now ACEhp) Award for Outstanding CME Collaboration in 2009 (see PDF pages 15-18 of www.acehp.org/d/do/150), and was presented in a 2012 CME Congress poster (P50: http://www.cmecongress.org/wp-content/uploads/2012/05/CME-Congress-2012-Abstracts.pdf). The study boasts collaboration among universities, professional societies, accredited CME providers, ACEhp presidents and conference chairs, CME directors at academic medical centers, the JCEHP Editor-in-Chief, and other published researchers [1,10] who carefully define the educational program’s framework and collaboration model in the new quality improvement paradigm of CME called for by the Institute of Medicine in 2001 [11].

The CS2day initiative is so big that this blog post cannot feature just one article reporting it. I will focus on the introductory editorial [10] and 2 study articles that focus on (a) developing competencies to assess needs and outcomes [3] and (b) the educational and patient health outcomes data themselves [4]. The medical education expert Donald Moore introduces the supplement and one article therein reports the outcomes data. I hope you will do as Moore recommends, when you question what you can take from articles describing “a huge project with significant funding,” which is to ask, “What are the general principles that I can identify in these articles and how can I use them in my CME practice[?]” [10].

In my previous post, I noted the difficulties of using PI-CME to change patient health outcomes in a condition posing a major public health challenge: the COSEHC study addressed cardiometabolic risk factors and saw performance and patient health improvements. The CS2day initiative faced the same challenge, and happily also reported performance change and a change in patient health outcomes: smoking cessation. Moore nicely summarizes the challenge of connecting Level 5 performance changes among clinicians to Level 7 changes in public health outcomes: “All of us want to improve the health of the public in some way, but our approaches … may prevent us from having the impact that we wish to have. The [CS2day] articles … suggest there might be another approach that we should consider to address the important public health issues that surround but do not seem to be impacted by our CME programs” [10; emphasis added].

The articles in the JCEHP supplement are organized around 4 themes [10], to which I have added themes from the articles: 

a) Collaboration is challenging but worth doing if guidelines are set and a formative evaluation of the collaboration against known success factors is carried out [1,2,5]
b) Best-practice CME includes an outcomes orientation that connects learning and performance objectives from the needs assessment to the outcomes assessment in a valid framework to support content in all educational activities [3-6]
c) A public health focus can lead to development of CME/CEhp activities with a translational or implementation science function that transcends what can happen when education addresses only a practice gap [7]
d) Standards and competencies for CEhp and members of the CEhp profession help initiatives meet the principles and characteristics of the IOM report’s expectations [8,9,11] 

The two featured research articles [3,4] function together as the Methods and Results sections of a typical IMRAD-structured paper, but each is extensive enough to stand alone and inform CEhp professionals. McKeithen et al describe the following: the need for establishing clinical competency statements related to supporting smoking cessation; the clinical guidelines that informed performance expectations; “the 5 A’s” of support for smoking cessation (Ask, Advise, Assess, Assist, and Arrange); the 14 competencies or the 8 performance outcomes measures that fit into the 5 A’s algorithm being assessed; and collaboration of clinical and educational experts on outcomes tools to develop “a comprehensive set of measures at Levels 3 through 6” [3].

The summative outcomes data are extensively reported by Shershneva et al, where “evaluation of a collaborative program” is presented as “translating” the outcomes framework into practice [3,4]. Defining desired outcomes of the program across Levels 1 to 6* was seen as useful in facilitating agreement among stakeholders; guiding the evaluation process; gathering data from multiple activities and collaborators in a central repository; and studying the effects of mechanisms that link education to outcomes [4]. Thanks to effective planning, the researchers were also able to add to the literature on instructional design in CEhp by distinguishing performance outcomes from two groups of activity types: a) live PI activities with either a collaborative or practice-facilitator model and b) self-directed learning PI activities.

Also worth reading are additional insights about using the Success Case Method (SCM) to determine whether and why educational interventions succeed [6]. In CS2day reporting, using the SCM allowed the research team to conclude remarkably confidently, stating, “the PI activities were a primary and proximal cause of improvement in clinical practice” [4]. Moore notes that “the results were impressive: physicians integrated a new guideline into their practices and many patients stopped smoking” [10]. The guideline integrated into practice through the CS2day initiative was a “heavily researched evidence-based practice guideline published by the U.S. Agency for Healthcare Research and Quality,” due to be updated in 2008, the year after this collaborative initiative was begun [1].

Finally, a comment: In CEhp, change data are often seen as valid only when educational and program interventions do not change before activity expiration, nor even when a formative assessment shows changes to be necessary. This attitude can leave participating clinicians with suboptimal educational opportunities and stakeholders in the educational design frustrated. The use of the formative program evaluations that improved the CS2day initiative, with acknowledgements of changes, is in my opinion better than a pure pre/post comparison on an activity where valuable investments are not updated when indicated. If the CME/CEhp profession helps clinicians link medical care to public health through disease prevention, accountability to quality, and more, then educational design should respond to data collected in lengthy and large interventions.

The CS2day initiative is a model study in educational and performance improvement methods for a challenging public health problem. Please read the study articles if you have print or online access to JCEHP, for I have only touched the surface of the initiative's methodology, results, and rationales in the limited confines of this space. 

* Note: In this study, “Learning” was used as Level 3 and included knowledge and clinical skill (competence) measures, while “Performance” including commitment to change (CTC) queries was used as Level 4. Thus Level 5 was “Patient Health Status” and Level 6 was “Population Health Status.”

References cited: 

1. Olson CA, Balmer JT, Mejicano GC. Factors contributing to successful interorganizational collaboration: the case of CS2day. J Contin Educ Health Prof. 2011;31(Suppl 1):S3-S12.
2. Ales MW, Rodrigues SB, Snyder R, Conklin M. Developing and implementing an effective framework for collaboration: the experience of the CS2day collaborative. J Contin Educ Health Prof. 2011;31(Suppl 1): S13-S20.
3. McKeithen T, Robertson S, Speight M. Developing clinical competencies to assess learning needs and outcomes: the experience of the CS2day initiative. J Contin Educ Health Prof. 2011;31(Suppl 1):S21-S27. http://www.ncbi.nlm.nih.gov/pubmed/22190097. [Featured Article]
4. Shershneva MB, Larrison C, Robertson S, Speight M. Evaluation of a collaborative program on smoking cessation: translating outcomes framework into practice. J Contin Educ Health Prof. 2011;31(Suppl 1):S28-S36. http://www.ncbi.nlm.nih.gov/pubmed/22190098. [Featured Article]
5. Mullikin EA, Ales MW, Cho J, Nelson TM, Rodrigues SB, Speight M. Sharing collaborative designs of tobacco cessation performance improvement CME projects. J Contin Educ Health Prof. 2011;31(Suppl 1):S37-S49.
6. Olson CA, Shershneva MB, Brownstein MH. Peering inside the clock: using success case method to determine how and why practice-based educational interventions succeed. J Contin Educ Health Prof. 2011;31(Suppl 1):S50-S59.
7. Hudmon KS, Addleton RL, Vitale FM, Christiansen BA, Mejicano GC. Advancing public health through continuing education of health care professionals. J Contin Educ Health Prof. 2011;31(Suppl 1):S60-S66.
8. Balmer JT, Bellande BJ, Addleton RL, Havens CS. The relevance of the Alliance for CME competencies for planning, organizing, and sustaining an interorganizational educational collaborative. J Contin Educ Health Prof. 2011;31(Suppl 1):S67-S75.
9. Cervero RM, Moore DE. The Cease Smoking Today (CS2day) initiative: a guide to pursue the 2010 IOM report vision for CPD. J Contin Educ Health Prof. 2011;31(Suppl 1):S76-S82.
10. Moore DE. Collaboration, best-practice CME, public health focus, and the Alliance for CME competencies: a formula for the new CME? J Contin Educ Health Prof. 2011;31(Suppl 1):S1-S2. http://www.ncbi.nlm.nih.gov/pubmed/22190095. [Featured Editorial]
11. Institute of Medicine (IOM) Committee on Planning a Continuing Health Professional Education Institute. Redesigning Continuing Education in the Health Professions. Washington, DC: The National Academies Press; 2010. http://books.nap.edu/openbook.php?record_id=12704. Accessed September 17, 2015.

MeSH “Major” Terms for the 3 Featured Articles (common items italicized): 

McKeithen et al [3]: Benchmarking; Clinical Competence; Education, Medical, Continuing/methods; Needs Assessment; Outcome and Process Assessment (Health Care)/organization & administration; Practice Guidelines as Topic/standards; Smoking Cessation/methods; Tobacco Use Disorder/prevention & control
Shershneva et al [4]: Benchmarking/methods; Clinical Competence/standards; Health Personnel/classification; Health Personnel/psychology; Health Personnel/statistics & numerical data; Interprofessional Relations; Outcome Assessment (Health Care)/organization & administration; Program Evaluation; Smoking Cessation/methods; Tobacco Use Disorder/prevention & control
Moore [11]: Benchmarking; Clinical Competence; Delivery of Health Care, Integrated; Education, Medical, Continuing/methods; Interinstitutional Relations; Public Health

Personalized MD Curriculum in Personalized NSCLC Treatment Produces High, “Clinically Significant” Educational Effect Size

In non-small cell lung cancer (NSCLC), evidence points to the benefits of tumor biopsy for biomarker analysis, which in turn may allow individually targeted therapy [e.g., 1-3]. In the last five years of this age of pharmacogenomics and prognostic markers, the clinical excitement for individualized medicine has produced a robust count of 256 review articles indexed in PubMed found with a search on “non small cell lung cancer treatment biomarker review,” even with additional filtering to “Abstract [available], English, [and] Humans.” But diagnostics in surgery and pathology, as well as personalized treatment for cancer are expensive, so the societal context of the Affordable Care Act enacted five years ago (March 23rd, 2010, with its emphases on quality measures, patient-centered care, and accountability in care decisions) cannot be ignored.

Individualized intervention is not just important to cancer biology and treatment: it is important to clinical education, as well. Not only do clinicians caring for patients with cancer have their own knowledge and competence gaps—mainly because of the discovery of new evidence in this rapidly changing therapeutic area—they have the healthcare system context to work within, from local to national levels. The newly published, featured articleby Hermann et al focuses on NSCLCeducation in the quality-driven system environment of the ACA, titled, “EducationalOutcomes in the Era of the Affordable Care Act: Impact of PersonalizedEducation About Non-Small Cell Lung Cancer.” The authors argue for timely opportunities for immediate, practical, and translatable education for individual clinicians, as follows: “Quality medical education must be available when the health care provider is ready to learn, provide feedback, and maximize translation of knowledge from desk to clinic” [4].

The educational methods and instructional design are of greatest interest. Oncologists completed a pre-intervention self-assessment of knowledge, skills, and attitudes. This was used to develop an individualized learning plan and a personalized curriculum, which included up to 5 distinct activities selected to address identified knowledge and practice gaps. The activities were distributed online, and learners received feedback at the completion of each activity. Learners were tested on 5 knowledge and decision-making areas relevant to NSCLC treatment.  

The results of education were dramatic: “Completion of the learning plan was associated with a high effect size (d = .70),” a Cohen’s d that indicates that the educational intervention was much more meaningful than the statistically significant differences between learners’ pre- and post-intervention testing would suggest. (Remember that p values tell the statistician only how likely it is that the hypothesis could be accepted or rejected in error.) If one reviews the Effect Size (ES) lecturenotes provided by Dr. Lee Becker on his University of Colorado webpages, this translates to what Cohen himself (reluctantly) defined as a medium-to-large effect but which has become standard usage where historical data from research teams are not published with current results. This effect size surpasses even what Wolf (1986) identified as the lowest benchmark for change results that are “clinically significant,” not just educationally meaningful, at d = .50.

Looking at this educational study’s effect size more simply at Becker’s site, Cohen’s d = .70 means that 43.0% of participating learners (oncologists) had posttest scores that did not overlap with pretest scores, indicating learning that facilitates change. This is a big percentage when one considers that even an effect size of .20 (small) is difficult to achieve in one initiative. In other words, personalized education on NSCLC affected quality care. Kudos to the researchers.

P.S. For additional reading on Cohen's d and effect sizes in CEhp, check out the AssessCME blog written by my outcomes colleague, Jason Oliveri: assesscme.wordpress.com/category/effect-size.

References cited: 

1. Remark R, Becker C, Gomez JE, et al. The non-small cell lung cancer immune contexture. A major determinant of tumor characteristics and patient outcome. Am J Respir Crit Care Med. 2015;191(4):377-90.
2. Cagle PT, Allen TC, Olsen RJ. Lung cancer biomarkers: present status and future developments. Arch Pathol Lab Med. 2013 Sep;137(9):1191-8.
3. Raparia K, Villa C, DeCamp MM, Patel JD, Mehta MP. Molecular profiling in non-small cell lung cancer: a step toward personalized medicine. Arch Pathol Lab Med. 2013;137(4):481-91.
4. Herrmann T, Peters P, Williamson C, Rhodes E. Educational outcomes in the era of the Affordable Care Act: impact of personalized education about non-small cell lung cancer. J Contin Educ Health Prof. 2015;35(Suppl 1):S5-S12. [Featured Article]
5. Becker L. Effect size (ES). University of Colorado—Colorado Springs Website. http://www.uccs.edu/lbecker/effect-size.html. Accessed September 16, 2015.
MeSH *Major* terms: This study [4] is so new, NLM librarians have not yet assigned Medical Subject Headings. Check for updates at http://www.ncbi.nlm.nih.gov/pubmed/?term=26115247. 

Medical education with EMR-based reminders reduces antibiotic prescribing and dispensing for respiratory tract infections in Norway

It is known that British guidelines for otitis media support delayed antibiotic prescribing [1], and other countries have guidelines to reduce certain antibiotic prescribing for otitis media, for example, France [2]. Conversely, Finnish guidelines do not [3]. A 2013 Norwegian study published in the British Journal of General Practice compares the varying effectiveness of 2 interventions in delaying primary care antibiotic prescribing for respiratory tract infections, including otitis [4].

Notwithstanding a complicated design for recruiting and assigning general practitioners across multiple sites, this article offers several interesting features. First, it compares an education-only intervention with the same education enhanced by pop-up reminders of a physician’s own prescribing patterns in the electronic medical record (EMR), a nice reinforcement of the educational intervention for participating physicians. While not a focus of this post, I would like to mention a new Penn study of adherence to guidelines on otitis media using EMRs for decision support at Children’s Hospital of Philadelphia [5]. This shows interest in implementation science combined with continuing medical education (CME) for changing physicians’ practice patterns.

The Norwegian study featured here [4] data collected and linked data on prescribed and dispensed antibiotics from (a) 1 year before and (b) 1 year during the intervention, which allowed prescribing practice patterns to be displayed to physicians in the EMR at the point of prescribing antibiotics for a respiratory tract infection. It also collected pharmacy fill rates by patients, which I find interesting because it may offer insights into patients’ (or parents’) agreement with the need for the prescription, after any access barriers to medication adherence. 

Both study arms showed slightly reduced antibiotic prescribing from baseline (pre-intervention) rates: 1% reduction vs. 4% reduction in “approximated risk” (risk ratio, RR) in the education-only vs. education-plus-EMR study arms, respectively. Both results report very tight ranges around a 95% confidence interval (CI), increasing confidence in the findings. (It is further nice to see the CI reported instead of the p value, for those who often hesitate to report CI because of many readers’ greater familiarity with the p value.) While reporting of “risk ratio” may be used as simply a convenient and appropriate way of reporting epidemiological data, it seems to me that its use for reporting educational outcomes with practice data is unusual and perhaps a comment on antibiotic prescribing for these infections as a risk.

The authors find that upper respiratory tract infection, sinusitis, and otitis “gave highest odds for delayed prescribing and lowest odds for dispensing,” which led them to conclude that the greatest potential for “savings” is greatest for these infections, a comment that brings this CME study with implementation science into the context of health utilization research. The article offers freely accessible full text, so enjoy reading the study.

References cited:

1. Centre for Clinical Practice at NICE (UK). Respiratory Tract Infections - Antibiotic Prescribing: Prescribing of Antibiotics for Self-Limiting Respiratory Tract Infections in Adults and Children in Primary Care. London: National Institute for Health and Clinical Excellence (UK); 2008 Jul. http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0010014/.
2. Levy C, Pereira M, Guedj R, et al. Impact of 2011 French guidelines on antibiotic prescription for acute otitis media in infants. Médecine Mal Infect. 2014;44(3):102-106. http://www.ncbi.nlm.nih.gov/pubmed/24630597.
3. [Update on current care guidelines: acute otitis media]. Duodecim. 2010;126(5):573-4. Finnish. http://www.ncbi.nlm.nih.gov/pubmed/20597310.
4. Hoye S, Gjelstad S, Lindbaek M. Effects on antibiotic dispensing rates of interventions to promote delayed prescribing for respiratory tract infections in primary care. Br J Gen Pract. 2013;63(616):e777-e786. http://bjgp.org/content/63/616/e777.full.pdf. [Featured Article]
5. Fiks AG, Zhang P, Localio AR, et al. Adoption of electronic medical record-based decision support for otitis media in children. Health Serv Res. 2015;50(2):489-513. http://www.ncbi.nlm.nih.gov/pubmed/25287670.  

MeSH *Major* terms: Anti-Bacterial Agents/therapeutic use*; Education, Medical, Continuing*; General Practice/statistics & numerical data*; Physician's Practice Patterns/statistics & numerical data*; Respiratory Tract Infections/drug therapy* 

Mixed-Methods Study Improves Team Communication After Non-Didactic Interprofessional Education on Cardiac Surgical Crisis

It would be hard to imagine a more crucial setting for effective interdisciplinary clinical interactions than the cardiac surgery operating theater. Stevens and colleagues published this 2012 pilot study on interprofessional education to “sharpen performance of experienced cardiac surgical teams in acute crisis management.”

The educational methods support existing effectiveness research for non-didactic education, incorporating both interactive workshops for an entire care unit and computer-based, crisis-case simulations (whose “high-realism” scenarios improved over time). Researchers found that 82% of the 79 participants recommended repetition of case simulations every 6 – 12 months. Workshop participants identified priorities in “encouraging speaking up about critical information and interprofessional information sharing,” particularly early communication of the surgical plan.

The mixed-methods outcomes assessment methodology is also noteworthy because of its appropriateness to this study of human communications and behaviors during a patient crisis: the structured interviews with study participants added context and insights to the quantitative data that could be gathered from periodic surveys. The surveys that were administered before, just after, and 6 months after the educational activities noted that the “concept of working as a team improved between surveys,” as well as “trends for improvement in gaining common understanding of the plan before a procedure and appropriate resolution of disagreements.” The qualitative arm of the study found that interviewees valued the initiative’s “positive effect on their personal behaviors and patient care, including speaking up more readily and communicating more clearly.”

In the continuing medical education field, we often see the Canadians leading educational research, standards, and innovative methods. In fact, looking only at the U. S. National Library of Medicine’s assignment of Medical Subject Headings (MeSH terms) to this indexed article shows the relevance of this study for medical education methods for promoting competence in decision-making, performance-in-practice change, and quality improvement (see “major” MeSH terms listed below, and others on the PubMed page). One hopes to see a follow-up on this pilot study at the Centre Hospitalier Universitaire de Montréal (Quebec, Canada).  

PubMed:  http://www.ncbi.nlm.nih.gov/pubmed/22502966

MeSH *Major* terms: Cardiac Surgical Procedures/education; Clinical Competence; Critical Care/standards; Education, Medical, Continuing/methods; Patient Care Team/organization & administration