Study Design and Paired Comparisons: Individualized Education Fails to Change Practice—Or Was It Only Poor Matching?

We should commend Malone et al for submitting this AHRQ-supported* study [1] for publication when a flaw in its design or execution could be the authors’ main reason for concluding that “the current study was not able to demonstrate a significant beneficial effect of the educational outreach program on [the primary performance outcome measure].” This blog’s “Back-to-School” service campaign did not exclude studies reporting negative outcomes because these studies can potentially inform continuing education in the health professions (CEhp) as much as positive studies can.

CEhp/CME educational proposals, audience-generation strategies, and outcomes reports now specify relevant “target audiences,” recognizing that not all practitioners with a certain degree, specialty, or other professional demographic description would benefit from the same educational activity or design. With this more recent recognition of the importance of targeting specific clinicians and learning about their needs has come greater recognition that many CE participants should not be included in aggregated data. This is even truer in studies with matched pairs, where the step of greatest importance lies in setting match criteria. On September 15th, I discussed an opioids-education study where matching criteria were so stringent that the authors were not able to match certain participants (physicians in the intervention group), and these participants’ data and group assignments were handled nicely and reported clearly in the paper [2] (see post at http://fullcirclece.blogspot.com/2015/09/eight-year-canadian-study-on-opioid.html).

Conversely, the first result listed in this study’s abstract indicates a matching flaw for a study on education on drug-drug interactions (DDIs): “The 2 groups were significantly different with respect to age, profession, specialty, and geographic region.” This finding undermines other benefits to the study, namely, that large samples (19,606 prescribers) were recruited to both groups (educational intervention vs. control) and matched on prescribing volume. Individualized education (also known as academic detailing) was delivered by trained pharmacists as clinical consultants who met with prescribers to “provide one-on-one information … promote evidence-based knowledge, create trusting relationships, and induce practice change.” This study’s performance (behavioral) measure was a reduced rate of prescribing potential DDIs. The prescribing of 25 clinically important, potential DDIs increased more in the intervention group than it did in the control group.

In conclusion, when we look at this presumably negative finding, we are left to wonder whether the educational intervention was not effective—or whether a better matching process might have revealed different results on reducing potential DDIs and improving health care quality and utilization. One could argue that with nearly 20,000 prescribers in both samples, more matching criteria could have been applied without sacrificing so many data points that results would be inconclusive. The study’s design as a retrospective study could also explain recruitment and matching practices. In social sciences research (including educational outcomes research), a core expectation is generalizability of a sample to a population of interest; when reasonably achieved, generalizability lets us apply findings to practical needs and future decisions. 

Recall the study conclusion quoted above: “The current study was not able to demonstrate a significant beneficial effect …” (emphasis added). A secondary analysis with different pair-matching practices might yet inform national initiatives in improving quality while reducing costs through academic detailing, both of which help patients. Now let’s remember to thank Malone, Liberman, and Sun for sharing their data and methods with the healthcare quality and educational research communities in the Journal of Managed Care & Specialty Pharmacy.

* AHRQ = United States Agency for Healthcare Research and Quality

References cited:
1. Malone DC, Liberman JN, Sun D. Effect of an educational outreach program on prescribing potential drug-drug interactions. J Manag Care Pharm. 2013;19(7):549-557. http://www.ncbi.nlm.nih.gov/pubmed/23964616. [Featured Article]
2. Kahan M, Gomes T, Juurlink DN, et al. Effect of a course-based intervention and effect of medical regulation on physicians’ opioid prescribing. Can Fam Physician. 2013;59(5):e231-e239. http://www.cfp.ca/content/59/5/e231.full.pdf+html.
Free Full Text: http://www.amcp.org/JMCP/2013/September_2013/17103/1033.html
MeSH “Major” Terms: Drug Interactions; Drug Prescriptions; Education, Medical, Continuing; Health Education; Physician's Practice Patterns; Prescription Drugs/administration & dosage

Eight-year Canadian study on opioid prescribing among regulator- and self-referred physicians to intensive course

This educational study in a clinical journal by Kahan et al at the University of Toronto examined “the effects of an intensive 2-day course on physicians' prescribing of opioids” [1]. The most impressive feature of this study is its eight-year-plus data-gathering period of opioid-prescribing levels among participating physicians, most of whom were family physicians. Other interesting features are worth mentioning, in both instructional design and study design.

The study design grouped participants into self-referred physicians vs. physicians who were referred by medical regulators, and added a control (nonparticipant) group. Undertaking a challenging matching procedure, researchers matched nonparticipants according to specific variables, including quarterly rates of opioid-prescribing, expressed as milligrams of morphine equivalent. Subgroups of participant groups with very high opioid-prescribing patterns were also identified; unfortunately, nonparticipants to match these participants were difficult to find. Yet this targeted approach to matching is appropriate and represents a significant investment of the researchers’ time, allowing the comparative group findings shown below. Nonparticipants were added to the study concurrently with their matched participants, per an “index date” defined as “the date of course completion for participating physicians. Control physicians were assigned the same index date as their matched pair.” In one deviation from the primary outcome measure, matching was done by number of opioid prescriptions rather than milligrams of morphine equivalent. Another study design feature is the specific comparison of opioid-prescribing rates for 2 years before vs. 2 years after the educational intervention, again by group and subgroup vs. nonparticipants; participants who could not be matched were analyzed separately from participants with matched pairs.

The instructional design of the 2-day course incorporated several educational settings and modalities. Planners used didactic presentations but added problem-based case discussions and mock-interview learning interactions with standardized patients who offered feedback. Pros and cons of changing prescribing patterns were discussed in a session at the end of the course, featuring a faculty interview with a patient. The course also provided a detailed syllabus with notes and references before the course, as well as office materials. It should be noted that benzodiazepine-prescribing was also addressed in course content. Finally, each 2-day course enrolled up to 12 participants, a limit that would confer an individualized learning environment and some professional privacy in what might be a sensitive concern among participating physicians.

The authors noted in the introduction, “Medical education has been suggested as one strategy to improve opioid prescribing among physicians” [2,3] and “Educational interventions focused on opioid prescribing lead to positive improvement in physicians’ knowledge and self-reported practices” [4]. Let's look at results by reported subgroup.

Among physicians referred by medical regulators, “the rate of opioid prescribing decreased dramatically in the year before course participation compared with matched control physicians,” and “the course had no added effect on the rate of physicians' opioid prescribing in the subsequent 2 years.” It seems that these physicians might have changed their behavior by arbitrarily reducing prescribing rates because of the regulatory investigation, even without an educational intervention to inform their clinical decision-making. In fact, the authors acknowledge this, noting, “We measured only the quantity of opioids prescribed, not the quality of opioid prescribing.” The regulatory concerns may have created a false baseline for the educational study that measured only quantity of opioid prescribed rather than patient-selection or other measure of competence.

Among the self-referred physicians who were matched to nonparticipants, “there was no statistically significant effect on the rate of opioid prescribing observed” from baseline to 2-year follow-up, although there had been a temporary decrease, particularly in prescribing for patients aged 15 – 64 (here’s a nice graph with patient ages: http://www.cfp.ca/content/59/5/e231/F4.expansion.html). On the other hand, “the rate of opioid prescribing decreased by 43.9% in the year following course completion” among self-referred physicians with high prescribing rates who could not be matched, suggesting that these physicians “might have responded to what was taught in the course.”  

References cited:
1. Kahan M, Gomes T, Juurlink DN, et al. Effect of a course-based intervention and effect of medical regulation on physicians’ opioid prescribing. Can Fam Physician. 2013;59(5):e231-e239. http://www.cfp.ca/content/59/5/e231.full.pdf+html.
[Featured Article]
2. College of Physicians and Surgeons of Ontario. Avoiding Abuse, Achieving a Balance: Tackling the Opioid Public Health Crisis. Toronto, ON: College of Physicians and Surgeons of Ontario; 2010.
3. National Opioid Use Guideline Group. Canadian Guideline for Safe and Effective Use of Opioids for Chronic Non-Cancer Pain. Hamilton, ON: National Opioid Use Guideline Group; 2010.
4. Midmer D, Kahan M, Marlow B. Effects of a distance learning program on physicians’ opioid- and benzodiazepine-prescribing skills. J Contin Educ Health Prof. 2006;26(4):294-301.
Free full text PDF: http://www.cfp.ca/content/59/5/e231.full.pdf.
MeSH *Major* terms: Analgesics, Opioid/therapeutic use*; Drug Prescriptions/standards*; Education, Medical, Continuing*; Physician's Practice Patterns/standards* 

Medical education with EMR-based reminders reduces antibiotic prescribing and dispensing for respiratory tract infections in Norway

It is known that British guidelines for otitis media support delayed antibiotic prescribing [1], and other countries have guidelines to reduce certain antibiotic prescribing for otitis media, for example, France [2]. Conversely, Finnish guidelines do not [3]. A 2013 Norwegian study published in the British Journal of General Practice compares the varying effectiveness of 2 interventions in delaying primary care antibiotic prescribing for respiratory tract infections, including otitis [4].

Notwithstanding a complicated design for recruiting and assigning general practitioners across multiple sites, this article offers several interesting features. First, it compares an education-only intervention with the same education enhanced by pop-up reminders of a physician’s own prescribing patterns in the electronic medical record (EMR), a nice reinforcement of the educational intervention for participating physicians. While not a focus of this post, I would like to mention a new Penn study of adherence to guidelines on otitis media using EMRs for decision support at Children’s Hospital of Philadelphia [5]. This shows interest in implementation science combined with continuing medical education (CME) for changing physicians’ practice patterns.

The Norwegian study featured here [4] data collected and linked data on prescribed and dispensed antibiotics from (a) 1 year before and (b) 1 year during the intervention, which allowed prescribing practice patterns to be displayed to physicians in the EMR at the point of prescribing antibiotics for a respiratory tract infection. It also collected pharmacy fill rates by patients, which I find interesting because it may offer insights into patients’ (or parents’) agreement with the need for the prescription, after any access barriers to medication adherence. 

Both study arms showed slightly reduced antibiotic prescribing from baseline (pre-intervention) rates: 1% reduction vs. 4% reduction in “approximated risk” (risk ratio, RR) in the education-only vs. education-plus-EMR study arms, respectively. Both results report very tight ranges around a 95% confidence interval (CI), increasing confidence in the findings. (It is further nice to see the CI reported instead of the p value, for those who often hesitate to report CI because of many readers’ greater familiarity with the p value.) While reporting of “risk ratio” may be used as simply a convenient and appropriate way of reporting epidemiological data, it seems to me that its use for reporting educational outcomes with practice data is unusual and perhaps a comment on antibiotic prescribing for these infections as a risk.

The authors find that upper respiratory tract infection, sinusitis, and otitis “gave highest odds for delayed prescribing and lowest odds for dispensing,” which led them to conclude that the greatest potential for “savings” is greatest for these infections, a comment that brings this CME study with implementation science into the context of health utilization research. The article offers freely accessible full text, so enjoy reading the study.

References cited:

1. Centre for Clinical Practice at NICE (UK). Respiratory Tract Infections - Antibiotic Prescribing: Prescribing of Antibiotics for Self-Limiting Respiratory Tract Infections in Adults and Children in Primary Care. London: National Institute for Health and Clinical Excellence (UK); 2008 Jul. http://www.ncbi.nlm.nih.gov/pubmedhealth/PMH0010014/.
2. Levy C, Pereira M, Guedj R, et al. Impact of 2011 French guidelines on antibiotic prescription for acute otitis media in infants. Médecine Mal Infect. 2014;44(3):102-106. http://www.ncbi.nlm.nih.gov/pubmed/24630597.
3. [Update on current care guidelines: acute otitis media]. Duodecim. 2010;126(5):573-4. Finnish. http://www.ncbi.nlm.nih.gov/pubmed/20597310.
4. Hoye S, Gjelstad S, Lindbaek M. Effects on antibiotic dispensing rates of interventions to promote delayed prescribing for respiratory tract infections in primary care. Br J Gen Pract. 2013;63(616):e777-e786. http://bjgp.org/content/63/616/e777.full.pdf. [Featured Article]
5. Fiks AG, Zhang P, Localio AR, et al. Adoption of electronic medical record-based decision support for otitis media in children. Health Serv Res. 2015;50(2):489-513. http://www.ncbi.nlm.nih.gov/pubmed/25287670.  

MeSH *Major* terms: Anti-Bacterial Agents/therapeutic use*; Education, Medical, Continuing*; General Practice/statistics & numerical data*; Physician's Practice Patterns/statistics & numerical data*; Respiratory Tract Infections/drug therapy* 

Implementation Science Extends Multidisciplinary Education on VTE Prophylaxis at World-Class Children’s Hospital

Let’s continue yesterday's thread with venous thromboembolism (VTE) education in a new patient population: children. In this quality improvement article, Raffini et al point out that while risks of VTE are far lower in children than in adults, children still need clinicians to maintain an appropriate level of suspicion and take action on VTE risk with prophylaxis.

What a nice way to phrase a practice gap and the need to narrow that gap: “VTE prophylaxis for patients at risk is often overlooked in pediatric health care institutions, which provides an opportunity to improve patient care.” Even at the world-class Children’s Hospital of Philadelphia (CHOP) at the University of Pennsylvania, children and adolescents were not receiving an appropriate level of risk assessment and prophylaxis, leading researchers to undertake this four-year quality improvement study.

CME/CEhp initiatives often do not extend into local facilities, but this study used reinforcing methods of implementation science to communicate and support the desired behaviors and tasks. These extended “multidisciplinary educational forums” into patient-care settings. Here’s a summary of the intervention’s rollout of CHOP’s locally established guidelines to “encourage timely initiation of thromboprophylaxis." These interventions included 1) VTE risk assessment in the nursing admission intake forms; 2) a VTE-prophylaxis order set implemented into the computerized ordering system; 3) ICU nurses assessing VTE risk and preventive practice daily during team rounds; 4) trials of various pneumatic compression devices with appropriate sizing for children; 5) acquisition of more compression devices with storage near high-risk areas; and 6) development of a protocol for perioperative nurses to initiate pneumatic compression before surgery for certain patients, later expanded to all inpatient settings.

So was extension of multidisciplinary education into care settings, using methods of implementation science, effective? YES. The study’s primary outcome measure was “compliance with thromboprophylaxis guidelines in patients at risk for VTE.” Clinically meaningful improvements were seen: “Over the 4-year study period, the observed rate of VTE prophylaxis in patients at risk increased from a baseline of 22% to an average rate of 82%, and there were intermittent improvements up to 100%.” Implementation science methods, when added to multidisciplinary clinical education, quadrupled or even quintupled guidelines-based care in an underserved population.    

There’s also a nice feature to point out to those watching IRB “requirements” that peer-reviewed journals increasingly expect of performance-change initiatives: “This project was a quality-improvement activity and exempt from review from the institutional review board at the Children's Hospital of Philadelphia.” The QI initiative was not seen as human subjects research (HSR).

Reference cited: Raffini L, Trimarchi T, Beliveau J, Davis D. Thromboprophylaxis in a pediatric hospital: a patient-safety and quality-improvement initiative. Pediatrics. 2011;127(5):e1326-32. PMID: 21464186.
See also the ACCP and ICSI guidelines linked from the September 7, 2015 post (http://fullcirclece.blogspot.com/2015/09/pharmacy-education-for-hospital.html)
PubMed: http://www.ncbi.nlm.nih.gov/pubmed/21464186
Free full text (Pediatrics final version): http://pediatrics.aappublications.org/content/127/5/e1326.full.pdf+html
MeSH *Major* terms: Anticoagulants/administration & dosage*; Guideline Adherence*; Patient Selection*; Primary Prevention/organization & administration*; Venous Thromboembolism/prevention & control*
(And for those of you who love MeSH and UMLS, many quality-improvement and program change-related MeSH terms were assigned by NLM but not starred as “major” terms.) 

Mixed-Methods Study Improves Team Communication After Non-Didactic Interprofessional Education on Cardiac Surgical Crisis

It would be hard to imagine a more crucial setting for effective interdisciplinary clinical interactions than the cardiac surgery operating theater. Stevens and colleagues published this 2012 pilot study on interprofessional education to “sharpen performance of experienced cardiac surgical teams in acute crisis management.”

The educational methods support existing effectiveness research for non-didactic education, incorporating both interactive workshops for an entire care unit and computer-based, crisis-case simulations (whose “high-realism” scenarios improved over time). Researchers found that 82% of the 79 participants recommended repetition of case simulations every 6 – 12 months. Workshop participants identified priorities in “encouraging speaking up about critical information and interprofessional information sharing,” particularly early communication of the surgical plan.

The mixed-methods outcomes assessment methodology is also noteworthy because of its appropriateness to this study of human communications and behaviors during a patient crisis: the structured interviews with study participants added context and insights to the quantitative data that could be gathered from periodic surveys. The surveys that were administered before, just after, and 6 months after the educational activities noted that the “concept of working as a team improved between surveys,” as well as “trends for improvement in gaining common understanding of the plan before a procedure and appropriate resolution of disagreements.” The qualitative arm of the study found that interviewees valued the initiative’s “positive effect on their personal behaviors and patient care, including speaking up more readily and communicating more clearly.”

In the continuing medical education field, we often see the Canadians leading educational research, standards, and innovative methods. In fact, looking only at the U. S. National Library of Medicine’s assignment of Medical Subject Headings (MeSH terms) to this indexed article shows the relevance of this study for medical education methods for promoting competence in decision-making, performance-in-practice change, and quality improvement (see “major” MeSH terms listed below, and others on the PubMed page). One hopes to see a follow-up on this pilot study at the Centre Hospitalier Universitaire de Montréal (Quebec, Canada).  

PubMed:  http://www.ncbi.nlm.nih.gov/pubmed/22502966

MeSH *Major* terms: Cardiac Surgical Procedures/education; Clinical Competence; Critical Care/standards; Education, Medical, Continuing/methods; Patient Care Team/organization & administration

Back to School: See How Clinical Continuing Education Has Improved Health Care Quality – Proven in Peer-Reviewed Journals!

We often hear that there is little evidence of CME’s success in teaching clinicians relevant strategies for better patient care, for changing practice through education, and for connecting with the quality improvement (QI) movement. Witness the current initiatives of the Alliance for Continuing Education in the Health Profession (ACEhp), “Why We Matter” and the 10-year Roadmap and 10 complementary Building Blocks of the Quality Improvement Education (QIE) Initiative. While Davis and others have noted historical effectiveness of CEhp, such as in this 1992 article, later articles by Bloom, Davis, and Ratanawongsa have questioned CME effectiveness (e.g., in 1995 and 2005) and identified reporting inconsistencies that reduce the validity of CME outcomes reports.

It is true that our reporting of CEhp methods and outcomes has need of greater readership and better reporting. Yet CME, performance improvement (PI), interprofessional education (IPE), and education-driven QI projects already have many achievements reported, in both initiative-level outcomes reports and the published, peer-reviewed literature. Going beyond JCEHP, where articles feature methods and case studies for improving clinical education, and CE Measure, which reports on-the-ground initiatives and specific outcomes data, we see that clinical journals and meetings are publishing increasing numbers of CEhp studies. What’s more, reporting of these achievements is certain to grow after the ACEhp QIE Initiative launches its custom version of the SQUIRE tool this month, at September 2015’s Alliance Quality Symposium. The SQUIRE tool, created by the group developing Standards for Quality Improvement Reporting Excellence, will help all of us doing CEhp research design our studies and publish our findings in a standardized manner … for later meta-analyses of into CME/CEhp effectiveness as a mechanism for quality improvement.

Today our goal is to show the past effectiveness of CEhp initiatives in published educational outcomes studies, as we prepare—as a profession—to begin reporting CME and IPE initiatives with greater rigor, and to illustrate the effectiveness of certified clinical education by accredited providers on a greater scale. Therefore, Don Harting, my friend and colleague who specializes in medical education needs assessments, and I have embarked on a “Back-to-School” campaign to highlight published articles where CME, IPE, PI, and QI initiatives worked to change clinicians’ behavior or improve performance in routine, clinical practice. Don featured 15 of our 30 articles on Twitter (@CME_Scout) and his blog from August 17 – August 31, 2015, and I will be posting to publicize another 15 articles at @SHB_CMEedit and this blog from September 1 – September 15, 2015. You can sign up to receive a summary report of the published studies we feature by following fullcirclece.blogspot.com, or find me in person at the Alliance Quality Symposium at the end of September.

We want you to take away these messages from this campaign, so let me encourage you to put yourself in the first person and say these motivational statements:

• I can talk about CEhp initiatives that were effective in improving the quality of health care
• There is evidence in the peer-reviewed literature to prove CEhp effectiveness
• I am ready to read CEhp outcomes studies, to help me learn to prepare my own (if you are not sure you’re ready, check out the 12-article research and statistics series in the ACEhp Almanac, which started in February 2015)
• My group’s CEhp outcomes data do not have to show change to be worth publishing, i.e., it’s okay to publish negative findings because I am a dispassionate researcher of clinical education
• I can find clinical and educational journals to whom to submit my CME outcomes data
• My CEhp work in content and faculty development, instructional design, and outcomes analysis matters

Now don’t you feel better? Not sure? Ready to go? Follow our tweets, read the articles or their abstracts, and then say these points to yourself again. 

Finally, to promote articles showing the achievements of the continuing education profession in improvement health care quality, we have a contest where you can Retweet to Win (see rules) by sharing our featured articles with your network of CEhp professionals. Harting Communications LLC is offering a $100 Amazon gift card as first prize, and Full Circle Clinical Education, Inc, is offering a $50 Amazon gift card as second prize!

Remember to follow us on Twitter @SHB_CMEedit and @CME_Scout so that you can notified via direct message if you win! And tell Don and me whether this service campaign supporting awareness of CEhp effectiveness was helpful to you. 

Good luck, happy reading, and enjoy! 

Best wishes, and thanks for viewing my updates, 

Sandra