By Denise Myshko
Outcomes assessment of continuing education can help determine whether there is a change in participants’ knowledge and performance. March 2007 Barbara Winkelman CME LLC Over time, there is going to be a change in how CME is delivered and how outcomes are measured; there will be more collaboration between CME providers. The goal is to create positive patient outcomes. The use of outcomes measures to study the impact of continuing medical education is gaining momentum, especially as new accreditation standards require CME providers to assess the impact of their programs. Providers must now provide activities and educational interventions that are designed to change competence, performance, or patient outcomes. Providers must also demonstrate that the program addresses a knowledge or practice gap. The Accreditation Council for Continuing Medical Education (ACCME) updated its accreditation standards in June 2006. The new standards are designed to align the mission of the educational program with the impact on physician behavior and practice. “Measuring outcomes is a hot topic,” says Barbara Winkelman, VP of marketing and multimedia at CME LLC. “Understanding educational outcomes helps CME providers understand whether the knowledge gap was bridged, to what degree, and whether providers are headed in the right direction to aid physicians and aid physician care.” Outcomes projects are becoming more sophisticated; different methodologies and different study designs are being used for question development, says Mazi Abdolrasulnia, Ph.D., VP of business development at Outcomes Inc. “Outcomes studies include more case-based questions and CME providers are thinking about how to appropriately scale the questions and how to measure the outcomes,” he says. There are two objectives with an outcomes approach, says Anne Goodrich, director of physician insights at Pri-Med. “One is to learn from the information and identify future clinical educational needs,” she says. “The other is to provide supporters with robust information about the impact of the educational grant on practitioners.” Although experts agree that the impact of CME on physician learning should be measured, there are no standards for conducting outcomes measurements or assessing changes in physician behavior. “There is a great deal of variation in how the industry defines learning, performance, and patient health,” Ms. Goodrich says. “There are different tools, methods, and methodologies that various organizations employ. Gathering the data also can be very costly, depending on what methodologies are being employed.” Heidi Liston, Pharm.D., general manager of DiMedix LLC, agrees that the challenge is deciding which educational outcomes should be measured and how. “There is no consistent measurement to follow from a performance improvement and quality-improvement standpoint,” Dr. Liston says. “This creates a dilemma regarding which quality improvement standards should be adopted. There needs to be agreement on one national standard by all stakeholders.” Outcomes measurement is not an exact science, says Heidi Chandonnet, senior director, accreditation and compliance, at Institute for Continuing Healthcare Education Inc. “It takes a team approach with differing opinions and differing philosophies to get the best results and have the educational program be effective for the person who’s being educated,” she says. Ms. Winkelman says one of the biggest barriers is getting the physicians to participate in the outcomes assessment. “Everyone should be involved in assessing outcomes, but there is an even bigger burden on physicians to complete the questionnaires,” she says. “Physicians are going to become inundated with questionnaires, case studies, and focus groups; these are things they won’t always have time for.” Assessing Outcomes Those involved in continuing medical education are trying a number of approaches. Survey-based studies are the most simple approach for assessing outcomes, says Mario R. Nacinovich Jr., senior VP of Fission Communications. “While some have warned of the danger of physician overload, both the supporters and participants of continuing education have become familiar and most comfortable with survey-based studies,” he says “We’re approaching more novel ways of doing outcomes, but the tried-and-true surveys are still working effectively. And these are still one of the most cost-effective ways to manage outcomes. Successful outcome measurement is not without barriers to success, but experienced providers know how to navigate getting the audience to provide feedback.” Fission is currently conducting an outcomes project with Outcomes Inc. based on a Postgraduate Institute for Medicine-accredited, Epocrates’ mCME activity launched in late-December in the area of multiple myeloma. Supported by an unrestricted grant from Millennium, Fission developed a real-time CME program based on content from the American Society of Hematology (ASH) annual meeting. In the first eight days of being available, there were more than 80 completes for the lesson, and within the first month of being available, almost 450 certificates have been generated for the lessons. Mr. Nacinovich says the outcomes assessment is not yet complete. The study will include a control group of those who didn’t participate in the education as well with attendees and nonattendees to the annual meeting also being identified to measure the impact of the activity. Ms. Winkelman says the data derived from the surveys are relevant for assessing CME outcomes. “Physicians are highly educated individuals, and they understand their weaknesses,” she says. “If the questions are worded appropriately, we can understand whether they are answering the question just for the sake of answering or whether they truly understand the changes in behavior they need to make.” Experts say surveys also can be used to evaluate not just physician knowledge, but also their confidence in their own abilities to treat a particular disease. “These surveys show that when a physician’s practice is not up to the highest standards, it’s usually because the physician or clinician is not confident enough in his or her skills to use good information or use it consistently,” says Sherri Foster, senior VP and general manager of Scienta Healthcare Education. “The assumption is that physicians would act on knowledge that they’re confident in.” Another metric is adherence to a clinical standard, Ms. Goodrich says. “There are well-established clinical standards and guidelines, and we can look at how often physicians are adhering to the guideline before and after the educational event,” she says. Another way to measure the outcomes of a CME program is to provide participants with case vignettes. “Case vignettes are a very cost-effective and valid way to obtain information about what a healthcare professional does,” Dr. Abdolrasulnia says. “What they self-report through a case vignette is very similar to what they do in their practices.” DiMedix recently participated in an educational outcomes assessment using this technique. The assessment included clinicians who participated in an educational activity related to hypertension, as well as a control group of practitioners who didn’t participate. They were given a case and asked to treat the person based on the seventh report of the Joint National Committee on the prevention, detection, evaluation, and treatment of high blood pressure. Some improvement was noted following the educational activity, however what does this mean in terms of population health? “Doctors may say they understand JNC 7, however, despite countless hours devoted to education on hypertension management in the United States, we know from population assessments that still only about one-third of patients are treated to goal,” Dr. Liston says. “That’s a very large disconnect and representative of the fact that a single educational activity cannot effectuate true behavioral change. Learning and behavior change take place over time through multimodality and sequential educational activities that meet the learners’ needs.” While the Institute for Continuing Healthcare Education has conducted successful outcome measurement programs of their own, one successful outcomes program was through a collaboration with an outcomes vendor using case vignettes. The analysis was conducted after a live symposium, with enduring materials, on the topic of kidney disease-related anemia. Activity participants and a control group of nonparticipants were asked to complete the outcomes survey. From the survey, the Institute for Continuing Healthcare Education was able to determine that education on CKD-related anemia is still warranted. Questions remained about how to manage patients in this disease state, and there was low familiarity with recent clinical study data regarding new formulations and agents, which supported the need for additional educational activities. “We were able to evaluate the impact of the education on those who participated versus the control group,” Ms. Chandonnet says. “We were able to determine that education on the topic is still something that was needed. Participants had ongoing questions about the disease state and that provided us with the needs-assessment information to move forward and develop additional educational initiatives.” Ms. Goodrich says Pri-Med is starting to evaluate the impact of program content and format on how well physicians retain the information. “For some sessions, we will bring in case-based activities into the presentations,” she says. “On the other hand, some sessions will be more didactic while others will be more interactive. We can evaluate the type of impact the number of cases included in the presentation or the level of interaction in the session have on persistent learning. The 2006 trend analysis is under way.” Pri-Med recently completed an in-depth outcomes analysis across all of its programs, which included feedback from almost 50,000 medical professionals who participated in 893 live clinical sessions in 2005. The study includes data covering 48 therapeutic areas, including asthma, diabetes, stroke, heart failure, depression, and erectile dysfunction. From this, they were able to identify physician learning gaps, which help determine where additional education should be focused. Moving the Learnings Forward Experts agree that a single educational event, regardless of the format or channel, is not likely to have a long-term impact on physician behavior. Outcomes studies of various CME programs are bearing this out. For example, Scienta Healthcare Education conducted an outcomes assessment of a program about migraine. This assessment included pre- and post-education surveys asking physicians how confident they are in their ability to treat migraines. The company then did a survey eight weeks later and found physicians had retained little of the learning. “Learning and knowledge had dropped off quite a bit,” Ms. Foster says. “The conclusion we were able to draw is that reiterative learning yields better results than one iteration, and this conclusion is validated in the CME literature.” She says the company is now thinking in terms of umbrella initiatives with multiple interventions over time as opposed to single symposia with enduring materials. “We’re developing programs that include follow up with the physicians who participate from the first event to give them additional learning and an additional chance to reiterate the key learnings from the previous event,” Ms. Foster says. From its outcomes assessment, CME LLC was able to see the impact of a longer-term educational effort. CME LLC created a custom outcomes measurement program to determine how well its initiative on bipolar education met the learning objectives. The initiative was an eight-month program geared toward primary-care physicians to address advances in screening tools. CME LLC found that primary-care physicians demonstrated immediate learning of the program content, and they had some degree of retention three months later. The assessment also was able to identify unmet goals and further educational needs, including increasing primary-care physicians’ use of the mood disorder questionnaire, increasing the knowledge of the American Diabetes Association’s recommendations for monitoring patients taking antipsychotic drugs, and addressing the pharmacological management of patients with bipolar disease. “Flexibility across channels and across formats leads to better outcomes,” Ms. Goodrich says. “The key is giving the practitioner a choice of which channel, which type of activity, and which format is most conducive for learning.” Dr. Liston says there needs to be an initiative to improve healthcare that is focused on quality and performance improvement. “For example, a tool that tracks sequential learning so clinicians can apply knowledge and get a report card of how they’ve performed based on agreed upon national standards,” she says. “That is how behavior change occurs. For this to happen, however, all stakeholders must be aligned, including the provider community, the managed care community, hospital and institution quality improvement/quality assurance departments, and government agencies. This would be a huge academic undertaking.” PharmaVOICE welcomes comments about this article. E-mail us at [email protected]. ACCME update In July 2006, the Accreditation Council for Continuing Medical Education’s (ACCME) board of directors adopted updated compliance criteria for the accreditation elements. The revised model represents a change in emphasis for the ACCME. The ACCME will focus on rewarding accredited CME providers for moving through levels of accreditation while changing and improving their practice of CME. Learning and change will be the goals — both for the learners and for the providers. Within the present framework of essential areas and the standards for commercial support, the ACCME will ask accredited providers: To set a CME mission that focuses CME programs on improving one or more physician competencies, physician’s performance, and/or the physician’s patient outcomes; To strive to meet that mission through their program of CME activities based on practice-based needs; and then To evaluate their success at meeting their change mission; and if possible To engage with their environment to enhance the role of their program, and of CME, in promoting quality and safety. The focus now is on contributing to enhancements to one or more of the physician’s toolbox of strategies for patient care (competence), their actual performance-in-practice, and/or their patient outcomes. The ACCME will be asking providers to establish a specific enhancement mission — providing education interventions to meet that mission — and then to focus on assessing their program’s impact at meeting that mission and improving their program, using internal and/or external strategies. ACCME has updated its compliance criteria for the elements and grouped them into three levels that support this process of program improvement. All providers will need to be in compliance under this revised model by 2012. The ACCME expects that providers will transition their programs to the revised model over the next few years. Source: Accreditation Council for Continuing Medical Education (ACCME), Chicago. For more information, visit accme.org. Heidi Chandonnet Institute for Continuing Healthcare Education Outcomes are one tool that helps an individual, an organization, and an educational provider measure the success and effectiveness of educational initiatives. Sound Bites From the Field PharmaVOICE asked providers of continuing medical education what akes a successful CME program, why outcomes of CME should be studied, and the barriers to studying outcomes. Michael Barnett, M.D., Ph.D., is Executive VP and Director of Medical and Scientific Affairs, Dorland Global Health Communications, Philadelphia, which is a business unit of Dorland Global Corp. For more information, visit dorland.com. “When my car is serviced by the dealership, I get a phone call to survey the level of my satisfaction with their work. My children get report cards, and I get a performance evaluation to elucidate strengths and determine areas in need of attention. In each instance, outcomes are measured to determine impact, assess satisfaction, and solicit feedback to improve service. Not only should the same be true in medical education, but outcomes measurements are becoming more important as CME providers and commercial supporters attempt to advance programming by driving content toward targeted audiences. This is the first of many new approaches designed to further improve patient care by enhancing healthcare providers’ knowledge and skills. But this progression is only useful if outcomes measurement becomes a customary element in educational initiatives. The next big step will be to link measurable outcomes to behavioral change at the patient level. This will undoubtedly cost more but is a requisite to track actual educational impact and determine success.” Lynn Rosen is Senior VP of Professional Relations at ProCom, Parsippany, N.J., part of CommonHealth, a medical education company that creates science-rooted strategic solutions to achieve client objectives. For more information, visit commonhealth.com. “A successful CME program requires three equally important elements: faculty, content, and execution. Ideally a successful CME program is one that entertains the audience while meeting educational objectives. Careful faculty selection involves recruiting experts in the identified therapeutic area that have chemistry with the other faculty and can present well. Placing the well-chosen faculty into a program format such as didactic, roundtable, or point/ counterpoint that allows the faculty to educate in their preferred manner is another important consideration. The faculty enhances the program content by using materials that are new, clinically relevant, and unbiased in nature. Finally, it is important to execute the program flawlessly by recruiting an audience or distributing the program through wide-reaching enduring channels. ” David Toub, M.D., MBA, is VP, Medical Director, of SciFluent Communications, Yardley, Pa., the division of MedErgy HealthGroup company that focuses on medical education. For more information, visit medergygroup.com. “As a physician, I believe that CME programs ideally provide education that can make a measurable and significant impact on patient outcomes. Learning about new therapies, along with filling in gaps regarding disease states and established interventions, are critical so that physicians can provide care that is consistent with best practices. Not surprisingly, best practices are a moving target, as medicine undergoes a constant process of review through clinical research. By analyzing which educational activities provide the most effective learning as reflected in enhanced patient outcomes, CME outcomes measurement enables such programs to be fine-tuned to provide the clinical database and motivation for physicians to provide evidence-based care.” Reggi Veatch is Director of Editorial and Accreditation Services at Innovia Education Institute, Columbia, Md., a medical education firm. For more information, visit innoviaeducation.com. “Continuing medical education is one of the primary ways that physicians stay current with the most timely clinical data, emerging therapies, and clinical practice guidelines after they leave medical school. A few short years ago, high physician attendance at, and satisfaction with, CME activities signaled success. Increasingly, providers and supporters alike want to measure retention, decision-making confidence, and changes in learners’ knowledge, skills, or competencies, track their real-time application in clinical practice, and demonstrate improvements in patient and population health. New metrics and data collection, reporting, and analysis tools are on the horizon to measure the latter. Shorter term, our focus needs to be on understanding and measuring the links between educational objectives, methods of knowledge transfer, and learner preferences in the design of effective, relevant, and independent CME programs.” Meg Webster, R.N., is Managing Director at Excellence in Medical Education (XME), Newtown, Pa., which is dedicated to the advancement of patient care through the development and execution of scientifically rigorous accredited medical education programs for healthcare professionals. For more information, visit xmedicaleducation.com. “A successful CME program requires several things, including a thorough needs assessment that involves digging systematically into the literature to build the foundation. It doesn’t stop there; CME providers have to talk with clinicians and find out what their educational interests and needs are and then incorporate the findings from applicable outcomes measurement activities. An exceptional faculty also is very important. Not only must the faculty members be considered experts in their field, they also need to be able to relate to and communicate with the intended learners. CME should be delivered through meaningful educational formats and be case based. Literature demonstrates that clinicians’ learning is more effective when patient cases are incorporated into the activity. Metrics should provide important information that justifies the completed activity and yields valuable input that can be incorporated into future needs assessments. CME programs should be fiscally responsible and budgets need to be managed responsibly. This is the glue that holds everything together — from the science to the delivery. It is all critical. This is what sets apart great CME programs and medical education companies.” Sherri Foster Scienta Healthcare Education Our charge is to improve the quality and safety of healthcare in America. The question is, though, how do we know we’re doing that if we’re not measuring the impact of our programs? Dr. Mazi Abdolrasulnia Outcomes Inc. One of the trends is a continuous pushing of the frontier of measurement into patient health measurement. Experts on this topic Mazi Abdolrasulnia, Ph.D. VP of Business Development, Outcomes Inc., Birmingham, Ala.; Outcomes Inc. is a full-service evaluation company dedicated to providing quality and objective measurement for providers of continuing healthcare education and patient education. For more information, visit ceoutcomes.com. Heidi Chandonnet. Senior Director, Accreditation and Compliance, Institute for Continuing Healthcare Education Inc., Philadelphia; The Institute provides certified continuing education activities for physicians, pharmacists, nurses, dietitians, and other allied health professionals. For more information, visit iche.edu. Sherri Foster. Senior VP, General Manager, Scienta Healthcare Education, Durham, N.C.; Scienta collaborates with national and international faculty to develop clinically relevant, evidence-based continuing medical education in a variety of therapeutic areas. For more information, visit scienta-edu.com. Anne Goodrich. Director of Physician Insights, Pri-Med LLC, Boston; Pri-Med is a leading provider of accredited continuing medical education programs and educates physicians through a network of programs that are linked across channels. For more information, visit pri-med.com. Heidi Liston, Pharm.D. General Manager, DiMedix LLC, Warren, N.J.; DiMedix is a full-service medical education company that exclusively develops and executes a new standard of CME programs that educate healthcare professionals and improve patient care. For more information, visit dimedix.com. Mario R. Nacinovich Jr. Senior VP, Fission Communications, New York; Fission is a global, full-service, medical education and communications company. For more information, visit fissioncommunications.com. Barbara Winkelman. VP, Marketing and Multimedia, CME LLC, Irvine, Calif.; CME LLC delivers high-quality continuing medical-educational activities developed by influential clinicians and leading healthcare researchers. For more information, visit cmellc.com.