Using a Simulated Practice to Improve Practice Management Learning

640 October 2009 Family Medicine Practice Management Using a Simulated Practice to Improve Practice Management Learning Leigh LoPresti, MD; Patrick Ginn, MD, MBA; Robert Treat, PhD Background and Objectives: Practice management education is required in family medicine residencies, and requirements have recently been expanded. Surveys show that graduates feel unprepared to address practice management after graduation, so it is an aspect of training in need of improvement. Methods: We substituted a simulated practice for part of an established didactic program in practice management. The curriculum included 20 modules with specific practice management tasks to be completed by the residents. An objective examination covering a broad range of practice management topics was created and given twice to residents, once early in their second year and again at the end of third year. One group of residents participated in the simulated practice curriculum, and a comparison group participated in a standard didactic curriculum. Results: Our pre-test and post-test examinations each produced reliable data. Residents with simulated practice training had statistically significant increases in exam scores while the comparison group did not. The simulated practice group also increased scores on every subsection of the exam, while the comparison group increased scores on only half of the subsections. However, only one resident (in the intervention group) achieved >60% correct answers on the post-test examination. Conclusions: The increased learning demonstrated by improved test performance suggests that a simulated practice approach is more helpful than a standard didactic curriculum for teaching residents about practice management, but it still does not yield knowledge scores at an optimal level. (Fam Med 2009;41(9):640-5.) Family medicine was the first specialty to require practice management training for residents. 1 There are a number of recommendations about what practice management topics should be taught and the amount of time that should be devoted to that training. For example, the Accreditation Council for Graduate Medical Education (ACGME) recently increased the requirement for practice management by 67%, to 100 hours, and the ACGME lists specific knowledge and skills to be included in that training. 2 The American Academy of Family Physicians (AAFP) also outlines desired knowledge, skills, and attitudes in practice management. 3 Halpern et al combined nine reports on curricular reform for the emerging practice environment and included yet another list of recommended topics From the Department of Family and Community Medicine (Drs LoPresti and Ginn) and Office of Educational Services (Dr Treat), Medical College of Wisconsin. in practice management. 4 Despite this attention to the content of practice management training, surveys show that graduates of family medicine residency programs feel poorly prepared to address practice management issues. 5,6 While they are the best prepared among all specialties in medicine, only 12% felt well prepared. 7 In addition to diverse recommendations for what should be taught about practice management, there are also no universally accepted methods for how to teach this topic. Lectures are still the predominant form of practice management education for most family medicine residents. However, lectures alone may not be optimal because lectures often result in non-maintained increases in knowledge, 8,9 and they do not appear to change behavior, even in the short run. 10 Research has shown that students clearly prefer interactive approaches, but even interactive learning has not been proven superior for achieving knowledge increases. 6,11-13 One study of an interactive format showed improvements over a 1-month testing interval, 14 but studies looking at retention of knowledge show deterioration over 4 8 months. 15,16

Practice Management Vol. 41, No. 9 641 Several programs have reported a more hands-on approach for practice management education. 17-22 A limitation in assessing the success of many of these approaches, however, is a reliance on subjective outcomes. Surveys of residents, program graduates, and program directors are the most commonly reported evaluation tools in these studies. For example, Bayard et al reported on residents learning practice management by developing mock practices. They evaluated their approach with pre- and post-experience resident self-assessments of practice management knowledge by topic. All of the residents agreed or strongly agreed that the course had been useful in increasing their knowledge, but there was no objective assessment of whether the residents knowledge actually did increase. 20 Indeed, there are few high-quality studies that include objective evaluations of residents knowledge gains following any form of practice management education. Babitch developed a 9-month curriculum and compared pretests and immediate posttests, but that study reported neither reliability nor any other psychometric properties of the test methods used. 23 Crites created a 12-month curriculum and used a 12-question true/false test given as a pretest with a posttest immediately after the last session. They reported face validity for their exam but were not able to report reliability due to sample size. 21 Given these limitations, the effectiveness of the many recommended practice management curricula is unclear. Practice management is not reported as a separate category for the in-training or board certification exams, and there are no widely accepted objective measurements of success. Thus, prior literature provides little evidence that any approach to practice management education has succeeded. Four family medicine residencies, including our program, share a lecture series on practice management. All of the programs provide additional practice management education. We converted our residency s additional education time to a simulated practice approach. We created a more extensive experiential curriculum than any that had been previously evaluated. Bayard et al reported work equivalent in scope, but they did not fully evaluate their curriculum. 20 We report here on both our simulated practice curriculum and the first reliable examination of practice management knowledge. The exam fills a gap in the literature: a reliable, objective test for the subject of practice management. Using the exam in a pretest/ posttest design, we compare the new curriculum in our program, which includes both the lecture series and the simulated practice, to the curriculum used by the other three programs in our group that shared the lecture series. Methods Overview The four residency programs involved in this study shared a core lecture series on practice management. Each program then added its own materials and teaching sessions to reach the required total of 100 hours of practice management education. Second-year residents in our program (the intervention group) used the simulated practice curriculum. Second-year residents in the three other programs (the comparison group) received their additional hours through course offerings provided by their individual residencies, none of which involved a simulated practice experience. Our research methods were designated as exempt by our Institutional Review Board. The Simulated Practice Experience The simulated practice part of our practice management curriculum involved 60 hours of education packaged into 20 modules (Table 1). Each module included a lecture of about 1 hour, and 19 of the 20 modules also involved 2 hours that were dedicated to projects. For 11 of the modules, the projects were completed by residents who worked in teams of two. Six modules had individual projects, and two had in-class exercises conducted by the entire group of residents. The projects were intended to simulate decisions they would make in practice, in accordance with the principles of adult learning 24 in which projects are designed around reallife issues. Some references were provided, but full answers required the residents to search out their own resources to answer the question(s). Module leaders (faculty members) provided support as needed. Participants Residents from all four programs were recruited to participate in the study during their second year of residency, and participation continued into the third year. Consent was sought from 24 residents, of whom 20 consented (six in the intervention-group program and 14 in the three comparison-group programs). We paid residents $25 for completing a pre-course examination, $25 for completing a post-course examination, and an additional $25 if they completed both examinations. Examination We created a practice management examination with 46 questions involving 14 topic areas (Table 2). Item construction used the guidelines of the National Board of Medical Examiners (NBME). 25 The examination contained both multiple choice and pick N questions. Per the NBME guidelines, test items were designed to assess application of knowledge, not recall of an isolated fact. One author who has 15 years of full-time practice with practice management responsibilities in a variety of settings wrote the examination. It was re-

642 October 2009 Family Medicine Table 1 Project Topics for the 20 Modules in the Simulated Practice Curriculum Topic Project Assignment Question Category on Test Projects completed by two-resident teams Market analysis and strategic planning Office space and office design Office equipment Information systems Policy and procedure manuals Staffing an office and personnel management Leadership Continuous Quality Improvement (CQI) Marketing Scheduling and triage Reading insurance contracts Alphabet soup (government agencies and your practice) Professional relationships Interviewing Paying providers and reading employment contracts Financial planning and personal finance Personal and professional success Pick a specific site in which to practice, and do a market analysis and strategic planning for your practice. Given a shell of set dimensions, design an office layout that maximizes use of space and customer comfort. Choose equipment for your office and choose buying versus leasing. Do return on investment analysis on three items of capital equipment in both purchase and lease scenarios. Choose an electronic medical record vendor and discuss the positives and negatives of your choice; determine your network configuration and what security it will require. Write procedures on payroll, telephone calls, medication refills, and lab and X-ray results for your office manual. Decide what kind of staff and how many of each you will hire for your office. Defend your choices. Identify a problem in the residency practice and identify: What leadership position you hold with regard to the issue(s) Who the stakeholders are What are the specific issues Who has to change to successfully resolve the issue The level of stress the problem is creating In what order you would address the issues Use CQI approaches to map out the process and issues for the problem chosen in leadership (if appropriate). Design a data collection you would do to get the most information. Develop a seven-word or less marketing core idea. Develop a marketing strategy to convey that core idea (identify media to be used, frequency and reach if applicable). State how this fits your practice and strategic plan. Choose one of the strategic items and prepare either a Web page, a radio ad (actually record it!), or a print ad. Design a schedule for your office. Consider scheduling style (eg, open access versus fixed appointments), types/lengths of appointments, use of limited facilities (procedure room, single pieces of equipment purchased earlier), hospital rounds/ deliveries if appropriate to strategic plan, and call schedules. Analyze a provided contract, pick out a defect, and outline your approach to negotiating this Projects completed by individual residents Choose an agency and report on agency and its impact on medical practice. Pick a professional from the list provided (eg, accountant, attorney, etc) and report on criteria and mechanisms for choosing as well as estimated costs. Each resident prepares a curriculum vitae and is interviewed for a job by a faculty member. Analyze a provided contract, pick out a defect, and outline your approach to negotiating this. Generate a 5-year financial plan. Find an approach to this that appeals to you. Market analysis and strategic planning Office space and office design Office space and office design; practice finance Computer technology Staff hiring and management; policies and procedures Staff hiring and management; policies and procedures Leadership Continuous Quality Improvement Marketing Office scheduling Negotiation and contracting Federal agencies, laws, and regulations Insurance and personal finance No questions on this Negotiation and contracting Insurance and personal finance No questions on this Topics with in-class exercises Negotiation Conduct a faculty supervised negotiation, followed by lecture, then re-do negotiation. Negotiation and contracting Practice finance Using an internally developed spreadsheet, explore the parameters of practice finance: Practice finance how many patients per day do you need to see to make certain levels of income? How do extra staff impact your income? How does hiring a partner or a physician extender affect your income? Topics with no project Taxation None Taxation

Practice Management Vol. 41, No. 9 643 viewed by family physicians with knowledge of practice management to validate the content of the test. The examination was challenging, and we estimated that the average pre-test score would be 30% 35% correct responses. The pretest was given between September 1 and October 17, 2005 (early in second year of residency). The posttest was given between April 13 and June 10, 2007 (late in third year). During the inter-exam interval, the answers to one computer technology question changed, and an additional answer was accepted as correct on the posttest. The test was administered using an online electronic testing system available to all residents (ANGEL Learning Management Suite by ANGEL Learning, Inc, Indianapolis). Using ANGEL allowed both the test questions and the answers to questions to be presented in random order. Data Analysis The original examination had 46 questions. Test questions were examined for internal consistency with Cronbach s alpha. The original Cronbach alphas on the pretests and posttests were.67 (n=17) and.66 (n=11), respectively. Six questions with negative item discrimination values were found to decrease the Cronbach alpha and were removed from analysis, and the Cronbach alpha values increased to satisfactory levels of 0.72 for both pretests and posttests. More detailed pretest/posttest reliability testing was precluded by the anonymous nature of the testing. The six questions eliminated were all multiple-choice questions. Examinations were re-scored with the six questions removed, and we report the results for the 40-question exam. Of the 40 questions, 25 were multiple choice, and 15 were Pick N type questions. There were one to five questions per topic on the exam (Table 2). All topics Table 2 Comparison of Pretest and Posttest Scores in the Intervention and Comparison Groups Question Category # of Questions Intervention Group Comparison Group Pretest (n=6) Posttest (n=5) Difference Pretest (n=11) Posttest (n=6) Difference Topics with only team projects Market Analysis and Strategic Planning 4 54% 83% +29% 47% 46% -1% Office Space and Design 2 47% 60% +13% 36% 47% +11% Computer Technology 3 38% 44% +6% 27% 26% -1% Staff Hiring and Management; Policies and Procedures 5 69% 71% +2% 60% 59% -1% Leadership 2 39% 40% +1% 42% 39% -3% Continuous Quality Improvement 2 17% 30% +13% 23% 25% +2% Marketing 2 33% 50% +17% 32% 25% -7% Office Scheduling 1 67% 70% +3% 55% 25% -30% Topics with only individual projects Federal Agencies, Laws, and Regulations 1 17% 100% +83% 0% 17% +17% Insurance and Personal Finance 3 8% 45% +37% 23% 25% +2% Mixed approaches Negotiation and Contracting 5 52% 63% +11% 39% 43% +4% Practice Finance 4 43% 53% +10% 31% 24% -7% No project Taxation 2 33% 35% +2% 36% 46% +10% Not taught in simulated practice curriculum Coding and Billing 4 50% 65% +15% 32% 33% +1% Overall exam Overall exam 40 41% 56% +15% 35% 36% +1% Percentages indicate the percentage of correct responses to questions on the examination.

644 October 2009 Family Medicine had at least two questions before the six questions were eliminated. Data were analyzed with SPSS 15.0. The pretest and posttest exam scores were treated as independent groups as the data were collected anonymously, and pretest and posttest scores were not linked to individual participants. Statistically significant differences in scores were ascertained via Mann-Whitney U-test since there were small deviations in the scoring distributions from normality (Shapiro-Wilks W tests). The data analysis compared performance on pretests and posttests for the intervention group versus the three other residencies aggregated as the comparison group. Results Intervention group residents attended the seminars on an afternoon otherwise reserved for lectures. However, patient care responsibilities (eg, call, continuity deliveries) and administrative issues (eg, work-hour restrictions) took precedence and resulted in less than complete attendance. Attendance for individual residents ranged from 40% 80%, with an overall mean of 66%. Seventeen residents completed the pretest, and 11 completed the posttest. All 11 who completed the posttest had completed the pretest (five in the intervention group and six in the comparison group). One person in the intervention group completed the pretest, but not the posttest, because of maternity leave at the time of the posttest. The dropout rate (ie, failure to complete posttest after completing the pretest) was high in the comparison group, despite each resident receiving up to three reminders to complete the test and the offer of compensation. Reasons for noncompletion included early Table 3 Additional Comparison of Scores on the Pretest and Posttest Group and Test Between-group Comparison: Pretest Intervention Group (pretest; n=6) 41.0 (6.7) Comparison Group (pretest; n=11) 35.5 (12.3) Between-group Comparison: Posttest Intervention Group (posttest; n=5) 55.8 (3.6) Comparison Group (posttest; n=6) 36.1 (4.1) Within-group Comparison: Pretest/Posttest Comparison Group (pretest; n=11) 35.5 (12.3) Comparison Group (posttest; n=6) 36.1 (4.1) Within-group Comparison: Pretest/Posttest Intervention Group (pretest; n=6) 41.0 (6.7) Intervention Group (posttest; n=5) 55.8 (3.6) SD standard deviation Mean Test Score (SD) (% correct answers) P Value graduation from residency and nonresponse. Dropouts included both male and female residents. The intervention group and the comparison group performed similarly on the pretest. The intervention group s posttest performance exceeded their own performance on the pretest and also the performance of the comparison group (Table 3). Residents spent comparable amounts of time on each test; they averaged 22 minutes and 23 minutes to complete the pretest and the posttest, respectively. In pretest/posttest comparisons, the intervention group improved their overall scores in all of the tested topic areas. The comparison group improved in only seven of 14 areas. In the seven areas where they did better, their improvement exceeded that of the intervention group in only one topic (Taxation). For the topic not covered in our simulated practice curriculum (Coding and Billing), both groups improved. Resident scores in the intervention group improved more in the individual project group, suggesting this may be a better approach (Table 2). We believe this is an artifact of their poor pretest scores in those areas. The combined posttest scores for the team project, individual project, and mixed methods topics were each 59%..546.006.615.006 Discussion Our practice management examination was judged to have content validity, and it demonstrated an adequate level of reliability in both pretest and posttest data. The intervention group, which took part in the simulated practice curriculum, had significant improvements in overall examination performance while the comparison group did not. Our findings are important to family medicine educators for several reasons. One is that ours is the first study to evaluate the outcomes of a practice management curriculum with a knowledge assessment that has good reliability and content validity. Another is that our simulated practice approach appeared to have resulted in better knowledge acquisition than standard teaching approaches. These findings suggest that the simulated practice approach to practice management education may have value and can be considered for implementation in other residency programs. Limitations Despite the positive findings, there are several limitations to our study methods. The limitations should be considered when interpreting the results of our study. First, while our simulated practice curriculum led to a statistically significant

Practice Management Vol. 41, No. 9 645 improvement in performance on a valid and reliable examination, only a single resident exceeded a score of 60% on the posttest. Thus, knowledge acquisition was not as great as we had hoped. Still, two of the three largest improvements in our residents test scores came in modules taught at least 15 months prior to the posttest. This is a substantial amount of time for knowledge retention based on prior experience. 15,16 We believe this may be our most significant finding, as acquisition of knowledge is not as important as acquiring and retaining knowledge. A second limitation is the small sample size and high drop-out rate in our study. This reduces generalizability and precludes the use of more powerful statistical tests. Further, the anonymous participation of residents in this study prevented the pairing of pretest and posttest datasets and reduced statistical power. Despite this limitation, however, we were able to demonstrate significant improvement in knowledge acquisition, suggesting that the simulated practice approach was generally successful though again, absolute levels of knowledge, with only one resident exceeding a score of 60%, were less than optimal. A third limitation is that while our examination questions had good content validity, a more powerful, data-dependent form of validity, which should be sought in a future analysis, is construct validity. Our review of the literature found two other published studies on practice management education that used a pretest/posttest design, 21,23 and neither of those studies report on the reliability or construct validity of their tests. Thus, no test of practice management knowledge in the literature, including ours, has reported construct validity of the data. Fourth, we did not perform a detailed review of the curriculum of the programs in the comparison group. We know that those programs did not use a simulated practice approach but do not know if the specific items taught to residents in those programs were a match for topics on the post-course examination. However, all of the topics on the test are found in the AAFP Recommended Curriculum Guidelines for Practice Management. 3 Conclusions Using our simulated practice approach to teach practice management improved residents knowledge and appeared to yield better results than standard educational approaches based on assessment with a knowledge examination that has good reliability and content validity. Residents knowledge, however, despite improving as a result of the simulated practice approach, was still not at an optimal level. Further work is needed to identify additional effective methods for teaching practice management. Corresponding Author: Address correspondence to Dr LoPresti, Medical College of Wisconsin, Waukesha Family Medicine Residency Program, 210 NW Barstow Street, Waukesha, WI 53188. 262-513-7310. Fax: 262-928-4075. leigh.lopresti@phci.org. Re f e r e n c e s 1. Rose EA, Neale AV, Rathur WA. Teaching practice management during residency. Fam Med 1999;31(2):107-13. 2. Accreditation Council for Graduate Medical Education. Family Medicine Program Requirements. www.acgme.org/acwebsite/downloads/ RRC_progReq/120pr07012007.pdf. Accessed February 13, 2008. 3. American Academy of Family Physicians. Recommended curriculum guidelines for family practice residents in practice management. Reprint number 268. Leawood, Kan: American Academy of Family Physicians, 2003. 4. Halpern R, Lee MY, Boulter PR, Phillips RR. A synthesis of nine major reports on physicians competencies for the emerging practice environment. Acad Med 2001;76(6):606-15. 5. Norton P, Jones JE, Wang MQ, Tulli CG. Incongruence of existing practice management curricula content and actual medical practice need. Fam Med 1994;26(2):98-100. 6. Stone MA. Family physicians evaluation of the practice management education received during residency training. Fam Med 1994;26(2): 101-5. 7. Cantor JC, Baker LC, Hughes RG. Preparedness for practice: young physicians views of their professional education. JAMA 1993;270(9): 1035-40. 8. Winter R, Picciano A, Birnberg B, et al. Resident knowledge acquisition during a block conference series. Fam Med 2007;39(7):498-503. 9. Picciano A, Winter R, Ballan D, Birnberg B, Jacks M, Laing E. Resident acquisition of knowledge during a noontime conference series. Fam Med 2003;35(6):418-22. 10. Wendling A, Green B. The effect of a didactic lecture on resident behavior. Fam Med 2005;37(6):386-7. 11. Costa ML, van Rensburg L, Rushton N. Does teaching style matter? A randomized trial of group discussion versus lectures in undergraduate teaching. Med Educ 2007;41:214-7. 12. Fischer RL, Jacobs SL, Herbert WNP. Small-group discussion versus lecture format for third-year students in obstetrics and gynecology. Obstet Gynecol 2004;104:349-53. 13. Beers GW, Bowden S. The effect of teaching method on long-term knowledge retention. J Nursing Educ 2005;44(11):511-4. 14. Schackow TE, Chavez M, Loya L, Friedman M. Audience Response System: effect on learning in family medicine residents. Fam Med 2004; 36(7):496-504. 15. Kaczorowski J, Levitt C, Hammond M, et al. Retention of neonatal resuscitation skills and knowledge: a randomized controlled trial. Fam Med 1998;30(10):705-11. 16. Curran VR, Aziz K, O Young S, Bessell C. Evaluation of the effect of a computerized training simulator (ANAKIN) on the retention of neonatal resuscitation skills. Teach Learn Med 2004;16(2):157-64. 17. Lawson JG, McConnell JW. Teaching practice management in a family practice residency. J Med Educ 1976;51:858-60. 18. Aluise JJ. The physician as manager: what and how of practice management education. J Fam Pract 1977;4(2):305-11. 19. Werblun MN, Martin LR, Drennan MR, Parker CE, Berggren RE. A competency-based curriculum in business practice management. J Fam Pract 1977;4(5):893-7. 20. Bayard M, Peeples CR, Holt J, David DJ. An interactive approach to teaching practice management to family practice residents. Fam Med 2003;35(9):622-4. 21. Crites GE, Schuster RJ. A preliminary report of an educational intervention in practice management. BMC Med Educ 2004;4:15. 22. Taylor ML, Mainous AG III, Blue AV, Carek PJ. How well are practice management curricula preparing family medicine residents? Fam Med 2006;38(4):275-9. 23. Babitch LA. Teaching practice management skills to pediatric residents. Clin Pediatr 2006;45:846-9. 24. Kaufman DM. Applying educational theory in practice. BMJ 2003; 326:213-6. 25. Case SM, Swanson DB. Constructing written test questions for the basic and clinical sciences, second edition. Philadelphia: National Board of Medical Examiners, 1998.