submit a Training Paper on MPAC s
CONTENTS 1 EXECUTIVE SUMMARY 3 Background 3 Solution 3 Conclusions 3 2 BACKGROUND 5 Training Objectives 6 Training Challenges 6 3 TRAINING SOLUTION 8 Blended Learning Approach 8 Training Resources 10 Development Process Overview 14 4 FEEDBACK AND CONTINUOUS IMPROVEMENT 16 Package 1.01 Results 16 Package 1.02 Results 17 Package 1.03 Results 17 5 CONCLUSION AND LEARNINGS 19 Page 2
1 EXECUTIVE SUMMARY Background The (MPAC) undertook a significant change management initiative several years ago when it was decided that the current computer infrastructure would not meet the organizational need in the future. A large, multi-year information technology project - was launched to create and rollout new functionality affecting every employee and many of MPAC s stakeholders. Recognizing the change management issues, MPAC created a Training Team focused on identifying and developing an education and support strategy that would enable MPAC staff to conduct their business with minimal impact on their stakeholders. The Training Team quickly realized that a traditional education and support strategy would be highly risky. Obstacles that had to be addressed to meet an aggressive timeline included: creation of educational resources in parallel with the system development; a shortage of subject matter experts; and employees in different sectors of the organization who had unique education needs and were geographically distributed. The training challenge was to ensure the IPS rollout encouraged all staff to fully utilize the system both during and after their training. Solution The Training Team decided to execute a blended learning strategy which incorporated the following elements a quality Train-the-Coach program, an engaging training kick-off event, a robust and tracked elearning self-study curriculum, and a highly experiential facilitator-led event followed by one-on-one coaching when required. Conclusions The results were exceptional and the rollout achieved all of the objectives established by the Training Team at the outset of the project. Based on extensive employee feedback, it was concluded that the blended learning strategy was more effective than an approach using only elearning or facilitator-led components. Although it was challenging during a system development project, it was critical that employees had the opportunity to work with the new system early and frequently to ensure their training was real and experiential. Building confidence and enthusiasm for the new functionality was critical to answer the question Why are we changing?. Page 3
Finally, a key to the success of the rollout was the establishment of a separate Training Team that was closely connected with the system development activity, but focused on how the system would be integrated into each employee s function. Since the processes and tools were going to change for everyone, it was critical for a highly skilled team to ensure the change would occur as smoothly as possible from a change management perspective. Page 4
2 BACKGROUND (MPAC) is progressing through a multi-year information system development project which involves the development of a new information technology platform (functionality and database) called the Integrated Property System (IPS) to support the current business processes of 1200 knowledge workers. To ensure the changing system does not have a negative and disruptive impact on the current business, a considerable focus is being placed on the rollout and education strategy and effort. The rollout must achieve the objective of ensuring everyone fully utilizes the new system. The development effort has involved the creation of discrete packages of functionality synchronized to the Ontario Government needs and integrated with MPAC s annual business activities. The rollout of functionality was divided into three specific cycles. Enquiry (Package 1.01) was piloted in April and May 2005; Enquiry and Reassessment (Package 1.02) was rolled out in February and March 2006; and Maintenance (Package 1.03) in March and April 2007. The Training Team recognized that this development and rollout strategy would present a number of training challenges. These included: 1. The requirement to develop training resources in parallel with the system development activity to meet the timing objectives of the rollout. 2. There would be no single source of knowledge about the functioning of IPS in different business units. As an example, the Business Analysts would have a very good understanding of the functionality of IPS and the business Subject Matter Experts (SMEs) would understand how the processes were being conducted in the individual business units. It would be necessary to capture and integrate these disparate sources of information to ensure the effective use of IPS for the continuity of the business processes. 3. Different MPAC employee audiences would require different IPS training information to allow them to carry on their responsibilities. In addition, it was expected that the sequence in which training content would be presented to the audiences would differ from one group to another. 4. To help ensure a successful rollout of the new system, the Training Team was asked to evaluate and employ leading edge training approaches. Equally important was the need to ensure that these training approaches would work within the MPAC environment and culture. Page 5
5. Finally, it was important for the Training Team to develop a training plan and delivery strategy to satisfy both the tight scheduling requirements and the training needs of a geographically distributed workforce for not only the initial training but for ongoing reference. Training Objectives The Training Team must ensure the IPS rollout encourages all staff to fully utilize the system both during and after their training. Objectives include to: 1. Ensure the continuity of the business is maintained over the transition period; 2. Ensure the introduction of the new system is a success; 3. Offer individuals the opportunity for individual self-paced learning as well as to support group learning events; and 4. Provide training resources that could be sustained. Training Challenges With these objectives in mind, the Training Team determined that their training strategy would have to satisfy the following needs: 1. To establish a mindset on the part of the users that this system would make their jobs easier and assist them in doing their jobs better; 2. To support the adoption of new standard operating procedures enabled by the functionality of the new system; 3. To present detailed information on approximately 250 use cases organized into lessons of 30 to 60 minutes of instruction time; 4. To enable the content to be organized into curricula for eight target audiences; 5. To allow students to practice the application of the business procedures using simulation exercises; 6. To supply supporting documentation to compliment the elearning lessons and simulations; 7. To capture competency information about each student to support the design and execution of custom curricula and assess student comprehension; and Page 6
8. To define the learnings from each phase of the training and continuously improve the training approach for the subsequent stages. Page 7
3 TRAINING SOLUTION The Training Team considered the following three alternative training strategies: 1. Conduct facilitated training sessions with groups of 15 to 20 students. This solution was rejected because of the large number of lengthy facilitated sessions that would be required over a very short time frame. In addition, there were insufficient SMEs who were knowledgeable about both the functioning of the system and the business processes being executed within the different business units. 2. Develop a set of elearning resources that the students could access and complete on a self-study basis. Students who participated in a pilot of this method indicated that they needed the opportunity to interact with peers and with facilitators to address specific questions and that a strictly self-study option did not satisfy that important requirement. This conclusion was reached after reviewing the results and the feedback of a pilot elearning course which was developed to support the Guidelines for the Release of Assessment Data (GRAD) program. 3. Adopt a blended learning approach that employed the strengths of both the facilitated approach and an elearning supported self- study approach. In addition, it was decided to structure specific feedback mechanisms to ensure the experience from each stage of the blended learning approach was factored into the next training stage. A pilot of the Enquiry Package provided concrete feedback that the blended learning approach was the best approach for supporting the IPS training of MPAC employees. A key to the blended learning approach was the use of a Learning Management System. This was an on-line environment that allowed the coaches to assign lessons to students and to track the student s interaction with the various lessons and evaluations. Blended Learning Approach The blended learning approach consisted of these five key steps: 1. Conduct a Train-the-Coach event - This session was designed to provide the coaches with all of the tools that were required to deliver the Coaching Event (Step 4). The objective was not to make the coaches SMEs on IPS, but instead, to instil comfort and confidence in their task of delivering the presentations and activities. The training group demonstrated each element of the agenda and then provided the coach with the opportunity to execute that same element on their own and receive peer feedback. 2. Setting the expectations and clarifying roles and responsibilities - This was typically structured as an hour and a half kick-off or preview event, in which line management, IPS project management and the coaches provided the background Page 8
information for IPS and explained the training approach and timing. This also included a question and answer session designed to allow the participants to deepen their understanding of IPS, why the organization was adopting it and how MPAC staff would be supported in using it. 3. Self-study using elearning resources - The students were assigned curriculum that outlined the one-hour elearning lessons that they would take and the sequence in which the material was to be reviewed. Typically, the student was provided one to two weeks (depending on the number of lessons assigned) to cover the material in preparation for the prescheduled facilitated event. The coaches were responsible for monitoring the Learning Management System that tracked which students had completed the assigned lessons. Often, the coaches would problem solve with the student to ensure the material was covered appropriately. Based on early feedback of needing active involvement with the system, students were provided with an IPS training environment that allowed them the opportunity to gain hands-on experience using the system. 4. Clarification and hands-on practice in facilitator-led sessions - The facilitated session ran from half a day to one day depending on the amount of material being covered. Since the students had already reviewed the background information in the elearning lessons, it was possible to structure the facilitated events to focus on discussing and responding to quality questions and to completing a variety of exercises that allowed the student to apply the knowledge developed in the self-study phase of the program. 5. One-on-one or small group Coaching - Following the facilitated event, the students were encouraged to review the elearning resources and use the IPS training environment to deepen their understanding and to build confidence in the use of IPS. Local coaching sessions were arranged, as required, to provide this in depth and personal support for any individuals and or small groups. The blended learning process is depicted in the following graphic, which also indicates the relative amount of time the student devoted to each portion of the blended learning approach. Page 9
One-on-one Facilit ated event Self-study Kick-off / Preview Train-the-Coach Figure 1: Blended Learning Process Training Resources The training strategy required the creation of a variety of training resources including the following components: 1. elearning lessons At the outset of the project, a team of SMEs identified the target groups for the training and outlined the topic areas that needed to be developed in support of the training. The topics were structured into a set of 55 one-hour lessons. Each lesson had a very specific structure including the learning objectives, an overview of the new IPS functionality compared to the previous method and a detailed walkthrough (text and screen snaps) of how to complete specific tasks within IPS. Page 10
Figure 2: Sample lesson page 2. Interactive evaluations For each elearning lesson, a set of 10 to 15 evaluation questions were authored to enable students to self evaluate their understanding of the lesson material. Figure 3: Sample evaluation page 3. Interactive simulations For targeted areas, elearning simulations were developed. These were scenarios that enabled the student to interact with what appeared to be a Page 11
live IPS system; with the exception that they were asked to execute a specific set of actions and received feedback if they did not execute the step correctly. Figure 4: Sample simulation exercise page 4. Curriculum The lessons and evaluations were organized into eight curricula for targeted audiences. This approach was used to ensure that students review only the material relevant to their department. 5. Learning Management System The lessons and evaluations organized by curricula were hosted to an on-line environment that enabled the assignment of specific lessons and evaluations to targeted student groups by department. In addition, this environment tracked the interaction of the student to the material, i.e., did they access the lesson, how long did they review it, what test score did they obtain, etc.? Page 12
Figure 5: Learning Management System - student assignments page 6. Training Resource Documents Based on feedback from the students, it was determined that a number of documents would be required to support the use of IPS on the job. For example, a Summary Document was created for targeted topic areas. These documents were typically two pages with the first page providing a high level overview of the steps required to complete an IPS task. This overview was cross-referenced to a screen snap of a key IPS screen. The second page provided a more detailed step-by-step walkthrough of the task. An IPS Data Guide was developed, which mirrored the IPS application, and included an explanation of all of the tabs, fields and codes within the application. An IPS Glossary was created describing all new terminology within the application. An inventory of IPS reports and extracts was developed including information on where to access, generate and save reports extracts. Page 13
Figure 6: Sample 2-page summary document 7. Coaches Support Resources The Train-the-Coach event, conducted at the outset of each of the package rollouts, was supported with a coaches binder. This binder contained all of the resources that the coaches would require to conduct a quality facilitated session, i.e., Event agenda, PowerPoint slides, Summary Documents, and Exercise scripts). In total, the Training Team managed the development of 166 elearning resources, the IPS Data Guide, the IPS Glossary, an inventory of IPS generated reports and 11 Summary Documents in support of the rollout of packages 1.01, 1.02 and 1.03. Development Process Overview Each of the elearning components was developed with the efficient input of multiple SMEs and often in parallel with the development of the IPS system. The steps included the following: 1. Facilitated videotaped lesson interview. This was typically a two-hour session in which the learning objectives were developed for the lesson and a walkthrough of the IPS system was conducted and videotaped. 2. The videotape was used by an Instructional Designer to produce a Storyboard document. This Word document was augmented with screen snaps from IPS. This document typically captured approximately 60% of the content required for the lesson. 3. Conducted a facilitated edit session with the SMEs. This session involved a detailed review and update of the draft storyboard. At this point, terminology was confirmed and interactive quizzes were defined. Page 14
4. The edited storyboard, quizzes and simulations were used to create the first release of the elearning lesson. 5. The SMEs provided feedback and the storyboards were adjusted to create a second version of the elearning component. This process efficiently utilized the busy SME s time, as it required only three hours of their involvement for each hour of developed content. Page 15
4 FEEDBACK AND CONTINUOUS IMPROVEMENT The blended learning approach evolved with each step of the rollout as a result of the feedback from the participants. In total, there were three specific steps taken by the Training Team to evolve the training strategy and ensure that it met the needs of the students. These steps included: 1. GRAD elearning and Feedback - This involved the development of a single elearning lesson and the implementation of a Learning Management System. This enabled the team to test the functionality of the application and gauge the reaction of the students. This test confirmed that an elearning approach functioned from a technical perspective; however, the usage and feedback suggested that elearning on a stand-alone basis would not meet the objectives the Training Team had established. It was concluded that a blended learning approach should be piloted. 2. Pilot 1.01 Feedback and Learnings - This pilot included many of the blended learning components discussed previously. Coach and student feedback was collected and evaluated. The results confirmed the value of this training approach. In addition, the feedback suggested that students should be provided access to IPS during the selfstudy period and the blended learning approach was adjusted accordingly. 3. Rollout 1.02 and Rollout 1.03 Feedback and Learnings Again, coach and student feedback, as well as, usage statistics from the Learning Management System were used to confirm that the training strategy was working well. Package 1.01 Results The average feedback rating for the overall value of the program was 7.5 with 91% of the participants indicating that they found the training program to be considerably helpful. The average feedback rating for the degree to which the training program met the participants expectation was 7.8. Discussion Forty percent of the participants indicated they found the elearning most helpful while thirty percent indicated the follow-up coaching session was very helpful. Feedback suggested the exercise developed for the coaching event was logistically difficult because of the number of participants and the length of the exercise. There was also an indication that the testing and scoring of results was somewhat intimidating. The changes Page 16
implemented for the next release were to redesign the coaching exercise to enable more participants to become actively involved and to reduce the size of the breakout teams. In addition, it was decided to provide student access to the system as part of their self- study program, thus allowing everyone to develop greater comfort for the system Package 1.02 Results Participants rated the overall blended learning program to be 8.1 out of a possible 10. Discussion The participants indicated that they found the training program to be of value and a similar percent indicated the training program met their expectations. They also indicated that the group interaction, the coaching and the elearning self-study all were very helpful in advancing their learning. This suggested that it was the blended approach that was very appealing. The limited functionality of IPS was somewhat frustrating to the users. The key recommended change to the program was to adjust the Kick-off event and use the time to provide a detailed live demonstration of the IPS system and alleviate resistance to change or the unknown. Package 1.03 Results Participants rated the overall blended learning program to be 7.0 out of a possible 10. Discussion Learning Management System statistics gathered from April 19, 2005 to November 2, 2007 for IPS related lessons: 1. Total number of students who took elearning 1193 students 2. Total number of computer courses provided there were 23 lessons, 22 evaluations, 15 exercises produced for IPS (60 total) users can go back into the lessons for review, etc. without necessarily taking the evaluation again. 3. 98.75% of all accesses were tracked as completed courses. 4. From empirical observation, it appears that the majority of students accessed at least one of their assigned courses more than once. 5. 56.70% of completed evaluations (2,624 of the 22,266) achieved a score of 80% or more. 6. Managers and Supervisors also accessed the elearning IPS courses 86 of the 165 supervisors (52.12%) spent on average a total elapsed time of 8.81 hours. Page 17
7. Over the reported period there have been 59,800 accesses 7.19% of these occurred after the formal training period had ended (i.e. after April 23, 2007). The elearning resources were available for use as a review; however, it is felt that the large proportion of the staff were able to smoothly transition to using IPS and there was little need for additional training. 8. The following statistics were based on removing data for elapsed times less than 0.02 hours (or spent less than 1.2 minutes on a course), thereby reducing the number of accesses to 52030 (89% of original data): a) Each student accessed an average of 25.77 lessons and exercises and 18.67 evaluations. b) Each student spent an average of 8.20 hours on lessons and exercises and 2.03 hours on evaluations. c) Each student spent an average of 7 minutes on each evaluation and 20 minutes on each lesson and/exercise. Page 18
5 CONCLUSION AND LEARNINGS Based on the feedback, the training strategies and program that were employed to roll out each of the packages was successful. The continuous improvement strategy implemented by the Training Team enabled timely adjustments to be made to the program as the packages were rolled out. Based on the concrete data and evolution of the program, a number of key conclusions and learnings are outlined below: 1. The blended learning approach proved to be a highly effective way of transferring large amounts of detailed information about the new IPS system to different target audiences and to geographically dispersed employees. The feedback was clear: elearning did not provide the level of interaction that the students needed. The elearning approach was not well received by some staff. It was felt that it is more of a resource tool than a learning tool. It is a good refresher to go back to in the future but not a good learning tool for starters. However, a 100% facilitator lead approach on its own would have been too limiting. elearning exposure prior to the actual training sessions afforded the users with basic, background knowledge, enhancing their understanding of the material covered during training sessions. Too much time would have been invested in communicating the necessary background information on IPS without having enough time for discussion and interactive exercises. It was when these two approaches were combined, building on the strength of each approach, that a high quality training solution was created. 2. Providing access to a hands-on IPS environment was highly useful in building understanding and confidence in the use of IPS and quick adoption of the technology. Again the elearning resources provided the necessary background information to allow early usage of IPS, to deal with the reluctance by some to use IPS initially and to minimize frustration. 3. Assembling a quality rollout team with clearly defined responsibilities provided timely support to the students. As an example, utilizing both Technical Support Analysts (TSAs) and coaches allowed the TSAs to focus on the technical aspect while the coaches focused on the training and responding to questions. Involving the managers and supervisors in the kick-off and preview event was critical in setting the appropriate mood and focus for the subsequent training activities. In another example, the selection of coaches should have been based on specific criteria, as the coaching varied according to office. Some coaches offered very in-depth training while others provided more high level, rushed training. Greater IPS team on-site involvement was required for the Page 19
first few days, particularly to aid in the training and understanding of the functionality of the system. This would have increased the confidence in IPS by these new users and alleviated some of the user apprehension and misunderstandings of the new system. 4. The training program is only effective if staff have the time to execute the program. MPAC management must ensure all staff has the time to take the required training. Arrangements and work-arounds must be found to handle the subsequent heavier workload that occurs when staff is away from their regular routine. 5. A key to the success of the program was effective coaching. More support mechanisms needed to be put in place to support the coaches. The role of the coach was difficult and they required additional support so that they did not feel alone or overwhelmed from the stress. During the initial rollouts, the training demands on the coaches were significant; therefore, their regular job responsibilities should have been reduced to allow them to focus on their role as a coach. In the future, coaches must be provided with additional tools and support to provide training for a product that is not working 100%. Coaches need good communication regarding work-arounds and problems with the system, to enable them to respond and train more effectively. There is also a need to provide a wider variety of training examples as the majority of examples were geared towards residential properties. Other property types must be included as well, even though the majority of properties are residential. Finally, provide a more comprehensive training session to the coaches, including both the tools to conduct presentations and training as well as functionality training within IPS (provide them with additional background functionality exposure to better equip them to respond to questions). 6. Some staff required more coaching than others. Additional time should be set aside for coaches to offer training to those staff who require it. More hands-on exercises would have been very useful as an integral part of the training or as a resource for ongoing training back on the job. Additional training and support are important for such a largescale project. 7. In many cases, the IPS application initiated a change in the business processes, which the coaches were not equipped to handle. These changes in business processes should have been communicated to staff during their training sessions by the business unit or, at the very least, a list of contact person(s) should have been made available during the coaching session to address these issues that the coaches could not. 8. Management support is critical to the success of the training and the implementation of the system. In hindsight, it is important for managers to be trained first and to be fully Page 20
engaged in all aspects of the rollout. Although they received training in managing the change and had a supporting role, some managers were not as involved, resulting in additional challenges. This management support should include giving both staff and coaches sufficient time for training and support. It also reinforces the future necessity for managers to ensure that new hires receive adequate training in IPS. All managers must understand the system themselves, enabling them to guide staff through the training with the assistance of internal resources. 9. Upgrades to the LMS tool would be beneficial in the future. For the addition and deletion of users, it would be beneficial to have a link to an ERP system so the Manager information is always up to date. From a file management perspective, it would be beneficial to have the ability to group the lessons into content type buckets and have the ability to drag and drop them into the appropriate location. From a report perspective, it would be nice to be able to break down the report curricula rather than exporting all lessons to a.csv format, then sorting through to locate the item you require. Page 21