NERC Training Systematic Approach EMS User Group Meeting September 21, 2010 Chicago, IL Presented By Robin Podmore IncSys, Issaquah, WA
Overview Introduction Systems S t Approach to Training i & Analysis of Training Needs Design of Simulator Scenarios Development of Simulator Scenarios Implementation ti of Simulator Scenarios Evaluation of the Simulator Training Program How to iterate the steps
Objectives- At the end of this course you will be able to: Describe Steps of Process Apply and RPD to developing Training Scenarios Describe appropriate role of Generic and simulators Custom Describe appropriate role of personal and team training ta
Systems Approach to Training & Foundation Military Training i Job & Tasks Gained Wide Acceptance
Instructional Design or Systems Approach to Training Foundation laid in World War II U.S. military faced the need to rapidly train large numbers of people to perform complex technical tasks. E.g. Field-stripping a carbine Navigating across the ocean Landing a plane on a carrier deck Define and Train for the Job and Tasks to be Performed Teach Everything That is Important Teach Nothing That is Superfluous SAT has recently gained acceptance for power system operators http://en.wikipedia.org/wiki/instructional_design
Analyze Design Develop Implement Evaluate Resource: The Model, Mike Molenda, Indiana University http://en.wikipedia.org/wiki/_mo del
Analysis
Analysis Instructional Analysis Learner Analysis Environment Analysis
Analysis - Types Job Analysis What tasks and responsibilities are associated with a job? Task Analysis What capabilities (knowledge, skills, abilities) and resources are needed to perform the task as required? Needs (Performance) Analysis If there is a performance gap, what is the root cause?
Instructional Analysis Identify Jobs and Tasks to be performed Identify skills, knowledge, performance level to be achieved e.g. Monitoring System Security Generation Control Voltage Control Switching System Restoration Other?
Important Soft Skills Source J. Theotonio NERC
Crew Resource Management Foundation for Soft Skills Situational Awareness Assertiveness Decision Making Communication Leadership Adaptability/Flexibility Mission Analysis
Learner Analysis Identify the characteristics of those who will be trained Operator in Training Transmission Operator Control Area Operator Generator Operator Substation Operator Experience Education Other?
Environment Analysis Environment where skills will later be performed: National Control Center North, Central, South Regional Control Centers Substations? Learning environment where skills will be taught: Camp Kanun classroom - group National Control Center classroom group Office on-line self study Other?
Job Analysis What is the business need for a job? What jobs are associated with a position? What responsibilities, duties, and tasks are associated with a job? What people. data, and things are associated with job performance?
Task Analysis PER-005 Tasks R1.1. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall create a list of BES company-specific reliabilityrelated tasks performed by its System Operators. R1.2. Each Reliability Coordinator, Balancing Authority and Transmission Operator shall design and develop learning objectives and training materials based on the task list created in R1.1.
Task Analysis PER_005 Requirements R1.1 and R1.2 Both R1.1 and R1.2 can be addressed at the same time Develop (modify) task statements in a format that gives you both task statements and learning objectives for training No need to do both as separate projects and allocation of resources Follow a suggested format for task statements and you will have learning objectives when you finish the task statements
Needs (Performance) Needs Analysis Is there a performance gap? 1. What performance is required at what level? l? 2. How is required performance measured? 3. Does the individual (s) have the capability to perform the task as required? 4. What is the root cause for the performance deficiency?
Needs (Performance) Needs Analysis Is there a performance gap? 1. Can the gap be corrected with training? i 2. Is the training need based on new tool(s), procedure(s), condition(s), etc.? 3. Is the training mandated by a regulatory body or management? 4. Will the new capability be supported in the workplace?
Analysis Root Cause Non-training causes for performance deficiency 1. Is there a deficiency i of physical resources? 2. Is there a deficiency in work processes or procedures? 3. Is there a lack of required information? 4. Is there a lack of motivation to perform the task to the level required? 5. Is personal illness the cause? 6. Is there a deficiency of tools? 7. Is there a lack of management support?
Analysis Outcomes List of tasks for each job Identification of a performance gap Root cause of performance gap Performance gap that is at least partially addressed by training. Criteria for performance of task Selection of appropriate intervention
Design
Training On Tasks What tasks will the training address? Do the task descriptions contain: 1. Performer performs the task? 2. Ti Trigger when is the task performed or what event or condition requires action? 3. Action what action is required by the performer? 4. Criteria what outcome is required; when is the task complete? Is the task performance criteria measurable or observable?
Learning Objectives What are the learning objectives? Are the learning objectives on the same taxonomy level as the related task performance criteria? What prerequisite knowledge, skills, abilities of participants are required? Are the learning objectives knowledge objectives, performance objectives, or a mix of both?
Design - Instructional Strategy How is the content is delivered? Instructor presentations Q&A data collection Participant table top work groups On-line tutorials Self-paced simulator exercises Team based simulator exercises Role playing Level of instructor activity vs. level student activity Factors: Abilities & interests of students Amount of time available Content to cover Activity pattern or routine established in the classroom Teacher s repertoire of activities Type of learning involved
Instructor Lead and CBT Instructor Lead instructor is on-line with students locally or remotely instructor sets time and place of training and determines pace of scenario. CBT Computer Based Training student works without an instructor. The computer provides the training. Student has more control of time and place of training and determines pace. Instructor can review results and coach student in follow up session.
Design Identify Assessments How we evaluate whether training was successful? - For the Team - For the Individual Operator Recognition Primed Decision Model provides an objective and clear foundation for assessments. - Were correct cues monitored? - Were correct patterns identified - Was an appropriate action script identified? - Was a correct mental model used? - Was the action correctly implemented? - Did the simulator move to a more secure operating point?
Design Assessment Tools Feedback must be provided in a timely manner; before the event is forgotten. An interrupt t process into the scenario allows the team and individual operators to develop a deeper sense of situational awareness by focusing on specific conditions in the operating environment. A Trainer with a keen sense of intuition can spot when an operator is struggling or having difficulty and then can ask a few pointed questions to help coax the operator back onto track.
Design Assessment Tools A Checklist with Instructor Notes can be designed into the scenario with an interrupt or pause at strategic points. The Trainer or designated staff are responsible for monitoring activity and performance during the scenario and documenting activity, performance, events, conditions and operator actions. The lead Trainer may decide to allow the scenario to unfold or continue rather than interrupt or pause the simulator. Capturing Operator performance may become a very arduous task when notes are taken during the scenario. The operators must know that something did NOT go right and then how to fix it correctly!
Decision Making Review Describe the situation (cues, patterns) What actions did you consider How did you evaluate the action chosen? What Other actions were rejected? Why? What Would You Do Differently Next Time?
Outline the Training Plan Based on the Training Criteria List Objectives Establish Training Methods List a Bill of Training Materials Outline Simulation Events & Scenarios Outline Support (personnel) Requirements Outline Assessment Tools
Develop
Develop Develop instructional ti materials Develop Scenarios Develop assessment tools
in Simulation Development Individual/Team Scenarios Generic & Custom Simulators Condition/Event Based Training for Simulation Assessment Tools Develop & Initiate Scenarios De-brief & Evaluate Performance
Personal and Team Scenarios Personal scenario one operator is in control of the entire scenario. One operator plays all the roles. Team scenario A team of students t is in control of the scenario. Control Room positions are assigned to each operator.
Possible Combinations Personal scenario with Instructor Personal scenario with CB Trainer Team scenario with Instructor Team scenario with CB Trainer
Generic Simulator Lowest cost method to accomplish many learning objectives Entry level training Review of principles for entry level and incumbents Identify gaps for entry level and incumbents Many scenarios are available from day 1. Wealth of tacit knowledge from industry leading instructors has been converted to explicit knowledge Students can train at own pace Virtual instructor is available Competence to operate Generic is necessary but not sufficient to operate a specific system with similar equipment.
Custom Model Benefits: Knowledge and experience directly applies to operators system. Quantitative results carry across to real-time system Support: Effort to capture tacit knowledge as explicit scenarios and solutions Effort to build the custom model Effort to build a virtual instructor t Scenarios need updating as power system changes
Complementary Usage Generic Used for entry level Annual Gap Assessment while custom being developed Used for remedial training when gap is discovered Used for Job transitions Used to Complete Annual CEH Requirements Custom Replaces Generic as a richer library of scenarios is built. This may take several years from initial delivery Use Lowest Cost Method to Meet the Learning Objective!
Types of Operating Conditions Normal (Basic) Operating Conditions Emergency Operating Conditions Restoration ti Operating Conditions
Scenario Design
Applications Operator in Training Power System Principles Operating Procedures Technical Crew Emergency Operations Technical Crew System Restoration Technical Crew
Scenario Preparation Analyze needs and select scenarios. Prepare design specifications. Identify required OTS model modifications and upgrades for selected scenarios. Develop assumptions on external conditions; load levels, weather, line and transformer outages, past/present/future Set up base cases and events for planned scenarios. Test and document scenarios. Ensure that the scenario creates credible situations on the power system. Develop any scenario materials such as handouts for operators and scripts for the instructor and others who may be role-playing.
Implementation
Phases Pre-Brief Conduct the Scenario De-Brief
Pre-Brief Explain the Objectives Present Background Information Present Specific Operating Criteria (Time Limits for Voltage Control, Overloads, etc.) Demonstrate Proper Operating Technique if the operations are unfamiliar or new Coordinate personnel required to conduct each scenario Assign operating roles
How To Pre-Brief Imagine you are coming on shift - what questions would you have about the system? Make sure these questions are covered in your Pre-Brief. Load conditions Weather Transmission outages Generation outages/de-rates
Conduct Scenario Log students onto their consoles Initiate the Operating Scenario Observe Operator Performance, note strengths and weaknesses Offer Coaching and/or Counseling, provide hints or challenges as needed. Provide updates on environmental factors Instructors act roles of external operators Adapt scenarios with additional events Monitor performance of PowerSimulator
Role of the Lead Trainer Coordinate All Activities for the Program Assign Support Staff Responsibilities Designate Environment boundaries Orchestrate the Movement of the Scenario -Initiate Event Timing Occurrences -Allow Events to Evolve -Call Time When Appropriate
Debrief Debrief trainees after the session providing any corrective or reinforcing instruction relevant to the scenario We must think in terms of performance improvement For training to be effective we must evaluate the performance of the Operators. If you are not holding the Operators to any performance standard then you are merely disseminating information and not conducting training.
With the Simulator, mistakes have Real Time Consequences that are obvious to the students Overloaded lines Overloaded transformer banks High voltages Low voltages Generators on high MVAR limits Generators on low MVAR limits Insufficient operating reserves Collapse of an island due to voltage collapse Cascading outages due to line sags Collapse of island due to high/low frequency
Decision Making Review Describe the situation (cues, patterns) What actions did you consider How did you evaluate the action chosen? What Other actions were rejected? Why? What Would You Do Differently Next Time?
Assessing Operator Performance Instructor and student can jointly review the operating logs. They can make the following assessments: Was the event correctly recognized? Were the correct cues detected? Were the correct patterns recognized? Was an appropriate action script identified? Was the effect of this action simulated with an appropriate mental model? Was the selected action implemented correctly? Did each member of the team play his/her role? What mistakes were made? What lessons were learned?
Assessing Operator Performance Compare operator knowledge & skill prior to training with results after training Review or grade Q&A; review evaluation from notes taken during the simulation Rank operator performance above or below competency level Provide feedback (de-brief) bi f)to operators as appropriate (Group discussion vs. individual confidential counseling) Praise in public and criticize in private
Evaluation
Evaluate Level 1: Reaction - Immediately after class Level 2: Learning - Immediately after class Level 3: Behavior in Workplace - A few days to a couple of months after training. Enough time has to pass in order for supervisor to determine if the person is performing tasks learned in the class on the job Level 4: Business Results - Usually a few weeks after all the training has taken place and all the level 3 assessments have been recorded Reference: Kirkpatrick Model for Evaluation http://www.e-learningguru.com/articles/art2_8.htm
Evaluation Form Name of Course: Course Number:
- An Iterative Process Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate
Development Iterations Analyze Design Develop Analyze Design Develop Typical cycle time - one month between prototype versions
Implementation Iterations Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate Typical cycle time - one week between classes
Feedback for New Product Releases Analyze Design Develop Implement Evaluate Analyze Design Develop Implement Evaluate Typical cycle time - one year
Develop Design Loop Back Design: Identify High Level Learning Objectives Design: Identify High Level Learning Assessments Develop: Develop a base case condition Develop: Develop a list of severe contingencies Develop: Simulate the contingencies Develop: Identify what power system phenomena are significant ifi Design: Refine the Learning Objectives Design: Refine the Learning Assessments Analyze Design Develop Implement Evaluate
Summary Systems Approach to Training i & Analysis of Training Needs Design of Simulator Scenarios Development of Simulator Scenarios Implementation ti of Simulator Scenarios Evaluation of the Simulator Training Program How to iterate the steps