: Process and Tools for Prioritizing Quickly Gus Prestera, PhD, CPT President, effect, Inc. 2004 ISPI Instructional Systems Design Conference Chicago, September 30, 10:15 AM 11:45 AM 2004 ISPI Instructional Systems Design Conference Slide 1 Look familiar?! Improved Problem or Opportunity Black Box of Intuition Throw Training At It! No accountability for performance outcomes Instructional Task Learner Design Development Implementation Evaluation 2004 ISPI Instructional Systems Design Conference Slide 2 Human Technology Improved What do actual and optimal performance look like? Is there a gap? Is the gap significant? What interventions will address the gap? Problem or Opportunity Gap Stop: Not Significant Intervention Design Formal Training Non-Training: Ergonomics, job redesign, clarify expectations, introduce consequences for performance and nonperformance, job aids, performance support tools, provide additional resources, etc. If there is a skill gap, how should the training be designed? Instructional Task Learner Informal Training: Coaching, OTJ training and cross-training, mentoring, job rotation, exchange programs, short-term assignments Design Development Implementation Evaluation 2004 ISPI Instructional Systems Design Conference Slide 3 1
Instructional Analyze tasks (the work) Analyze learners Analyze learning needs Analyze work context Why does this occur so infrequently? Instructional Task Learner Design Development Implementation Evaluation 2004 ISPI Instructional Systems Design Conference Slide 4 (TNA) Once you know training can help address the gap What specific skills can/should be addressed through training? Which will require more resources? Which should receive priority? Which are in greatest demand? Skill gaps are caused by gaps in: - Knowledge - Skills - Attitudes Instructional Task Learner Design Development Implementation Evaluation 2004 ISPI Instructional Systems Design Conference Slide 5 A TNA Process 1. Identify critical skills 2. Prioritize skill set - Difficulty of implementation - Potential of impact - Type of cognitive process - Type of knowledge (Krathwohl, 2002) 3. Survey skill needs TNA (Prestera, 2004b) 2004 ISPI Instructional Systems Design Conference Slide 6 2
Answer these questions: - What are the tasks and sub-tasks involved? - What skills are needed to perform those tasks? Suggestions: - Observe real workers, not just SMEs - Avoid job descriptions because they typically don t describe the job very well - Avoid documentation skills are embedded in work, not in manuals and slide decks - Avoid topics mentality focus on performance tasks and skills, not on subjects 2004 ISPI Instructional Systems Design Conference Slide 7 Brainstorming Sub-task 3 Spider Diagram Sub-task 3 Sub-task 2 Sub-task 4 Task 1 Task 2 Sub-task 2 Sub-task 1 Sub-task 4 Sub-task 1 2004 ISPI Instructional Systems Design Conference Slide 8 IRC Worksheet Tool Task Skill Difficulty of Potential of Type of Type of Type of IRC Implementation Impact Skill Cognitive Process Knowledge Rating 5 Very High 5 Very 4 Attitudinal 7 Alter Attitude 5 Attitude 5.50 Hitting a baseball Remains positive even when down in the count High Hitting a baseball Anticipating the pitch correctly 5 Very High 5 Very High 1 Cognitive 3 Apply 3 Procedures 4.00 Hitting a baseball Making solid contact with the ball 5 Very High 5 Very High 3 Psychomotor 2 Understand 3 Procedures 3.75 Hitting a baseball Using appropriate hitting stance consistently 3 Moderate 4 High 3 Psychomotor 3 Apply 3 Procedures 3.25 Hitting a baseball Beating the throw at first base 4 High 4 High 3 Psychomotor 2 Understand 3 Procedures 3.25 Hitting a baseball Running fast 5 Very High 4 High 2 Motor 0 Motor Skill 0 No Knowledge 2.25 Hitting a baseball Translating batting coach's hand signals 2 Low 3 Moderate 1 Cognitive 2 Understand 1 Facts 2.00 Hitting a baseball Remember the pitch count (e.g., 2 strikes, 1 ball) 1 Very Low 2 Low 1 Cognitive 1 Remember 1 Facts 1.25 Populate the worksheet with the tasks/skills and sub-tasks/subskills you ve brainstormed. 2004 ISPI Instructional Systems Design Conference Slide 9 3
Answer these questions: - Which skills are most critical to have? - Which skills require the most resources to develop? Suggestions: - Use a discussion panel - Represent different stakeholders - Go through multiple iterations 2004 ISPI Instructional Systems Design Conference Slide 10 IRC Worksheet Tool Task Skill Difficulty of Potential of Type of Type of Type of IRC Implementation Impact Skill Cognitive Process Knowledge Rating 5 Very High 5 Very 4 Attitudinal 7 Alter Attitude 5 Attitude 5.50 Hitting a baseball Remains positive even when down in the count High Hitting a baseball Anticipating the pitch correctly 5 Very High 5 Very High 1 Cognitive 3 Apply 3 Procedures 4.00 Hitting a baseball Making solid contact with the ball 5 Very High 5 Very High 3 Psychomotor 2 Understand 3 Procedures 3.75 Hitting a baseball Using appropriate hitting stance consistently 3 Moderate 4 High 3 Psychomotor 3 Apply 3 Procedures 3.25 Hitting a baseball Beating the throw at first base 4 High 4 High 3 Psychomotor 2 Understand 3 Procedures 3.25 Hitting a baseball Running fast 5 Very High 4 High 2 Motor 0 Motor Skill 0 No Knowledge 2.25 Hitting a baseball Translating batting coach's hand signals 2 Low 3 Moderate 1 Cognitive 2 Understand 1 Facts 2.00 Hitting a baseball Remember the pitch count (e.g., 2 strikes, 1 ball) 1 Very Low 2 Low 1 Cognitive 1 Remember 1 Facts 1.25 Gain agreement around how difficult the skill/task is to perform in the workplace and the potential to impact positively or negatively the success of the organization. Classify the type of skill. Some types require more instructional resources than others. 2004 ISPI Instructional Systems Design Conference Slide 11 IRC Rating Instructional Resource Commitment Helps to prioritize list of skills Most resource-intensive* on top Most difficult to implement on top Most potential impact on top * Resources = Costs associated with direct instruction, practice, assessment, and feedback 2004 ISPI Instructional Systems Design Conference Slide 12 4
Low-IRC Skills What do we do about low-irc skills? 2004 ISPI Instructional Systems Design Conference Slide 13 Answer these questions: - What do the workers feel they need? - What do supervisors feel their workers need? Suggestions: - Sample different groups and compare results - Narrow the list of skills to the most resource-intensive ones (high IRC) - Compare to performance metrics & benchmarks - Trust your data - Disregard learners at your own peril! 2004 ISPI Instructional Systems Design Conference Slide 14 TNA Survey Tool needed but No training Need this Training written needed but right aw ay to Need Training but instructions and written No training or transition to my the need is not coaching would instructions learning support Unsure new role! urgent be helpful would be helpful is needed Mean n= 22 0 5 4 3 2 1 Rating 16 Enter Task or Topic Here 14% 32% 50% 5% 0% 0% 3.73 3 Enter Task or Topic Here 5% 18% 36% 27% 0% 14% 3.32 6 Enter Task or Topic Here 5% 5% 55% 18% 9% 9% 3.23 8 Enter Task or Topic Here 14% 14% 50% 9% 9% 5% 3.18 21 Enter Task or Topic Here 5% 5% 59% 9% 9% 14% 3.18 This tool automates the survey 14 Enter Task or Topic Here 5% 0% 50% 23% 18% 5% 3.09 development and analysis, 9 Enter Task or Topic Here 9% 18% 27% quickly informing 23% you 9% of which 14% 3.00 skills have high perceived training need and which have low perceived training need and can be addressed through nontraining interventions, if at all. 2004 ISPI Instructional Systems Design Conference Slide 15 5
Summary 1. Identify critical skills - Brainstorm, gain agreement 2. Prioritize skill set - Attribute each skill 3. Survey skill needs - Who needs it and how badly? TNA (Prestera, 2004b) 2004 ISPI Instructional Systems Design Conference Slide 16 Slides and tools available at: http://www.effectperformance.com/html/library.htm effect Instructional design solutions for your learning and performance needs Gus Prestera, Ph.D., CPT President, effect, Inc. E-mail: gprestera@effect.com Voice 610.449.2060 Fax 610.449.2061 1513 Fairview Avenue, Havertown, PA 19083 www.effect.com 2004 ISPI Instructional Systems Design Conference Slide 17 References ASTD. (2001). Benchmarking Report on e-learning. Bloom, B. S., Engelhart, M. D., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956). Taxonomy of educational objectives: The classification of educational goals. Handbook 1: Cognitive Domain. White Plains, NY: Longman. Clark, R.E. (2004, March). The 10 Most Wanted motivation killers. Xpress. Clark, D. (2003, August). How effective is training? A new summary of the past 40 years of training field research and evaluation. Xpress. Cronbach, L. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334. Dick, W., & Carey, L. (1990). The Systematic Design of Instruction. Glenview, IL: Scott, Foresman. Kirkpatrick, D. (1998). Evaluative training programs: The four levels (2nd ed.). New York, NY: Barrett-Kohler. Krathwohl, D. R. (2002). A revision of Bloom's Taxonomy: An overview. Theory into Practice, 41(4), 212-218. Kuder, G. F., & Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2, 151-160. Nitko, A. J. (1996). Educational of Students (2nd Ed). Englewood Cliffs, NJ: Prentice-Hall. Prestera, G.E. (2004a). Are your e-learners learning? A rapid prototyping process and tool for test development. effect White Papers. Retrieved from the effect, Inc. web site: http://www.effect.com/html/library.htm. Prestera, G.E. (2004b). Training needs assessment: Process and tools to help you identify and prioritize training needs quickly. effect White Papers. Retrieved from the effect, Inc. web site: http://www.effect.com/html/library.htm. Prestera, G.E. (2004c). Understanding ADDIE: A foundation for designing instruction. effect White Papers. Retrieved from the effect, Inc. web site: http://www.effect.com/html/library.htm. Rossett, A. (1999). First things fast: A handbook for performance analysis. San Francisco, CA: Jossey-Bass. Sullivan, R. L., Wircenski, J. L., & Major, M. J. (1999). Analyzing knowledge-based tests. In D. L. Kirkpatrick (Ed.), Another Look at Evaluating Training Programs (pp. 113-118). Alexandria, VA: ASTD. Thiagarajan, S. (1999). Rapid Instructional Design. Workshops by Thiagi, Inc. Retrieved 11/18/2003, from the World Wide Web: http://www.thiagi.com/article-rid.html. 2004 ISPI Instructional Systems Design Conference Slide 18 6