Improvisation in Safety Critical Situations: An Analysis of Improvisation Incidents in Led Outdoor Activities

International Conference on Naturalistic Decision-making 2013, Marseille, France Improvisation in Safety Critical Situations: An Analysis of Improvisation Incidents in Led Outdoor Activities Margaret TROTTER a, Paul SALMON b and Michael LENNÉ a a Monash Injury Research Centre, Monash University, Melbourne, VIC 3800, Australia b University of the Sunshine Coast Accident Research (USCAR), Faculty of Arts and Business, University of the Sunshine Coast, Maroochydore, QLD 4558, Australia ABSTRACT Introduction: Appropriate, effective improvisation can enhance organisations resilience in safety critical situations; however, conditions across work systems that influence improvisation are not well understood. This PhD aims to identify the factors influencing improvisation from a systems perspective. Method: 20 Critical Decision Method interviews were conducted with led outdoor activity leaders and the transcripts analysed using Leximancer software. Results and discussion: 35 concepts across 12 major themes were identified. The themes confirmed influencing factors identified in the literature (e.g. decision making, knowledge, situation awareness) as well as several important new themes (e.g. time and students). The concept map analysis demonstrates the important links between concepts and themes that will provide guidance for the construction of a systems model of improvisation. KEYWORDS Decision-making; outdoor recreation, improvisation, systems safety, critical decision method. INTRODUCTION Improvisation the spontaneous real-time conception and execution of a novel response to a situation that is beyond the boundaries for which a system has prepared (Trotter, Salmon & Lenné, 2012) provides a means by which organisations can adapt in response to safety critical situations (Grøtan, Størseth, Rø & Skjerve, 2008). Improvisation can therefore enhance organisational resilience; however, how to support appropriate and effective improvisation remains ambiguous. While improvisation can mitigate catastrophic systems failure, such as in the ditching of US Airways flight 1549 (National Transportation Safety Board, 2010), it can also produced less favourable outcomes, such as in the Mangatepopo Gorge incident in which seven people died (Brookes, Smith & Corkill, 2009). Determining how organisations can enhance appropriate, effective improvisation while preventing inappropriate, ineffective improvisation requires an understanding of the factors that influence improvisation in safety critical situations; however, as yet there is no comprehensive model of these factors and their interactions (Trotter et al., 2012). The existing improvisation research examines influencing factors independently rather than from a systemsbased perspective as would be in line with contemporary ergonomics approaches (e.g. Rasmussen, 1997; Leveson, 2011). As a result, the conditions across the system that influence improvisation are not well understood. This represents a significant research gap, which currently prevents the concept from being realised within safety critical systems. To address this I devised a five stage research agenda aimed at identifying and modelling the factors and their interactions influencing improvisation from a systems perspective. Stage 1 critiques existing knowledge of improvisation in safety critical situations and the extent to which it has been investigated from a systems perspective (Trotter et al., 2012). Stage 2 examines the appropriateness of a particular systems-based framework for the study of improvisation using a case study methodology to examine two improvisation incidents (Trotter, Salmon & Lenné, submitted 2012). Stage 3 uses a survey methodology to determine the relevance of the factors identified from the literature to a particular complex sociotechnical system, the led outdoor activity (LOA) domain (Trotter, Salmon & Lenné, submitted 2013). Stage 4 identifies the factors influencing improvisation across the LOA system using in-depth analysis of Critical Decision Method interviews (CDM; Klein, Calderwood & MacGregor, 1989) focused on LOA improvisation incidents in order to produce a systems-based model improvisation in this domain. Stage 5 establishes how LOA organisations can best support appropriate, effective improvisation. This paper outlines a preliminary analysis of the CDM data from Stage 4 of the research program. Authors retain copyright of their work H. Chaudet, L. Pellegrin & N. Bonnardel (Eds.). Proceedings of the 11th International Conference on Naturalistic Decision Making (NDM 2013), Marseille, France, 21-24 May 2013. Paris, France: Arpege Science Publishing. ISBN 979-10-92329-00-1

Trotter M. et al. Improvisation in safety critical situations METHOD Participants Participants were eleven male and nine female employees of a major Australian LOA organisation with a mean age of 29 years (SD 4.6). Participants were outdoor leaders currently working in the field with on average 5.7 years of LOA experience (SD 3.7). Procedure The CDM method (Klein et al., 1989) was adapted to study improvisation by having the interviewees focus specifically on instances of improvisation and including a series of improvisation-specific probes. Interviews lasted approximately 2 hours and were conducted individually by one interviewer. Each interview consisted of four stages: 1. Identification: Interviewer determined that participants incidents fit the improvisation definition. 2. Summary: Participants summarised their improvisation; 3. Timeline: Interviewer created a timeline of the improvisation; 4. Probes: Participants responded to a series of probes about the improvisation; All interviews were recorded and then transcribed using Microsoft Word. Data Analysis A content analysis of the transcripts was performed using the text analysis software tool Leximancer. Leximancer identifies key concepts and relationships, and then creates semantic networks from raw data. Concept names and themes are derived based on co-occurrence in the texts. A statistical algorithm is used to create a two dimensional concept map. The initial Leximancer analyses were exploratory so that concepts were extracted from the transcripts automatically. Following this, similar concepts were combined manually and the analysis run again based on these revised concept lists. Themes were automatically produced and mapped by Leximancer. Analyses were produced for each individual case as well as for the aggregate of all cases. This paper focuses on the aggregate analysis. RESULTS Incidents ranged in potential severity from minor (e.g. crossing a ford safely without appropriate equipment) to life threatening (e.g. facing a hurricane during a kayaking trip). All improvisation incidents recounted by participants resulted in positive safety outcomes for participants. The aggregate analysis yielded 35 emergent concepts (Table 1) with Knowledge and Time the most relevant. These concepts were also relevant in 18 and 17 individual cases respectively. Leximancer identified 12 dominant themes within the aggregate text, labelling them according to the central concept within each (Table 2). Decision was the most highly connected theme, followed by Situation and Knowledge. These themes also appeared in seven, five, and nine of the case-specific concept maps respectively. The connections between themes and concepts are illustrated in the concept map (Figure 2). Table 1. Concepts identified from the aggregate analysis Concept Count Relevance* Concept Count Relevance Knowledge 282 100% Use 106 38% Time 272 96% Option 94 33% Students 210 74% Someone 90 32% Decision 183 65% Day 90 32% People 175 62% Person 86 30% Situation 168 60% Risk 76 27% Ability 164 58% Level 68 24% Work 159 56% Area 68 24% Experience 150 53% Rescue 64 23% Group 141 50% Thinking 58 21% Training 140 50% Trip 56 20% Happen 132 47% Comfortable 52 18% Organisation 124 44% Plan 52 18% Water 123 44% Place 51 18% Different 116 41% Better 46 16% Felt 114 40% Everything 38 13% Take 112 40% School 37 13% Look 111 39% * The relevance percentage represents the percentage frequency of text segments coded with that concept, relative to the most frequent. 304

Trotter M. et al. Improvisation in safety critical situations Table 2. Themes and their component concepts from the aggregate analysis Theme Decision Situation Knowledge Training Experience Time Students Organisation Option Look Trip Day Connectivity 100% 99% 84% 63% 62% 59% 25% 23% 22% 20% 16% 20% Concepts Decision, happen, group, felt, comfortable, thinking Situation, work, different, person, place Knowledge, people, someone, better Training, water, rescue, level Experience, ability, use Time, take, everything Students Organisation, risk Option, area Look, plan Trip, school Day Figure 2. Aggregate analysis concept map (theme names in large font). Note: The size of each node represents the centrality of the concept, while the pathways denote the most likely connections between concepts. Key Themes Decision Theme The Decision theme includes concepts associated with participants decision-making processes. The prominence of this theme is unsurprising given than decision-making is key component of improvisation (Mendonc a, 2007). The concepts Thinking and Felt are included in this theme. Many participants reported thinking through the consequences of their improvised solutions; however, many also referred to unconscious processes such as intuition, particularly in relation to determining when a situation required improvisation, for example: My gut feeling, intuition, which was saying this situation isn t right, it isn t good, I don t feel that these students are safe Interestingly, the concepts Decision and Thinking are directly connected to the Time concept in the network whereas Felt is not. This indicates that the time available affects improvisers ability to exhaustively think through options and consequences in order to make a decision is more than it effects intuitive processes. In very time constrained situations, participants may rely more on intuitive decision-making. Situation Theme The Situation theme reflects the Person, Place and Work factors apparent in a situation; for example, 305

Trotter M. et al. Improvisation in safety critical situations It was a combination of elements that made the situation different: the weather the slipperiness, the particular person and the location that meant you couldn t see. These concepts resemble the three internal factors identified in models of outdoor decision-making (e.g. Boyes & O Hare, 2003). A key differentiating concept within this theme is Different, which relates to aspects of the situation that are different to what participants have previously experienced. The Situation theme overlaps with, Knowledge, and Training, in particular, with knowledge of leaders and students abilities, and rescue training. Knowledge Theme The Knowledge theme reflects knowledge of the people involved in the situation, those available to provide assistance, the location, different activities and techniques. It also represents knowledge of the reasons behind the rules, for example: Having that good sound knowledge base allows you to know the rules and where you have got room to move. This theme overlaps with the Experience theme, particularly through the concept of ability. Many participants reported that their knowledge of student abilities develops over the course of programs and that their knowledge of fellow leaders abilities came from working with them previously. Also overlapping is the Options theme, particularly around knowledge of the area and the options that it affords, for example, the location of access points. DISCUSSION This research forms part of a program aimed at identifying and modelling the factors influencing improvisation in safety critical situations from a systems perspective. This research represents preliminary analysis of CDM data from Stage 4 of this program. The Leximancer analysis identified 35 concepts and grouped these into 12 major themes, the most connected of which were Decision, Situation and Knowledge. These themes are consistent with those identified in the literature as influencing improvisation (Trotter et al., 2012). The concept map enhances our understanding of improvisation by identifying the dominant connections between the concepts and themes, a vital element of any systems-based model. Another important characteristic of a systems-based model is that it reflects factors and interactions from across different systems levels. While this analysis identifies the higher-level themes Organisation and School, nothing at the regulatory or government levels of the system is identified. This is most likely the consequence of all CDM participants being at the leader level and therefore potentially unaware of how external factors affect their organisation. To address this, semi-structured interviews with all management level staff at the same organisation were conducted and will be included in the subsequent in-depth interview analysis. This analysis also identified a number of factors that not identified in previous research, the most dominant of these being Time and Students. Examination of the Time theme revealed variations in the timescales over which improvised responses were executed. While some improvisation incidents required only one decision, then execution of this response over a number of minutes (e.g. rescuing a capsized kayaker), others involved multiple decisions and executions over a longer timescale (e.g. protecting kayaking students from a hurricane). This analysis indicates that time is not just a defining characteristic of improvisation but an influencing factor. This has significant implications for the definition of improvisation and will be explored further through in-depth analysis of the incident timelines. The Student theme was also identified in the Stage 3 survey and may be specific to LOA. Boyes and O Hare (2003) include student ability in their model of outdoor adventure decision-making; however, in a more generalised model of improvisation, this factor may be subsumed by the concepts People or Ability. Leximancer analysis is based on word frequency and co-occurrence within raw data. It provides a useful interpretive guide for an exploratory examination of influencing factors, indicating potential factors and interactions. Meaning, however, can lie not just within specific words, but also in the spaces between. In the next phase of analysis manual coding will be conducted to identify and understand factors in sufficient detail to allow the development of a comprehensive model of improvisation from which recommendations can be devised. The structure of this context map will guide model construction. ACKNOWLEDGMENTS Ms. Trotter s contribution to this research was supported by an APA Postgraduate Research Scholarship. Dr. Salmon s contribution was funded through the Australian NHMRC Post doctoral training fellowship scheme. The authors would like to acknowledge Dr Natassia Goode for her advice on the draft manuscript. 306

REFERENCES Trotter M. et al. Improvisation in safety critical situations Boyes, M. A. & O Hare, D. (2003) Between Safety and Risk: A Model for Outdoor Adventure Decisionmaking, Journal of Adventure Education and Outdoor Learning, 3, 1, 63-75. Brookes, A., Smith, M. & Corkill, B. (2009) Report to the Trustees of the Sir Edmund Hillary Outdoor Pursuit Centre of New Zealand, Mangatepopo Gorge Incident, 15 April 2008. Grøtan, T., Størseth, F., Rø, M. H. & Skjerve, A. B. (2008) Resilience, Adaptation and Improvisation Increasing Resilience by Organising for Successful Improvisation, Proceedings of the 3 rd Symposium on Resilience Engineering. Antibes, Juan-Les-Pins, France. Available from: http://www.resilienceengineering.org/re3/papers/grotan_storseth_ro_skjerve_text.pdf (21 August 2011). Klein, G., Calderwood, R. & MacGregor, D. (1989) Critical Decision Method for Eliciting Knowledge. IEEE Transactions on Systems, Man, and Cybernetics, 19, 3, 462 472. Leveson, N. G. (2011) Applying Systems Thinking to Analyze and Learn from Events, Safety Science, 49, 1, 55 64. Mendonc a, D. (2007). Decision support for improvisation in response to extreme events: learning from the response to the 2001 World Trade Centre attack, Decision Support Systems, 43, 952 967. National Transportation Safety Board (2010) Loss of Thrust in Both Engines After Encountering a Flock of Birds and Subsequent Ditching on the Hudson River, US Airways Flight 1549, Airbus A320-214, N106US, Weehawken, New Jersey, January 15, 2009. Aircraft Accident Report NTSB/AAR-10/03. Washington, DC. Rasmussen, J. (1997) Risk Management in a Dynamic Society: A Modelling Problem, Safety Science, 27, 2/3, 183 213. Trotter, M. J., Salmon, P. M. & Lenné, M. G. (2012) Improvisation: Theory, Measures and Known Influencing Factors, Theoretical Issues in Ergonomics Science, Online, 1-24. Trotter, M. J., Salmon, P. M. & Lenné, M. G. (Submitted Nov 2012) Impromaps: Applying Rasmussen s Risk Management Framework to Improvisation Incidents, Safety Science. Trotter, M. J., Salmon, P. M. & Lenné, M. G. (Submitted Mar 2013) Improvisation in Led Outdoor Activities: An investigation into its occurrence and influencing factors. Australian Journal of Outdoor Education. 307