The effects of a scientifically-based team resource management intervention for fire service teams

196 Int. J. Human Factors and Ergonomics, Vol. 2, Nos. 2/3, 2013 The effects of a scientifically-based team resource management intervention for fire service teams Vera Hagemann* Department for Computer Science and Applied Cognitive Science, School for Engineering, Institute for Business and Organizational Psychology, University of Duisburg-Essen, Lotharstr. 65, LE 206, 47057 Duisburg, Germany E-mail: vera.hagemann@uni-due.de *Corresponding author Annette Kluge Department for Computer Science and Applied Cognitive Science, School for Engineering, Institute for Business and Organizational Psychology, University of Duisburg-Essen, Lotharstr. 56, LE 207, 47057 Duisburg, Germany E-mail: annette.kluge@uni-due.de Abstract: Team resource management training for fire service teams has become increasingly popular. However, the effectiveness of this training, adapted from the well-established crew resource management training for flight crews, has not yet been evaluated for this audience. Ninety-one firefighters participated in a Team Resource Management seminar (experimental and control group) and in platoon leader training for conducting After Action Reviews (experimental group only). The intervention was evaluated, and its impact on knowledge acquisition, attitude, and behavioural changes at three different measurement times was assessed. The seminar was perceived as enjoyable and useful and participants stated an increase in knowledge, which was supported by an objective knowledge test assessment. Both the knowledge increase and three positively changed attitudes remained stable over the following seven months. Positive behavioural changes toward Team Resource Management were found in the experimental group only. Overall, the intervention proved to be effective according to several criteria. Keywords: after action reviews; debriefings; high responsibility teams; HRTs; training design; evaluation; skill acquisition; crew/team resource management; fire service. Reference to this paper should be made as follows: Hagemann, V. and Kluge, A. (2013) The effects of a scientifically-based team resource management intervention for fire service teams, Int. J. Human Factors and Ergonomics, Vol. 2, Nos. 2/3, pp.196 220. Biographical notes: Vera Hagemann was working as a Research Assistant and PhD student from 2008 until 2010 at the Institute for Organizational Psychology at the University of St. Gallen, Switzerland. She studied the field of human factors and crew resource management in cooperation with a European Copyright 2013 Inderscience Enterprises Ltd.

The effects of a scientifically-based team resource management intervention 197 airline. In January 2011, she joined the Institute for Business Psychology at the University of Duisburg-Essen, Germany, as a Research Associate and received her PhD in Psychology (training development for high responsibility teams) in the same year. Her research is concerned with identifying factors that influence team effectiveness and with designing, implementing, and evaluating team resource management training in high responsibility teams such as aviation, fire service, healthcare or the military. She teaches courses in statistics, organisational psychology and consumer behaviour at the University of Duisburg-Essen. Annette Kluge is a Full Professor in Business and Organizational Psychology at the University of Duisburg-Essen, Germany, since July 2008. Before, she was an Assistant Professor for Psychology at the University of St. Gallen, Switzerland, (2002 2008) and a Research Assistant at the Technical University Aachen (1996 2001). She received her Diploma in Psychology at the Technical University of Aachen and her Doctor s degree of Economics and Social Sciences at the University of Kassel, at the Department of Ergonomics and Vocational training. Her research is concerned with training and learning in organisations, training for knowledge and skill acquisition and retention in complex systems especially process control tasks, the use of cognitive task analysis for training design and simulator training, and the development of job performance aids based on cognitive error analysis methods. 1 High responsibility teamwork Many industries have realised that human error and teamwork problems such as breakdowns in communication or coordination processes or failures in decision-making have been major causes of accidents and incidents. Examples of this are the Tenerife airport accident in 1977, which resulted in a loss of 583 lives, or the explosion of the Deep Water Horizon in spring 2010, which claimed 11 lives (Flin et al., 2002; Helmreich et al., 1999). Teamwork professions such as hospital anaesthesia teams or disaster management and first responder teams in the fire service are also recognising the human contribution to errors and incidents. Examples in this regard include the South Canyon Fire ( storm king fire) near Glenwood Springs (Colorado, USA) in 1994 with a loss of 14 firefighters, the Black Saturday Bushfires (Victoria, Australia) around Saturday, 7th February 2009, in which 164 people (including 1 firefighter) died, or the incident in an old timbered house in Tübingen (Germany) in 2005, in which two firefighters lost their lives. Therefore, fire service teams are also trying to apply team training interventions in order to counteract these teamwork errors (c.f., Gaba et al., 2001; Okray and Lubnau, 2004). Teams in aviation, fire services or healthcare work in so-called high reliability organisations (HROs; Weick and Sutcliffe, 2003). They are named high responsibility teams (HRTs; c.f., Hagemann et al., 2011) due to their dynamic and often unpredictable working conditions and demanding work contexts, in which technical faults and slips have severe consequences for human beings and the environment if they are not identified and resolved within the team immediately (Kluge et al., 2009). HRTs bear responsibility regarding their own lives and those of third parties. In order to adapt to the dynamic und unpredictable working conditions, they are confronted with specific requirements regarding information sharing and coordination their non-technical skills.

198 V. Hagemann and A. Kluge It is highly important for these teams to collect relevant information from the environment and from other team members as quickly as possible and when appropriately managed by the team, this leads to successful teamwork. They have to interpret this information based on their current mental model to anticipate future events and problems and to collect new information continuously in order to gain an up-to-date picture of the current situation and build good situation awareness (SA) (Endsley and Bolstad, 1994; Salmon et al., 2008). The collected information has to be shared within the team in order to foster shared mental models among the teammates for more efficient explicit and implicit coordination and to enable decisions and forecasts and the evaluation of their own actions (Cannon-Bowers et al., 1993; Mathieu et al., 2000; Waller et al., 2004). 1.1 Team training and effectiveness Some team training approaches already exist to support non-technical (e.g., communication or decision making) skill acquisition for successful operation in HRTs such as flight crews. In the USA, since the end of the 1970s, team training for aviation teams has been developed around the concept of crew resource management (CRM; Helmreich et al., 1999; Kanki et al., 2010; Salas et al., 2006a). CRM-based training concepts are instructional strategies for crews and teams in HROs in order to: a b c train them to use all available resources efficiently (i.e., people, equipment, and information) enhance their teamwork and therefore enhance their performance diminish the likelihood of possible human error with severe consequences for people and the environment (Salas et al., 2006b). This CRM-based training concept has been well established within commercial aviation for over 30 years, and has become mandatory worldwide. Since then, several HROs such as the healthcare sector have applied CRM-based training concepts (c.f., Davies, 2001; Gaba et al., 2001) as well as the fire service with a concept called team resource management (TRM) (c.f., Flin and Yule, 2005; Okray and Lubnau, 2004). So far, nothing is known about the effects of TRM training on prerequisites for successful teamwork of fire service teams. Training effectiveness is usually assessed by using Kirkpatrick s (1998) training evaluation hierarchy to categorise the training outcomes. The first level of the evaluation hierarchy is the evaluation of reactions, such as subjectively perceived enjoyment and perceived usefulness of the TRM training. The second level is learning and contains the participant s attitudinal changes and knowledge gain after TRM training. Behavioural changes are the hierarchy s third level and refer to the application of acquired knowledge and skills to the job. The organisational level (not applied in this study) refers to effects such as improvement in teamwork efficiency or decrease in errors or accidents. Due to the research gap regarding training effectiveness of TRM training, a TRM intervention for fire service teams and its impact on team members reactions, teamwork safety-relevant knowledge, attitudes, and behaviours was investigated in the present study.

The effects of a scientifically-based team resource management intervention 199 2 TRM intervention: objectives and implementation A training needs assessment was conducted in advance by applying the teamwork context analysis inventory, which is described in detail in Hagemann (2011) and Hagemann et al. (2012). Based on the resulting team-specific profiles, non-technical skills for training and multidimensional training objectives were defined on a cognitive, affective, and behavioural level (c.f., Kraiger et al., 1993), for which the TRM intervention was developed (see Hagemann, 2011). The identified teamwork competencies of the TRM intervention for the fire service teams were feedback behaviour, communication, shared mental model building (c.f., Cannon-Bowers et al., 1993; Johnson and Lee, 2008; Mathieu et al., 2000), interpositional knowledge (c.f., Smith-Jentsch et al., 2001) and SA (c.f., Endsley and Bolstad, 1994; Salmon et al., 2009; Wickens and McCarley, 2008). The training objectives were threefold evaluated at the three levels mentioned above (c.f., Kraiger et al., 1993): 1 the team members exchange information considering all members perceptions of building up a shared mental model within the team, even under stressful conditions 2 the team members possess interpositional knowledge for passing relevant information to others and supporting them in critical situations, if necessary 3 the team members collect all important information about the new situation locally and try to identify possible risks, even under stress, in order to gather good SA. The intervention was designed based on instructional design theory (c.f., Reigeluth, 1999) and considered Merrill s (2002) five first principles of instruction, including, for example, activation of existing knowledge, problem-based learning, or application of new knowledge, which is beneficial for achieving the training outcomes. The TRM intervention consisted of two modules, the first being a half-day TRM seminar and the second being platoon leader (team leader) training to prepare him/her for a guided after action review (AAR). The guided AAR was considered to be important for training transfer and knowledge retention. 2.1 TRM seminar The TRM seminar took half a day. It was conducted with four fire service teams on four different days during their standby times at the firehouses. The main focus of the seminar was on attitude changes and declarative knowledge acquisition, so that the first two levels of Kirkpatrick s evaluation criteria and two levels (cognitive and affective) of the multidimensional training objectives were covered. The seminar was a mixture of theoretical inputs, exercises, discussions, and reflections. The TRM seminar included five units, each of which was relevant to achieve the multidimensional training objectives. The five units are explained in Table 1. The seminar units were developed based on research-based literature, interviews with firefighters and the proposition made by the theoretical framework of the Duisburger CRM-Model. This model represents an idealised teamwork process including aspects such as SA, information exchange or decision making while considering impairing factors, for example, from the environment (see Hagemann, 2011; Ritzmann, 2012). For instance, regarding SA, the unit started with two exercises which aimed to make

200 V. Hagemann and A. Kluge participants aware of possible misperceptions in a situation. Subsequently, the participants were asked to name similar examples taken from their teamwork context. Following this, the relevance of SA for missions and for team performance were discussed within the group, guided by the trainer, and Endsley s (1995) model was introduced. Factors challenging the SA which might occur during each of the three phases were discussed with the participants and demonstrated with a video. The video demonstrated, for example, the effect of tunnel vision (problem in level 1). Furthermore, cues indicating the loss of SA were named by the participants and completed by the trainer. Finally, the participants were asked to reflect on past situations in which they had experienced good and bad SA. The other seminar units were designed accordingly. Table 1 Units 1 Communication/ assertiveness Explanation of seminar units The aim of the unit was...to provide the participants with an understanding of rules for successful communication during missions, recognising problems in, and the influence of stress on, communication. 2 Feedback techniques...to give the participants an understanding of the dos and don ts for feedback, practising the techniques and learning to receive feedback. 3 Situation awareness (SA)...to explain the three levels of SA and possible problems in each level with SA, the influences of stress on SA, and what, for the participants, the SA is in detail during their missions. 4 Shared mental models (SMM) 5 Effective team competencies (c.f., Smith-Jentsch et al., 1998a)...to provide the participants with an understanding of the different aspects and functions of SMM, the various SMM building strategies, and what the SMM is during their missions....to explain the four teamwork competencies information exchange, communication, supporting behaviour, initiative/leadership to the participants. They should be able to differentiate the significant behaviours regarding these competencies and to classify their own behaviours into the model. The four effective team competencies were also relevant for the team members and platoon leaders to enable them to conduct effective after action reviews. 2.2 After action review An AAR is a kind of debriefing method after missions (c.f., Gabelica et al., 2012), and supports effective teamwork of HRTs, especially subsequent to TRM seminars (Salas et al., 2006c). An AAR contains aspects of guided team self-correction and team dimensional training (c.f., Smith-Jentsch et al., 1998a, 2008), and supports learning from on-the-job feedback. It fits into the already existing debriefings after real missions and supports informal learning (c.f., Tannenbaum et al., 2010). The AAR is a problem-solving-oriented process and focuses on teamwork as a core topic rather than on technical details of the mission (Salas et al., 2006c). It supports acquisition and maintenance of SMM, SA and interpositional knowledge (Rasker et al., 2000; Salas et al., 2006c; Smith-Jentsch et al., 2008). The main focus of the AAR implemented in the present study was on attitude changes and changes regarding the safety-relevant behaviours, in order to also fulfil the third level of Kirkpatrick s evaluation criteria and all three levels of the multidimensional training objectives.

The effects of a scientifically-based team resource management intervention 201 The four effective team competencies of the team dimensional training (c.f., Smith-Jentsch et al., 1998a) were included as an important topic in the developed AAR of this study. Finally, the AAR for the fire service consisted of eight subsequently discussed questions (see Table 2): Table 2 The eight questions to be discussed in the AAR for fire service teams 1 What was the intended goal of the mission? (The primary goal, not the one achieved) Who recognised what during the turbulent beginning of the mission and the following 2 phases? (Everyone reports the mission from his/her perspective) 3 What was the actual result of the mission? (This might differ from the primary goal) 4 Reactivation of team members knowledge regarding the team competencies (important for the next steps; application of poster (see below)) Which behaviours supported reaching the mission goal? Which did not? (focus on what and 5 why, not who; everyone reports positive and negative behaviours shown by oneself, classified in the scheme of the four teamwork competencies) 6 Platoon leader gives his/her feedback (addition of the team members comments) Collecting the identified strengths and weaknesses of the team and defining goals for 7 improvement (future results of missions) 8 Which behaviours support reaching the defined goals for future? 2.2.1 The platoon leaders training An important prerequisite for an effective AAR after a mission is the competence of the platoon leader to guide the AAR. Therefore, platoon leader training was developed and conducted. The training took one day and the platoon leaders were off duty to avoid external interruptions (e.g., alarms, etc.). The aim of this training was to familiarise the platoon leaders with the sequences and advantages of an AAR and to prepare them for skilfully conducting AAR after missions. The training consisted of two modules. During the first half of the day (module 1), theoretical input was presented, followed by simulations and discussion phases. The four effective teamwork competencies and the feedback techniques were presented once more, a case study was conducted for applying the teamwork competencies, facilitation techniques for effective debriefing sessions (c.f., Cuper et al., 2009; McDonnell et al., 1997; Tannenbaum et al., 1998) were introduced, and the sequence of actions of the AAR was presented and discussed. The platoon leaders also received posters, checklists and worksheets developed by the author to write down their observations after missions in a structured way and guide the AAR with the team in a standardised and correct manner. During the second half of the day (module 2), the facilitation techniques and the AAR applying all the checklists and worksheets were practised in simulations with corrective feedback for the platoon leaders. Relating to the CRM training research literature (e.g., Gregorich et al., 1990; Helmreich and Wilhelm, 1991; O Connor et al., 2003; Salas et al., 1999a, 2001, 2006b) and the specified training objectives, the following hypotheses were developed: 1 The TRM seminar will have a positive impact on team members reactions and their subjectively rated learning success. 2 The TRM seminar will have a positive impact on declarative knowledge acquisition. 3 The TRM seminar will have a positive effect on safety-relevant attitudes.

202 V. Hagemann and A. Kluge 4 The AAR will have a positive impact on the team members reactions and their subjectively rated learning success. 5 The AAR will support maintaining declarative knowledge. 6 The AAR will have a positive effect on safety-relevant attitudes. 7 The AAR will have a positive effect on the team members safety-relevant behaviours. 3 Method 3.1 Sample Ninety-one experienced firefighters (four female) from a German professional fire brigade participated in this study from June 2010 to February 2011. Most of the participants (37%) were between 31 and 40 years old. 31% were between 20 and 30 years old, 22% between 41 and 50 years, and the residual (10%) was older than 50 years. The firefighters belonged to two different firehouses and two different shifts. Participation in the study was voluntary, but the shifts and firehouses were selected by the department chief. 3.2 Design of the field study The study included a within- and a between-group comparison. Due to the natural division of the firefighters by firehouses and shifts, the participants formed four teams. Thus, the developed TRM seminar was conducted four times in the same manner during standby times by the same instructors within one week (within-group comparison). The platoon leader training followed two weeks later. This training was only conducted with the platoon leaders of the two teams building the experimental group (EG, n = 44), because only the EG received the AAR. The two teams of the control group (CG, n = 47) should still apply their standard mission debriefings (between-group comparison). The TRM seminar was conducted with all teams of the EG and the CG. The two teams of the EG belonged to one shift, but they worked in two different firehouses. Similarly, the CG belonged to one shift (the other shift to the EG) and also worked in the same two firehouses as the EG did. Questionnaires measuring the team members reactions to the TRM seminar and learning success, teamwork safety-relevant declarative knowledge, attitudes, and behaviours as well as reactions to the AAR and the quality of the platoon leaders facilitation techniques were distributed one week before the TRM seminar (T0), one day after the seminar (T1), and a third time seven months later (T2) (see Table 3). Due to this long time span and because of holidays, shift changes, and absenteeism, not all firefighters were able to participate at all three measurement times, but they all participated at a minimum of two measurement times.

The effects of a scientifically-based team resource management intervention 203 Table 3 Instruments Overview of instruments and measurement times One week before the intervention One day after the TRM seminar Seven months later T0 T1 T2 Knowledge Knowledge Knowledge Attitudes Behaviours Attitudes Attitudes Behaviours TRM seminar half day (within-group comparison) Reactions to the TRM seminar Platoon leader training and practice of AARs (only EG) (between-group comparison) Reactions to the AAR Evaluation of platoon leaders facilitation techniques 3.3 Assessment instruments Reactions to the TRM seminar. The training evaluation inventory (TEI; Ritzmann et al., 2013) for evaluating the TRM seminar and the team members reactions, respectively, at T1 consisted of 18 items (five-point Likert scale from 0 to 4). It covered training outcomes based on the first (reaction) and second (learning) level of Kirkpatrick s (1998) four levels of evaluation. Based on the work of Alliger et al. (1997), Phillips and Phillips (2001) and Salas et al. (2006b), the first level (reaction) was further divided into three scales: reported enjoyment (e.g., I enjoyed learning ), perceived difficulty (e.g., I understood all technical terms ) and perceived usefulness (e.g., The training is useful for my profession ). In particular, perceived usefulness is assumed to support the motivation to apply acquired knowledge and skills to the trainees field of work (Helmreich and Wilhelm, 1991; Phillips and Phillips, 2001; Salas et al., 2006d). Furthermore, it enhances the probability of the trainees work performance improving. The second level (learning) was divided into learning knowledge (e.g., I think my knowledge has been expanded in the long term ) and learning attitudes (e.g., I would recommend this training to my colleagues ). This questionnaire operationalised the cognitive- and affective-based learning objectives. The TEI was used as it was developed for training evaluation and has been applied in various CRM training evaluation studies (see Ritzmann et al., 2013). Reactions to the AAR. The survey for evaluating the AAR and the team members reactions (only the EG), respectively, at T2 consisted of 14 items (five-point Likert scale from 0 to 4). This survey was based on the TEI for evaluating the TRM seminar and therefore covered training outcomes based on the first (reaction) and second (learning) level of Kirkpatrick s (1998) four levels of evaluation. Again, the first level (reaction) was further divided into three scales: reported enjoyment (e.g., The learning environment during the AAR is convenient ), perceived difficulty (e.g., I am able follow all discussions in the AAR ) and perceived usefulness (e.g., It is very useful to spend time on the AAR ). The second level (learning) was divided into learning knowledge (e.g., I am continuously learning new things about the missions or my teammates in the AARs ) and learning attitudes (e.g., I would recommend this AAR to my colleagues ). Again, this questionnaire operationalised the cognitive- and affective-based learning objectives.

204 V. Hagemann and A. Kluge Declarative knowledge. Acquired knowledge within the TRM intervention was measured three times (T0, T1, T2) with a questionnaire containing five multiple-choice questions (e.g., Please indicate if the following assumptions are right or not ) and eight open questions (e.g., Please name influencing factors that could impair your SA ) (c.f., O Connor et al., 2003; Salas et al., 1999a, 2001). For each seminar module (communication, feedback techniques, SA, SMM, and effective team competencies), items were developed. An overall knowledge score was calculated over all items (max. 60 pts.). Five separate knowledge scores were also calculated for each single module. This questionnaire operationalised the cognitive-based learning objectives. Safety-relevant attitudes. To measure a change in safety- and teamwork-supporting attitudes, a 20-item five-point Likert scale (0 to 4) questionnaire was developed and applied three times (T0, T1, T2). The instrument is called the Fire Service Management Attitudes Questionnaire (FSMAQ). The items were adapted from existing attitude questionnaires [ORMAQ (surgery); Yule et al., 2004; ORMAQ (anaesthesia); Sexton et al., 2000; CAQ; McDonald and Shadow, 2003) and covered the seven most frequently investigated safety-relevant attitudes: command roles and responsibilities (e.g., Team members should not question the decisions or actions of senior staff ), speak up (e.g., I always ask questions when I feel there is something I do not understand ), debriefing (e.g., A regular debriefing of procedures and decisions after a mission is an important part of teamwork ), feedback and critique (e.g., Disagreements in the team are appropriately resolved, i.e., it is not who is right, but what is best for the mission ), realistic appraisal of stress (e.g., Personal problems can adversely affect my performance ), denial of stress (e.g., Even when fatigued, I perform effectively during critical times on a fire ), and handling errors (e.g., I am more likely to make errors in tense or hostile situations ). An attitude score for each scale was calculated and used in the analyses. The FSMAQ operationalised the affective-based learning objectives. Behaviours. A change in the team members safety-relevant behaviours was rated by the platoon leaders twice (T0, T2) with a 14-item six-point Likert scale survey from 0 (The team never showed this behaviour) to 5 (The team constantly showed this behaviour). The development of the 14 items was based on the concept of four effective team competencies (c.f., Smith-Jentsch et al., 1998a) and the anti-air teamwork observation measure (ATOM) scale for measuring SA (Smith-Jentsch et al., 1998b). Platoon leaders had to rate the teams behaviours shown in a mission regarding the following six aspects: information exchange (e.g., Team members share information at the right time with the right person, without being asked to do this ), communication (e.g., Team members articulate properly ), supporting behaviour (e.g., Team members observe each other, recognise errors and correct each other ), initiative/leadership (e.g., Team members give advice to each other and support each other ), shared mental model (e.g., Team members report thoroughly all relevant information in the correct order ) and SA (e.g., Team members identify potential future problems ). This survey operationalised the behavioural-based learning objectives. Platoon leaders facilitation techniques. The items of this questionnaire were based on the facilitation techniques, which were taught during the platoon leader training and support effective team debriefings while considering all team members (Cuper et al., 2009; McDonnell et al., 1997; Tannenbaum et al., 1998). The aim of this instrument was to check whether there was a difference in the behaviours between the trained (EG) and non-trained (CG) platoon leaders during debriefings and to show whether the trained platoon leaders applied their acquired skills at all. The questionnaire consisted of 14

The effects of a scientifically-based team resource management intervention 205 items, which were rated by the team members on a six-point Likert scale from 0 (never) to 5 (always). Minor differences in expression existed for one item between the EG and the CG (EG: I knew how the new mission debriefing works and what was expected from me and my teammates vs. CG I knew how the mission debriefing works and what was expected from me and my teammates ). The following item was identical in both groups: The discussion parts of the team members were longer than that of the platoon leader. An overall score was calculated over all 14 items. All instruments for measuring the specific training outcomes the questionnaire for knowledge acquisition, the FSMAQ, the behaviour ratings, and the questionnaire regarding platoon leaders facilitation techniques were developed for this study and tailored to the specific training objectives specified here. The instruments were adapted from instruments that are well established in research and have been pretested in a small sample in order to identify problems and misunderstandings within the target group (c.f., Hagemann, 2011). Therefore, firefighters other than those included in the final study answered the questionnaires and were additionally interviewed. Control variables (age and duration of team membership) and, for the first time, the dependent variables were measured before the intervention. Age and duration of team membership were also measured at T1 and T2, because it was assumed that due to the quasi-experimental design, the group composition would not always be identical, and the number of participants ranged between n = 56 (T0), n = 49 (T1) and n = 47 (T2) because some participants dropped out while others joined in over time. 4 Results Regarding the control variables, for all three measurement times, there were no significant differences between the CG and EG with regard to age (except for T2, EG = 4 (Mdn), CG = 3 (Mdn), U = 142.00, p <.01, r =.43) and duration of team membership. Table 4 M, SD, and Cronbach s α of training outcome scales at T1 (seminar) and T2 (AAR) Scales TRM seminar, both groups combined, n = 49 AAR, only EG, n = 17 α M SD α M SD Reported enjoyment.79 2.79 0.65.84 2.71 0.73 Perceived usefulness.90 2.41 0.80.94 2.49 0.91 Perceived difficulty +.72 2.90 0.53.73 2.57 0.71 Learning knowledge.86 2.22 0.79.86 2.65 0.87 Learning attitudes.85 2.51 0.87.89 2.86 1.03 Notes: + A high score means that the training was not difficult; range from 0 to 4. In order to test hypothesis 1, the subjectively perceived outcomes of the TRM seminar were evaluated by applying the TEI at T1. The internal consistencies and means of the evaluation scales are displayed in Table 4. The mean values of the five scales indicated an overall positive evaluation of the seminar regarding reactions and subjectively rated learning success. Furthermore, there were no significant differences between the EG and the CG with regard to the scales, except for reported enjoyment (F (1,39) = 11.56, p <.01, η 2 p = 0.23; EG: M = 3.15, SD = 0.54; CG: M = 2.51, SD = 0.66). Thus, the results

206 V. Hagemann and A. Kluge supported Hypothesis 1, namely that the TRM seminar was assessed positively and equally in both groups. In order to test the second hypothesis, whether declarative knowledge was acquired after the seminar, univariate analyses of variance (ANOVA) for analysing between- and within-group effects were conducted for the overall test score and for the five subparts. The main effects regarding measurement time reached significance for the overall score as well as the five subparts, with moderate to high effect sizes (see Table 5). Both groups, EG and CG, acquired safety-relevant knowledge within the TRM seminar, meaning that Hypothesis 2 was supported. Table 5 Results regarding knowledge test and attitudes at T0 compared to T1 (SD in brackets) α Knowledge test Knowledge overall (0 60 pts.) Shared mental model (0 11 pts.) Communication (0 11 pts.) Situation awareness (0 16 pts.) Team competencies (0 13 pts.) Feedback (0 9 pts.) Safety-relevant attitudes Realistic appraisal of stress EG CG T0 T1 T0 T1 17.18 (1.18) 0.00 (0.00) 4.71 (0.33) 4.35 (0.56) 2.53 (0.49) 5.59 (0.49).71 2.44 (0.98) Denial of stress +.69 2.33 (0.83) Handling errors.68 1.94 (0.66) 37.4 (2.06) 4.5 (0.60) 7.8 (0.64) 10.4 (0.83) 6.5 (0.92) 8.1 (0.35) 3.02 (0.46) 1.98 (0.75) 2.53 (0.67) 15.55 (1.47) 0.00 (0.00) 4.82 (0.41) 3.55 (0.70) 1.46 (0.61) 5.73 (0.61) 2.43 (0.58) 2.38 (0.62) 2.17 (0.56) 35.73 (2.56) 3.18 (0.74) 7.55 (0.80) 9.55 (1.03) 7.36 (1.14) 8.10 (0.43) 2.50 (0.71) 1.82 (0.59) 2.30 (0.56) Notes: EG = experimental group, CG = control group, T0 = pre-test, T1 = post-test 1; + Low values indicate a positive attitude; range from 0 to 4. Significance (for measurement time) F (1, 26) = 108.86, p <.001, η 2 p = 0.81 F (1 26) = 65.40, p <.001, η 2 p = 0.72 F (1, 26) = 25.79, p <.001, η 2 p = 0.50 F (1, 26) = 61.45, p <.001, η 2 p = 0.70 F (1, 26) = 41.29, p <.001, η 2 p = 0.61 F (1, 26) = 27.57, p <.001, η 2 p = 0.52 F (1, 28) = 5.79, p <.05, η 2 p = 0.17 F (1, 29) = 18.53, p <.01, η 2 p = 0.39 F (1, 29) = 12.06, p <.01, η 2 p = 0.29 ANOVAs with repeated measures were conducted in order to test whether the safety-relevant attitudes changed positively after the TRM seminar (Hypothesis 3). Group was the between-subject factor and the attitudes at T0 and T1 were within-subject factors. There were no significant results for command roles and responsibilities, speak up, debriefing and feedback and critique. Regarding realistic appraisal of stress and denial of stress, the main effects for measurement time reached significance (see Table 5). Thus, both attitudes changed significantly from T0 to T1, both for the EG and the CG. For handling errors, the interaction between measurement time and group (F (1,29) = 4.84, p <.05, η 2 p = 0.14; semi-disordinal) as well as the main effect only regarding measurement time reached significance (see Table 5). Because the parameter identifications at T0 and T1 did not show any significant results (no differences between

The effects of a scientifically-based team resource management intervention 207 the two groups), the attitude changed significantly from T0 to T1 in both groups. Thus, Hypothesis 3 could be partially supported; the TRM seminar had a positive impact on three safety-relevant attitudes. To test Hypothesis 4, that the AAR has a positive impact on the team members reactions and their subjectively rated learning success (knowledge and attitudes), the subjectively perceived outcomes of the AAR (only EG) were measured at T2. The internal consistencies and means of the evaluation scales are displayed in Table 4. The mean values of the five scales indicate an overall positive evaluation of the AAR regarding reactions and subjectively rated learning success (0 = completely disagree, 2 = partly agree, 4 = completely agree). Thus, the results supported Hypothesis 4. To test Hypothesis 5, it was analysed whether the groups maintained significantly more safety-relevant declarative knowledge at T2 than at T0. Six ANOVAs with repeated measures (for T0, T1, T2) were conducted for the overall test score as well as the five subparts of the knowledge test. The main effects regarding measurement time reached significance for the overall score and for the five subparts (see Table 6). Table 6 Results regarding knowledge test at T0, T1 and T2 (SD in brackets) Knowledge overall (0 60 pts.) Shared mental model (0 11 pts.) Communication (0 11 pts.) Situation awareness (0 16 pts.) Team competencies (0 13 pts.) Feedback (0 9 pts.) EG CG T0 T1 T2 T0 T1 T2 17.75 (4.06) 0.00 (0.00) 4.50 (1.07) 5.13 (1.73) 1.75 (1.05) 6.38 (1.77) 40.63 (9.93) 5.25 (2.49) 7.88 (2.03) 11.63 (3.29) 7.50 (4.21) 8.38 (0.92) 28.75 (8.73) 2.25 (2.38) 7.00 (2.20) 7.75 (2.66) 4.13 (2.30) 7.63 (1.06) 16.33 (4.93) 0.00 (0.00) 4.67 (0.58) 3.33 (2.16) 1.33 (1.53) 7.00 (2.00) 40.67 (10.02) 4.33 (2.79) 10.00 (1.73) 9.33 (6.51) 8.33 (4.16) 8.67 (0.58) 28.33 (12.70) 1.67 (2.89) 8.67 (2.08) 8.00 (6.56) 2.67 (2.31) 7.33 (0.58) Significance (for measurement time) F (2, 18) = 25.15, p <.001, η 2 p = 0.74 F (2, 18) = 12.05, p <.001, η 2 p = 0.57 F (2, 18) = 17.24, p <.001, η 2 p = 0.66 F (2, 18) = 9.72, p <.01, η 2 p = 0.52 F (2, 18) = 13.53, p <.001, η 2 p = 0.60 F (2, 18) = 6.53, p <.05, η 2 p = 0.42 Paired comparisons (Bonferroni-corrected) for each measurement time showed that the increase in knowledge acquisition from T0 to T1 was significant for the overall test score as well as all five subparts (see also Table 5; sample is now smaller due to third measurement time). A significant decrease in declarative knowledge from T1 to T2 was only found for the overall test score (p <.05), team competencies (p <.05) and feedback (p <.01). At T2, the participants still had significantly more knowledge than before the training (T0) regarding the overall test score (p <.05) and communication (p <.01). The distribution of the scores is similar for both groups. Thus, it can be stated that the EG and the CG acquired safety-relevant knowledge within the TRM seminar and maintained their knowledge over a period of seven months. However, Hypothesis 5 the AAR (only EG) supports the maintenance of declarative knowledge was not confirmed.

208 V. Hagemann and A. Kluge In order to test Hypothesis 6, whether the three attitude changes from T0 to T1 were stable, or whether one of the other four safety-relevant attitudes changed over a time period of seven months, ANOVAs with repeated measures (T1/T2) were conducted to compare the two groups. The three attitudes realistic appraisal of stress, denial of stress, handling errors remained stable over time, as no significant decrease was identified. For feedback and critique, the main effect for measurement time was now significant (F (1,17) = 4.89, p <.05, η 2 p = 0.22). The (semi-disordinal) interaction between group and measurement time was not significant; nevertheless, the effect size was medium to large (F (1, 17) = 3.54, p >.05, η 2 p = 0.17). Summing up, the results indicate that the three positive attitude changes from T0 to T1 were stable over a period of seven months for both groups. The attitude feedback and critique decreased more strongly in the CG (T1: M = 2.44, SD = 0.42; T2: M = 1.88, SD = 0.69) than in the EG (T1: M = 2.14, SD = 0.42; T2: M = 2.09, SD = 0.63). The other three attitudes did not change at all in either of the two groups. Thus, Hypothesis 6 the AAR has a positive effect on safety-relevant attitudes could not be supported. To test Hypothesis 7, after seven months, it was analysed whether the safety-relevant behaviours changed from T0 to T2 within the EG this group underwent the AAR and had time to practise the four effective team competencies. In order to test this hypothesis and the between-group effects, six ANOVAs with repeated measures were conducted each for one of the behaviour categories rated by the platoon leaders. Information exchange: results of the F-tests showed a significant main effect for group and a significant interaction between group and measurement time (see Table 7). This behaviour also changed significantly from T0 to T2, as the main effect for measurement time reached significance. As shown in Figure 1, due to the semi-disordinal interaction, information exchange increased in the EG from T0 to T2. Within the CG, this behaviour decreased from T0 to T2. At T2, the EG showed significantly more information exchange than the CG. Communication: results of the F-tests showed a significant main effect for group and a significant interaction between group and measurement time (see Table 7). This behaviour also changed significantly from T0 to T2, as the main effect for measurement time reached significance. As shown in Figure 2, due to the ordinal interaction, the EG showed significantly more communication than the CG and both groups significantly increased this behaviour from T0 to T2 additionally tested with post-hoc t-tests for one sample (EG and CG: p <.01). Figure 1 Information exchange

The effects of a scientifically-based team resource management intervention 209 Figure 2 Communication Supporting behaviour: results of the F-tests showed a significant interaction between group and measurement time. This behaviour changed significantly from T0 to T2, as the main effect for measurement time also reached significance (see Table 7). As shown in Figure 3, due to the disordinal interaction, supporting behaviour increased in the EG from T0 to T2. However, within the CG, this behaviour decreased from T0 to T2. At T2, the EG showed significantly more supporting behaviour than the CG, t (71) = 6.30, p <.001, η 2 p = 0.36. Initiative/leadership: due to missing variance within the EG at T0 and T2 the platoon leaders rated every team with the same values no ANOVA or t-test for one sample could be conducted. However, parameter identification at T2 showed a significant difference between the EG and the CG, t (71) = 28.08, p <.001, η 2 p = 0.92. The descriptive values in Figure 4 indicate that only the EG significantly changed their initiative/leadership from T0 to T2, while the CG did not do so. Figure 3 Supporting behaviour

210 V. Hagemann and A. Kluge Figure 4 Initiative/leadership SA: results of the F-tests showed a significant main effect for group and a significant interaction between group and measurement time (see Table 7). As the main effect for measurement time also reached significance, this behaviour changed significantly from T0 to T2. As shown in Figure 5, due to the disordinal interaction, SA significantly increased in the EG from T0 to T2. Within the CG, this behaviour significantly decreased from T0 to T2. At T2, the EG showed significantly more SA than the CG, t (71) = 39.42, p <.001, η 2 p = 0.96. Shared mental model: results of the F-tests showed a significant main effect for group and a significant interaction between group and measurement time (see Table 7). As the main effect for measurement time also reached significance, this behaviour changed significantly from T0 to T2. As shown in Figure 6, due to the semi-disordinal interaction, the shared mental model behaviour increased in the EG from T0 to T2. However, within the CG, this behaviour decreased from T0 to T2. At T2, the EG showed significantly more shared mental model-building behaviour than the CG, t (71) = 39.99, p <.001, η 2 p = 0.96. In summary, Hypothesis 7 the AAR has a positive effect on the team members safety-relevant behaviours could be supported. Figure 5 Situation awareness

The effects of a scientifically-based team resource management intervention 211 Table 7 Results regarding behaviours at T0 and T2 (SD in brackets) EG T0 3.33 (0.13) EG T2 4.00 (0.00) CG T0 3.18 (0.17) CG T2 2.67 (0.00) Significance for group Significance for measurement time Significance for interaction Information exchange Communication Supporting behaviour Initiative leadership Situation awareness Shared mental model F (1,71) = 2,577.41, p <.001, η 2 p = 0.97 F (1,71) = 29.04, p <.001, η 2 p = 0.29 F (1,71) = 1,599.24, p <.001, η 2 p = 0.96 3.73 (0.35) 3.98 (0.35) 3.16 (0.63) 3.28 (0.51) F (1,71) = 31.45, p <.001, η 2 p = 0.31 F (1,71) = 280.02, p <.001, η 2 p = 0.80 F (1,71) = 35.47, p <.001, η 2 p = 0.33 3.15 (0.23) 3.80 (0.47) 3.29 (0.76) 3.26 (0.25) F (1,71) = 3.78, p =.056, η 2 p = 0.05 F (1,71) = 43.01, p <.001, η 2 p = 0.38 F (1,71) = 50.15, p <.001, η 2 p = 0.41 3.00 (0.00) 4.00 (0.00) 2.76 (0.25) 2.76 (0.25) 3.33 (0.12) 4.00 (0.00) 3.64 (0.38) 3.13 (0.13) F (1,71) = 37.41, p <.001, η 2 p = 0.35 F (1,71) = 11.47, p <.01, η 2 p = 0.14 F (1,71) = 617.24, p <.001, η 2 p = 0.90 3.67 (0.00) 4.00 (0.16) 3.00 (0.00) 2.83 (0.17) F (1,71) = 3,928.81, p <.001, η 2 p = 0.98 F (1,71) = 29.04, p <.001, η 2 p = 0.29 F (1,71) = 299.32, p <.001, η 2 p = 0.81 Notes: EG = experimental group, CG = control group, T0 = pre-test, T2 = post-test 2; range from 0 to 5.

212 V. Hagemann and A. Kluge Figure 6 Shared mental model As a post-hoc analysis, the relationship between perceived usefulness of the TRM seminar (indicator of transfer motivation, evaluated by applying the TEI, subjective measure) and the acquisition of declarative knowledge (knowledge test, objective measure) was analysed by hierarchical regressions. As displayed in Table 8, perceived usefulness of the seminar was a significant predictor of knowledge acquisition at T1 as well as maintaining knowledge at T2. Table 8 Hierarchical regression of perceived usefulness on knowledge acquisition at T1 and maintaining knowledge at T2 Knowledge acquisition at T1 (n = 27) 1 Maintaining knowledge at T2 (n = 10) 2 B SE B β B SE B β Step 1 Constant 38.88 6.05 31.79 13.75 Knowledge overall T0 0.11 0.35.07 0.09 0.75.04 Step 2 Constant 26.41 6.75 30.79 9.92 Knowledge overall T0 0.16 0.30.09 1.21 0.67.57 Perceived usefulness 5.20 1.75.52** 7.40 2.56.90* Notes: 1 R 2 =.004 for step 1; R 2 =.27 for step 2 (p <.01). **p <.01. 2 R 2 =.002 for step 1; R 2 =.54 for step 2 (p <.05). *p <.05. 5 General discussion The aim of the study was to investigate whether a TRM intervention for fire service teams has positive effects on team members reactions as well as teamwork safety-relevant knowledge, attitudes, and behaviours. We conducted a within-group (TRM seminar) and a between-group study (AAR) with an EG and a CG. This procedure is often lacking within this field of training evaluation (e.g., Helmreich and Wilhelm, 1991; O Connor et al., 2012; Salas et al., 2001).

The effects of a scientifically-based team resource management intervention 213 According to the team members reactions, the TRM seminar was assessed as enjoyable, easy to follow, and useful for the work. Furthermore, positive attitudes towards TRM and the teamwork-relevant topics were established. The reactions of the participants of the EG regarding the AAR were similarly positive. The AAR was assessed as enjoyable, useful for the work, and recommendable to colleagues. With regard to the predefined multidimensional training objectives, the outcomes were assessed on an affective and subjectively rated cognitive level. The present results confirm findings of previous evaluation studies of CRM training within aviation and military fields, as demonstrated in studies and literature reviews by Salas et al. (1999b, 2001, 2006b). Regarding the teamwork safety-relevant knowledge, results showed that the TRM seminar had a significant positive impact on declarative knowledge acquisition for the overall test score as well as the five modules separately. Directly after the TRM seminar (T1), the team members had significantly more knowledge about the topics, such as SA, shared mental model or supporting team competencies. Similar results regarding CRM training outcomes for other HRTs within civil and military aviation were also found in previous studies and literature reviews by O Connor et al. (2003) or Salas et al. (1999a). Moreover, it was demonstrated that over a time period of seven months, the level of acquired knowledge could not be held constant, but the team members of the EG and the CG still had significantly more knowledge than before the training. Thus, a long-term effect of the TRM seminar could be demonstrated, but no stand-alone effect of the AAR was found. Regarding the predefined multidimensional training objectives, the outcomes were objectively assessed on a cognitive level. The post hoc analyses of perceived usefulness of the seminar rated by the team members as a predictor of knowledge acquisition and maintenance were in accordance with the results by, for example, Alliger et al. (1997), Helmreich and Wilhelm (1991), and Warr et al. (1999). These authors reported positive relationships between perceived usefulness of an intervention and motivation to learn, transfer motivation, subjectively rated learning success and objective measurement of knowledge acquisition. Their findings are in line with those of the present study and stress the importance of evaluating trainee reactions in a differentiated manner by focusing on perceived usefulness as postulated by Helmreich and Wilhelm (1991), Phillips and Phillips (2001) as well as Salas et al. (2006d). Regarding the positive changes in teamwork safety-relevant attitudes, results showed that after the TRM seminar (T1), three of the seven attitudes changed. Positive changes were found for realistic appraisal of stress, denial of stress, and handling errors. The present results are in accordance with some previous studies within other HRTs in aviation, such as the studies by Gregorich et al. (1990) and Helmreich and Wilhelm (1991). Even though Röttger et al. (2013) did not report effects of CRM training on attitude changes, however they found significant positive relationships between negative attitudes and teamwork behaviour and performance in the maritime domain. In the present study it was also demonstrated that after a time period of seven months, the positive attitude changes were stable. For feedback/critique, a significant decrease from T1 to T2 was identified. Even though the interaction did not reach significance (in view of the small sample size), the effect size (n 2 p =.17) and the descriptive data indicated that only the CG showed a significant decrease, and not the EG. In particular, the EG received the AAR, which focuses on discussing problems and errors made within missions and receiving feedback. Thus, the EG recognised the importance of giving and receiving