Available online at ScienceDirect. Procedia Manufacturing 3 (2015 )

Available online at www.sciencedirect.com ScienceDirect Procedia Manufacturing 3 (2015 ) 2589 2596 6th International Conference on Applied Human Factors and Ergonomics (AHFE 2015) and the Affiliated Conferences, AHFE 2015 Assessing navigational teamwork through the situational correctness and relevance of communication Kjell Ivar Øvergård a, *, Astrid R. Nielsen b, Salman Nazir a, Linda J. Sorensen a a Department of Maritime Technology and Innovation, Buskerud and Vestfold University College, Postboks 4, 3199 Borre, Norway b Department of Psychology, University of Oslo, Postboks 1094 Blindern, 0317 Oslo, Norway Abstract Investigations of accidents occurring in complex sociotechnical systems have found that communication and coordination among system elements are major contributing factors. The distributed situation awareness of a system is inherently dependent on the level of interaction between system elements. Previous studies have found that the distributed situational relevance of team communication is highly correlated with team performance. This study investigated the impact of the combination of both relevance andsituational correctness of communication during simulated high-speed craft navigation. Anatural experiment with navigation in the inner Oslo Fjord was performed with a three-person team in a maritime desktop simulator. Participants were required to communicateverbally to ensure safe and efficient navigation. All statements communicated by the participants during the experiment was rated for relevance (relevant/irrelevant) and for situational correctness (correct or incorrect) and these scores was combined to form a Communication score ranging from +1 (good) to -1(bad). Measures of the relevancy, correctness and the combined communication score were then correlated with the navigational team performance (measured as cross-track error). Findings indicate surprisingly that the relevance of communication (irrespective of its correctness) did not have any relationship to navigational performance (r ) while the ratio of wrong communication showed a small to moderate positive correlation (r =.237) to cross-track error. Most interestingly the combined communication score had a moderately large negative correlation with cross-track error (r =-.349) thus indicating that at least for dynamic navigation tasks it is not enough to only look on the relevance of communication but that also the situational correctness of information must be measured. 2015 The Authors. Published by by Elsevier Elsevier B.V. B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference. Peer-review under responsibility of AHFE Conference Keywords:Situation awareness; Communication; Teamwork; Navigation; Situational relevance; Situational correctness * Corresponding author. Tel.: +47-986-48-233; fax: +47-330-31-100. E-mail address: koe@hbv.no 2351-9789 2015 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/). Peer-review under responsibility of AHFE Conference doi:10.1016/j.promfg.2015.07.579

2590 Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 1. Introduction Collaboration and coordination are identified as critical factors for acquisition and maintenance of Situation Awareness (SA) [1]. SA refers to an individual s dynamic awareness of a specific external situation [2]. The concept focuses on how individuals develop and maintain an adequate understanding of what is really going on [2] in order to successfully perform their task. SA is dependent on situational characteristics and is tightly interwoven with situational changes[3]. Additionally, SA is considered as a critical commodity for successful performance in teams working in collaborative systems [4, 5]. Today, teams are widely used in time-critical and complex environments [6] and such settings place greater demands on teams ability for dynamic adaptation to their external environment [7]. Free and interactive communication, cooperation and coordination enhance teamwork [8]. However, due to the increased complexity of sociotechnical systems and work procedures, the presence of teams is required to a greater extent [9].The complexity and interconnections among different elements in the maritime industry calls for effective teamwork. Well-functioning teams can perform complex tasks more effectively than that of individual operators [10].Research investigating causes of vessel accidents show that between 62 and 96% of marine accidents are attributed, at least in part, by some form of human error [11]. Furthermore, researchers claim that SA in collaborative environments is closely connected to performance increased level of SA is associated with increased level of performance [2, 7, 12, 13]. Despite the presumed importance of Distributed Situation Awareness (DSA) to navigational teamwork, there exist few systematic studies of the relationship between these constructs in the maritime domain. Research on aviation, road traffic, and rail accidents is comparatively widespread, allowing a better understanding of the problems related to their causes [14, 15]. 1.1. Navigational teamwork and SA A common definition of a team is that a team consists of two or more individuals, who have specific roles andperform interdependent tasks in order to accomplish a common goal [8]. Efficient teamwork require that the team members possess specific knowledge, skills, and attitudes, such as knowledge of their own and other team members tasks and responsibilities, skill in monitoring each other's performance, and a positive attitude towards working in a team. Although the team members have access to the same information, they utilise the information differently based on their goals, roles and tasks in a compatible manner [16]. Good teamwork plays a vital role in safe and efficient navigation. Efficient and successful bridge team communication is acknowledged as a key to safe operations of sea-going vessels. Communication coordinates teamwork [7, 12, 17]by ensuring that the required information is given to the appropriate agents at the correct time. Accordingly, communication enables team members to perform their specific task and thus supports effective team performance. Furthermore, efficient communication is considered as a key aspect of the development of situation awareness in collaborative settings.indeed, Orasanu[18] found that information exchange between a systems agents (both human and non-human) was associated with high levels of SA and that high levels of SA was related with high levels of performance in teams. A considerable amount of marine accidents involve communication errors that cause lack of situation awareness and decreased decision making abilities [19]. Recent studies on marine incidents have recognized communication as a critical factor [20, 21]. A total of 41.2% of accidents involving groundings were accounted for by poor communication [20]. Also, Pyne and Koester [21] found that communications errors played a vital role in maritime accidents. The theory of DSA [22] is tightly related to the communication of information. Stanton et al[22] define DSA as: activated knowledge for a specific task, at a specific time within a system (p. 51). In other words, the theory of DSA points to communicated information as a key ingredient that can establish and improve team SA and hence also affect team performance.

Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 2591 1.2. Evaluation of team communication A recent approach to evaluating teamwork was given by Sorensen and Stanton [12]. They evaluated the connection between relevancy of team communication and team performance using thepropositional Network Method [23]to define relevant information concepts for the experimental task. The Propositional Network method represents DSA as a network of information nodes and their interconnection [22], which is based on the assumption that knowledge comprises concepts and the association between them [24]. The nodes represent knowledge, agents, source of information and artefacts that are linked through specific paths [23]. The map represents information that has developed or emerged within a system, which are illustrated by nodes and the relationship existing between them. The technique gives indications of what knowledge the agents require in order for them to successfully perform their task [16].The relevancy of communication was defined by the ratio of relevant statements divided by the total number of statements (e.g. relevant + irrelevant statements). Sorensen and Stanton [12] found a high positive correlation (r =.923) between task success and the Distributed Situational Relevancy (DSR) of team communication. Similarly, a negative correlation was found between DSR and team failures (r =-.52), indicating that the relevancy of team communication could be a good indicator of good (and bad) team performance. 1.3. Aim of this paper The aim of this study isto identify the association between the distributed relevancy of communication and team performance during realistic simulator training sessions involving high-speed craft navigation in congested coastal waters. Building on Sorensen and Stanton s work [12] the DSR method was extended to include situational correctness of communication in order to consider their relation to navigational performance. This is done because navigation is a highly context dependent activity where incorrect statements can lead to disastrous consequences. 1.4. Hypotheses We aim to give answers to the following research hypotheses: H 1 : There is a positive relationship between the combined relevance/correctness score of communication and team (navigational) performance. H 2 : There is a negative relationship between the ratio of wrong communication items and team performance. H 3 : There is positive relationship between the ratio of relevant communication and team performance. 2. Method 2.1. Participants, ethical consent and sampling In total, 9 teams consisting of three persons(22 male, 5 female, see table 1) were recruited. The participants were second and third year students attending the bachelor program in maritime studies located at the University of Buskerud and Vestfold, Norway. Their age ranged from 20 to 38 years (x = 24.22, = 4.47). The participants were familiar with maritime simulators, as simulator training was part of their degree course. The participants were recruited on the basis of their navigational education and experience of navigation in simulators. All participants were informed of the intention of the study and the use of the recorded data. All participants signed an informed consent form.the Norwegian Social Science Data Service (NSD) approved our application for making video recordings of the experiments. Noldus Software was used for the synchronisation ofrecordings from one audio recorder and two video cameras.

2592 Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 2.2. Experimental task and equipment An experienced navigator instructor at the Buskerud and Vestfold University College (HBV) mapped out a route of total4,63nautical miles in the inner Oslo fjord. This section is characterized bydemanding and confined waters with numerous islets, making for particularly challenging navigation. The navigation task was designed to reflect scenarios of different complexity levels intended to engage the whole navigation team to ensure interdependency in accordance with Salas [8] definition of teams. The experimental setting was performed during daytime in fairweather and visibility conditions. A navigation teacher at HBV selected and prepared the route in paper charts. The route was divided into ten segments with varying density and difficulty and it also contained navigational information regarding turning points and courses. Table 1 shows the longitude and latitude of the ten segments of the navigation scenario. Table 1. Position of segments in the navigation scenario. Leg Latitude start Latitude end Longitude start Longitude end Distance 1 59 52.928 59 52.648 E010 41.385 E010 42.552 0.65 nm 2 59 52.648 59 52.750 E010 42.552 E010 42.598 0.54 nm 3 59 52.750 59 53.317 E010 42.598 E010 43.694 0.57 nm 4 59 53.317 59 53.629 E010 43.694 E010 44.689 0.58 nm 5 59 53.629 59 53.904 E010 44.689 E010 44.615 0.28 nm 6 59 53.904 59 54.108 E010 44.615 E010 44.010 0.38 nm 7 59 54.108 59 53.904 E010 44.010 E010 43.083 0.51 nm 8 59 53.904 59 53.890 E010 43.083 E010 42.173 0.46 nm 9 59 53.890 59 53.612 E010 42.173 E010 41.212 0.55 nm 10 59 53.612 59 53.502 E010 41.212 E010 41.224 0.11 nm 2.3. Simulator setup and recording of information of vessel and navigation characteristics The experiment was conducted using desktop simulator running Kongsberg Polaris Simulator software. The simulator provides a virtual environment during sailing recreating the visual environment surrounding the vessel as well as mathematically simulating the hydrodynamics of the vessel. The simulator was equipped with a visual system (120degrees forward view) that enabled realistic and detailed re-creations of vessel movement, environments, weather and sea conditions. It was also equipped with Conning, which provided information regarding speed, course and rudder angle. A RADAR (Radio Detection and Ranging) -systemwas utilized as an object-detection system and provided the navigation team with information regarding altitude, range, direction and movement of other vessels. The vessel model was a missile-torpedo-boat (MTB) similar to the Norwegian Royal Navy s Super-Hauk MTB which was a 150 gross ton high-speed craft that allows for a maximum speed of ca. 33 knots (see [25]). The simulator recorded information about the vessel s position (latitude and longitude) as well as technical information about the vessel such as rudder angle, rate of turn, heading and speed. Recordings was at 2 second intervals e.g. with a rate of 30 times a minute. 2.4. Experimental set-up Three workstations where arranged in three cubicles sectioned with blue foam boards (see Fig. 1). The participants were not able to see each other during the experiment. Hence, each participant had access to the information in their cubicle but not the information the other participants had. This was done to necessitate efficient and correct verbal communication to achieve successful navigation performance. No other forms of communication between the participants where possible. Teams consisted of three team members who are assigned to each their distinct roles and tasks: the commanding officer (CO), the navigator and the helmsman. The CO s overall responsibility was to take the vessel to its destination in an efficient and safe manner.the CO s workstation was equipped with a screen displaying the visual lookout. Hence, the CO was in charge of visually monitoring and reporting the environment from the vessel s bridge to the other team members.the CO have the

Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 2593 Fig. 1. Overview of the experimental set-up. Navigator to the left, CO in the middle and helmsman to the right. Picture by the Authors. executive role and thus made decisions regarding direction and speed based on information provided by the navigator and the helmsman. The navigator was responsible for handling the map during sailing and was exclusively equipped with a paper chart, having no other information available. He/she is accountable for monitoring the vessels position during sailing. Furthermore, he/she is responsible for providing the CO with information from the paper chart. The helmsman was provided with a steering wheel, throttle and a conning display presenting information regarding rudder angle, water depth, heading, course and speed. The helmsman is in charge ofsteering the vessel based on commands from the CO. Commands from the CO to the helmsman wasgiven verbally and the helmsman executed and verbally confirmed the command. 2.5. Transcription of communication All statements were transcribed from the audio/video-recording. The person (CO/Helmsman/Navigator) and the time of onset of each statement were recorded. All statements were transcribed in the two-second interval it started. Only a small number of statements were longer than two seconds. 2.6. Evaluating relevancy of communication Identification of relevant communication followed the propositional network method [23]. The propositional network was based upon an Interview with a subject matter expert in coastal navigation using the Critical Decision Method [23]. The propositional network for based upon a content analysis of this interview (see Fig. 2). 2.7. Evaluating correctness of communication The evaluation of situational correctness of communication was done by cross-checking the audio/video with the map and the current position of the vessel to investigate if the navigational team s relevant statement also included statements that were in accordance with the situation or with what they were seeing or would do in the near future. Most of the errors stemmed from miscalculations of position and resulting misinterpretation of navigational objects. 2.8. Combined score of relevance and situational correctness The combined scores of the relevancy and situational correctness are simply called Communication scores. Statements that where rated as relevant and correct where labelled as Good and given a score of +1, the relevant and incorrect statements where labelled Wrong and scored as -1. Finally, all irrelevant statements where scored 0 points.

2594 Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 Fig. 2. Propositional Network for Coastal Navigation In addition, instances of silence (no communication) lasting 10 seconds or more was given a score of -1 for each 10-second interval. This was in recognition of the experimental tasks that were high-paced and dynamic as caused by the short (navigational) segments, the abundance of shallow waters, rocks and other vessels requiring a high frequency of communication between team members to produce successful navigation. 2.9. Navigational performance Team performance was measured based on the distance between the planned route (see Table 1) and actual sailed route. The deviation between the planned track and the observed track was measured as cross-track error (XTE). The XTE is given as the shortest distance between the actual position and the closest point on the planned track as measured at a 90 degrees angle from the pre-planned track to the actual position (see [26]). 2.10. Preparation of data for analysis Communication scores and XTE were averaged over each of the 10 segments by calculating the arithmetic average of each of these variables. The number of relevant, irrelevant and wrong statements was counted for each of the 10 segments and the counts used to calculate the ratio of relevant statements ( good + wrong statements/( good + irrelevant + wrong statements)), the ratio of irrelevant statements ( irrelevant /( good + irrelevant + wrong statements)), and the ratio of wrong statements ( wrong /( good + irrelevant + wrong statements)). 3. Results Pearson correlation was used to test H 1 -H 3. Statistical analyses where done with IBM SPSS 22. Confidence intervals for r was calculated by bootstrapping procedure in SPSS repeated 1000 times. A significant negative correlation of medium size (Cohen, 1988) was observed between the average Communication score and the XTE (r(89)= -.349, 95% CI [-.521, -.226], 2-tailed p=.001), thus corroborating H 1 by showing that there is a correlation between high communication scores and lower deviations from the planned track. This means that teams that had higher instances of relevant and correct statements supported navigation along the pre-planned route.

Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 2595 H 2 was corroborated by a significant positive correlation (r(87)=.237, 95% CI [.060,.410], 2-tailed p =.027) by showing that higher levels of wrong statements were associated with higher XTE. Thus, the segments where the teams had higher ratio of errors also tended to have higher XTE. Contrary to expectations shown in H 3 there was no significant relationship between the ratio of relevant statements (irrespective of whether the statements are correct or not) and XTE (r(87) =.010, 95% CI [-.166,.181], p =.928) thereby finding no support for H 3. To follow up on this rather unexpected finding we calculated the ratio of good statements to all statements ( good /( good + irrelevant + wrong statements)) and checked the association between the ratio of good statements with XTE. The indication showed a near significant finding (r(87) = -.191, 95% CI [-.381, 013], p =.076) thereby showing that good communication is somewhat associated with lower XTE as a measure of better performance. 4. Discussion Communication and DSA within teams is a vital component for successful team or system performance [17, 22]. This study investigated how the relevance and correctness of statements during high-speed coastal navigation affected navigation performance. It was based upon Sorensen and Stanton s [12] DSR method. Evaluation of results indicates that it is necessary to measure the combination of situational relevancy and correctness of statements in order to have a measure that is associated with team performance. A sole focus on relevant statements as identified in a propositional network do not show any association with team performancein coastal high-speed navigation. When considering only the correctness of communications we found an association between ratio of wrong (relevant and incorrect) statements and performance. The association between good (relevant and correct) statements and team performance was tenuous (e.g. not significant) but evaluation of the effect size and the confidence interval give reason to believe that a larger sample size would have revealed a significant association. The lack of support for the hypothesized association between the ratio of relevant statements and team performance was somewhat surprising as it did not correspond with the large effect size (r =.923) observed by Sorensen and Stanton [12].However, this might be due to the highly contextualized activity of coastal high-speed navigation (see e.g. [25, 27]). In high-speed navigation the team and vessel must adapt quickly to situational changes and this requires good DSA [17, 27], so wrong communication leads to deviations from planned route and hence reduce navigational safety and efficiency. This association is exactly what we have observed with the association between wrong statements and XTE. These early, but promising results give reason to believe that it can be possible to create a simple but comprehensive method of observation that allows for a thorough assessment of team performance in highly complex and context-dependent work tasks such as coastal high-speed navigation. The findings further indicate that the method can be improved by refining the weighting of good, irrelevant and wrong statements with the aim of increasing the reliability of the method to assess DSA and communication in team performance. 4.1. Limitations The participants in this study werenautical students and as such not as experienced as real navigators however, we believe that the association between correctness and relevance of statements and team performance would still be present in groups of experienced navigators. The statistical analyses of this paper only allow us to state the existence of an association between variables and we cannot make causal inferences. Similarly, we cannot say anything of the causal direction or of the timedependency between these measures. Further studies will allow us to delve deeper into these challenges. 5. Conclusion It seems to be necessary to consider the combination of both the relevancy and the situational correctness of statements when assessing the association between DSA (as measured by team communication) and team performance. A measure that only relies on the relevance of communication (irrespective of the correctness of communication) is not associated with team performance in near-coastal high-speed craft navigation.

2596 Kjell Ivar Øvergård et al. / Procedia Manufacturing 3 ( 2015 ) 2589 2596 References [1] D. D. Woods, M. Branlat. J. Basic patterns of how adaptive systems fail. In E. Hollnagel, J. Pariés, D. D. Woods and J. Wreathall (eds.). Resilience engineering in practice: A guidebook.ashgate, Aldershot, England, 2011, pp. 127-144. [2] M. R. Endsley, Toward a theory of situation awareness in dynamic systems. Human Factors 37(1) (1995) 32-64. [3] P. M. Salmon, N. A. Stanton, G. H. Walker, & D. P. Jenkins. Distributed Situation Awareness. Ashgate, Farnham, England, 2009. [4] Y. Shu, K. Furuta. An inference method of team situation awareness based on mutual awareness. Cognition Technology & Work, 7 (2005) 272-287. [5] S. Nazir, S. Colombo, D. Manca. The role of situation awareness for the operators of process industry. Chemical Engineering Transactions, 26 (2012) 303-308. [6] A. Worm, J. Jenvald, M. Morin. Mission efficiency analysis: evaluating and improving tactical mission performance in high-risk, time-critical operations. Safety Science, 30 (1998) 79-98. [7] S. Nazir, L. J. Sorensen, K. I. Øvergård, D. Manca, D. Impact of training methods on Distributed Situation Awareness of industrial operators. Safety Science 73 (2015) 136-145. Doi: http://dx.doi.org/10.1016/j.ssci.2014.11.015. [8] E. Salas (2005). Team Methods. In N. A Stanton, A. Hedge, K. Brookhuis, E. Salas and H. Henrick (eds.) Handbook of Human Factors Methods. CRC Press, Boca Raton, 2005, pp. 429-433. [9] S. M. Fiore, E. Salas, H. M. Cuevas, C. A. Bowers. Distributed coordination space: toward a theory distribute team process and performance. Theoretical Issues in Ergonomic Science,4(2003) 340-364. [10] J. Orasanu, U. Fisher, Finding decisions in natural environments: The view from the cockpit. In C. E. Zsambok and G. Klein (eds.), Naturalistic Decision Making, Lawrence Erlbaum, Mahwah, NJ, 1997, pp. 343-57. [11] B. S. Dhillon. Human Reliability and Error in Transportation Systems. Springer, New York, 2007. [12] L. J. Sorensen, N. A. Stanton. Y is best: How Distributed Situational Awareness is mediated by organisational structure and correlated with task success. Safety Science, 56 (2013) 72-79. [13] M. Leonard, S. Graham, D. Bonacum. The human factor: the critical importance of effective teamwork and communication in providing safe care. Quality and Safety, 13 (2004) 85-90. [14] J. R. Wilson, L. Cordiner, S. Nichols, L. Norton, N. Bristol, T. Clarke, S. Roberts. On the right track: Systematic implementation of ergonomics in railway network control. Cognition, Technology & Work, 3 (2001) 238 252. [15] N. B. Sarter, R. Amalberti. Cognitive engineering in the aviation domain. Lawrence Erlbaum Associates, Inc, Mahwah, NJ, 2000. [16] P. M. Salmon, N. Stanton, G. H. Walker, D. P. Jenkins, L. Rafferty. Is it really better to share? Distributed situation awareness and its implications for collaborative system design. Theoretical Issues in Ergonomics Science, 11 (2010) 58-83. [17] P. M. Salmon, N. A. Stanton, G. H. Walker, D. P. Jenkins. Distributed Situation Awareness. Ashgate, Farnham, England, 2009. [18] J. Orsanu.. Evaluating team situational awareness through communication. In Garland, D.J. (Eds.), Proceedings of an International Conference on Experiment Analysis and Measurement of Situational Awareness. Embry-Riddle Aeronautical University Press, Daytona Beach, FL, 1995, pp. 283-288. [19] P. John, B. Brooks, C. Wand, U. Schriever, U. Information density in bridge team communication and miscommunication a qualitatively approach to evaluate maritime communication. WMU J Marit Affairs, 12 (2013) 229-244. [20] C. McCrae. Human Factors at sea: Common patterns of errors in groundings and collisions. Maritime Policy and Management, 36(1) (2009) 21-38. [21] Pyne, R. & Koester, T. Methods and means for analysis of crew communication in the maritime domain. The Archives of Transportation, Vol. XVII, No. 3-4, 2005. [22] N.A. Stanton, R. Stewart, D. Harris, R. J. Houghton, C. Baber, R. McMaster, R. M. Salmon, G. Hoyle, G. H. Walker, M. S. Young, M. Linsell, R. Dymott, D. Green, D. Distributed situation awareness in dynamic systems: theoretical development and application of an ergonomics methodology. Ergonomics, 49 (2006) 1288-1311. [23] N. A. Stanton, P. M. Salmon, L. A. Rafferty, G. H. Walker, C. Baber, D. P. Jenkins. Human Factors Methods, 2nd edition. Ashgate, Farnham, England, 2013. [24] Salmon, P.M. What is really going on? Review of situation awareness models for individuals and teams. Theoretical Issues in Ergonomics Science, 9 (2008) 297-323. [25] K. I. Øvergård, C. A. Bjørkli, B. K. Røed, T. Hoff. Control strategies used by experienced marine navigators: observations of verbal conversations during navigation training. Cognition Technology & Work, 12 (2010) 163-179. [26] S. Han, Q. Zhang, H. Noh, B. Shin. A dynamic performance evaluation method for DFPS receivers under linear parallel-tracking applications. American Society of Agricultural Engineers, 47 (2004) 321-329. [27] C. A. Bjørkli, K. I. Øvergård, B. K. Røed, T. Hoff. Control situations in high-speed craft operation. Cognition Technology & Work, 9 (2007) 67-80.