Hollnagel s test: being in control of highly interdependent multi-layered networked systems

Transcription

1 DOI /s ORIGINAL ARTICLE Hollnagel s test: being in control of highly interdependent multi-layered networked systems David D. Woods Matthieu Branlat Received: 21 February 2010 / Accepted: 10 March 2010 Ó Springer-Verlag London Limited 2010 Abstract Advances in technologies for networking, sensing, and automation have resulted in multi-layered networked systems that extend information gathering, interactions across roles, and the potential for control over wider ranges. But these systems also represent a scale shift in complexity in terms of the density of interdependencies across processes and activities. In the new systems, coupling has run amok introducing new challenges about how to control processes when they are part of such highly interdependent webs. Based on the joint cognitive systems perspective, Hollnagel examines, or tests, technology changes by asking two key questions: what does it mean to be in control and how can control be amplified? Hollnagel has shown that the answers to these questions are not inherent in technology itself but rather point to emergent system properties that can and should be supported to produce success and avoid failures. This paper applies Hollnagel s test to the reverberations of technology change that are producing multi-layered networked systems. The paper shows how being in control of multi-layered networked systems requires the ability to navigate interdependencies and shows how amplifying control then consists of tools that help reveal/track relevant interdependencies and help anticipate how projected actions will propagate (resonate) across interdependencies relative to D. D. Woods (&) Cognitive Systems Engineering Laboratory, Ohio State University, 290 Baker Systems Engineering, 1971 Neil Avenue, Columbus, OH 43210, USA woods.2@osu.edu M. Branlat Cognitive Systems Engineering Laboratory, Ohio State University, 372 Baker Systems Engineering, 1971 Neil Avenue, Columbus, OH 43210, USA goals. The end result is that a shift is underway from supervisory control to polycentric control architectures. Keywords Cognition Control Cognitive systems engineering Joint cognitive systems Complex adaptive systems Polycentric control Supervisory control 1 Complexity and control Technology advances in networking, sensing, and robotics have produced a scale shift in modern systems. Technologies for connecting people at a distance from each other provide new opportunities to coordinate activity across large distances and to integrate activities that go on at different places or at differing time scales. Advances in sensing and robotics allow fusion of feeds from multiple sensors to allow people in remote locations to look at and explore aspects of the world as if they were standing in places that are too difficult, expensive, dangerous, or impossible to occupy. High-level managers can attempt to control from a distance as technology allows them in real time to examine and intervene in remote activities carried by subordinates. The new sensors, networks, and automation expand the spatial and temporal scales that can be monitored and controlled (Doyle 2000; Kulathumani et al. 2008). These new technologies have resulted in multi-layered networked systems that extend information gathering, interactions, and the potential for control over wider ranges. But these systems also represent a scale shift in complexity in terms of the density of interdependencies across processes and activities. In the new systems, coupling has run amok introducing new demands to keep track of more interconnected processes, new difficulties in

2 assessing and anticipating how highly interconnected situations will evolve or cascade, new risks for fragmentation across activities that inadvertently work at cross-purposes, and new challenges to inject control actions that produce only the desired results while avoiding unintended consequences. The scale shift expands the range of potential control but also has produced the risk of increasingly brittle systems and the need to harness the complexity through new control architectures. Hollnagel (1992, 1993, 1998, 2001) has approached such points of technology change from the perspective of joint cognitive systems (Hollnagel and Woods 1983, 2005) by asking two key questions about systems (1) What does it mean to be in control? (2) How to amplify control? Technology shifts can be analyzed then in terms of how the answers to these questions change and constitute Hollnagel s test (Hollnagel and Woods 2005, Chap. 7). Hollnagel (1999, 2001) has shown that the answers to these questions are not inherent in the technology itself but rather point to emergent system properties that can and should be supported to produce success and avoid failures. This paper applies Hollnagel s test to the technological changes producing multi-layered networked systems. The paper shows how being in control of multi-layered networked systems requires the ability to navigate interdependencies and shows how amplifying control then consists of tools that help reveal/track relevant interdependencies and help anticipate how projected actions will propagate (resonate) across interdependencies relative to goals. 2 Being in control : supervisory control of automated resources Shifts in technology change the answers to the questions that make up Hollnagel s test. Consider the shift from manual control to supervisory control (e.g., note the content focus shifts from Edwards and Lees 1974 to Rasmussen and Rouse 1981 to Hollnagel et al. 1986). In manual control, the controller sees and touches key base parameters blood pressure in cardiovascular anesthesia or vehicle path in driving. The controller directly adjusts effector mechanisms that change the parameter under control to meet targets and stay within limits (e.g., a safe braking time from the vehicle ahead). The target for control is an instantiation of general system goals defined at the level of the parameter itself at a narrow spatial, temporal, and functional scale. The control loop is direct and tangible. Being in control and amplifying control were questions that could be answered in terms of control theory and the practical engineering of control loops depending on the dynamics of the process to be controlled, e.g., avoid delays in feedback, provide anticipatory information. The shift to supervisory control changed the answer to the questions of what it means to be in control and how one amplifies control. Now the supervisory controller manages a set of automated subsystems and cognitive functions come to dominate the new role e.g., goal monitoring, exception handling, recognizing anomalies, and modifying plans in progress in the face of disrupting events (e.g., Woods and Sarter 2000). What is under control shifts from trends in key parameters of the monitored process (blood pressure, vehicle path) to higher level properties that relate to expectations, plans, and goals (Hollnagel 2001; Hollnagel and Woods 2005; Woods and Hollnagel 2006). The supervisor controller monitors the lower-order controllers and their progress, not simply process state; monitors progress toward goals and shifts goal priority, not simply targets; manages anomalies and constraints, not simply avoiding limits on a parameter. Potential breakdowns are different from direct control as the supervisor can get stuck on an initial assessment and discount discrepant data, as communication can break down between supervisor and automation, as attention can be misdirected or misfocused, as goal conflicts can be missed or misprioritized (Woods et al. 2010). The new joint cognitive system of people, visualizations, automation, and intelligent support systems changed what it meant to amplify control, shifting the focus to anomaly response and replanning in the face of disruptions (Woods and Shattuck 2000). What is valuable feedback changes (Woods and Sarter 2000). Now interfaces need to highlight the behavior of automated resources under supervision and become more Transition oriented to capture and display events and sequences, Future oriented to reveal what should/can happen next, Pattern based to support quick recognition of unexpected or abnormal conditions Abstract to capture the state and trends in higher order system properties. As a result, being in control now refers to the ability to pick up and take advantage of these new forms of feedback in order to better anticipate trouble ahead. Amplifying control then to designers ability to provide these forms of feedback by analyzing tasks characteristics and constraints and in building appropriate dynamic information visualizations. In direct control, design concerned the controller s ability to handle variations and dynamics in the process to be controlled. In supervisory control, there are multiple roles, and the relationships across roles are dynamic depending on the demands of the situation and the ability of various control roles to handle those demands. Depending on the match between demands and control

3 resources, roles change supervisors may delegate more autonomy or authority to lower-order controllers or reassert their authority to decide on responses or even take over the task (Dekker and Woods 1999; Miller and Parasuraman 2007). Smooth transfer of control becomes critical as authority and autonomy relationships shift depending on how well one configuration is able to handle the demands of the situation at hand. Amplifying control now refers to the ability to adjust relationships between roles in synchrony with the changing demands of situations. Also note that in supervisory control, the focus is on the supervisory role as the key center of control. It is this center that delegates and can reassert authority; it is this center that manages goals and goal priorities; it is this center that possesses responsibility for outcomes relative to goals (Woods and Hollnagel 2006). There are debates about how the supervisory role maintains its ultimate authority and responsibility, but in the end this role is seen as in command. In the process of studying supervisory control systems, the question of control morphed into questions about coordination of distributed cognition (Hutchins 1995; Hollnagel 2001; Klein et al. 2005; Nyssen 2007). Coordinating work over distributed roles invoked new concepts about interaction and collaboration such as building common ground, aligning subgoals across different roles and groups. Breakdowns took the form of coordination surprises and going sour incidents where, through a series of miscommunications and misassessments, the system was managed into trouble. Interestingly, a basic finding from these investigations paralleled the findings on supervisory control: coordinating activity depends on being able to see how hard others are working to maintain or regain control within their scope of authority and responsibility, as situations change. 3 Being in control : multi-layered networked systems The technology advances that enable and empower multilayered networked systems have shifted the potential for control to wider spatial, temporal, and functional ranges. This scale shift, in the pursuit of extending the range of control, also changes what it means to be in control and what it means to amplify control within the new range of possibilities. Distributed cognitive systems form one kind of layered networked systems, and a common theme emerged in studies of such systems across settings ranging from space mission control (Patterson and Woods 2001) to critical health care settings (Cook and Rasmussen 2005; Cook 2006; Miller and Xiao 2007; Wears and Woods 2007) to fire fighting (Woods and Branlat 2010) to aviation (Knecht and Smith 2001; Bergström et al. 2010) and others (Flach et al. 2003; Morel et al. 2008). In all of these settings, a key factor emerged work was organized around how to manage the capacity to handle future demands or contingencies. The work system controlled its margin of maneuver based on anticipating upcoming demands. This process was seen in how trauma units adapt to maintain a surge capacity (Miller and Xiao 2007); how intensive care units avoided bed crunches (Cook 2006); how mission control maintained the ability to call in extra expertise quickly when anomalies occurred (Patterson and Woods 2001); how urban fire ground commanders attempted to maintain a reserve capacity to respond to the next event and avoid all hands situations where all available resources were committed (Woods and Branlat 2010). In part, control of margin for maneuver constitutes one kind of tactic for managing uncertainty given the potential for surprise in these settings (Grote 2004; Norros 2004; Cuvelier and Falzon 2010). Being in control relates to the ability to assess how margins of maneuver are expanding or contracting relative to uncertainties and the potential for surprise. Another control theme has emerged from studies of multi-layered networked systems such as software intensive systems (Jackson 2009), safety management (Hollnagel 2004), air traffic management (Chapman et al. 2001; Smith et al. 2007), and financial systems (Sundström and Hollnagel 2010). In all of these studies, the key capability was managing interdependencies across roles and parts, especially tracking how changes create new interdependencies to be managed. For example, the investment in software intensive systems is valuable because it connects parts or layers that previously had been too separated. But these connections create interdependencies that pose risks for software reliability and system safety (e.g., the Ariane 501 rocket launch failure). Interdependencies within software intensive systems are easy to miss and hard to reveal (Jackson 2009). Advances in software engineering attempt to help developers assess how modules depend on one another when designing a program (Rae et al. 2005). Basically, a module A depends on a module B if A will not work properly without B working properly. These couplings or dependencies make it hard to anticipate the side effects of local modifications to a module. Changing B may trigger a change to A and then to other activities or structures that depend on A. If the software engineer has tools to track dependencies, they can reduce them in design or anticipate and evaluate side effects when changes occur over the software s life cycle (e.g., software reuse). Similarly, studies of how joint activity is synchronized have emphasized the need to manage dynamic interdependencies between activities to achieve broader system goals. Studies of coordination in health care reveal the

4 critical role of synchronizing interdependent activities in pace with events (Nyssen and Javaux 1996; Nemeth et al. 2007; Xiao et al. 2007; Nyssen 2010). The synchronization process depends on being able to see how hard others are working to gain and maintain control within their scope of authority and responsibility, as situations change (Klein et al. 2005; Woods and Branlat 2010). These research patterns point to an answer to Hollnagel s test for multi-layered networked systems: such systems need the ability to navigate and coordinate interdependencies across roles, activities, and levels. This aspect of control is exemplified by Sundström and Hollnagel (2010) at the scale of financial institutions and regulatory bodies, though they analyze cases where interdependencies were not handled successfully (attenuation of control). Similarly, Smith et al. (2007) discuss how changes in the US air traffic management system required tools to navigate and coordinate interdependencies across roles, activities, and levels when disrupting events occurred (capacity constrictions). In this case, new support systems were developed that have proved successful for managing interdependencies across multiple groups with different expertise, perspective, and primary goals (amplification of control). Methods to trace and reveal interdependencies to new events or prospective changes become basic tools for designing for control of multi-layered networked systems. For example, Hollnagel (2004) introduced the functional analysis and resonance method (FRAM) to reveal interdependencies for proactive safety management Sundstrom and Hollnagel (2010) use the FRAM method to analyze the financial system s inability to navigate interdependencies. Work on resilience and adaptive systems has revealed another aspect of control in the context multi-layered networked systems: such systems are able to recognize when to shift priorities across goal trade-offs. Studies of complex adaptive systems have revealed the existence of fundamental and inescapable trade-offs: optimalitybrittleness, efficiency-thoroughness, acute-chronic (Doyle 2000; Brown 2005; Woods and Wreathall 2008; Hollnagel 2009; Alderson and Doyle in press). Hollnagel (2009) poses the trade-off in terms of how a system knows when to sacrifice efficiency for thoroughness. For example, margins of maneuver and reserves can be seen as inefficiencies to be eliminated or as key sources of resilience (Woods 2006). How does an organization know when to develop or sustain the reserves and when to emphasize efficiency? How does an organization know what cues provide early indicators of the need to shift how they have been making the trade-off e.g., when uncertainties are increasing, is increasing margins for maneuver a prudent investment? Thus, multi-layered networked systems exist in the space defined by these fundamental trade-offs: does a system know where it is positioned in the trade-off space, can the system assess whether this position is appropriate for the context, and can the system shift its position in the trade-off space to move to better region? When one is part of a system in a changing, interconnected, interdependent environment, it is difficult to step back and reflect on how that system works, to identify the weaknesses, and to begin to develop new ways to work. Yet human adaptive systems learn to modulate their adaptive capacities to continuously update their fitness relative to an environment of changing pressures and opportunities. Attempts to define control architectures for this process are underway based on fundamental findings about complex adaptive systems (Alderson and Doyle in press). For multi-layered networked systems, being in control refers to navigating interdependencies to adjust behavior in one role (its knowledge, field of view and range of adaptive behavior) to take into account interdependencies with other roles, activities, and events (e.g., Kulathumani et al. 2008). Each role in the system adjusts what they are doing to fit what others are doing relative to the changing situation and the changing priority over goals (trade-off space). Coordinating interdependencies allows the network to achieve a diverse interconnected set of goals and manage the inevitable goal trade-offs. Amplifying control consists of developing tools that help reveal/track relevant interdependencies and help anticipate how projected actions will propagate (resonate) across interdependencies relative to goals. 4 The quest for polycentric control architectures Work has begun to develop new polycentric control architectures that dynamically manage and adapt the relationships across diverse but interdependent roles, organizations, processes, and activities (Ostrom 1999; Andersson and Ostrom 2008; Hollnagel et al. 2006, 2010). The concept of polycentric control architectures was developed by Ostrom (1990, 1999) through studies of how complex systems avoid the tragedy of the commons. The tragedy of the commons concerns shared physical resources (among the most studied examples of common pools are fisheries management and water resources for irrigation). It describes a baseline adaptive dynamic whereby the actors, by acting rationally in the short term to generate a return in a competitive environment, deplete or destroy the common resource on which they depend in the long run. In the usual description of the dynamic, participants are trapped in an adaptive cycle that inexorably overuses the common resource; thus, from a larger systems view the

5 local actions of groups are counterproductive and lead them to destroy their livelihood or way of life in the long run. Note that this breakdown is one subpattern of one of the basic forms of maladaptive behavior in highly interdependent, multi-layered systems: working at cross-purposes where behavior is locally adaptive, but globally maladaptive (Woods and Branlat 2010). Managing physical common pool resources is a specific example of a general type of goal conflict where different groups are differentially responsible and affected by different subgoals, even though there is one or only a couple of commonly held over-arching goals (Woods et al. 2010). The standard view of how to manage common pool resources is to create a higher level of organization responsible for the resource over its entire range and over longer periods of time. This organization then needs authority to compel individuals or local groups to modify their behavior, i.e., sacrificing short-term return and autonomy in order for the higher-level organization to analyze and plan behaviors that sustain or grow the resource over the long term a command organization. Ostrom (1999; see also Andersson and Ostrom 2008) reviews the empirical results on how people actually manage common pool resources and finds the standard view unsupported by the evidence. Basically, she found that overuse by local actors is not inevitable and command style relationships across levels of organizations do not work well (similarly, complete decentralization also does not work well). Instead, she finds from research on co-adaptive systems that common pool resources can be effectively managed through polycentric governance systems. Polycentric systems provide for multiple levels of governance with overlapping authority in a dynamic balance but where there is no single governance center that directs or commands unilaterally. Her synthesis of research identifies a variety of conditions and properties for polycentric control architectures (such as cross-communication, shared norms, and reciprocity). In polycentric control, there are multiple centers that are interdependent, but each possesses partial authority and autonomy. All of the centers are responsible for subgoals of the total system, but they have differential responsibility for some goals relative to others. Polycentric control seeks to sustain a dynamic balance across these multiple centers of governance some closer to the basic processes under control, but with narrower field of view and scope of action, and others farther removed but with larger fields of view and scopes of action. The basic control scheme is as follows: Empower decentralized initiative (at more local layers) Coordinate over emerging trends to meet changing priorities (by more distant, regional supervisory roles) These two layers are in constant interplay as situations evolve in themselves and as a result of activities and progress at each center. This kind of control architecture is evident particularly in effective disaster response and in military operations (Woods and Shattuck 2000) because of the need for adaptation to local conditions, the high uncertainty and potential for surprise, and the need to coordinate local units over wider scales and multiple echelons (classically these issues have been referred to as commander s intent, more recent treatments use the label netcentric operations). Note the shift in authority structures that accompanies the shift from supervisory to polycentric control architectures. Instead of a single center with final authority and responsibility, there are multiple centers with partial autonomy, partial authority, and responsibility for a subset of goals relative to others. 5 Results from Hollnagel s test Each technology shift manual to automated control to multi-layered networks extends the range of potential control, and in doing so, the joint cognitive system that performs work in context changes as well. For the new joint cognitive system, one then asks the questions of Hollnagel s test: What does it now mean to be in control? How to amplify control within the new range of possibilities? The answers define the critical requirements for control architectures for that technology shift. As technology becomes more powerful, note the answers to Hollnagel s test become more abstract. For example, being in control always requires the ability to anticipate; but the form of anticipation changes dramatically in the shift from direct to supervisory to polycentric control anticipate a trend in a parameter or upcoming disturbances on that parameter; anticipate when a local controller will be unable to keep up with cascading demands and shift tactics before a critical bottleneck; anticipate when the system is exhausting its adaptive capacity or when centers are working at crosspurposes before system performance collapses in a failure (Woods 2010). For multi-layered networked systems, the answers to Hollnagel s test include to be in control any role or center needs to be able to navigate and coordinate over interdependencies; to amplify control then consists of tools that help reveal/track relevant interdependencies and help anticipate how projected actions will propagate (resonate) across interdependencies relative to goals. To accommodate

6 the demands of the scale shift represented by the growth of multi-layered networked systems, a shift is underway from supervisory control for managing automated resources, to polycentric control architectures that balance multiple interdependent centers working across multiple echelons and over wide spatial, temporal, and functional scales. References Alderson DL, Doyle JC (in press) Contrasting views of complexity and their implications for network-centric infrastructures. IEEE Syst Man Cybern Part A Andersson KP, Ostrom E (2008) Analyzing decentralized resource regimes from a polycentric perspective. Policy Sci 41:71 93 Bergström J, Dahlström N, Dekker SWA, Petersen K (2010) Training organisational resilience in escalating situations. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot, UK (in press) Brown JP (2005) Key themes in healthcare safety dilemmas. In: Patankar MS, Brown JP, Treadwell MD (eds) Safety ethics: cases from aviation, healthcare, and occupational and environmental health. Ashgate, Adelshot, pp Chapman R, Smith PJ, Billings CE, McCoy CE, Heintz Obradovich J (2001) Collaborative constraint propagation as a planning strategy in the national airspace system. Paper presented at the 2001 annual meeting of the IEEE society on systems, man and cybernetics, Tucson, AZ Cook RI (2006) Being bumpable: consequences of resource saturation and near-saturation for cognitive demands on ICU practitioners. In: Woods DD, Hollnagel E (eds) Joint cognitive systems: patterns in cognitive systems engineering. Taylor & Francis/ CRC Press, Boca Raton, pp Cook R, Rasmussen J (2005) Going Solid : a model of system dynamics and consequences for patient safety. Qual Saf Health Care 14: Cuvelier L, Falzon P (2010) Coping with uncertainty: resilient decisions in anaesthesia. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot (in press) Dekker SWA, Woods DD (1999) To intervene or not to intervene: the dilemma of management by exception. Cogn Technol Work 1(2):86 96 Doyle JC (2000) Multiscale networking, robustness, and rigor. In: Samad T, Weyrauch J (eds) Automation, control, and complexity: an integrated approach. Wiley, New York, pp Edwards E, Lees FP (eds) (1974) The human operator in process control. Taylor & Francis Ltd, London Flach JM, Smith MRH, Stanard T, Dittman SM (2003) Collisions: getting them under control. In: Hecht H, Savelsbergh GJP (eds) Theories of time to contact. Elsevier, North-Holland, pp Grote G (2004) Uncertainty management at core of the system design. Annu Rev Control 28(2): Hollnagel E (1992) Coping, coupling and control: the modelling of muddling through. In: Booth PA, Sasse A (eds) Mental models and everyday activities. Proceedings of second interdisciplinary workshop on mental models, Cambridge, UK, March Hollnagel E (1993) Human reliability analysis: context and control. Academic Press, London Hollnagel E (1998) Context, cognition, and control. In: Waern Y (ed) Co-operation in process management cognition and information technology. Taylor & Francis, London Hollnagel E (1999) From function allocation to function congruence. In: Dekker S, Hollnagel E (eds) Coping with computers in the cockpit. Ashgate, Aldershot, pp Hollnagel E (2001) Extended cognition and the future of ergonomics. Theor Issues Ergon Sci 2(3): Hollnagel E (2004) Barriers and accident prevention. Ashgate, Aldershot Hollnagel E (2009) The ETTO principle: efficiency-thoroughness trade-off, why things that go right sometimes go wrong. Ashgate, Farnham, Surrey Hollnagel E, Woods DD (1983) Cognitive systems engineering: new wine in new bottles. Int J Man Mach Stud 18: Hollnagel E, Woods DD (2005) Joint cognitive systems: foundations of cognitive systems engineering. CRC Press, Boca Raton Hollnagel E, Mancini G, Woods DD (eds) (1986) Intelligent decision support in process environments. Springer, New York Hollnagel E, Woods DD, Leveson N (eds) (2006) Resilience engineering: concepts and precepts. Ashgate, Aldershot Hollnagel E, Paries J, Woods DD, Wreathall J (eds) (2010) Resilience engineering in practice. Ashgate, Aldershot (in press) Hutchins E (1995) How a cockpit remembers its speeds. Cogn Sci 19: Jackson D (2009) A direct path to dependable software. Commun ACM 52:78 88 Klein G, Feltovich P, Bradshaw J, Woods DD (2005) Common ground and coordination in joint activity. In: Rouse W, Boff K (eds) Organizational simulation. Wiley, Chichester, pp Knecht WR, Smith K (2001) The manoeuvre space: a new aid to aircraft tactical separation. In: Harris D (ed) Engineering psychology and cognitive ergonomics, vol 5. Ashgate, Aldershot, pp Kulathumani V, Sridharan M, Ramnath R, Arora A (2008) Weave: an architecture for tailoring urban sensing applications across multiple sensor fabrics. Proceedings of the international workshop on mobile device and urban sensing (MODUS), April 2008 Miller C, Parasuraman R (2007) Designing for flexible interaction between humans and automation: delegation interfaces for supervisory control. Hum Factors 49:57 75 Miller A, Xiao Y (2007) Multi-level strategies to achieve resilience for an organisation operating at capacity: a case study at a trauma centre. Cogn Technol Work 9:51 66 Morel G, Amalberti R, Chauvin C (2008) Articulating the differences between safety and resilience: the decision-making process of professional sea-fishing skippers. Hum Factors J Hum Factors Ergon Soc 50(1):1 16 Nemeth CP, Nunnally M, O Connor M, Brandwijk M, Kowalsky J, Cook RI (2007) Regularly irregular: how groups reconcile crosscutting agendas and demand in healthcare. Cogn Technol Work 9: Norros L (2004) Acting under uncertainty. VTT Publications, Espoo Nyssen AS (2007) Coordination in hospitals: organized or emergent process? Cogn Technol Work 9: Nyssen AS (2010) From myopic coordination to resilience in sociotechnical systems: a case study in a hospital. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot (in press) Nyssen AS, Javaux D (1996) Analysis of synchronization constraints and associated errors in collective work environments. Ergonomics 39: Ostrom E (1990) Governing the commons: the evolution of institutions for collective action. Cambridge University Press, New York Ostrom E (1999) Coping with tragedies of the commons. Annu Rev Polit Sci 2: Patterson ES, Woods DD (2001) Shift changes, updates, and the oncall model in space shuttle mission control. Computer supported cooperative work. J Collab Comput 10(3 4):

7 Rae A, Jackson D, Ramanan P, Flanz J, Leyman D (2005) Critical feature analysis of a radiotherapy machine. Reliab Eng Syst Saf 89(1):48 56 Rasmussen J, Rouse W (1981) Human detection and diagnosis of system failures. North Holland, New York Smith PJ, Spencer A, Billings C (2007) Strategies for designing distributed systems: case studies in the design of an air traffic management system. Cogn Technol Work 9:39 49 Sundström GA, Hollnagel E (2010) The importance of functional interdependencies in financial services systems. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot (in press) Wears RL, Woods DD (2007) Always adapting. Ann Emerg Med 50(5): Woods DD (2006) Essential characteristics of resilience for organizations. In: Hollnagel E, Woods DD, Leveson N (eds) Resilience engineering: concepts and precepts. Ashgate, Aldershot, pp Woods DD (2010) Resilience and the ability to anticipate. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot (in press) Woods DD, Branlat M (2010) How adaptive systems fail. In: Hollnagel E, Paries J, Woods DD, Wreathall J (eds) Resilience engineering in practice. Ashgate, Aldershot (in press) Woods DD, Hollnagel E (2006) Joint cognitive systems: patterns in cognitive systems engineering. Taylor & Francis, Boca Raton Woods DD, Sarter N (2000) Learning from automation surprises and going sour accidents. In: Sarter N, Amalberti R (eds) Cognitive engineering in the aviation domain. Erlbaum, Hillsdale, pp Woods DD, Shattuck LG (2000) Distant supervision local action given the potential for surprise. Cogn Technol Work 2: Woods DD, Wreathall J (2008) Stress-strain plots as a basis for assessing system resilience. In: Hollnagel E, Nemeth C, Dekker S (eds) Remaining sensitive to the possibility of failure. Ashgate Publishing Company, Aldershot, pp Woods DD, Dekker SWA, Cook RI, Johannesen LL, Sarter NB (2010) Behind human error, 2nd edn. Ashgate, Aldershot (in press) Xiao Y, Kiesler S, Mackenzie CF, Kobayashi M, Plasters C, Seagull J, Fussell S (2007) Negotiation and conflict in large scale collaboration: a preliminary field study. Cogn Technol Work 9: