Planning in Air Traffic Control: Impact of Problem Type

THE INTERNATIONAL JOURNAL OF AVIATION PSYCHOLOGY, 15(3), 269 293 Copyright 2005, Lawrence Erlbaum Associates, Inc. Planning in Air Traffic Control: Impact of Problem Type Scott D. Gronlund Department of Psychology University of Oklahoma Michael R. P. Dougherty Department of Psychology University of Maryland Francis T. Durso Department of Psychology Texas Tech University John M. Canning White Oak Technologies Silver Springs, Maryland Scott H. Mills The MITRE Corporation McLean, Virginia To characterize the planning activities of en route air traffic controllers, 12 certified professional controllers (CPCs) were placed in the role of planners and verbalized a plan for controlling traffic to a confederate tactician. The tactician, another CPC, implemented the plan. Planning, which is typically tacit, was made explicit by distributing it across these 2 individuals. The sequencing problem, which required the sequencing of aircraft going to a common destination, had a distinct environmental analysis phase (bottom-up) followed by a distinct plan development (top-down) phase. In the crossing problem, which involved aircraft en route to many different Requests for reprints should be sent to Scott D. Gronlund, Department of Psychology, University of Oklahoma, Norman, OK 73019. Email: sgronlund@ou.edu

270 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS destinations, environmental analysis melded into a phase of balanced top-down guidance and bottom-up analysis. Implications for interface development are discussed. Planning is a complex and challenging activity that engages many and varied aspects of cognition (see Mumford, Shultz, & Van Doorn, 2001, for a review). It encompasses many different domains ranging from vacation planning (Stewart & Vogt, 1999) to the administration of anesthesia (Xiao, Milgram, & Doyle, 1997). Furthermore, the study of planning is nontrivial because it appears to be a cognitive task almost invisible to the outside observer (David, 1997, p. 13). One method frequently used to make planning visible is verbal protocol methodology. For example, Byrne (1977) had six experienced cooks plan a three-course meal. The cooks talked aloud while they planned, yielding a verbal protocol. Patterns in the protocols were classified to provide a description of the type of planning in which the cooks engaged. The planning was characterized as hierarchical: The cooks first set up a list of goals (e.g., main course, appetizer). Next, they subdivided the main goals into subgoals (main course should have a protein, a vegetable, etc.). However, not all planning proceeds in a top-down manner involving successive refinements at lower levels. Hayes-Roth and Hayes-Roth (1979) had participants plan a sequence of errands around town, producing verbal protocols while they planned. The planning in this situation was characterized as opportunistic. Detailed sequences of specific actions were planned in conjunction with high-level modifications to the plan. Hayes-Roth and Hayes-Roth argued that planning would not benefit from the discipline and structure imposed by a hierarchical structure if general solutions did not exist or if they were computationally intractable. Although they were speaking of computer models of planning, computational intractability also describes the dilemma humans face in complex dynamic domains. Planning is difficult in complex dynamic domains (see Bainbridge, 1997). Unavailable or ambiguous information regarding the state of some parts of the system routinely contributes to tractability concerns. Consequently, operators must actively search for information to keep ahead of the task. This conclusion was reinforced by Mumford et al. (2001), who reviewed several models of planning. The initial activity characteristic of these planning models was plan generation, and the first step in plan generation involved an analysis of the environment. However, the resulting initial plan only served as a prototype for a more detailed plan to follow. A more detailed plan could not be developed due to a second characteristic of complex dynamic domains difficulty anticipating situations that might arise (Bainbridge, 1997). Therefore, operators frequently must adapt their behavior to the changing situation. Environmental analysis and adapting to changing situations typifies planning in many complex dynamic domains (e.g., medicine: Kuipers, Moskowitz, & Kassirer, 1988; Xiao et al., 1997; military operations: Pew & Mavor, 1998; project management systems: Pietras & Coury, 1991).

PLANNING IN AIR TRAFFIC CONTROL 271 The present research sought to describe the degree of environmental analysis and adaptation that characterized en route air traffic control. En route controllers are responsible for the high-altitude, high-speed portion of a flight within a volume of airspace called a sector. The controllers job is to route aircraft safely and expeditiously through their sector to the next sector. The task of routing aircraft through a sector requires a variety of planning activities, such as lining up streams of aircraft that have common destinations, anticipating future aircraft, and solving traffic-level problems. An investigation of planning in en route air traffic control is timely given proposals that have called for the establishment of a more strategic controller position (e.g., sometimes called a multisector D-side controller; Celio, Bolczak, Newman, & Viets, 2004). Similarly, the NASA Ames Research Center has proposed creating an airspace coordinator position (Vivona, Ballin, Green, Bach, & McNally, 1996). These positions have one person responsible for a multiple-sector airspace, making planning decisions about traffic in these sectors, and delegating responsibility for time-critical decisions to sector-level controllers. Thus, more needs to be learned about the planning behavior of en route air traffic controllers so that recommendations can be made regarding how best to implement more strategic air traffic control and what interface tools can best support that planning (see Canning, Johansson, Gronlund, Dougherty, & Mills, 1999). Of particular interest in this study was how environmental analysis and adaptation responded to two types of air traffic control problems: crossing and sequencing problems. Both are typical of the problems faced by en route controllers, yet they differ in important respects. The crossing problem included aircraft en route to many different destination airports, with most aircraft crossing one another s route of flight. The sequencing problem included aircraft coming from many different locations but needing to be sequenced to land at the same destination airport. To enhance the difference between the two types of problems, we varied the routing of the aircraft. Currently, most aircraft fly along routes (so-called highways in the sky). This was the case in the sequencing problem. As a result of flying along established routes, there are particular points in a controller s sector where routes intersect and merging aircraft will conflict (like intersections at street corners). This was not the case in the crossing problem. Most aircraft in the crossing problem flew straight-line or direct routes through the sector. This meant that aircraft could intersect at any point in the sector. This simulated aspects of free flight 1 1 Free flight has other implications for the air traffic controller, including decentralized decision making and changing the controller s role to be more of a monitor, that were not examined in this experiment. The advantage of free flight, from the perspective of the airlines and their passengers, is shorter, more direct routes between departure and destination airports. Although we believe that it was the crossing traffic more than the use of direct routing that made the crossing scenario less predictable, additional research should be conducted to determine the contribution of each factor.

272 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS (Federal Aviation Administration [FAA], 1995; Radio Technical Commission for Aeronautics, 1995). Carlson, Rhodes, and Cullen (1996) argued that direct routing was likely to result in a significant increase in the amount of time-critical separation actions and a corresponding decrease in longer term (or strategic) separation actions. Problem states were easier to anticipate in the sequencing problem because the conflict points, the intersections, were known in advance. In contrast, the variety (and unfamiliarity) of intersection points in the crossing problem made it difficult for the controllers to anticipate problem states. Consequently, a crossing problem resulted in more information being unavailable or ambiguous. Another way to conceptualize the differences involved the number of plans required to resolve the situation. In the crossing problem, a few aircraft were headed to each of many different destinations; controllers needed many different plans to resolve these situations. On the other hand, in the sequencing problem, the primary goal was to sequence a large number of aircraft to a common destination airport. Although controllers differed in how this was accomplished, a single plan could be constructed to handle most of the aircraft in the sector. As a result of these differences between sequencing and crossing problems, environmental analysis was expected to play a different role. For the sequencing problem, once initial environmental analysis was complete, controllers would have a good understanding of the situation and could make a plan with little additional environmental analysis. In other words, planning would be guided from the top down when the problem was predictable. If an emergency arose such that the original plan could no longer be followed, environmental analysis (bottom-up processes) would again become necessary. For the crossing problem, on the other hand, the less predictable situation together with the conflicts that would arise among the multiple plans would result in a greater need to adapt to the situation. As a result, the coordination of planning would shift between top down, when a miniplan was being put in place, and bottom up, when environmental analysis was required prior to developing the next miniplan. METHOD Despite the widespread belief that operators in complex dynamic domains engage in planning, research with air traffic controllers often characterized their activities as largely tactical in nature (Durso & Gronlund, 1999; Hutton, Olszewski, Thordsen, & Kaempf, 1997). A tactical decision is required for the resolution of time-critical conflicts. It occurs in the current moment and typically involves the separation among a small number of aircraft. Planning occurs further in advance than tactics and involves a larger number of aircraft. To ensure that the controllers in this study had the opportunity to engage in a suffi-

PLANNING IN AIR TRAFFIC CONTROL 273 cient amount of planning, several design decisions were made. These decisions are detailed later. Participants Twelve en route air traffic controllers participated in this research study as planners. All the planners were instructors at the FAA Academy and were familiar with the fictional AeroCenter airspace used in the experiment. All were certified professional controllers (CPCs), which meant that they were certified to work a sector independently, in contrast to a trainee, who must work with a CPC. They had been CPCs for an average of 6.3 years. They last worked in the field an average of 2.5 years ago, with a range of 0.5 to 6.2 years. The subject matter expert, another CPC, served as the tactician for all 12 planners. Although the tactician functioned as a confederate, he worked as hard as he could to implement a planner s plan. Because we were interested in the planner s behavior, not in the ability of the tactician to fix faulty plans, the tactician did not inform the planner of any shortcoming of a plan until it became a tactical concern. In other words, the tactician s job was to implement the plan as best he could, until such time that the aircraft involved were in danger of losing separation, at which time a tactical action was taken to keep them separated. The planner was in charge, and the tactician implemented the plan as proficiently as possible. Materials The experiment was conducted at the Radar Training Facility at the Mike Monroney Aeronautical Center in Oklahoma City. The Radar Training Facility has high-fidelity air traffic training simulators used to provide radar training. Communications between the tactician and the aircraft took place in the same manner as in the field, although the aircraft were piloted by ghost pilots who controlled the simulated aircraft based on the tactician s instructions. The equipment consisted of the circular radar display (the plan view display), two bays of paper strips, the computer readout display, and a keyboard and trackball. The plan view display indicated the two-dimensional position of the aircraft with an attached data block containing information including the aircraft s call sign, altitude, and ground speed. One flight progress strip for each aircraft was stacked vertically in a strip bay adjacent to the plan view display. Flight strips are 20 cm 3cmrectangular pieces of paper that include a variety of information, including an aircraft s call sign, aircraft type, requested altitude, requested speed, route of flight, and so on. The tactician sat in front of the plan view display. The planner usually sat in front of the strip bay, al-

274 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS though he or she could move around behind the tactician if desired. The planner was given full access to all the functionality of the computer readout display and plan view display. In particular, this included calling up an aircraft s destination or flight plan using the computer readout display, and displaying an aircraft s route of flight and projecting a straight-line vector for all aircraft 1 to 8 min into the future on the plan view display. Three air traffic scenarios were developed with the help of the subject matter expert/tactician. The subject matter expert judged the scenarios to exceed the workload level typically experienced in the field by a team of two controllers. Two of the scenarios were designed to be similar. In both, the primary problem to be solved involved sequencing aircraft for landing at Dallas/Fort Worth airport. The aircraft flew on preestablished routes, which meant that the aircraft intersected at limited and predictable locations in the sector. As was standard procedure, the strip bay was organized with a column of Dallas/Fort Worth arrivals and a column of other aircraft. Within the two columns, the strips were organized by increasing time relative to Tulsa (i.e., all strips included information on the expected time that they would reach Tulsa). In the third scenario, aircraft were crossing traffic for one another (e.g., an aircraft heading north and an aircraft heading west might cross over Tulsa). Aircraft flew direct routes, which meant that they could intersect at any point in the sector. There were four orders of scenarios, which resulted from randomly selecting one of the two sequencing scenarios to be third and then randomly assigning the other sequencing scenario to be either first or second, with the crossing scenario filling the remaining position. The 12 participants allowed us to rotate through the four counterbalanced orders three times. Procedure Tactics and planning are normally confounded because both types of decision making typically lie within the same head, even when a team has responsibility for a sector. This natural confounding led to the development of the distributed verbal protocol method. The distributed verbal protocol method made explicit that which was tacit by distributing the cognition across individuals. In other words, the role of the planner and the tactician (the implementer of the plan) were put into different heads, and planning was assessed by examining the verbalizations made by the planner to the tactician. The planner s responsibility was to make the tactician s job as easy as possible by giving the tactician a plan for managing the flow of traffic in the sector. The planner s efforts were to be directed at solving problems that might arise in the relatively distant future and that had a relatively long-term impact on the traffic flow. In contrast, the tactician s actions were directed at solving problems that arose in the near future and that had little long-term impact. The tactician maintained separation between

PLANNING IN AIR TRAFFIC CONTROL 275 aircraft and made whatever altitude, speed, and heading changes were necessary to implement the plan. During pilot testing of the experiment, it was observed that the planner immediately fell into a tactical mode when both sat down for the first time at an active traffic situation. This was consistent with the findings of Liberman and Trope (1998) and Penningroth and Rosenberg (1995), who found that time pressure and heavy workload limited the amount of planning. Because an examination of planning was critical to our goal of developing an interface for a strategic controller position, the experiment began with the scenario paused. This allowed us to observe what the planner did to analyze the environment. In many respects, what our planners did was no different from what would be accomplished in the field by a position-relief briefing delivered to the relieving controller by the controller in charge. The experiment began with a practice session that served to illustrate, in a concrete way, the roles of the planner and the tactician. The practice scenario had several key events embedded in it. First, the planner was shown a simple, isolated, two-aircraft conflict. This was proffered as an example of a tactical conflict; no matter what solution to the conflict was implemented by the tactician, these aircraft would not affect the overall plan. Therefore, the planner was told that this was the kind of conflict that could be left to the tactician. A second example involved a two-aircraft conflict that required a particular solution (i.e., some tactical solutions to the problems affected the plan). A third example involved another two-aircraft conflict, but some tactical solutions created a conflict with a third aircraft. In this situation, the planner was told to look for a single solution that would solve both problems simultaneously. These types of conflicts were common to both types of scenarios. The planner was given a sheet with the altimeter-indicated altitude, the traffic flow (either the Dallas rush was coming in the sequencing problems or the aircraft were on direct routings in the crossing problem), and flow restrictions (10 miles between aircraft heading to Dallas/Fort Worth; 5 miles was normal). The planner then described what he or she saw and verbalized a plan. The tactician took notes on the plan because he would have to implement it. The scenario began once the planner was finished. Thereafter, the planner was instructed to modify the plan as necessary. At the 10-min mark the scenario was paused, and the planner was asked to verbalize the plan as it now stood. The scenario then ran for an additional 10 min. This was the procedure for the first two scenarios, with a 15-min break following each. The third scenario, which was always a sequencing scenario, began like the others, with the following change at the 10-min mark. After pausing the scenario, we handed the planner a note that stated that an accident had occurred at Dallas/Fort Worth and one half of the Dallas/Fort Worth arrivals had to be rerouted to Oklahoma City (via a certain airway, at or above a certain altitude, with 10 miles separation between aircraft), and one half had to be rerouted to Houston (with similar restrictions). The planner was asked to verbalize a plan for dealing with this

276 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS event. No prior warning was given about closing the airport. After completion of the third scenario, the planners were debriefed and released. The entire experiment took approximately 2.75 hr per planner. RESULTS AND DISCUSSION The primary data collected were the verbalizations from the planner to the tactician. 2 The first step in analyzing these data involved subdividing the verbalizations into idea units. This was done by one of the experimenters (MD) who was familiar with air traffic control jargon. It was completed prior to the development of the coding scheme. The process was straightforward, and few disagreements were uncovered during the subsequent coding. Any disagreements were discussed and settled during coding. Coding Scheme Table 1 defines the major categories and the subcategories of the coding scheme and gives an example for each. Collect data was assigned whenever the planner identified a piece of information without performing an action on it. The two subcategories were data (DAT; information came directly from a source) and inference (INF; information inferred from the problem or from domain-specific knowledge). The collect data verbalizations signaled environmental analysis. Monitor (M) was assigned whenever the planner rechecked what he or she had done and acknowledged that specific aspects of the situation were not a problem. The four subcategories were: environment (ENV) the planner checked jetways, airports, or altitudes; plan (PL) the status of the plan was checked; tactics (TAC) the planner commented on the control actions of the tactician; and aircraft (AC) the planner checked on the status of an aircraft. Problem (P) was assigned whenever a planner identified a problem. It had two subcategories: The planner could identify (ID) a specific problem or detect (DET) a problem without identifying its exact nature. Monitor and problem verbalizations gauged adaptability. For example, a plan that could be put in place with minimal interruptions due to double-checking (monitor) or correcting misjudgments (problems) signaled little need to adapt the plan to the changing situation. Develop plan (DP) was assigned whenever the planner was constructing plans for actions 2 All the planner tactician teams safely and expeditiously moved the traffic through the sector with no separation violations. Dougherty, Gronlund, Durso, Canning, and Mills (1999) summarized these aspects of the data.

PLANNING IN AIR TRAFFIC CONTROL 277 TABLE 1 Definitions and Examples of Verbalization Categories Major Category Subcategory Example Collect data (CD) Planner identified piece of information without performing an action on it. Monitor (M) Planner rechecked what had been done and acknowledged that specific aspects of the situation were not a problem. Problem (P) Planner checked what had been done and acknowledged that specific aspects of the situation were a problem. Develop plan (DP) Planner articulated action to be taken in the future. Data (DAT) Information read from a source (e.g., strips or plan view display). Inference (INF) Information inferred from the scenario or from domain-specific knowledge. Environment (ENV) Aspect of the environment checked. Plan (PL) Plan checked. Tactics (TAC) Tactics checked. Aircraft (AC) Aircraft status checked. Identify (ID) Problem identified. Detect (DET) Problem detected but not identified. Unconditional action (UA) Future action issued, no constraints. Conditional action (CA) Future action issued, with constraints. AAL123 goes to Dallas/Fort Worth. DAL123 is an overflight. a Flight level 330 (33,000 ft) is available. Looks like DAL123 and AAL123 have sufficient spacing. AAL123 will need more of a vector (to maintain separation). DAL123 is looking good. AAL123 is overtaking DAL123. DAL123 and AAL123 are not going to work Take DAL123 direct Amarillo. After DAL123 passes Tulsa, take him direct Amarillo. a An overflight was deduced by determining that an aircraft neither departed from nor landed in the sector. to take in the future. DP had two subcategories: A future action could be either an unconditional action (UA) or a conditional action (CA). The former should be carried out regardless; the latter had some temporal, spatial, or logical constraint placed on it. The first two authors independently coded each idea unit extracted from the planner verbalizations. To check on the level of agreement, we compiled the data from three randomly chosen planners for each of the three scenarios. Agreement was high, with no notable differences as a function of type of scenario, phase of the experiment, or category being coded (overall agreement = 81%, κ =.75, z = 14.15, p <.001, signaling that the level of agreement was significantly greater than expected by chance). Disagreements were resolved in the following manner: If one

278 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS of the raters failed to code an idea unit and the other assigned it a code, the assigned code was used. In cases where the two raters assigned different codes to an entry, the item was discussed and the difference resolved. The verbalization data were studied at three levels of analysis. The first focused on the four major categories and their proportion of occurrence. The next used the Pathfinder algorithm (Schvaneveldt, 1990; Schvaneveldt, Durso, & Dearholt, 1989) to examine the latent structure underlying the verbalization transitions (for all 10 subcategories). The final level of analysis examined summary measures extracted from the Pathfinder graphs, which showed the degree to which planning was coordinated from the top down or bottom up. Proportion of Occurrence The first compilation of the data examined rolling blocks of 10 verbalizations for the four major categories. The data were from the initial planning period, before the scenario became active. In Figure 1, the first point on the x axis shows how the proportion of verbalizations in the four major categories were distributed among the first 10 verbalizations; the second point on the x axis gives the same for Verbalizations 2 through 11, and so on. This method is a generalization of data smoothing (see Velleman & Hoaglin, 1981), although a span of 10 verbalizations is larger than what is typically used because averages of four variables were computed simultaneously. In both scenarios, the proportion of CD was initially greater than the proportion of DP, although at some point this relation reversed. Because this reversal occurred at a different point for each planner, the average shown in Figure 1 was created by matching each individual s graph according to the point that CD first exceeded DP, and then tallying backward and forward from this point as far as the data allowed (i.e., until planners ceased contributing data). The top panel in Figure 1 gives the data for the crossing scenario and the bottom panel for the two sequencing scenarios (averaged together). In the left portion of the both graphs, which will be referred to as the picture-building phase, planners began by collecting data about the situation (controllers refer to getting the picture with regard to their situation awareness; Durso & Gronlund, 1999). This was expected given prior research noting the importance of bottom-up environmental analysis as a precursor to planning. However, despite the necessity of this initial environmental analysis, there were differences between the two types of scenarios. The sequencing scenario began with a high constant level of CD with rare occurrences of DP, M, and P verbalizations. This was followed by a rapid decrease in CD, which traded off with a rapid increase in DP, signaling the end of the picture-building phase. The situation was different in the crossing scenario. The initial data collection phase was less distinct, and CD began decreasing immediately whereas DP, M, and P verbalizations all increased.

PLANNING IN AIR TRAFFIC CONTROL 279 FIGURE 1 Proportion of verbalizations for the four major categories as a function of rolling blocks of 10 verbalizations. Crossing scenario on the top and sequencing scenario on the bottom. The gap in the middle of the graphs shows the point where DP > CD. Error bars show 1 SEM. The scenario was paused during this phase. There also was a marked difference between the sequencing and crossing scenarios in the right portion of these two graphs. A distinct plan development phase was evident in the sequencing scenario: As CD rapidly decreased, DP increased just as rapidly. M and P verbalizations continued to be rare. In other words, the more predictable sequencing problem was coordinated from the top down by a plan once the initial environmental analysis was complete. However, in the crossing scenario, DP was no more likely than CD during this phase. Rather, the coordination of planning shifted between top down, when a miniplan was being put in place, and bottom up, when environmental analysis was required to develop the next miniplan. Dividing the data into discrete intervals rather than rolling averages changed none of these conclusions, as the reader can judge by looking at intervals 0, 11, and 22, or 1, 12, and 23, which have no overlapping data. These conclusions were supported by a 2 2 4repeated-measures analysis of variance (ANOVA) conducted on the proportion of verbalizations collapsed within a phase (crossing or sequencing scenarios picture-building or plan development phases 4 major verbalization categories). Of primary importance was the three-way interaction, F(3,33) = 9.27, p <.001, MS =.007 (means presented in Table 2). There was a greater proportion of CD in the picture-building phase of the sequencing scenario than in the crossing scenario. In addition, CD in the picture-building phase exceeded CD in the plan development phase in the sequencing scenario, but not in the crossing scenario. The proportion of DP verbalizations was greater in the plan development phase than in the picture-building phase in the se-

280 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS TABLE 2 Mean Proportion of Verbalizations Collapsed Within a Phase Crossing Sequencing Picture Building Plan Development Picture Building Plan Development Collect data (CD) 0.52 0.41 0.72 0.41 Monitor (M) 0.17 0.21 0.05 0.10 Problem (P) 0.16 0.16 0.03 0.04 Develop plan (DP) 0.15 0.23 0.21 0.44 quencing scenario, but there was no difference in the crossing scenario. Finally, the proportions of M and P verbalizations were always greater in the crossing scenario for both phases; the difference was significant for the P verbalizations in the picture-building phase, although not for the remaining three comparisons (ps <.06,.08, and.13). The data showed evidence of initial bottom-up environmental planning in both scenarios although the picture-building phase was more pronounced for the sequencing scenario. Thereafter, the sequencing problem exhibited a top-down plan development phase as distinct as its initial picture-building phase. However, there was no distinct plan development phase for the crossing scenario; instead, plan development vied with data collection. Was this the result of random alternation between these two verbalizations, consistent with a reactive stance, or did it reflect a more principled alternation from one miniplan to another? A detailed look at the latent structure underlying the verbalizations can tell us which. Pathfinder Graphs Strings of verbalizations were summarized as transition matrices from verbalization n to verbalization n + 1. For example, if a planner uttered three DP verbalizations in succession, the transition matrix would have a count of 2 corresponding to the number of times a DP verbalization was followed by another DP verbalization. 3 The subcategory level was considered, which resulted in a 10 10 matrix of transitions. The transition matrices were normalized by the total number of verbalizations uttered by a participant before averaging across partici- 3 Vortac, Edwards, and Manning (1994) suggested weighting the transition between successive behaviors as an exponential function of the temporal interval between the behaviors. That would mean that behaviors that occurred in close temporal proximity would count more than would two behaviors that did not occur in close temporal proximity. We did not exponentially weight the transitions because the planners talked almost continuously, at least when the scenario was paused, and the majority of the data came from compressed time spans.

PLANNING IN AIR TRAFFIC CONTROL 281 pants. As a result, transition matrices consisted of proportions, rather than raw frequencies, which ensured that each participant made the same contribution to the average irrespective of how verbose they were. Averaging across participants was preferred because it eliminated idiosyncratic patterns of verbalizations in favor of those produced by a preponderance of the participants. Many procedures have been developed to reveal the latent structure underlying a set of data like these transition matrices (e.g., multidimensional scaling, Shepard, 1962; clustering, Johnson, 1967). These procedures share the assumption that the observed data reflect latent ( true ) structure plus statistical noise, and that the two can be separated by mathematical means. We chose Pathfinder rather than multidimensional scaling because Pathfinder can represent asymmetric transitions that were likely in this experiment. Pathfinder reduces a transition matrix to a graph by eliminating those transitions between verbalization categories that do not satisfy metric properties (e.g., the triangle inequality). The k transitions chosen represent the shortest distance between all verbalizations i and j, given k transitions, which means that every link in the resulting graph is a link on some minimal path between two nodes. A family of graphs can be created, depending on the metric used to compute path distance. We chose the parameter values that created the sparsest Pathfinder graphs (i.e., the minimum number of links): q was set to 9 (10 verbalization categories 1), and r (the value of the Minkowski distance metric) was set to (reflecting only ordinal assumptions about the data). The two sequencing scenarios (averaged together) were across the top of Figure 2; the crossing scenario was across the bottom. The picture-building phase corresponding to when CD was greater than or equal to DP was on the left. The plan development phase when DP exceeded CD was on the right. Each node represented a type of verbalization. The size of a node signaled the proportion of that verbalization that occurred during that phase. For example, 72% of all verbalizations in the sequencing scenario in the picture-building phase were CD DAT; therefore, the CD DAT node represents 72% of the total area of all the nodes depicted in the top left graph of Figure 2. Links connect verbalizations that tended to co-occur. The more frequent the transition between two verbalizations, the thicker the link joining them. Some of the links were loops a verbalization of a particular type followed by another of the same type. The resulting Pathfinder graphs function as a planning grammar because they can be used to generate the sequence of verbalizations that typically occurred. For example, a planner in the picture-building phase of the sequencing scenarios usually began with a verbalization involving data collection (CD DAT). This typically was followed by several more CD DATs (the loop from CD DAT to CD DAT): The planner first had to establish where the aircraft were going to determine which aircraft needed to be sequenced to Dallas/Fort Worth and which did not. At some point, a piece of the plan would be constructed (DP UA), after which the planner returned to more data collection (CD DAT). A DP UA verbalization

282 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS FIGURE 2 Pathfinder graphs for q = n 1 and r =. The sequencing scenarios (averaged together) are across the top, the crossing scenario is across the bottom. The picture development phase is on the left and the plan development phase is on the right. Nodes were placed in the same location across graphs to facilitate comparisons (M on the left side, DP near the top center, P near the bottom, and CD near the center). When the position of a node interfered with a link, these links were drawn with dashed lines. The distances between nodes in the depiction are not meaningful. was seldom followed by anything besides another CD DAT (hence the link back to CD DAT and no links to anything else). M or P verbalizations were rare, triggered by a preceding CD DAT verbalization and, typically, followed by another CD DAT verbalization. Pathfinder graphs were used to determine if there was structure or random alternation among verbalization categories. The degree of structure was assessed by the number of links in the graphs. Random alternation among verbalizations would result in a higher number of links. For example, M TAC might be followed by CD DAT on one occasion, M AC on the next, and so on, resulting in links from M TAC to many different nodes. However, that was not the case. Although 100 links were possible, 20 to 24 were sufficient to represent the verbalizations. Even in the relatively unpredictable crossing scenario, plan development and data collection alternated in a principled way. The graphs highlight many similarities during the picture-building phase for the sequencing and crossing problems. Most verbalizations were triggered and followed by data collection. Also, there was a prominent loop on CD DAT and a strong link between CD DAT and DP UA. These links summarized much of what happened during this phase. These graphs also highlight differences between the two problems. One difference in the picture-building phase involved monitoring. The identification of a

PLANNING IN AIR TRAFFIC CONTROL 283 problem (P ID) preceded monitoring the plan (M PL) in the crossing scenario, but M PL was triggered by data collection (CD DAT) in the sequencing scenario. This signaled that problems in the crossing scenario involved the plan, which was consistent with the creation of several miniplans that conflict with one another. There were three noteworthy differences between the graphs in the plan development phase; all were indicative of a planner being able to rely on the internal environment (mental simulation of the plan) in the sequencing scenario, whereas the planner in the crossing scenario needed to refer to the external environment for verification regarding the implications of the plan. For the sequencing scenario, monitoring the plan (M PL) was part of the plan development process because it was preceded and followed by DP UA. However, in the crossing scenario, data collection (CD DAT) triggered M PL. Similarly, problem verbalizations were related to plan development (DP UA) in the sequencing scenarios but to data collection (CD DAT) in the crossing scenario. Finally, there was a prominent loop on DP UA in the sequencing scenario, whereas in the crossing scenario the strong loop was on CD DAT. Summary Measures Various summary measures can be derived from the Pathfinder graphs. These can be used to assess the extent to which planning was coordinated from the top down or balanced between top-down and bottom-up influences, and the degree of adaptation in the sequencing and crossing scenarios. The focal node or center of a graph denoted the activity that coordinated planning behavior. The center of each Pathfinder graph was determined by computing its median (see Durso, Rea, & Dayton, 1994). The median of a graph was the node that had the smallest average distance to all other nodes (medians are based on links, not on the size of the nodes). Medians were computed for the most influential and the most prestigious node. An influence median was the node from which every other node could be reached in the minimal number of transitions; a prestige median was the node that every other node could reach in the minimal number of transitions. Table 3 gives both medians for the two types of scenario crossed with the different phases of planning. The median values (the number of transitions) are given in parentheses. For example, an influence median value of 1.1 indicated that 1.1 transitions were needed to get from the median node to any other node. In other words, any verbalization followed the median verbalization immediately (in essentially one step). The first four rows of Table 3 give the medians for picture building and plan development for both scenarios. For picture building, CD DAT coordinated behavior in both scenarios, consistent with the idea that bottom-up environmental analysis preceded plan development. For plan development, DP UA usurped the

284 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS TABLE 3 Center of Graph or Median Influence Median Value Prestige Median Value Sequencing, picture building CD DAT 1.1 CD DAT 1.0 Crossing, picture building CD DAT 1.3 CD DAT 1.1 Sequencing, plan development DP UA 1.3 DP UA 1.2 Crossing, plan development CD DAT 1.3 DP UA 1.5 DP UA 1.5 DP UA 1.5 Sequencing, implementation DP UA 1.1 DP UA 1.1 Crossing, implementation CD DAT 1.4 CD DAT 1.4 DP UA 1.6 DP UA 1.4 Note. CD = collect data; DAT = data; DP = develop plan; UA = unconditional action. role of CD DAT in the sequencing scenario; planning was coordinated from the top down. However, in the crossing scenario, CD DAT and DP UA shared responsibility for coordinating planning. This would be expected if several miniplans had to be put into place, each requiring data collection before the next miniplan could be developed. Another summary measure extracted from the Pathfinder graphs was the number of cycles, which are enumerated in Table 4. A cycle was a sequence of three or more verbalizations (excluding loops) that did not repeat any verbalizations. For example, DP UA to M AC to DP CA was a cycle; DP UA to M AC to DP UA was not. The number of cycles was construed as a measure of the amount of adaptation required by a planner. At one extreme, a large number of cycles would result if planners were randomly spouting verbalizations as they reacted to an unpredictable situation. It might also result if planning was coordinated by a set of miniplans as opposed to one larger plan. At the other extreme, few cycles would signal that a particular verbalization orchestrated the flow of activity, and disruptions to that flow involved only a single-step departure from that primary activity. This would be the case if a single plan was sufficient to solve a problem. It might also be the case if departures from the norm could be anticipated and integrated into the flow of activities. Examination of Table 4 showed that planners adapted more in the crossing scenario, especially during the picture-building phase. Summary. The type of problem affected the air traffic planning process in several ways. In the crossing scenario, proportionally fewer of the verbalizations involved data collection. In addition, there was more need to adapt behavior to the situation. As a result, plan development shifted between top-down and bottom-up control. In the sequencing problem, planning began with an environmental analysis that resulted in such a thorough understanding of the situation that the planner coor-

PLANNING IN AIR TRAFFIC CONTROL 285 dinated the next phase of planning with little additional environmental analysis and little need to adapt. Once the plan was developed and conveyed to the tactician, the scenario ran for 10 min. During this time, the planners had to maintain the plan they had created, modifying or reworking it as new aircraft entered the sector. Mumford et al. (2001) called this the implementation phase because the plan was in place by this point. According to their analysis, implementation consisted of monitoring of various sorts, problem identification, and generating backup plans. Once again, the crossing scenario would be expected to result in more M and P verbalizations, and TABLE 4 Cycles, Excluding Loops Cycles Sequencing, picture building 1. CD DAT CD INF P ID CD DAT 2. CD DAT DP CA M AC CD DAT Crossing, picture building 1. CD DAT M ENV M TAC CD DAT 2. CD DAT CD INF M TAC CD DAT 3. M ENV M TAC CD INF CD DAT M ENV 4. CD DAT P ID M PL CD DAT 5. CD DAT M AC DP UA DP CA CD DAT 6. CD DAT M AC DP UA CD DAT 7. CD DAT DP UA DP CA CD DAT Sequencing, plan development 1. DP UA M AC CD DAT DP UA 2. DP UA M ENV CD DAT DP UA 3. DP UA DP CA CD INF DP UA 4. DP UA DP CA CD INF M TAC DP UA Crossing, plan development 1. CD DAT M PL DP UA CD DAT 2. CD DAT CD INF DP UA CD DAT 3. CD DAT M TAC DP CA CD DAT 4. CD DAT DP UA DP CA CD DAT 5. CD DAT M PL DP UA DP CA CD DAT 6. CD DAT CD INF DP UA DP CA CD DAT Sequencing, implementation None Crossing, implementation 1. CD DAT M PL DP UA CD DAT 2. CD DAT P ID DP UA CD DAT 3. CD DAT CD INF P DET DP UA CD DAT 4. CD DAT M TAC M AC DP CA DP UA CD DAT 5. CD DAT M TAC M AC CD DAT 6. DP UA M TAC M AC CD DAT DP UA 7. DP UA M TAC DP CA DP UA 8. DP UA M TAC CD DAT DP UA 9. DP UA M TAC M AC DP CA DP UA Note. CD = collect data; DAT = data; INF = inference; P = problem ; ID = identify; DP = develop plan; CA = conditional action; M = monitor; AC = aircraft; ENV = environment; TAC = tactics; PL = plan; UA = unconditional action; DET = detect.

286 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS would require a balance between top-down guidance and bottom-up analysis. On the other hand, the sequencing problem should continue to be guided from the top down by the original plan with little need to adapt. Implementation Phase A 2 (sequencing vs. crossing) 4 (type of verbalization) repeated-measures ANOVA was conducted on the proportions of the different verbalizations. There was a significant main effect of type of verbalization, F(3, 33) = 37.7, p <.05, MS =.021. According to a Tukey post hoc test, verbalizations about DP were as frequent as those indicating CD, which occurred more often than M verbalizations, which occurred more often than P verbalizations (DP =.44, CD =.36, M =.17, P =.04). This main effect was moderated by a marginally significant interaction, F(3, 33) = 2.86, p =.052, MS =.001. A follow-up Tukey post hoc test showed that within the sequencing scenario, utterances about DP exceeded those signaling CD; DP did not differ from CD in the crossing scenario. Figure 3 shows the Pathfinder graphs for the implementation phase for the sequencing and crossing scenarios. For the sequencing scenario, every verbalization except P DET was connected by a bidirectional link to DP UA. This was similar to the sequencing plan development graph (see Figure 2; 14 links in common). In the sequencing scenario, the structure of verbalizations was basically unchanged by starting the scenario because the planner did much the same thing while the plan was being developed as when the plan was being implemented. The median for the sequencinggraphwasdp UA(seeTable3).Aplanwasinplaceandchangestothesituation were incorporated into the normal flow of behavior without prolonged interruption to that flow. As shown in Table 4, there were no cycles in this graph. The flow of activity was coordinated from the top down by DP UA and was not interrupted by prolonged sequences of other verbalizations(little need for adaptation). The crossing implementation graph, like the crossing plan development graph in Figure 2, was dominated by the DP UA to CD DAT sequence, with loops on both these nodes. These two nodes also were the medians of the implementation phase (see Table 3). Starting the scenario resulted in more changes to the structure of verbalizations in the crossing scenario than in the sequencing scenario (only 10 links were shared with the plan development graph). Consistent with the idea that planners had difficulty anticipating what was going to happen, nine cycles occurred during the implementation phase (see Table 4). Replan Phase The final phase of the experiment was called the replan phase. After 10 min of running the final sequencing scenario, we forced planners to develop a new plan. The scenario was paused and the planner was informed that one half of the Dal-

PLANNING IN AIR TRAFFIC CONTROL 287 FIGURE 3 Pathfinder graphs for the implementation phase, q = n 1 and r =. The sequencing scenarios (averaged together) are on top, and the crossing scenario is on the bottom. The scenario was active during this phase. las/fort Worth arrivals had to be rerouted to Oklahoma City and one half had to be rerouted to Houston. Figure 4 gives the rolling blocks of 10 verbalizations for the replan phase. The pattern was similar to the sequencing scenario in Figure 1. An initial period of data collection, although much briefer than before, followed by a period of plan development. During the data collection portion of the replan phase, the controller typically checked an aircraft s current location to see if it was easier to take it to Oklahoma City or Houston. During the implementation phase, the more predictable sequencing problem continued to be guided from the top down by the original plan until the planner was forced to abandon that plan and make a new one. The recurrence of a distinct data collection period marked the creation of a new plan (although not as extensive because the planner already knew a lot about these aircraft). This again was followed by a sharp transition to a plan development phase. CONCLUSIONS The type of problem affected the en route air traffic planning process in several ways. The crossing problem necessitated increased adaptation to the situation, which was apparent even before the scenario started. The crossing problem also

288 GRONLUND, DOUGHTERTY, DURSO, CANNING, MILLS FIGURE 4 Proportion of verbalizations for the four major categories as a function of rolling blocks of 10 verbalizations (error bars show 1 SEM). Data from the replan phase, which immediately followed informing the planner about the closing of Dallas/Fort Worth airport. The scenario was paused during this phase. required the creation of many miniplans, which resulted in plan development and implementation phases that were balanced between bottom-up analysis while a miniplan was created and top-down guidance once a plan was put in place. According to a case-based approach (e.g., Hammond, 1990), the coordination of planning from both directions might signal that no single prior case was sufficient to solve the problem; rather, several prior cases needed to be retrieved in succession. In the sequencing problem, planning began with a thorough environmental analysis that resulted in a detailed understanding of the situation. This meant that a single plan could govern performance with little additional environmental analysis and little need to adapt because problem states could be anticipated. Drastic means (closing the primary destination airport) forced planners to create a new plan in the sequencing scenario. How is it possible that a single plan was sufficient to handle a complex dynamic situation like air traffic control? Single Plan In the sequencing scenario, planners appeared to maintain a single plan by fine-tuning it and finalizing prior indeterminate decisions rather than jettisoning an existing plan as the situation changed. The planners made definitive decisions about some aircraft (deciding which aircraft would be first, second, and third in line to Dallas/Fort Worth), but indeterminate decisions about other aircraft. For example, they would report the group of three aircraft that would be next, but would not yet commit to which would be fourth, fifth, or sixth. Presumably, there was a more optimal time to do that once uncertainty in the situation was reduced.