Problem and Design Spaces during Object-Oriented Design: An Exploratory Study

Problem and Design Spaces during Object-Oriented Design: An Exploratory Study Sandeep Purao 1,2 Ashley Bush 2 Matti Rossi 3 1: Institutt for Informasjonvitenskap, Agder University College, Kristiansand, Norway 2: Department of CIS, College of Business, Georgia State University, Atlanta, GA 30302 3: Helsinki School of Economics, P.O. Box 1210, FIN-00101 Helsinki, Finland Email: spurao@gsu.edu, aabush@gsu.edu, mrossi@hkkk.fi Abstract The information system design process is not well understood in spite of a rich literature stream dealing with the general notion of design. The few studies of IS designer behaviors that are reported in IS have focused on small tasks in single sessions, and have ignored the impact of modeling techniques on the design process. In this paper, we report preliminary results based on a study of two designers engaged in a multi-session design task using the object-oriented design modeling techniques. We use an extended version of the model proposed by Adelson and Soloway to analyze the verbal protocols. Our observations and interpretations focus on the existence of problem and design spaces and designers use of these during design. Specifically, we discuss the distinction between problem and design spaces during object-oriented design, the shifts in emphasis between problem and design spaces, and changes in designer behaviors between the two spaces. 1. Introduction The information system design process and the nature of expertise in IS design is not well understood in spite of a rich literature stream dealing with the general notion of design spanning almost a century [14, 24]. A number of prescriptive methodologies [3, 4] are available that represent simplified accounts of the intricate and esoteric act of designing. Over the years, a number of studies have also been carried out to better understand the design process (e.g. [16, 17, 27]). The principal aim of these studies has been to provide an account of the underlying structures associated with the rather private actions of seeking out system design solutions on the part of the designer. To facilitate analysis, researchers [1, 17, 26] have studied individual designers working on simple tasks, requiring short, single sessions 1. The important task of investigating the individual as s/he tackles a non-trivial design task through multiple sessions has not received adequate attention. Investigating design behaviors using categories provided by earlier researchers has not been attempted, particularly in studies of IS design processes. A third underlying theme in this research stream is the surprising non-consideration of representation scheme or modeling technique for the analysis of design behaviors. The notation or representation is probably more important in IS design than other design professions (such as architecture or engineering), because the artifact being created by IS designers has no direct correspondence to a physical realization. The topic of interest in this paper is, thus, narrowly defined. We are concerned with understanding and interpreting the interior situational logic, decisionmaking and related activities of individual IS designers tackling a non-trivial object-oriented design task over multiple sessions. Accordingly, the specific objective of this paper is to investigate the behaviors of IS designers for complex tasks requiring multiple sessions, explicitly considering the impact of the representation technique and modeling techniques used. We investigate behaviors of two IS designers, each engaged in three design sessions, lasting up to 90 minutes each. The non-trivial design task they tackle represents a larger, extended version of the design task used by other researchers [17]. The representation 1 The locus of investigation has shifted to teams or organizations [10, 29] for investigation of larger, more complex tasks. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 1

scheme and modeling technique the designers use during these sessions is the de facto standard, the Unified Modeling Language [4] for object-oriented design. The model of design behaviors suggested by Adelson and Soloway [1] provides the starting point for out analysis. Following [1, 14, 17, 24, 26, 30] and Ericsson and Simon [13], we use verbal protocols as the data collection mode and coded the protocols for analysis [20]. The features uncovered through data analysis are interpreted, using results from prior design studies. In this paper, we have elected to focus on the distinction between problem and design spaces and the designers activities and movements in these spaces. The paper is organized as follows. In section 2, we discuss prior studies of individual designers. The section also highlights essential elements of object-oriented design. In section 3, we describe the methodology employed for the multi-session design study, the data collection approach and specific decisions taken during data analysis. Section 4 presents our preliminary findings behaviors in problem and design spaces. In section 5, we discuss implications of our findings for practice and education and briefly discuss further research. 2. Prior Research 2.1. Individual Design Processes Design is a process in which the designer progresses from a description of requirements to a model of an IS artifact (e.g. data flow diagrams, entityrelationship diagrams or object-oriented diagrams [19] that will satisfy these requirements. It begins with a problem where the actions necessary to obtain a solution are not obvious. The actions that follow may, in turn, focus on solving this problem as well as additional problems that are brought up through the definition and redefinition of the problem [24 p.39]. As a preliminary, design can, thus, be defined as an iterative process of solving wicked problems [5] 2. One major obstacle to studying and understanding the design process is the messiness of the design task. For studies of architectural design, two recent papers represent the nature of research and interpretations obtained. First, Gero et al. [14] develop a multi- 2 A wicked problem [5] (a) defies the possibility of a definitive formulation, (b) exhibits no stopping rules for the design activity, (c) allows differing formulations to imply different solutions, and (d) allows alternative solutions that have only degrees of acceptance, unlike ill-defined problems that eventually lend themselves to definition of criteria for judging the solutions. dimensional model to describe design activities. These include different levels of design foci such as behavior or structure and different micro and macro design strategies. In another study, Goldschmidt [15] uses the reflective practice framing provided by Schön [25] to interpret different design tasks. Few studies of IS design have focused on the individual designer. An early study by Guindon et al. [17] focuses on breakdowns observed during early design activities. Based on a study of three designers engaged in creating a solution for an N-lift elevator problem, they identify a number of breakdowns in two broad categories knowledge-related and those due to cognitive limitations. Another study by Adelson and Soloway [1, 2] provides a model of design behaviors of both expert and novice designers in domains in which they had varying levels of familiarity. They identify five major categories of activities such as model formation, expansion and simulation, and planning activities such as note making and label retrieval. In another study, Guindon [16] investigates behaviors of three software designers for a similar N-lift elevator problem with a view to identifying opportunism during the design process. She concludes in favor of opportunistic instead of top-down decomposition during the early stages of design. Sen [26] confirms the role of opportunism in the software design process by proposing a cognitive model for examining demand-side reuse tasks during the design process. It is clear that use of think aloud protocols continues to be the dominant mode of data collection with a few studies in architecture beginning to use results from prior studies for data analysis (for example, [15]). The dominant type of task investigated, however, remains a simple one, requiring a single design session, particularly in the IS design studies (for example, [17, 26]). 2.2. Designing Object-Oriented Systems The object-oriented design paradigm is based on a message-object view of IS, instead of the traditional operator-operand model [8]. Excellent descriptions of important concepts in object-oriented technology are available in [3, 4, 18]. Our discussion here is focused on key aspects of object-oriented paradigm that can impact the design process. The first important consequence of using the O-O technique is that it requires that elements in the information system be defined as objects that can interact with each other by sending messages [32]. This view is claimed to be closer to the way human beings perceive and interact with real-world entities. Unlike the traditional, operator-operand view, it does not require 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 2

that we translate our understanding of a real-world phenomenon into an algorithmic view. As a result, designers can continue to use the users vocabulary during the design process 3, using similar constructs to describe concepts in the users domain as well as those in the software domain. The second key feature is the multiple-models that the O-O paradigm lends itself to. Like architectural drawings, blueprints and cross-sections, the objectoriented representation techniques can produce multiple views such as use cases and class diagrams. For example Unified Modeling Language suggests at least three complementary perspectives for specifying the information systems, structural, functional and dynamic [4, 12 p. 45]. Object-oriented development methods such as UML [4] call these (a) class diagrams, (b) use cases and (c) interaction diagrams respectively (and suggest others to supplement this set). Using multiple techniques allows checking for internal consistency of the specification and designing of a more complete specification of the solution. The representational modes for object-oriented design do not require either a top-down or a bottom-up approach to problem-space planning and design procedures [24 pp. 67-71]. Instead, an iterativeincremental approach [3] is considered more appropriate, which suggests growing and refining the design by taking into account additional scenarios, and/or considering progressively detailed accounts of given scenarios. It identifies a micro cycle that individual designers can iterate to perform the design task and a macro cycle that can be used to manage the overall design process [3]. The picture that emerges from the above demonstrates three important characteristics germane to the study of design behaviors. These include: (a) the close mirroring of real-world concepts in the design space, (b) the multiplicity of representational modes, and (c) the iterative-incremental approach to design that requires revisiting the problem with increasing depth. 2.3. Problem and Design Spaces The idea of problem and design spaces is important in artifact design processes. The problem space can be described as the metaphoric space that contains mental representations of the designer s interpretation of the user requirements. The design space, on the other hand, can be described as the metaphoric space that contains mental representations of the designer s specific solutions, based on which s/he 3 Though this mirroring is not exact, due to constraints such as platforms, performance and generalization, the benefits of the OO paradigm are not lost if these differences are documented [12 p. 3]. creates models and specifications for building the IS artifact. Newell and Simon [21] first used the term problem space, defining it as the metaphoric space wherein the problem solver moves from an initial state (problem) through intermediate states to a goal (solution) state by applying appropriate operators [15]. Their view, however, places the problem and the solution on the same plane. This is contrary to what we may intuit for IS design processes, where the development of a solution i.e. a design, proceeds through proposals, expansions and challenges. These activities clearly represent moves within the design space and continue after an initial goal state is reached suggesting the existence of a separate design space. Though such a separate design space has rarely been advocated in prior research, evidence that something like this exists abounds. For example, Rowe [24 pp.51-52] describes generative processes, or operations, that allow the designer to take knowledge states as input, or as starting positions, and produce new knowledge states as outputs. The representation of new, but incomplete states may resemble partial design solutions suggesting the existence of a separate design space. Gero and McNeil [14 p.23] observe designer navigation between different levels of problem abstractions and as different design strategies used by the designer. Oxman [22 p. 336], while interpreting behaviors of designers who are given a partial solution and asked to adjust it, comments that they could easily explore different solution spaces... [17 p. 457]. Guindon [16 pp. 285-86] clearly identifies the problem domain as a subset of the real world with which a computer system is concerned but not the design solution describing the computer system itself. The two spaces, thus, clearly represent a dimension orthogonal to the different activities that a designer may undertake. Previous research has been interested in problem formulation and design as somewhat separate activities, but they have not investigated into the separation itself. 3. A Multi-Session Design Study A multi-session design study was conducted to investigate the nature of the design activity in relatively complex tasks, spread over multiple sessions in which verbal protocol data was collected from two novice designers. 3.1. Research Setting and Data Collection The research setting engaged the subjects in multiple sessions performing a single design task. The 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 3

Designer Behaviors Model Formation Model Expansion Model Simulation Constraint Representation Note Making Table 2. Model of Design Behaviors [1] Explanation New ideas are identified and mental models are formed. The mental models are expanded to an acceptable level of detail. The mental models are simulated in a dynamic manner. Implicit constraints about boundary are made explicit. A note is made of issues that need to be addressed later. design task was to create object-oriented models for an Elevator Control System. The description of requirements [33] covered different aspects of an elevator system but was necessarily incomplete. Compared to the N-lift problem statement [16, 17], our problem statement was much larger, less specified and allowed many different interpretations. It was aided by a short list of important scenarios resembling names of use cases [4] such as: an elevator is called from a floor, expected to provide the designer a number of starting points. The deliverables required included a statement of scope, a structural (class) diagram, interaction diagrams, and decisions about the general software architecture. The subjects were completing a graduate level course in object-oriented system design as part of a masters degree program in Information Systems. The use of students as subjects is comparable to other recent design studies such as Sen [26] and Adelson and Soloway [2]. One issue in protocol studies is generalizability of the results when the number of participants is very small. There is simply no reliable way, given the current maturity of the field, to decide how representative our subjects are of the larger population of designers [16]. The subjects we selected had some prior experience in the IS industry and were obtaining a specialized master s degree in information systems while continuing work at IS professions. These designers were, thus, representative of at least some segment of the population of actual designers. During the design process, the subjects were given the problem statement, a reference book [3], a copy of the Unified Modeling Language Notation Guide Version 1.1 [4] and analysis patterns [7]. Each designer was allowed to work on the design problem over three sessions, each up to 90 minutes in duration. The sessions for each designer were spread over about a week, scheduled at the designer s convenience. The sessions were conducted in a room, occupied solely by the designer, who was asked to think aloud [13] as s/he proceeded through the design task. A tape recorder captured the verbal protocols. Of the three designers who participated in the study, two (identities concealed) provided complete verbal protocols. 3.2. Coding Procedures The transcribed verbal protocols were coded using the model of design behaviors suggested by Adelson and Soloway [1]. Their model consists of six categories: formation, expansion, simulation, constraint representation, label retrieval and note making. The choice of the model was based on a number of factors, both theoretically rigorous and pragmatic. First, this model provided a well-accepted and often-cited model of design behaviors that has been shown to be useful for describing both novice as well as expert designer behaviors. This ensured possible comparisons with other studies and a feasible path to facilitate reproducibility on subsequent occasions. Second, their model represents one of the few models specifically aimed at IS design and prima facie, covers the entire design process including problem definition (constraint representation), solution generation (model formation, expansion and simulation) and strategies for planning the design process (note making and label retrieval) [16, 17, 26]. Table 2 shows the categories suggested by Adelson and Soloway [1] along with a brief explanation of the activities. As a first pass, separate and independent coding was attempted by two of the co-authors on two different sessions, followed by an assessment of agreements and disagreements among the coders. Any disagreements on the granularity of units identified as well as their classification following the categories (Table 3) were resolved by a process of negotiation and logical argumentation. As a shared frame of reference was established for application of the model, the coders restarted and conducted the coding process for all the protocols jointly. This allowed immediate discussion of discrepancies and resolution of disagreements. During the coding process, the coders established the rules of coding, which included (a) identification of verbal protocols as a phrase or a sentence indicating a reasonably complete unit, and (b) classification of each unit into one or more categories. As the analysis progressed, the categories (table 2) were refined. This resulted in the extended model underlying our coding (see Table 3). 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 4

Table 3. Extended Model of Design Behaviors [6] Designer Behaviors Explanation Concept Formation New ideas are identified and mental models are formed. Concept Expansion The mental models are expanded to an acceptable level of detail. Concept Simulation The mental models are simulated in a dynamic manner. Representing Constraints Implicit constraints about boundary are made explicit. Retrieving Domain Knowledge Prior knowledge about problem domain is retrieved. Retrieving Technique Knowledge Prior knowledge about modeling technique/method is retrieved. Making Meta-Level Notes Making notes about managing the process or other meta-aspects. Retrieving from Experience Base Drawing on experience the designer is building during this design. Refocusing Refocusing on different parts of the current design. Validating Validation to check that the needs are met and solution is consistent. Note Making A note is made of issues that need to be addressed later. The context, indicated by the phrases preceding and following the phrase in question, often dictated the classification. In case of difficulties in coding, the coders appealed to first principles, that is, reverted to the source model and/or attempted to find analogies to the phrase in question in the phrases suggested by the authors [1]. Following the coding, the analysis proceeded as a hermeneutic process, marked by two broad stages. First, a number of descriptive analyses were performed. This stage yielded descriptions of surface features of the design process [6]. The second stage of the analysis involved use of different theoretical bases, published accounts of the design behaviors and literature on O-O design to arrive at interesting interpretations of designer behaviors. The next sections discuss these findings. 4. Behavior Patterns in Problem and Design Spaces Our observations suggest that IS designers move within and between a problem space and a design space. They interpret, elaborate upon and attempt to understand the requirements; and devise, expand upon and continually adjust a design. As an example, consider the two protocol fragments from our study. Andrea: The sensors know when to stop the elevator when it is approaching the floors. It has to illuminate when people know which floor they're going to. Stanley: it does that process weight limit and it has been exceeded it does not allow the doors, it does not allow another floor or pending floor pickup to come about until that weight limit has been reduced. Andrea demonstrates activities within the problem space such as exploring different concepts, considering action sequences, and visualizing a mental representation of concepts. On the other hand, Stanley demonstrates activities within the design space, such as considering action sequences among concepts that he has formulated as part of a design, and visualizing how the artifact created based on his design may behave. Activities in the problem space clarify the environment in which the artifact to be designed must operate and crystallizes concepts that must be represented as a part of the design. Activities in the design space, on the other hand, reflect design behaviors such as proposing new concepts, amplifying or adjusting these concepts and refining the design. 4.1. Distinguishing between Problem and Design Space during Object-Oriented Design The distinction between problem versus design spaces was, in fact, somewhat difficult as we interpreted and coded the behavior protocols. To a large degree, we found that this was due to the fact that under the objectoriented paradigm, distinction between concepts in the real-world (problem space) and those in the artifact being designed (design space) is somewhat blurred. Since the object-oriented design principles suggest use of real-world concepts for identifying classes, such blurring is understandable and may be expected. Object-oriented design can include several steps some of which may be in the problem space and others in the design space. While specification of use cases in this manner is considered by many (e.g. [18]) to be the strength of object-oriented design, it is starting to be questioned by others. For example, Skagstein [28] argues for an initial step during the design process to separate objects and their alter-ego s. Following his argument, an object or class identified in the models does not represent a real-world object, but rather, designates an augmentation of the corresponding realworld object. The significant overlap between the realworld objects and the objects-modeled also lead to blurring of boundaries between the problem and the design space. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 5

100 100 Andrea Percentage of Activities 80 60 40 20 Stanley Percentage of Activities 80 60 40 20 0 Session 1 2 3 Solution Space 73,12 91,82 97,89 Problem Space 26,88 8,18 2,11 0 Session 1 2 3 Solution Space 67,06 88,56 98,74 Problem Space 32,94 11,44 1,26 Figure 1. Designer Focus in Problem and Design Spaces A related concern was the difficulty this blurring appeared to pose for the designers in clearly identifying and deciding the scope of (placing a boundary around) the system. For example, the following statements from Andrea suggest that the concept of Floor Switch being considered here straddles the boundary between the problem space and the design space. Andrea, during session 3 (protocol fragments 515-518) states, And the floor switch has a stop attribute and its boolean... The floor switch records stop and records floor and it records elevator number. The floor switch is used by the elevator and there is one elevator, again there is one to many floor switches for one elevator. Here, she is verbalizing her decisions about the concept of Floor Switch in the class models. What she is describing, however, are behaviors and properties she would like to see added to the real-world concept of Floor Switch, that is, augmenting the realworld Floor Switch. Based on the decisions about these properties/behaviors, Andrea would decide where the boundary will lie between the artifact she is modeling (design space) and the real-world concepts represented in it (problem space). Deciding on the appropriate boundary will, then, lead to the appropriate sharing of responsibilities between the real-world things and those modeled within the system [28]. The distinction between problem space and design space we noticed, therefore, goes to the core of much system development efforts, particularly using objectoriented techniques. It seems to be true that it is easy for the developers to move between spaces, but especially for less experienced developers this has also dangers, such as easy sliding from analysis of the problem into details of the design. 4.2. Shifts in Emphasis on Problem and Design Spaces In spite of the difficulties in deciding between problem and design space, the emphasis on the latter is evident. The separation between the two spaces is also borne out by our observations across multiple sessions. We observe a trend where the designer moves from the problem space to the design space progressively across the sessions. As the designing proceeds, the designers' emphasis quickly shifts from the former to the latter (see figure 1). Averaged across sessions, as well as for each session, both Stanley and Andrea show an emphasis the design space (85.61% and 89.63%). The largest percentage of problem space activities for both designers occurs in session 1 (26.88% and 32.94%). In the final session, both designers show an almost exclusive bias towards the design space (97.88% and 98.73%). The behavior confirms what other researchers have indicated and what our personal experiences as an IS designer suggest. Given a deadline, a design emerges, instead of a continuous search for the perfect design. It supports the common rule of thumb that 'analysis paralysis' is best avoided by defining clear timeline and deadlines for subtasks. A less flattering interpretation suggests that the designers, having selected a plan of attack for analysis and design, do not return to reexamine their assumptions and instead, focus their efforts on making the initial designs more complete. This behavior pattern, design fixation [23], can lead to inferior designs. The rather quick change in emphasis into design space observed between sessions 1 and 2 - for both designers - indicates that design fixation may, in fact, be occurring. To further understand whether the designers engaged in each space early or late during each session, we recorded the first and last engagement in each space. Figure 2 shows this analysis. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 6

Note: Numbers refer to protocol fragments. e.g. In session 1, Andrea engages in Problem space till the 13 th protocol fragment, in both from 13 th to 22 nd, and exclusively in Design space from 22 nd through 76 th. 1 13 22 76 79 88 373 423 428 492 560 624 Andrea 1 14 253 254 260 498 500 501 503 600 Stanley Session 1 Session 2 Session 3 Legend: Problem Space Design Space Both Spaces Figure 2. Engagement with Problem and Design Spaces The figure shows that Andrea starts sessions 1 and 2 in the problem space and ends with the design space. In the figure, the arrows show protocol-fragment numbers for the beginning and end of each session, and the first and last engagement in each space. We may, therefore, characterize Andrea s first two sessions as a start-off in the problem space (protocols 1-12 in session 1), engagement with the design space along with the problem space (protocols 13-21 in session 1), and a final move to the design space (protocols 22-76 in session 1). The difference between the first and the second session may be that Andrea moves a little faster into the design space during session 2 than she does in session 1. Unlike the first two sessions, she starts the third session in the design space and also ends with the design space. If we consider the first move into the design space, Andrea s moves into the design space, then, occur more and more quickly within each session (after 12 protocol fragments in session 1, after 9 in session 2, and immediately in session 3). Stanley shows a similar pattern. His moves into the design space also occur faster with each session (13, 6, and 2 fragments respectively). These moves lend further plausible evidence to the notion of design fixation. Several other interesting observations may be made from figure 2 above. Andrea s behavior, for example, is characterized by exclusive attention to the design space towards the end of each session. On the other hand, Stanley continues to engage with both problem and design space till the end of each session. Andrea, therefore, appears to follow a breadth-first strategy, covering as much of the problem space as she can, before moving on to the design space to deal with the details. Stanley, on the other hand, follows a depthfirst strategy, going back and forth between problem and design spaces for each issue that demands attention. During analysis of the use of modeling techniques, this difference in styles of the two designers was also apparent [6]. 4.3. Changes in Designer Behaviors in Problem and Design Spaces Following our extended model of design behaviors (see Table 4 earlier), activities such as concept formation, concept expansion and simulation may be observed both within the problem space as well as the design space. Figure 3 shows, on the horizontal axis, the different activities (table 4), and on the vertical axis, the frequency with which each activity was observed. It characterizes the activities in each space, in each session as a separate series of connected points. Proceeding from the series closest to the front to the one in the back, these series show sessions 1 to 3, alternating between problem and design spaces. Thus, the first series shows activities in the problem space during session 1, the second, activities in the design space during the same session, the third, activities in the problem space during session 2, and so on. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 7

Andrea Stanley Legend: 250 200 150 100 From front to back, sessions 1 to 3 Legend: White = Problem Space Black = Design Space 140 120 100 80 60 From front to back, sessions 1 to 3 Legend: White = Problem Space Black = Design Space 50 0 40 20 0 Horizontal Axis: Behavior Categories (see table 3) Figure 3. Changes in Design Behaviors From figure 3, a few spikes in activity are clearly evident. The most prominent is clearly the second activity (Model Expansion), particularly in the second session for both designers. This activity is complemented by a spike in the third activity (Model Simulation). Expansion and simulation, particularly in the design space, therefore, appears to go hand-in-hand. For Stanley, this pairing is also evident in session 1. In both cases, the primary focus appears to be on Expansion, with Simulation used as a vehicle for further expansion. Often, expansion appears to lead to simulation, which in turn, leads to further expansion. For example, consider the following protocol from Andrea during session 2. She verbalizes (protocol fragments 156-159), Got to have some kind of pending array of elevators, one to forty. And it will place in pending a request, and it will process the request so the elevator can send the correct elevator there. Elevator will have a direction and it is boolean. Here, Andrea begins expanding her model by asserting that an array of pending elevators may be necessary. From here, she moves to simulate this decision, indicating that it will need to communicate with the Elevator. She then realizes that Elevator should, therefore, have a direction and decides that it will be a boolean attribute. Clearly, this interplay between expansion and simulation is more relevant for the design space than for the problem space. Another use of simulation is also observed in the design space in the final session of each designer. In figure 3, the series in the far back (session 3, design space) shows the occurrence of simulation. Expansion does not, however, dominate Simulation as it does in the first two sections. In fact, for Stanley, Simulation dominates Expansion. Overall, the emphases on behaviors in the problem space are quite different from that in the design space. The most significant behaviors in the problem space occur in session 1 for both designers. Specifically, both designers engage in Simulation in an effort to better understand the domain. For example, Stanley verbalizes in session 1 (protocol fragments 9-12), First of all I must understand what this use case is, what are the characters involved, a summary of the use case. Elevator summoned a person on a particular floor is gonna to press the button. Such use of simulation of the problem space is often observed, and other researchers have suggested designing tools to support this activity [31]. 5. Discussion and Implications In this paper we investigated the behavior of novice IS developers over multiple sessions of system analysis and design. We analyzed verbal think-aloud protocols from the sessions and found some interesting phenomena that are backed by prior research. The most important finding reported in the paper is the importance of separation between problem and design space. It appears that there are sufficient differences between the two spaces in terms of designer behaviors, and how designers engage in these, to warrant a separate consideration of problem and design spaces. This may have some interesting implications for ISD methods, process models, tools and education practices. First, we notice the importance of system boundary, which may be affected by the two spaces. This may be especially relevant for embedded systems designed with the OO paradigm, where the objects of interest represented in the design and those in the problem space may have significant overlap. Drawing on what we observe for the difficulties in distinguishing the problem and design space for the designers, we suggest that development methods be enhanced to give more precise guidelines for using different models and techniques, especially during problem formulation and design. The current spate of tools does not explicitly support movement between problem and design spaces. While they do support capture of requirements, this 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 8

support does not include notions such as expansion or simulation of concepts in the problem space. Tools should support problem space explorations and incorporate it into the design task. It may also be possible to use techniques such as hypertext for crossreferencing and freely moving between spaces and models. The second important point for systems development practice is the early design fixation we observed. This has been known in project management literature and folklore for a long time, but not reflected in the micro-processes embedded in development models and tools. This may have implications for both practice and education. Other ongoing research [9] suggests that education of IS developers may be improved to ensure that novices are less susceptible to design fixation. Third, the paired use of expansion and simulation shows that simulation is an important technique that designers use to expand and develop the design. So, what are now mostly static tools for recording designs (CASE), should be improved to allow for rapid generation of alternatives and simulation especially during the early conceptual stage, instead of waiting for the detailed specifications to emerge. A related issue is support for the validation activity with simulation. One implication of this may be to improve the available tools to incorporate design rationale capture as an integral part of the design process or forcing validation as a separate activity during design. The tools may also be enhanced to support peer review or allow translation of the design into a textual description that can be validated by mapping it back to the requirements. For practice, we should put greater emphasis on peer reviews as a means of validation, better integrating these concerns into the education of would-be system developers [11]. The observations and implications we have presented in the paper are still evolving. The selection of novice IS developers as subjects remains a limitation of our findings. The credibility of our findings is, however, supported by two sources. First, use and extension [6] a behavior classification suggested earlier [2] has provided our analysis has a rigorous foundation. Second, the observations we have reported in the paper were corroborated by multiple analyses and supported by expected or prescriptive suggestions from other researchers. Our future research plans include further analyses of these results and comparison of our interpretations from these analyses against behavior patterns of seasoned software developers, which may aid in discovering ways to provide adequate support for different styles and experience levels. 6. References [1]B. Adelson and E. Soloway, "The Role of Domain Experience in Software Design," IEEE Transactions on Software Engineering, vol. SE-11, no. 11, 1985, pp. 1351-1360. [2]B. Adelson and E. Soloway, "A Model of Software Design," The Nature of Expertise, 1988, pp. 185-208. [3]G. Booch, Object-Oriented Analysis and Design with Applications, 2nd ed., Redwood City: The Benjamin/Cummings Publishing Company, Inc., 1994. [4]G. Booch, I. Jacobson, and J. Rumbaugh, "UML Resource Center," http://www.rational.com/, 1999. [5]R. Buchanan, "Wicked problems in design thinking," Design Issues, vol. 8, no. 2, 1992, pp. 5-22. [6]A. Bush and S. Purao, ed., Mapping UML Techniques to Design Activities, Hershey: Idea Group Publishing, 2000. [7]P. Coad, Object Models: Strategies, Patterns, and Applications, Englewood Cliffs: Yourdon Press, 1995. [8]B.J. Cox, "Message/Object Programming: An Evolutionary Change in Programming Technology," IEEE Software, vol. 1, no. 1, 1984, pp. 50-61. [9]N. Cross, "Evidence from Protocol and Other Formal Studies of Design Activity," in Proceedings of Knowing and Learning to Design, Atlanta, GA, April 27-29, 1999. [10]B. Curtis, H. Krasner, and N. Iscoe, "A Field Study of the Software Design Process for Large Systems," Communications of the ACM, vol. 31, no. 11, 1988, pp. 1268-1287. [11]B. de Vries, J. van Leeuwen, and H.H. Achten, "Design Studio of the Future," in Proceedings of Proceedings of the CIB W96 Conference, IT Support for Construction Process Re-Engineering, Cairns, Australia, 1997, pp. 139-146. [12]D.F. D'Souza and A.C. Wills, Objects, Components, and Frameworks with UML: The Catalysis Approach, Reading: Addison Wesley Longman Inc., 1999. [13]K.A. Ericsson and H.A. Simon, Protocol Analysis: Verbal Reports as Data (Revised Edition), Cambridge: The MIT Press, 1993. [14]J.S. Gero and T. Mc Neill, "An Approach to the Analysis of Design Protocols," Design Studies, vol. 19, no. 1, 1997, pp. 21-61. [15]G. Goldschmidt, "Capturing Indeterminism: Representation in the Design Problem Space," Design Studies, vol. 18, no. 4, 1997, pp. 441-445. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 9

[16]R. Guindon, "Designing the Design Process: Exploiting Opportunistic Thoughts," Human-Computer Interaction, vol. 5, no. 1990, 1990, pp. 305-344. [17]R. Guindon, H. Krasner, and B. Curtis, "Breakdowns and Processes During the Early Activities of Software Design by Professionals," 1986, pp. 454-475. [18]I. Jacobson, M. Christerson, P. Jonsson, and G. Overgaard, Object-Oriented Software Engineering: A Use Case Driven Approach, 4th edition ed., Addison-Wesley, 1992. [19]S. Kelly and M. Rossi, "Differences in Method Engineering Performance with Graphical and Matrix Tools: A Preliminary Empirical Study," in Proceedings of 2nd CAiSE/IFIP8.1 International Workshop on Evaluation of Modeling Methods in Systems Analysis and Design, EMMSAD'97, Barcelona, Spain, 1997. [31]R. van Zutphen and J.M.M. Mantelers, "Computational Design: Simulation in Virtual Environments," in Proceedings of Proceedings of the 3rd Design and Decision Support Systems in Architecture and Urban Planning Conference, Spa, Belgium, 1996. [32]P. Wegner, "Why Interaction Is More Powerful Than Algorithms," Communications of the ACM, vol. 40, no. 5, 1997, pp. 80-91. [33]E. Yourdon and C. Argila, Case Studies in Object Oriented Analysis and Design, Upper Saddle River: Yourdon Press Computing Series, 1996. [20]M.C. Lacity and M.A. Janson, "Understanding Qualitative Data: A Framework of Text Analysis," Journal of Management Information Systems, vol. 11, no. 2, 1994, pp. 137-156. [21]A. Newell and H. Simon, Human Problem Solving, Englewood Cliffs, NJ: Prentice Hall, 1972. [22]R. Oxman, "Design by re-representation: a model of visual reasoning in design," Design Studies, vol. 18, no. 4, 1997, pp. 329-347. [23]T. Purcell, J. Gero, H. Edwards, and E. Matka, "Design fixation and intelligent design aids," in J. S. Gero and F. Sudweeks, ed., Artificial Intelligence in Design '94, Kluwer: 1994, pp. 483-495. [24]P.G. Rowe, Design Thinking, Cambridge: The MIT Press, 1987. [25]D. Schön, The Reflective Practitioner, New York: Basic Books, Inc., 1983. [26]A. Sen, "The Role of Opportunism in the Software Design Reuse Process," IEEE Transactions on Software Engineering, vol. 23, no. 7, 1997, pp. 418-436. [27]H.A. Simon, The Sciences of the Artificial, 3rd ed., Cambridge: The MIT Press, 1996. [28]G. Skagstein, "Are Use Cases Necessarily the Best Start of an OO Development Process?," in Proceedings of Information Systems Development (ISD 2000) Conference, Kristiansand, Norway, 2000. [29]M. Tan, "Establishing Mutual Understanding in Systems Design: An Empirical Study," Journal of Management Information Systems, vol. 10, no. 4, 1994, pp. 159-182. [30]R. Valkenburg and K. Dorst, "The Reflective Practice of Design Teams," Design Studies, vol. 19, no. 3, 1998, pp. 249-271. 0-7695-0981-9/01 $10.00 (c) 2001 IEEE 10