Leopold Franzens University of Innsbruck Institute of Computer Science Research Group Quality Engineering Dealing with Uncertainty in Business Process Modeling and Execution: Agile vs. Plan-Driven Approach Master Thesis of Verena Messner Bakk. techn Supervisor: Assoc.-Prof. Dr. Barbara Weber July 9, 2014
Acknowledgement At this point, I would like to thank all those who made this thesis possible and supported me on my way towards its completion. First and foremost, I would like to express my deep gratitude to my supervisor Assoc.- Prof. Dr. Barbara Weber for her guidance and persistent help as well as for her understanding and patience during this time. Her inspirations and constructive feedback were essential for writing this thesis. Finally and most importantly, this master thesis would not have been materialized without the confidence and support of my family, boyfriend and friends. I am deeply grateful to my parents for their encouragement and love throughout my studies. I would also like to express my gratitude to my boyfriend Christoph for supporting me in every possible way. 2
Abstract Process variability and uncertainty are significant factors in business process management causing difficulties in prediction and decision making and forcing deviations from the plans. A shortcoming of traditional business process modeling is inflexibility for uncertainty. In order to create a competitive advantage and to mitigate dynamically changing business environments, companies are increasingly required to focus on maneuverability within their business processes. Agile approaches have been recognized as promising in this regard and are increasingly gaining in importance. However, empirical insights into the applicability of these approaches and their ability to efficiently deal with process uncertainties are still lacking. The aim of this master thesis is to provide, first of all, a comparison between the agile and the plan-driven planning approaches, particularly with respect to their impact on the business process outcome. Furthermore, it is investigated to what extent agile principles are adopted by inexperienced users and if a higher agile adoption implies enhancement of the process outcome. Another objective is to investigate how the occurrence of uncertain events is managed by the user during process execution. Based on the evaluation results obtained, it can be stated in summary that agile planning outperforms the plan-driven approach. Furthermore, statistical tests show that a higher agile adoption has a positive impact on the process outcome. Moreover, the evaluation of events provides insights into the handling of events by the user during process execution. In particular, a classification of events is presented describing the effects of events depending on their type. In addition, common practices and pitfalls in dealing with events are derived. 3
Contents 1. Introduction 6 1.1. Motivation and Problem Statement...................... 6 1.2. Related Work.................................. 7 1.3. Research Question............................... 9 1.4. Research Methods............................... 11 1.5. Structure of the Thesis............................. 12 2. Background 14 2.1. Alaska Simulator-Toolset........................... 14 2.1.1. Alaska Simulator Meta-Model..................... 14 2.1.2. Graphical User Interface........................ 18 2.1.3. The Journey Metaphor........................ 20 2.2. Declarative Approach............................. 21 2.3. Decision Deferral Patterns........................... 22 2.4. Planning Approaches.............................. 23 2.4.1. Plan-Driven Mode........................... 24 2.4.2. Agile Mode............................... 24 2.5. Agile Practices................................. 24 3. Comparison of Planning Approaches 27 3.1. Setup...................................... 27 3.1.1. Experimental Setting.......................... 28 3.1.2. Journey Configurations........................ 29 3.1.3. Response Variable........................... 31 3.1.4. Data Preparation............................ 31 3.2. Evaluation of Business Value......................... 32 3.2.1. Research Questions........................... 32 3.2.2. Proceeding............................... 33 3.2.3. Data Analysis.............................. 34 4
3.2.4. Discussion of the Results....................... 37 3.3. Evaluation of Practices............................. 38 3.3.1. Research Questions........................... 39 3.3.2. Proceeding............................... 39 3.3.3. Data Analysis.............................. 40 3.3.4. Discussion of the Results....................... 50 4. Dealing with Events 53 4.1. Setup...................................... 53 4.1.1. Experimental Setting.......................... 53 4.1.2. Characteristics of Journey Configurations.............. 56 4.2. Evaluation of Events.............................. 59 4.2.1. Research Question........................... 59 4.2.2. Proceeding............................... 59 4.2.3. Classification Criteria for Events................... 60 4.2.4. Data Analysis.............................. 65 4.2.5. Descriptive Evaluation of Events................... 72 4.2.6. Discussion of the Results....................... 78 5. Summary and Outlook 81 A. Appendix 83 A.1. Raw Data.................................... 83 A.2. Classification of Events............................. 86 List of Figures 92 List of Tables 94 Bibliography 98 5
1. Introduction 1.1. Motivation and Problem Statement A Process-Aware Information System (PAIS) is specified as a software system that organizes and executes operational processes involving people, applications, and/ or information sources on the basis of process models [15]. Traditionally, supported process models are instantiable for multiple times (repetitive) and every instance is managed in a predefined way (predictable) [3] [24]. Classical examples for business processes with these characteristics can be found, amongst others, in processes of banking and insurance companies (e.g., opening a new bank account or granting a loan) [17]. Because enterprise environments might change rapidly, companies are forced to adapt their business processes more frequently [9]. Changing business processes might become necessary, for example, in cases where new laws become effective, optimized or restructured business processes need to be implemented, exceptional situations occur, or adaptations because of changing markets are required. However, the ability to flexibly manage business process changes is a critical success factor and yields sustainable economic benefits for companies [24]. Therefore, there is an increasing interest on the part of companies to improve the quality and efficiency of processes. Further efforts are focused on the alignment of information systems in a process-centered way [11]. Today s PAISs provide support for more flexibility in business processes [11]. Underlying business processes are now increasingly less structured and are no longer characterized by either full predictability or repeatability. Consequently, these processes cannot be fully pre-specified [13]. In practice, most business processes exhibit a combination of predictable and unpredictable elements. Examples for that can be found in the field of healthcare [17] [11]. The process of handling medical orders and examinations can be specified as being predictable. In contrast, the individual patient treatment is quite unpredictable and can only be fully determined during the process execution. In particular, 6
these processes are classified as non-repeatable (i.e., two process instances are not identical) and unpredictable (i.e., the process is determined exactly during execution based on situation-specific parameters). In addition, they are described as knowledge-intensive, hence knowledge is required from parts of the process (i.e., process participants) in order to determine the further process course [11]. Processes with these characteristics cannot be pre-specified at a fine-grained level and demand the support of a loosely specified process model [17]. The need for agility in business processes and ad hoc reaction to dynamic changes has initiated the development of appropriate paradigms in this context. Reichert et al. [10] introduces for example a set of change operations based on a formal workflow model which enables adapting the structure of a running workflow in compliance with its (structural) correctness and consistency. Case handling [1] is another paradigm that follows in contrast to traditional workflow management the central concept of the case. The focus is set on what can be done in order to achieve the business goal instead of using predefined process control structures which determine the course of action during process execution. A knowledge worker, who is responsible for a particular case, decides actively on the basis of available information how the goal of the case can be reached, assisted by the case handling system. Furthermore, process flexibility can be obtained by using declarative processes [22] and decision deferral patterns [21], which are described in more detail in Chapter 2, as they are the subject of this thesis. 1.2. Related Work Despite the development of corresponding approaches for enhancing flexibility in business processes modeling and execution, unfortunately there are only a few empirical studies investigating the suitability of the approaches mentioned in Section 1.1 [18]. Therefore, the demand for empirical insights in that field was an incentive for the Quality Engineering Research Group at the University of Innsbruck to conduct controlled experiments in order to resolve these knowledge gaps. For the purpose of conducting controlled experiments, the research group developed the Alaska Simulator Toolset which provides integrated support for different approaches for business process flexibility and facilitates a meaningful comparison in that field. In [18], the underlying concepts of the Alaska Simulator are introduced, including how the toolset provides support for different patterns for decision deferral. Besides, factors (e.g., 7
configuration and personal characteristics) are discussed which influence the suitability of particular approaches related to decision deferral. In addition, the paper provides a guidance for how controlled experiments can be conducted using the Alaska Configurator (e.g., designing a business process), the Alaska Simulator (e.g., planning and executing) and the Alaska Analyzer in order to support detailed evaluation of the data. Closely related to the research work in [18] are empirical tests, investigating the applicability of the declarative approach in business process execution [22]. These tests are performed initially in cooperation with the School of Industrial Engineering at the University of Technology in Eindhoven. A replication of the experiment was made with students of a comparable course at the University of Innsbruck. The objective was to evaluate if test subjects were able to handle different business cases (e.g., process instance) while applying the declarative approach. The focus was set on different levels of constraints which need to be satisfied within a particular business case. Particularly, the impact of constraints on the effectiveness of process execution was investigated. Beforehand it was assumed that the application of the declarative approach would be effective when only few constraints must be considered. The question of whether these expectations can be also met when the number of constraints increases was aimed to be tested. The major outcome of the analysis shows that users are able to apply agile planning as supported within the declarative approach, in order to handle effectively different levels of constraints. However, it is assumed that an incremental validation of constraint violations and the creation of a rough plan of the business process are significant in order to achieve a good performance in that context. Besides the alignment with constraints in business cases, the handling of the occurrence of unforeseen events was investigated in addition [19]. Further experiments conducted with students of the Management Center Innsbruck examine how effectively inexperienced users can manage exceptional situations during business process execution when performing a declarative process with considerable constraints. An argument in favor of applying the declarative approach results from its suitability for dynamic processes and handling of unexpected events. Analysis in paper [19] shows that the events occurring during process execution have a statistically significant influence on the process outcome. Handling constraints were not as sophisticated for the user because appropriate tool support was given in the experiments. In contrast, dealing with events presents a challenge for users, depending on how critical the event is for the process outcome (e.g., business value, number of failed processes). Generally, it turned out that talent and skills are crucial for effectively applying agile methods as supported in declarative 8
processes. Furthermore, adequate tool support is essential in order to align successfully with business process constraints. Another empirical test referring to declarative processes analyzes the agile planning approach to obtaining flexibility in business processes [17]. The experiments were conducted with students of the University of Innsbruck. The test involves modeling and executing declarative processes, applying the agile mode once and the plan-driven approach once in a second test run. The intention was to make a comparison of both planning approaches for modeling and executing business processes. Particularly, it was questioned how well inexperienced users can apply the varying planning approaches. The experiments show no statistically significant difference when performing both planning modes. Manual analysis shows that only some users fully apply the agile approach, which suggests that the only-partial adoption contributed to the non-statistically significant results. Furthermore, analysis shows that the adoption of a higher number of agile practices implies a better process outcome. Apart from this, it can be observed that the application of agile principles requires a greater level of experience than the performance of the plan-driven approach. 1.3. Research Question The results of the experiments described in [17] show no significant difference in the process outcomes of inexperienced users when varying the applied planning approaches (presumably because of the agile approach only being partially adopted). Consequently, one major goal of this thesis is to refine the analysis described in [17] in the context of Research Question 1 and 2, better considering the limited adoption of the agile approach. In particular, the first objective of this thesis is to investigate whether the adoption of different planning approaches has an influence on the process outcome in terms of the achieved business value. In order to provide a general comparison between the agile and the plan-driven planning approach in this regard, Research Question 1 is formulated (cf. Chapter 3). To investigate the real potential of both planning approaches, Research Question 1 only considers the best performers (i.e., the best 25 % and 50 % performers) of both the agile approach and the plan-driven approach. 9
RQ 1: Is there a difference in terms of mean business value between the best performers applying the agile and the plan-driven approach? The second research objective focuses on the evaluation of how good inexperienced users are in modeling and executing business processes when these two planning approaches are applied. In addition, an investigation is made (as suggested in [17]) of whether the number of agile practices adopted during planning and execution of the journey has an impact on the process outcome (i.e., business value). Research Question 2 is addressed in Chapter 3. RQ 2: Does a higher number of applied agile principles imply a higher achieved business value compared to the plan-driven approach? The third research objective of Chapter 4 is to evaluate the impact of the uncertain events on the process execution, extending the research presented in [19]. Additionally, challenges for users should be detected which might arise due to the occurrence of an event. Based on this common practices and pitfalls in handling events are derived. Research Question 3 is defined for this concern. RQ 3: How does the occurrence of events influence planning and executing a journey? 1. What kind of events might occur? 2. How do users handle these events during business process execution? 10
1.4. Research Methods The analytical process in this thesis and the selection of appropriate research methods are based on the methodologies introduced by C. Wohlin et al. [25]. First of all, actual challenges in the field of business process modeling (cf., Section 1.1) were identified. Based on the challenges identified, the research problem was formulated from which the research questions summarized in Section 1.3 were derived. In order to provide a summary of existing approaches in the field of dealing with uncertainty in business process modeling, a literature review was necessary as a natural part of the problem formulation. Useful background information is indicated in Chapter 2. Afterwards, the appropriate research method for analyzing and interpreting data was chosen. The data sets from the conducted controlled experiments in [17] [22] [19] were input for analysis as the main part of this thesis. In order to draw valid conclusions on the data sets, quantitative interpretation as illustrated in Figure 1.1 was used to answer research questions RQ 1 and RQ 2. In the first step, descriptive statistics were conducted in order to characterize the underlying experiment data. The goal of the descriptive statistics is to get an overview of how the data is distributed and of the nature of the data (e.g., outliers). The second step contains the reduction of the data sets to a set of valid and meaningful data (e.g., comparing the 50 % of best performers). The third and final step is analyzing data by hypothesis testing. Analysis and Interpretation Experiment Data Descriptive Statistics Data Set Reduction Hypothesis Testing Conclusions Figure 1.1.: Steps in quantitative interpretation based on [25] 11
Conducting quantitative analysis was supported by the application of the statistical software SPSS. Furthermore, the statistical analytical methods described refer to the analysis in Chapter 3 only. To answer research question RQ 3, in turn, evaluations of events in Chapter 4 were conducted without statistical tool support. Analysis in this part is of a descriptive nature, based on spreadsheets and graphical presentations. The actual analysis contains a classification of events according to a pre-defined classification scheme. Subsequently, the descriptive evaluation of events contains a survey of the frequencies of occurrence. Furthermore, handling of events by the users was investigated in this section. The evaluation of events is concluded with common practices and pitfalls when dealing with uncertainty in business processes. 1.5. Structure of the Thesis The remainder of this thesis is structured as follows: Chapter 2 provides background information mainly on the software used, the Alaska Simulator, in order to conduct the experiments. Furthermore, this chapter introduces declarative processes, decision deferral patterns as well as planning approaches which are supported in the simulator. In addition, some agile practices are presented which are treated in the following evaluations. Chapter 3 focuses on the comparison of the agile and the plan-driven planning approaches. First of all, the setup for the analysis is outlined. The main part of this chapter covers the evaluation on the basis of the process outcome (i.e., business value) and the application of agile practices. For each evaluation, research questions and the appropriate proceeding are defined. The actual data analysis highlights statistical outcomes. Evaluations are concluded with discussions on the findings. Chapter 4 addresses the handling of uncertain events by the user during the process execution. The chapter starts with the declaration of the experimental setting and characteristics concerning journey configurations. Before the real evaluation, research questions, proceeding and required criteria for classifying events are indicated. The data analysis provides a characteristic specification of events. Following up on this, a descriptive evaluation of events reports on the occurrence of events. This chapter closes with discussion on the obtained findings and common practices and pitfalls in dealing with events. 12
Chapter 5 concludes the thesis and summarizes the main findings. outlook for future research work in this field is provided. Furthermore, an 13
2. Background This chapter first introduces the Alaska Simulator Toolset (AST) in Section 2.1, which is applied in controlled experiments on process flexibility [18] using travel planning as a metaphor for a business process. Besides the core concepts, the graphical user interface of the simulator will be introduced and some explanations on the underlying journey metaphor follow. Section 2.2 provides comparisons of different process modeling paradigms. Further, concepts for obtaining more flexibility are described in Sections 2.3 and 2.4. The extent of implementing these concepts can be measured by defined practices introduced in Section 2.5. 2.1. Alaska Simulator-Toolset The research group Quality Engineering at the University of Innsbruck developed the interactive software tool Alaska Simulator Toolset (AST) [18] [8] [11] in order to conduct controlled experiments on process flexibility. Besides the support for planning and executing business processes, the tool enables analyses of planning behavior, comparisons of different planning approaches and decision deferral patterns. The following section gives an overview of the underlying meta-model and core concepts as well as a brief description of the graphical user interface of the AST. 2.1.1. Alaska Simulator Meta-Model The central object in the AST Meta-Model is a Journey. Each journey consists exactly of one Journey Plan into which multiple Plan Items can be inserted. In addition, a journey refers to at least one Location where several Events might occur. Events can be classified into NewActionEvent or ChangeActionEvent. Furthermore, diverse Constraints might be defined for a journey configuration which can either be termination or execution constraints (e.g., the journey can only be completed at the location San Francisco 14
or the action Golden Gate Bridge in San Francisco has to be executed at least once). Different types of plan items can be selected for composing a journey, such as Actions or PlaceholderActions. During planning and journey execution the user has to deal additionally with potential Uncertainty and Resource Scarcity in order to follow a predefined Goal. Correlations between the single components which are explained in the following are depicted in Figure 2.2. Goal: Test persons plan and execute a journey supported by the AST, primarily pursuing the objective of maximizing the overall business value. This value is also an indicator for a user s travel experience. In order to optimize the total business value several information is required concerning the expected business value, costs, duration, availability, and reliability of actions. Journey, Journey Plan, Location: The composition of a journey is based on a prespecified journey configuration (e.g., Alaska) which represent instances of the Alaska Simulator Meta-Model. The course of actions of a journey can be planned on a journey plan, which is displayed as a calendar. For planning the journey, users can choose from several plan items which include activities, routes or accommodations. During a journey different locations can be visited. The selection of plan items depends on the corresponding location (e.g., the activity Golden Gate Bridge is only available at location San Francisco). Plan Items: Plan items can be divided into atomic actions which refer either to activities, routes or stays in accommodations, whereas complex placeholder actions are used for a later refinement during journey execution. Generally, plan items have a name, a description, a starting time which depends on the position on the calendar and a state which enumerates if the action is, for instance, scheduled, booked, started or executed. Atomic actions are additionally determined by location, costs, duration, reliability, availability and their business value. Furthermore, booking options and cancellation fees are indicated for each atomic action. When integrating placeholder actions within the journey plan, users can use Late Binding Placeholder Actions or Late Modeling Placeholder Actions. The difference is that users can select different sets of Plan Item Sequences at run-time when applying Late Binding Placeholders. Alternatively, the user can choose from a set of predefined plan items at run-time in order to design a sequence of Plan Items using the Late Modeling variation. Besides the attributes of name, description, start time and state, Placeholder Actions contain further attributes such as duration, start and end location. 15
Figure 2.1.: Action Life-cycle adopted from [11] The state transition diagram in Figure 2.1 shows the life-cycle of an action where possible states and state transitions are depicted. The process is started when the user selects an activity and inserts it into the journey plan. Consequently, the state of the activity is set from Inactive to Scheduled. Optionally, scheduled activities can be booked causing a transition into state Booked. If all required pre-conditions for execution are fulfilled, scheduled or booked activities respectively turn into the state Enabled, from where users can initiate execution. This leads into the Running state. If execution is successful the processed activity will be transferred into the state Completed. Otherwise, the process results in the state Failed. During the planning phase or during run-time, scheduled, booked or enabled activities can be removed from the journey plan, leading to the state Skipped. In addition, reservations for activities can be canceled which effects a backward transition from state Booked to state Scheduled. Furthermore, scheduled, booked or enabled activities might become unavailable during run-time, for example, due to occurring events. Such occurrences might trigger a transition into the state Unavailable. Constraints: A further component integrated in the AST is the set of constraints which need to be considered when planning and executing a journey. The simulator provides support for two types of constraints such as execution and termination constraints. Execution constraints determine occurrences, dependencies or the exclusion of activities in the journey plan. In contrast, Termination constraints include preconditions for a successful termination of journeys. For example, process termination requires the execution of a particular activity to occur at least once, or a particular terminal point of the journey to be determined. In [16] es- 16
sential control-flow constraints are introduced in order to avoid chaotic behavior while planning and executing processes. The mentioned constraints are entirely supported in AST and need to be satisfied during process execution. In addition, AST provides restrictions related to resources, time and locations. Uncertainty: An essential characteristic of a journey in the AST is uncertainty due to incomplete information prior to the journey s execution. Uncertainty in this context refers to the reliability of an activity, which is strongly influenced by the weather condition at the location where the relevant action is available. To what extent an activity is subject to variation regarding weather conditions will be expressed by the action s reliability indicated by a percentage. For instance, a low percentage shows a high dependency on weather. In this way, the actual business value cannot be predicted because the user only receives an estimation of the expected business value of the specific activity before the journey execution which is evaluated on the basis of the reliability and the average weather situation in the specific location. Finally, at run-time the actual business value will be computed depending on the actual weather. The overall business value of a journey expressed by a numeric value representing user s travel experience will be obtained by summing up the individual business values of each executed activity in the journey plan. Resource Scarcity: For planning and executing a journey, users have a certain amount of budget available in order to make activity reservations and ensure their availability. If an activity contains the option for booking, additional information referring to booking deadlines and charged cancellation fees for cancellations within a certain period will be indicated. Bookings are especially necessary for activities with low availability combined with low reliability (cf., Figure 2.6). When bookings are made, the expected fees for the activity must be paid immediately. Users have the option to cancel their reservations. In such a case, cancellation fees might be charged if the definitive booking deadline has expired. As a consequence, early commitments to activities might become costly if a cancellation occurs subsequently. In this context resource scarcity has to be considered in relation to financial means which should raise user s awareness of making reservations too early, with the resulting cancellation of activities. Events: Besides changing weather conditions, events are a second factor that creates uncertainties during the execution of the journey. Primarily, AST provides for differentiation between ChangeActionEvents and NewActionEvents. Here, Change- 17
ActionEvents are changes of activities value either by increasing or decreasing business values, duration, cost or availability. NewActionEvents launch, new actions during run-time require users to respond during the journey execution. Chapter 4.2 is completely dedicated to the evaluation of events. Figure 2.2.: UML Class Diagram describing the Alaska Simulator Meta-Model adopted from [11] 2.1.2. Graphical User Interface The Alaska Simulator contains a graphical user interface [18] which facilitates the composition of individual journeys in a user-friendly and intuitive way. The graphical layer is divided into five frames which contain different views and deliver appropriate information. Figure 2.3 illustrates the components which facilitate the planning process. The largest view shows the Planning Editor (1) as a calendar display where the journey plan will be composed. Users are planning their journey by dragging activities from the 18
Available Actions View (3) and dropping them onto the journey plan. The user obtains supplemental information which are essential for planning activities in this view. In general, cost, duration, constraints, the expected business value, reliability and availability are required for choosing appropriate items from the displayed list to achieve the best possible results in total business value. In addition, the Actions View provides several filtering options according to costs, duration and business value. Furthermore, the user can restrict the display according to the type of action (e.g., activity, route or accommodation). Additionally, restrictions can be made referring to the location where activities are available. Finally, some searching options are supported, such as filtering specific actions or freely searching for activities. Figure 2.3.: Screenshot of the graphical user interface of the Alaska Simulator Toolset adopted from [18] The top frame at the right shows the Constraint View (2) which indicates all constraints 19
defined for the relevant configuration variant. The view gives a short description of the restriction, the affected activity and the assigned type for each constraint. A second tab in this frame contains the Event View, which is depicted in (6). During execution, the events are recorded. Entries in this view contain the name of the specific event, the affected activity and the slow-down time of the event. The map (4) in the bottom frame at the right gives an overview of all existing locations in the journey configuration. Additionally, weather information which includes trends and stability is indicated for these regions. For a better understanding of the symbols used, appropriate explanations are given. Furthermore, this view also provides a search option referring to locations, and zoom functionality. The last view (5) at the very bottom in the graphical user interface displays information during run-time which contains messages about problems that arise or constraint violations. In a second tab, actual weather information (7) is displayed. This is only available during execution. 2.1.3. The Journey Metaphor Motivation for the business process of travel planning can be found in the established fact that the majority of test subjects are familiar with journey planning (cf. [18]). In addition, this journey metaphor attracts people s attention more easily and consequently facilitates the access to business processes. Furthermore, planning and executing journeys signifies a procedure with a more intuitive characteristic. Due to these features, test persons show a higher readiness to participate seriously in the experiments and to use the provided simulation software. This leads also to higher validity of the experimental data. In order to justify the comparison of travel planning with business process modeling, the core concepts described in Section 2.1.1 are here discussed in the context of the fundamental elements of business processes. According to the definition of Weske [24] A business process consists of a set of activities that are performed in coordination in an organizational and technical environment. These activities jointly realize a business goal. Transforming this definition to the journey metaphor, journeys are composed by a set of actions which can either be atomic actions (activities, routes and accommodations) or placeholder actions (Late Binding and Late Modeling). Thereby, travel planning follows the overall goal of maximizing business value (travel experience). However, for 20
managing and executing business processes, performance targets are specified such as reducing costs or cycle time as well as optimizing quality or customer satisfaction [12]. Furthermore, the composition of a journey is based on a journey configuration (e.g., Journey to Alaska) which represents a process instance describing a concrete business case [24]. In addition, planning and executing a journey, as well composing a business process, are both undertaking which are subject to consideration of determined constraints. The Alaska Simulator provides support of basic control-flow constraints which can be compared with declarative process management systems like DECLARE [16]. Characteristics such as uncertainty, resource scarcity and unforeseen events which are integrated in a journey configuration (cf. Section 2.1.1) are also existent in business processes. In [4] authors exemplify these specifics of business processes using the example of healthcare service provision in Singapore. In this case, uncertainty relates to continuously changing environmental conditions (e.g., competing healthcare providers) which make it difficult to predict activities and resources precisely in order to perform specific processes. Resource scarcity is mentioned especially within surgical care (e.g., surgical professionals, surgical theatres, etc.). Unforeseen events that have to be handled involve changing medical requirements during the patient s treatment. 2.2. Declarative Approach Following the goal of achieving a higher degree of process flexibility declarative process models [7] were established as a suitable approach in that field. Currently, business process modeling is done while applying the imperative mode, where the process is strictly prescribed. However, dynamic process management requires modification of process models during work. So applying the imperative paradigm leads to an over-specification of processes in order to pre-capture potential process adaptations. The fundamental differences in comparing declarative and imperative models arise in the use of activities and constraints for specifying the control-flow of process models. Imperative processes exactly determine how things have to be done. This is implemented within the inside-tooutside approach [11] [16] [22] where the focus is set on the desired behavior. Therefore, all execution alternatives of the process are specified in the model. New alternatives have to be added to the model explicitly. Due to this key aspect of desired behavior this approach is well suited for guaranteeing compliance with existing business requirements. In addition, the inside-to-outside approach leads to process models which are rather inflexible and tend to be over-specified as well as over-constrained [11] (cf., imperative 21
approach in Figure 2.4). In contrast, declarative models follow the outside-to-inside approach. This method focuses initially on all possible combinations of activities which are based on an identified set of relevant activities. Activities can be executed in any sequence and as often as required. Then, constraints are added in order to approximate the desired behavior and to exclude traces of forbidden behavior. The declarative approach is used to build various models. These can range from processes which are defined in every detail, to very relaxed processes [16]. As regards constraints, there are three types specified [6]: selection of activities (e.g., minimal and maximal occurrence), ordering (e.g., pre-requisite) and usage of resources (e.g., budget). A contrast of the imperative and declarative approach is given in Figure 2.4. Figure 2.4.: Comparison of imperative and declarative approach for process modeling based on [19] 2.3. Decision Deferral Patterns In order to support flexible deviation from predefined processes Weber et al. introduce in [21] a range of change approaches in process-aware information systems. The application of patterns for predefined changes enables specification of determined process fragments during run-time and enhances flexibility in dealing with uncertainty by decision deferral in business process modeling and execution. A beneficial characteristic of these patterns is that the underlying business process model might contain parts or placeholder activities which remain unspecified before execution. For predefined changes at build-time three different patterns are distinguished: Late Selection, Late Modeling and Late Composition (cf. Figure 2.5). The distinction refers to the fragments for speci- 22
fication at run-time which leads to a different degree of flexibility. Among other reasons, the motivation for using the Alaska Simulator Toolset is the provision of integrated support for these types of Decision Deferral Patterns, which are explored and investigated in a controlled setting under varying circumstances [21]. Using Late Selection offers a choice of a set of predefined process fragments at run-time in order to specify placeholder activities. Thus, different implementations for an activity exist, and run-time information is required for the selection of the appropriate placeholder implementation from repository. By contrast, Late Modeling provides more freedom by modeling placeholder activities at run-time. The most flexibility is given when applying the Late Composition pattern. Users can iteratively assign selected activities from a repository to the business process at run-time considering the alignment with constraints. In this way, business processes are fully specified at run-time, when pre-defined process fragments will be combined. Late Selection Late Composition Late Modeling Figure 2.5.: Excerpt of Change Patterns adopted from [21] 2.4. Planning Approaches Dealing with uncertainties in business processes requires more flexibility within the planning and execution phase. For that purpose decision deferral patterns [18] are used to provide the option of deferring the precise specification of the process to the last responsible moment. The Alaska Simulator implements two different planning approaches 23
which support techniques for decision deferral. 2.4.1. Plan-Driven Mode This planning approach can be equated with traditional workflow management systems which provide the least process flexibility. In order to mitigate uncertainty, a very detailed predefined process acts up-front as the basis for the execution. This plan remains unchanged during its execution. Thus, planning and execution are strictly separated in this approach. An adverse effect of this pre-specification is that plans bear the risk of beeing imprecise especially the earlier they are created. Furthermore, full concentration on the planning phase might lead to the consequence of plans turning out to be useless due to changing conditions and requirements. To overcome these two shortfalls the two decision deferral patterns of Late binding and Late Modeling are integrated into the plan-driven planning approach. The provided placeholder activities increase flexibility because they can be specified during run-time and hence relax the strict differentiation between planning and execution. 2.4.2. Agile Mode In contrast to the plan-driven approach, the agile mode focuses on modeling as an ongoing activity. In order to realize this principle, the decision deferral pattern of late composition will be supported in this case. Applicants are able to refine a more coarsegrained model step by step during execution. However, the execution of a fully prespecified business process is supported, as well as the planning of the process ad-hoc during run-time (ad-hoc composition). Here, single process instances can be iteratively modeled and executed. Decision deferral like late composition allow users to integrate newly gained knowledge in order to react more flexibly to unpredictability. Therefore, the agile approach is well suited for highly-changing business processes whereas the plan-driven mode is intended for repetitive and predictable processes. 2.5. Agile Practices Thus, the agile mode enables iterative refinement, where two extreme cases are included ranging from pre-specifying everything to pre-specifying nothing. Planning in an agile manner requires the usage of different practices (i.e., Elimination of waste). Thus, the 24
degree of agile adoption can be operationalized by considering to what extent these practices are applied. Using these practices, it is possible to make observations referring to the efficient exploitation of the agile principles and the effective adoption of gained knowledge. In the following an explanation of the preconditioned practices is given: 1. Elimination of waste: During the execution of the planned journey the duration of an activity might be shorter than expected. Elimination of waste considers replanning by the participant (i.e. through shifting the subsequent activity to an earlier starting point) in order to reduce or rather to avoid idle times. 2. Defer commitment: Activities of the two configurations are different concerning their availability and reliability. Figure 2.6 summarizes the four possible categories of activities and indicates simultaneously the required strategy for their handling. This practice examines whether the participant defers his commitments to the last responsible moment, which varies from category to category. For example, highly available activities don t require any reservation because they would be on hand anyway. A reservation in this case would be needless and would not provide any benefits. The last responsible moment for this category would be immediately before executing the concerning activity. In contrast, activities with low availability combined with high reliability require a reservation before the deadline. The last responsible moment for these activities will therefore be at the reservation deadline. The handling of activities with low availability associated with low reliability is more complex and will be examined separately. 3. Create knowledge: This practice investigates how effectively participants apply their gained knowledge during the journey by means of continuous replanning. In this context, aspects to be focused on are whether changing conditions during the execution of the journey will be considered (e.g. weather conditions), and whether newly available activities will be included in the journey. 4. Create options: Creating options only refers to activities with low availability combined with low reliability, which are also depicted in Table 2.6. In this case the concerned activity will be reserved and simultaneously a more reliable activity is planned in parallel. This practice allows to defer the last responsible moment for choosing one activity during run-time. 25
High Reserve Defer Decision Reliability Establish Option To Further Defer Decision Defer Decision Low Low Availability High Figure 2.6.: Classification of activities referring to availability and reliability based on [17] 26
3. Comparison of Planning Approaches The evaluations in this chapter generally focus on the comparison of the agile and plandriven approaches. The investigation focuses on which planning mode is better concerning the effect on the process outcome (i.e., the journeys business value). The introduction of this chapter in Section 3.1 provides an overview of the data collection, basics of individual journey configurations and general data preparation for analysis. Comparisons of the total business values of the best performers applying the agile and plan-driven approach (cf. RQ 1) in Section 3.2 offer conclusions with respect to the performance of the investigated approaches. Section 3.3 investigates the effective application of the agile principles, particularly if a higher number of agile practices implies a higher business value (cf. RQ 2). 3.1. Setup Evaluations within this chapter refer to data sets which were obtained from experiments conducted at the University of Innsbruck [17]. Bachelor students of Computer Science served as test subjects in order to investigate how well the different planning approaches can be adopted by inexperienced applicants. The challenge of these experiments was modeling and executing a declarative business process while applying the agile and the plan-driven planning approach. Furthermore, uncertainties arising within the processes needed to be managed effectively. Below, Section 3.1.1 provides a summary of the experimental setup and data composition. Explanations referring to the different journey configurations are included in Section 3.1.2. Section 3.1.3 introduces the subject of analysis. Finally, Section 3.1.4 provides details on the filtering options for data preparation. 27
3.1.1. Experimental Setting As the basis for this thesis, several experiments were used which aimed at investigating differences in modeling and executing a declarative business process (see Section 2.2) in an agile versus a plan-driven mode (see Section 2.4). First, experiments were conducted in April 2008. Further experiments were performed one year later in April and May 2009. Finally, in May 2010 the last experiment was undertaken in that field. Generally, experimental sessions were organized in groups of 10-15 students. First, introductions were given on the basic concepts of the Alaska Simulator and the different planning approaches. Furthermore, students were informed about the course of actions, rules and goals of the experiment. Afterwards, users were assigned randomly to two groups of equal sizes. In the first run of the experiment, Group 1 was instructed to model and execute the Journey of California I using the plan-driven approach, whereas Group 2 was assigned to perform the Configuration California I adopting the agile planning mode. For the second run of the experiment, the planning modes were changed between the two test groups. Then Group 1 was expected to apply the agile approach and Group 2 to model and execute the business process while using the factor level plan-driven. The process configuration in the second run was based on the Alaska I Journey. The idea behind this experimental design, which is also depicted in Figure 3.1, is to avoid any learning effects. Each experimental run takes about one hour. For the first 25 minutes there is an introduction to the Alaska Toolset using a starter kit which includes screencasts explaining elementary features of the simulator and the appropriate journey configuration (e.g., California I in the first and Alaska I in the second run). In the remaining time participants were engaged to plan and execute the business process. Their task was to apply the different planning approaches with the aim to optimize the resulting business value. Further details on the experimental setup and performance of the experiments are given in [17]. The data obtained from the conducted experiments was stored in the log of the Alaska Simulator. For evaluation purposes, only data could be considered when participating students followed the predetermined experimental design. Otherwise, the data entries were discarded. From experiments conducted in 2008, 44 subjects out of 49 could be used for further analysis. Data entries referring to the test year 2009 count a total of 18 valid test subjects out of 23 per experimental run. For 2010 only one out of 8 subjects had to be discarded due to planning and executing the two journey configurations using the agile 28
Group 1 n/2 Participants Factor Level 1: Plan-Driven Configuration CALIFORNIA Group 1 n/2 Participants Factor Level 2: Agile Configuration ALASKA Group 2 n/2 Participants Factor Level 2: Agile Configuration CALIFORNIA Group 2 n/2 Participants Factor Level 1: Plan-Driven Configuration ALASKA First Run Second Run Figure 3.1.: Design of conducted experiments based on [17] planning approach both times. Table 3.1 summarizes the available data differentiated by testing year, journey configuration and the applied planning approach for the subsequent analyzing process. In addition, the appendix provides a listing of the complete raw data referring to the Alaska I and California I Configurations in Section A.1. Test Year Journey Configuration Approach N Total Agile 22 Alaska I 44 Plan-driven 22 2008 Agile 22 California I 44 Plan-driven 22 2009 2010 Alaska I California I Alaska I California I Agile 10 Plan-driven 8 Agile 8 Plan-driven 10 Agile 3 Plan-driven 4 Agile 4 Plan-driven 3 Table 3.1.: A break down of data by test year, journey configuration and approach 18 18 7 7 3.1.2. Journey Configurations Initially, Chapter 2 explains the core concepts of planning and executing a journey. Based on that two different journey configurations are derived which are instances of the Alaska Simulator Meta-Model. This section provides a comparison of the Alaska I (i.e., data set I) and the California I (i.e., data set II) Configuration. Besides, an overview of the fundamental settings related to the journey process will be given. Descriptions refer to publications in this context by the Quality Engineering Research Group at the 29
University of Innsbruck [18] [17]. Basically both journeys have the same number of plan items, whereas the number of activities and actions referring to accommodations and tours are slightly different. However, the number and type of constraints is similar. Activities in the Journey to California I are restricted by constraints referring to the journey s end-location, execution time and budget. Each journey plan based on the Configuration California I has to be finished at San Francisco. Thereby, users have 800 monetary units available for planning, which is limited by the budget constraint. During the planning and execution of the journey, users have to deal with several execution time constraints. For instance, the activity Visitor Center in Yosemite can be executed at least once in a journey. In comparison, for the Configuration Alaska I the end-location is set to the location Anchorage. In addition, users of planning the Alaska I Journey can use 1.800 monetary units. This amount is justified by the fact that the expenses for activities are generally higher than in the Journey to California I. When planning the Alaska I Journey a fourth constraint of mutual exclusion has to be considered additionally. For example, at Denali National Park, the user can choose either the activity Home Stead Boat Tour or the activity Wilderness Jetboat, but not both of them. Comparing the events that might occur, it can be remarked that the journeys have roughly the same number of different types. Both journeys include events which introduce new activities (e.g., Moose Hunting, Flightseeing, Jet Boat Trip), increase the business value of an activity due to their occurrence (e.g., grizzlies or elephant bulls can be seen) or make activities unavailable (e.g., camping equipment stolen, dogs become sick, Tioga pass closed). However, there is a considerable difference referring to the availability of activities within the journeys. 24 out of 25 activities are highly available when planning and executing the Journey of California I. In contrast, the Journey of Alaska I has only 12 highly available activities out of 28 activities in total. In this case, users were more challenged to handle the higher resource scarcity by deferring decisions and building options. The comparison of the two journey configurations shows that there are some differences referring to diverse activity settings (e.g., availability of activities). Consequently, it can be remarked that these differences in the journey configurations lead to differences in the achievement of the total business values. Table 3.2 gives a summarizing overview of the Alaska I and California I Configuration. 30
Journey to California I Journey to Alaska I Number of Plan Items: 43 43 Accommodations 8 7 Activities 25 28 Tours 10 8 Type of Constraints Times of action s execution End-location of journey Budget limitation Times of action s execution Mutual exclusion of actions End-location of journey Budget limitation Number of Events 4 5 Name of Events Jet Boat Trip Elephant Bulls in Monterey Tioga Pass closed Invitation for Flightseeing Moose Hunting Dogs sick Grizzlies Invitation for Flightseeing Camping Equipment stolen Table 3.2.: Characteristics of Journey Configuration California I and Alaska I 3.1.3. Response Variable The subject of analysis in Section 3.2 and 3.3 is the response variable. The response variable is defined as the maximum achievable business value when planning and executing a given journey configuration with a given planning approach [17]. The goal of achieving the highest possible business value, and the calculation thereof, has been explained above in the Alaska Simulator Meta-Model (cf. Section 2.1.1). Thus, the actual gained business value depends on the varying weather conditions. These uncertain conditions remain the same for each test subject in order to ensure comparability [22]. In addition, the response variable was additionally used in order to restrict the amount of data sets for specific analysis - for example considering only the 50 % highest performers on the basis of the total achieved business values. Settings for filtering of data are described in the next Section. 3.1.4. Data Preparation Since the journey configurations described in Section 3.1.2 are different in their level of difficulty, and participants can achieve varying total amounts in business value, it is necessary to prepare the available data sets for comparison. 31
Basically, a division according to the Journey Configuration Alaska I and California I is required. This grouping is also essential for the evaluation of practices. The Alaska I Journey enables the observation of four practices, which are explained in Section 2.5, while planning and executing the journey. However, the activity settings for the California I Journey require no creation of options for decision deferral. This fact additionally justifies a separation by the number of applied practices. Furthermore, journeys can be planned and executed using the agile and plan-driven modes. Consequently, the applied planning-approach must be considered separately. In order to investigate the defined research questions, several restrictions and adaptations of the criteria for filtering are applied. In the following analysis, the performed test configurations are described in more detail and summarized in table form. 3.2. Evaluation of Business Value The first analysis deals with the examination of the response variable Business Value in order to compare planning approaches. At the beginning, research question to be answered within this evaluation is indicated in Section 3.2.1. Subsequently, explanations of the procedure that was followed during evaluation are given in Section 3.2.2. Analysis of the data is made in Section 3.2.3, which comprises descriptive statistics and significance tests. The chapter ends with conclusions on the obtained findings in Section 3.2.4. 3.2.1. Research Questions Evaluations within this section focus on a general comparison between the two planning approaches (i.e., agile vs. plan-driven). The objective of this section is to evaluate the effect on process outcome in terms of business value due to the adoption of different planning approaches. For this purpose, the following research question was formulated: RQ 1: Is there a difference in terms of mean business value between the best performers applying the agile and the plan-driven approach? To answer this question it is necessary to evaluate whether the adopted planning approach has an impact on the response variable. For this purpose, significance tests [5] (i.e., the T-test) will be applied. The goal of these tests is to examine if the adoption of a planning approach has a statistically significant impact on the response variable Business Value. Therefore, the following hypotheses are formulated: 32
Null Hypothesis H 0 : There is no significant difference in the mean business values with respect to the adopted planning approach. Alternative Hypothesis H 1 : There is a significant difference in the mean business values with respect to the adopted planning approach. In order to test the defined hypothesis by applying the T-test, the data set must be normally distributed. This precondition can be tested in the Kolmogorov-Smirnov-test (KS-test). Subsequently, the Levene-test investigates whether the second precondition of homogeneous variances is also fulfilled. In case the preconditions are not fulfilled, the U-test according to Mann and Whitney must be conducted alternatively. The obtained p-value [14] from the appropriate significance test indicates statistical significance if it lies below the significance level 1 α of 5 %. This case leads to the rejection of the null hypothesis H 0 which assumes no significant differences. Otherwise, H 0 can be accepted at a confidence level of 95 %. 3.2.2. Proceeding Analysis in this context is based on data sets described in Section 3.1.1. The raw data that was used as the basis for this analysis is shown in Appendix A.1. Thus, the evaluation in this case considers only a certain number of best performers, while appropriate restrictions and filterings are incorporated. The reasoning for this test configuration is to find out if statistical results of the two planning approaches principally differ from each other when the data sets have been restricted. The intention for comparisons of the 25 % and 50 % best performers is to make sure that agile data sets are actually compared. Since the agile planning approach provides an increased scope for application (cf. Chapter 2) it is reasonably necessary to make restrictions on the best performers in order to see the real potential of this planning approach. In addition, the comparisons should identify how far the best adopters of the two planning approaches differ from each other measured in terms of business values. Therefore, the underlying data obtained from the logs of the Alaska Simulator is restricted according to the journey configuration and planning-mode. Additionally, a filtering is made considering the 50 % of best performances with respect to the total achieved business values. Accordingly, for the Alaska I Journey 18 valid records in the 1 is the interval, which contains the expected value with a probability of 95%. [5] [14] 33
agile mode can be assigned to the 50 % of best performers. For the plan-driven approach, 17 data sets are counted in this case. Samples for the Configuration California I comprise 17 entries for the agile approach, and 18 journeys are performed in the plan-driven mode. For the 25 % of best performances the indicated data sets are divided in half. 3.2.3. Data Analysis Statistical analysis in this section is supported by the application of the statistic software SPSS 18 2. The analytic process starts with descriptive statistics which summarize minimal, maximal and mean values of the defined samples. Subsequently, significance tests are conducted in order to investigate significant differences in mean values. 3.2.3.1. Descriptive Analysis In the following a descriptive report for two distinct test configurations is provided. Calculations refer to the variable Business Value. 1. Comparing the 50 % of best performances of the agile and plan-driven approach This test case depicts descriptive statistics for the Configurations of Alaska I and California I using the agile and plan-driven mode, where only the 50 % of best performers are considered in each case. The test results should give an indication of whether the 50 % best performers of Alaska I and California I applying the agile mode gain a higher business value compared to the groups using the plan-driven mode. Configuration Planning Mode N Min Max Mean Alaska I agile 18 3973.58 6080.23 4654.28 Alaska I plan-driven 17 3374.00 5881.00 4170.92 California I agile 17 4803.00 6128.90 5430.73 California I plan-driven 18 4549.67 6053.00 5176.41 Table 3.3.: Descriptive Statistics examining 50% of the best business values for Alaska I and California I Configuration Table 3.3 summarizes the obtained results. The statistical output shows the minimum, the maximum and the mean business value for each sample. 2 http://www-01.ibm.com/software/at/analytics/spss/ The highest value for 34
both configurations can be achieved when the journey is performed in the agile planning mode. 2. Comparing the 25 % of the best performances in the agile vs. plan-driven mode The second test case in this section shows descriptive statistics again for the Alaska I and California I Journeys in the agile and plan-drive modes. Additionally, restrictions on the data sets have been made while considering the 25 % of best performances with respect to the business value. The objective is to investigate if 25 % of best performers planning both journeys in the agile mode are better than the 25 % of the best plan-driven performers. Configuration Planning Mode N Min Max Mean Alaska I agile 9 4578.00 6080.23 5164.34 Alaska I plan-driven 9 4207.20 5881.00 4589.89 California I agile 9 5460.00 6128.90 5749.31 California I plan-driven 9 5154.73 6053.00 5456.06 Table 3.4.: Descriptive Statistics examining the 25% of the best business values for Alaska I and California I Configuration By analyzing the results in Table 3.4 it can be observed that the highest minimum, maximum and mean business value can be obtained when the journey is planned and executed in the agile mode. This observation is valid for both configurations in the test case. 3.2.3.2. Testing for Differences in Business Value Based on the descriptive statistics, investigation focuses here on whether the observed differences in mean business values are statistically significant. Subsequently, the two test cases show results of the conducted significance test where the best performers applying the agile approach and the plan-driven approach were compared. 35
1. Comparing the 50 % of best performances of the agile and plan-driven approach This test case investigates samples of factor levels agile and plan-driven which are compared for each journey configuration separately. Besides the restriction according to configuration and planning approach, only the 50 % of the best performances with respect to the business value are considered. Hypotheses to be tested within this examination are related to the impact of the application of different planning approaches. In this context two appropriate hypotheses are formulated a priori: Null Hypothesis H 0,0 : There is no significant difference in the 50 % best mean business values with respect to the adopted planning approach. Null Hypothesis H 1,0 : There is a significant difference in the 50 % best mean business values with respect to the adopted planning approach. Table 3.5 summarizes results for this test configuration. Considering the outcome for the Alaska I Journey it can be remarked that both samples referring to the factor level are normally distributed. Furthermore, the homogeneity of the data sets is checked by the Levene-test (0.542 > 0.05). The preconditions for the T-test have been fulfilled, therefore the T-test is used. The obtained p-value of 0.032 (< 0.05) indicates statistical significance. Based on that, the null hypothesis can be rejected at a confidence level of 95 %. For the California I Journey the required preconditions for performing the T-test are also given. Due to the obtained significance of 0.067 (> 0.05) the hypothesis H 0,0 cannot be rejected at a confidence level of 95 % in this case. Configuration Planning Mode N KS-test Levene-test T-test Alaska I agile 18 0.737 Alaska I plan-driven 17 0.655 0.542 0.032 California I agile 17 0.976 California I plan-driven 18 0.893 0.494 0.067 Table 3.5.: Test results of comparing only the 50 % of the best performances, divided into journey configuration and planning approach 36
2. Comparing the 25 % of the best performances in the agile vs. plan-driven mode The following test suit investigates if there is a possible difference between the mean business values of the 25 % best performers applying the agile and the plan-driven approaches. This examination was made for each configuration separately. For this purpose the following hypotheses are assumed: Null Hypothesis H 0,1 : There is no significant difference in the 25 % best mean business values with respect to the adopted planning approach. Alternative Hypothesis H 1,1 : There is a significant difference in the 25 % best mean business values with respect to the adopted planning approach. Considering test results for the Alaska I Configuration, from Table 3.6 it can be seen that the preconditions are met for conducting the T-test. With the obtained p-value of 0.039 (< 0.05) the null hypothesis H 0,1 can be rejected at a confidence level of 95 %. For California I, the samples for both factor levels show normal distributed data and the Levene-test approves homogeneous variances, therefore the T-test can be applied. The resulting p-value of 0.035 (< 0.05) justifies the rejection of the null hypothesis at a confidence level of 95 %. Configuration Planning Mode N KS-test Levene-test T-test Alaska I agile 9 0.975 Alaska I plan-driven 9 0.604 0.786 0.039 California I agile 9 0.986 California I plan-driven 9 0.566 0.299 0.035 Table 3.6.: Test series examining differences in mean business value of the 25 % best performances for Alaska I and California I Journey distinguished by planning mode 3.2.4. Discussion of the Results The outcome of this data analysis shows in three out of four test cases that the application of different planning approaches (i.e., agile and plan-driven) consequently leads to statistically significant differences in the total business value of a journey. The first test case comparing the 50 % best performers of both planning modes applying the California I Journey, however, does not yield any statistically significant differences. 37
Although the plan-driven approach is easier to handle for inexperienced users than the agile approach (cf. [17]) an explanation for the statistically significant results might be the limited flexibility during the execution of the journey in the plan-driven mode. Due to characteristics of the plan-driven approach (see Section 2.4), modifications of the journey plan during run-time are not permitted unless an event occurs. In contrast, the agile mode enables adaptation of planning of business processes which ranges from fully pre-specified processes to ad hoc planning and execution. Hence, both Configurations of Alaska I and California I contain significant uncertainty, which planners using the agile mode apparently address more effectively. In particular, both journey configurations contain a considerable number of activities which have low reliability values (cf. Fig. 2.6). Deferring decisions to run-time when more information is available (e.g., actual weather conditions) facilitates the handling of such activities. This practice is not supported by the plan-driven approach what explains why the results of the agile approach were significantly better. However, the absence of a statistical significant result in California I comparing the 50 % best performers (cf. 1. Test) might be explained by the fact that the Journey to California I is less challenging to perform than the Configuration Alaska I because of the configuration characteristics (cf. Section 3.1.2). The California I Configuration contains no activities of low reliability assigned with low availability. Therefore, the agile practice of option creation (cf. Section 2.5) is not applicable in this configuration. Due to this characteristic the user might not benefit from the agile planning to the same extent as in the Alaska I Configuration. Besides, the assumption that some users of the 50 % best performers have not adopted the agile concepts effectively cannot be excluded. 3.3. Evaluation of Practices This section deals with the evaluation of applied practices within the execution of the journey configuration. The objective of this analysis is to answer the defined research questions in Section 3.3.1. Afterwards, a brief overview of the general proceeding of this analytic process will be given in Section 3.3.2. The main part in Section 3.3.3 contains the textual and tabular representation of test results summarized in a descriptive analysis. The examination of potential differences related to business values follows. The evaluation of practices ends with conclusions in Section 3.3.4. 38
3.3.1. Research Questions Analysis in this section focuses on investigation of how well inexperienced users can perform process modeling and execution when applying these two planning approaches. Because the agile planning approach provides more freedom and flexibility in planning and executing business processes, the investigation should focus on to what extent agile principles are implemented by users. In particular, the research question below should be analysed: RQ 2: Does a higher number of applied agile principles imply a higher achieved business value compared to the plan-driven approach? For this purpose it is necessary to investigate whether, the number of adopted practices during planning and executing the journey has any influences on the variable Business Value. Significance tests [5] will be conducted as already used for testing differences in Section 3.2. The goal of conducting these tests is to reject the a priori determined null hypothesis, which assumes consistency of two samples, in order to accept the alternative, which supposes that there is a difference in the mean business values with respect to the number of the applied practices. Hence, for each test case a null and an alternative hypothesis were formulated a priori and are statistically checked in Subsection 3.3.3.2: Null Hypothesis H 0 : There is no significant difference in the mean business values with respect to the adopted number of agile practices. Alternative Hypothesis H 1 : There is a significant difference in the mean business values with respect to the adopted number of agile practices. 3.3.2. Proceeding Since agile planning cannot be enforced, the approach will be pursued that only journeys planned in the agile mode where agile principles (cf. Section 2.5) are adopted at least to a certain extent are compared with journeys planned and executed in the plan-driven mode. The extent of the agile adoption can be operationalised by the number of practices applied. That means that in the first step (cf. Figure 3.2) journeys are analyzed manually with respect to the application of practices. Due to different characteristics of journey configurations the analysis was different for the Configurations California I and Alaska I. For the California I Configuration, agile principles such as elimination of waste, defer 39
commitment and the creation of knowledge can be considered. However, for Alaska I additionally creating options is a relevant practice since Configuration California I contains no activities with low availability associated with low reliability. Data Collection Definition of Evaluation Criteria Definition of Measures Classification of Journey Examination and Valuation of Data Data Analysis with SPSS Discussion of Results Time Figure 3.2.: Graphical view of process sequence within the evaluation of practices After the definition of the criteria, a scaling must be determined in order to perform the evaluation. Each criterion can be valued in a range from 0 to 1, where the value 0 indicates that the particular practice was not applied. Otherwise it was valued with 1. In addition to whole points, half points can also be awarded. Journeys applying the plan-driven mode are valued with 0 because no re-planning at run-time is allowed and therefore a consideration of the defined criteria is not possible. Next, examination and valuation of the data according to the defined criteria and appropriate scaling occurs. For that purpose each single journey was replayed from the log of the simulator. The replay shows the initial planning of the participant and the subsequent execution of the concerned journey. Simultaneously an assessment of the planned and executed journey according to the criteria is made. The recorded observations are stored in an Excel-spreadsheet, which finally is transferred to the SPSS-statistics software for further processing. Data sets already described in Section 3.1 are used as the basis for evaluations in this context. conducted using SPSS and a final discussion of the findings. The following paragraphs describe the analysis 3.3.3. Data Analysis For statistical analysis of the findings referring to the evaluation of practices the statistics software SPSS is used. First, the descriptive analysis below gives some basic insights by offering frequency-based information and calculations on the minimum, maximum and mean value. In order to analyse if there are significant differences in the mean business values, the T-test or the U-test according to Mann-Whitney respectively are applied. In addition to a brief explanation on methods of analysis [2], this section further contains an evaluation of the obtained results, and discussion in this context. 40
3.3.3.1. Descriptive Analysis In the following, descriptive statistics are described. For comparisons between the agile and plan-driven approach, a different degree of agile adoption will be assumed (e.g., 1.5, 2.0 or 2.5 practices). 3. Adoption of at least 1.5, 2.0 and 2.5 practices compared with a lowered number of applied practices Based on the mentioned separation according to the journey configuration and the number of applied practices, these test series consider only data sets which contain at least 1.5, 2.0 and 2.5 practices. An additional differentiation between the planning mode agile and plan-driven is not made in this case, but for journeys in the plan-driven mode the application of 0 practices was assumed. Configuration Practices N Min Max Mean Alaska I < 1.5 41 0 5881.00 3038.47 Alaska I 1.5 28 0 6080.23 4054.95 California I < 1.5 46 0 6053.00 4021.65 California I 1.5 23 0 6128.90 4833.21 Alaska I < 2.0 45 0 5881.00 3133.62 Alaska I 2.0 24 0 6080.23 4045.95 California I < 2.0 47 0 6053.00 4007.51 California I 2.0 22 0 6128.90 4900.31 Alaska I < 2.5 52 0 5881.00 3141.35 Alaska I 2.5 17 2395.10 6080.23 4397.98 California I < 2.5 53 0 6053.00 3973.34 California I 2.5 16 4077.91 6128.90 5348.30 Table 3.7.: Descriptive statistics for variable Business Value related to the filtering by journey configurations and the number of practices Table 3.7 summarizes the obtained results of descriptive statistics calculated for the response variable Business Value. This statistical analysis is based on an overall population of 69 data sets for each journey configuration. Results show the minimum, the maximum and the mean business value for each group. The highest business value for both configurations can be achieved in the sample groups where more practices are applied. 41
4. Adoption of at least 1.5, 2.0 and 2.5 practices for agile planning mode compared to low adoption Besides the separation according to the journey configuration, this test suit considers only data applying the agile planning mode. Comparisons in this regard are made in order to contrast the application of a certain number of practices based on the agile approach. The reasoning here is to see how far adopters of the agile planning mode differ from those that did not properly adopt the agile planning approach. Configuration Practices N Min Max Mean Alaska I < 1.5 7 0 3293.65 1558.32 Alaska I 1.5 28 0 6080.23 4054.95 California I < 1.5 11 0 5272.47 2932.82 California I 1.5 23 0 6128.90 4833.21 Alaska I < 2.0 11 0 5326.00 2485.81 Alaska I 2.0 24 0 6080.23 4045.95 California I < 2.0 12 0 5272.47 2968.16 California I 2.0 22 0 6128.90 4900.31 Alaska I < 2.5 18 0 5326.00 2760.05 Alaska I 2.5 17 2395.10 6080.23 4397.98 California I < 2.5 18 0 5555.00 3213.99 California I 2.5 16 4077.91 6128.90 5348.30 Table 3.8.: Descriptive statistics exclusively for data sets of agile planning mode referring to the filtering by journey configuration and the number of practices The descriptive analysis in this section is based on raw data comprising 35 entries assigned to the Alaska I Journey and 34 to the California I Configuration. The respective test results are depicted in Table 3.8. Participants who perform the Journey of Alaska I or the Journey of California I in the agile planning mode can obtain a higher total business in the group of more applied practices. 5. Comparing data of agile planning mode applying at least 1.5 and 2.0 practices with data of plan-driven planning mode This sub section compares data sets of the plan-driven mode with journeys planned in the agile mode in which at least 1.5 or 2.0 practices are applied. Reasoning for this 42
test is that only those journeys are considered for the agile planning mode which, truly adopted agile principles - i.e., that applied at least a certain amount of agile practices. Configuration Practices N Min Max Mean Alaska I 1.5 28 0 6080.23 4054.95 Alaska I 0 34 0 5881.00 3343.20 California I 1.5 23 0 6128.90 4833.21 California I 0 35 0 6053.00 4363.86 Alaska I 2.0 24 0 6080.23 4045.95 Alaska I 0 34 0 5881.00 3343.20 California I 2.0 22 0 6128.90 4900.31 California I 0 35 0 6053.00 4363.86 Table 3.9.: Descriptive statistics for all plan-driven data assigned to the group of zero applied practices and data in agile planning mode applying at least 1.5 and 2.0 practices respectively Table 3.9 illustrates test results referring to the specified comparisons between journeys following the agile and plan-driven approach. On the basis of a total population of 69 data sets for each journey configuration, half of the data is covered by the plan-driven mode and 28 entries apply at least 1.5 practices with regard to the Alaska I trip. The distribution for the Journey to California I shows 35 valid data entries for the plan-driven mode and 23 data sets in the group agile 1.5. In the second test run at a level of 2.0 practices, 24 data sets are counted for the Alaska I and 22 entries for the California I Journey. By analyzing the obtained results it can be discovered that participants achieve a higher maximum and mean business value performing the agile mode for both journey configurations. 3.3.3.2. Testing for Differences in Business Value In order to investigate whether, the number of adopted practices during planning and executing the journey has any influence on the variable Business Value, significance tests [5] are conducted as already used for testing differences in 3.2. The goal of conducting the significance test is to reject the a priori-determined null hypothesis, which assumes consistency of two samples, in order to accept the alternative which supposes that there is a difference in the mean business values with respect to the number of the applied practices. As such, for each test case a null and an alternative hypothesis were formulated a priori which were subsequently tested statistically. 43
3. Adoption of at least 1.5, 2.0 and 2.5 practices compared with a lowered number of applied practices The test series for investigating a possible difference between the mean business values starts with comparing two groups of data sets, applying on the one hand fewer than - and on the other hand - at least 1.5 practices (2.0 and 2.5 practices respectively). This examination was made for each configuration separately and will be summarized in Table 3.10. The goal of this test suit is to examine the influence of the adoption of a certain numbers of practices irrespective of the applied planning approach. Based on that, the following hypotheses can be formulated: Null Hypothesis H 0,1 : There is no significant difference in mean business values of the Alaska I Configuration applying at least 1.5 practices. Alternative Hypothesis H 1,1 : There is a significant difference in mean business values of the Alaska I Configuration applying at least of 1.5 practices. Null Hypothesis H 0,2 : There is no significant difference in mean business values of the California I Configuration applying at least 1.5 practices. Alternative Hypothesis H 1,2 : There is a significant difference in mean business values of the California I Configuration applying at least 1.5 practices. Null Hypothesis H 0,3 : There is no significant difference in mean business values of the Alaska I Configuration applying at least 2.0 practices. Alternative Hypothesis H 1,3 : There is a significant difference in mean business values of the Alaska I Configuration applying at least 2.0 practices. Null Hypothesis H 0,4 : There is no significant difference in mean business values of the California I Configuration applying at least 2.0 practices. Alternative Hypothesis H 1,4 : There is a significant difference in mean business values of the California I Configuration applying at least 2.0 practices. Null Hypothesis H 0,5 : There is no significant difference in mean business values of the Alaska I Configuration applying at least 2.5 practices. 44
Alternative Hypothesis H 1,5 : There is a significant difference in mean business values of the Alaska I Configuration applying at least 2.5 practices. Null Hypothesis H 0,6 : There is no significant difference in mean business values of the California I Configuration applying at least 2.5 practices. Alternative Hypothesis H 1,6 : There is a significant difference in mean business values of the California I Configuration applying at least 2.5 practices. The results for the Alaska I Journey at the level of 1.5 practices, show that the error probability p of the KS-test for both samples exceeds the value of 0.05 and indicates that the samples are adequately normally distributed. The p-value of the Levene-test shows that the variances of the considered samples are homogeneous. Hence preconditions for conducting the T-test are fulfilled in this case. The obtained p-value of 0.001 (< 0.05) indicates a high statistical significance and leads to the rejection of the null hypothesis H 0,1 at a confidence level of 95%. By contrast, the result of the KS-test shows a deviation from normal distribution within the California I data set in group < 1.5 (0.039 < 0.05). Although the outcome of the Levene-test ensures variance homogeneity (0.358 > 0.05), the result of the T-test wouldn t be meaningful in this case. Alternatively, the U-test is therefore applied in order to compare the mean values of these samples. The obtained p-value of 0.007 (< 0.05) is significant and sufficient in order to reject H 0,2 at a confidence level of 95%. Considering test results at a level of 2.0 practices, the obtained p-value of 0.005 (< 0.05) when conducting the T-test for the Alaska I Configuration can be interpreted as significant and justifies rejecting H 0,3 at a confidence level of 95%. In contrast, an evaluation of the results for the California I Journey shows a deviation from normal distribution (0.030 < 0.05). Consequently, the non-parametric U-test must be applied and results in a p-value of 0.002 (< 0.05). This leads to the rejection of H 0,4 at a confidence level of 95%. Referring to the last level of 2.5 practices in this test series, results are similar compared to the previous outcomes. For the Journey to Alaska I, due to an obtained significance of 0.000 (< 0.05), hypothesis H 0,5 can be rejected at a confidence level of 95%. Hence, for the California I Configuration, the preconditions of normally distributed data (0.014 < 0.05) and variance homogeneity (0.022 < 0.05) are not fulfilled, and the U-test must 45
Configuration Practices N KS-test Levene-test T-test U-test Alaska I < 1.5 41 0.986 0.309 0.001 Alaska I 1.5 28 0.576 California I < 1.5 46 0.039 0.358 0.007 California I 1.5 23 0.332 Alaska I < 2.0 45 0.989 Alaska I 2.0 24 0.461 California I < 2.0 47 0.030 California I 2.0 22 0.293 Alaska I < 2.5 52 0.744 Alaska I 2.5 17 0.867 California I < 2.5 53 0.014 California I 2.5 16 0.978 0.310 0.005 0.285 0.002 0.183 0.000 0.022 0.000 Table 3.10.: Test series examining differences in mean business value when applying at least 1.5, 2.0 and 2.5 practices be conducted as an alternative. With the resulting significant difference of (0.000 < 0.05), hypotheses H 0,6 can be rejected at a confidence level of 95%. 4. Adoption of at least 1.5, 2.0 and 2.5 practices for agile planning mode compared to low adoption The test series in this section evaluates data sets exclusively planned in the agile mode which are grouped by the two journey configurations and the applied number of practices. Table 3.11 depicts the results based on this test specification. The hypotheses to be tested within this examination deal with the impact of adoption of a certain number of practices when applying the agile planning mode. Accordingly, the following hypotheses are formulated: Null Hypothesis H 0,7 : There is no significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 1.5 practices. Alternative Hypothesis H 1,7 : There is a significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 1.5 practices. 46
Null Hypothesis H 0,8 : There is no significant difference in business values for the agile planning mode of the California I Configuration applying at least 1.5 practices. Alternative Hypothesis H 1,8 : There is a significant difference in business values for the agile planning mode of the California I Configuration applying at least 1.5 practices. Null Hypothesis H 0,9 : There is no significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 2.0 practices. Alternative Hypothesis H 1,9 : There is a significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 2.0 practices. Null Hypothesis H 0,10 : There is no significant difference in business values for the agile planning mode of the California I Configuration applying at least 2.0 practices. Alternative Hypothesis H 1,10 : There is a significant difference in business values for the agile planning mode of the California I Configuration applying at least 2.0 practices. Null Hypothesis H 0,11 : There is no significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 2.5 practices. Alternative Hypothesis H 1,11 : There is a significant difference in business values for the agile planning mode of the Alaska I Configuration applying at least 2.5 practices. Null Hypothesis H 0,12 : There is no significant difference in business values for the agile planning mode of the California I Configuration applying at least 2.5 practices. Alternative Hypothesis H 1,12 : There is a significant difference in business values for the agile planning mode of the California I Configuration applying at least 2.5 practices. 47
As the preconditions of normal distribution and homogeneous variances are fulfilled for the Alaska I Configuration at a level of 1.5 practices, the conducted T-test results in a p-value of 0.000 (< 0.05). Due to this statistical significance, H 0,7 can be rejected at a confidence level of 95%. In contrast, the Levene-test for California I shows variance heterogeneity (0.048 < 0.05), which requires application of the U-test. The obtained p-value of 0.001 (< 0.05) indicates a statistically significant difference and leads to the rejection of H 0,8 at a confidence level of 95%. To test for differences at a level of 2.0 practices, as a precondition for conducting the T- test, are fulfilled for both configurations. Results show a statistical significance of 0.003 (< 0.05) for Alaska I and a p-value of 0.001 (< 0.05) for California I. Accordingly, H 0,9 and H 0,10 can be rejected at a confidence level of 95%. Analyzing results at a level of 2.5 practices shows that for testing differences in the mean business values, the U-test must be conducted for both journeys. With an obtained significance of 0.001 (< 0.05) for Alaska I and of 0.000 (< 0.05) for California I, hypotheses H 0,11 and H 0,12 can be rejected for the Configurations Alaska I and California I at a confidence level of 95%. Configuration Practices N KS-test Levene-test T-test U-test Alaska I < 1.5 7 0.987 0.952 0.000 Alaska I 1.5 28 0.576 California I < 1.5 11 0.238 0.048 0.001 California I 1.5 23 0.332 Alaska I < 2.0 11 0.975 Alaska I 2.0 24 0.461 California I < 2.0 12 0.146 California I 2.0 22 0.293 Alaska I < 2.5 18 0.707 Alaska I 2.5 17 0.867 California I < 2.5 18 0.067 California I 2.5 16 0.978 0.097 0.003 0.071 0.001 0.016 0.001 0.002 0.000 Table 3.11.: Test results of comparing only the agile data set divided by journey configuration and number of applied practices 48
5. Comparing data of agile planning mode applying at least 1.5 and 2.0 practices with data of plan-driven planning mode Within this test case a comparison was made which comprises the detection of differences between journeys planned in an agile manner where at least 1.5 and 2.0 practices were applied and data entries assigned to the group using the plan-driven planning mode. The respective results are summarized in Table 3.12. In order to investigate whether the adopted planning approach has an impact on the response variable Business Value, appropriate hypotheses are postulated: Null Hypothesis H 0,13 : There is no significant difference in business values between the agile mode applying at least 1.5 practices and the plan-driven mode of the Alaska I Configuration. Alternative Hypothesis H 1,13 : There is a significant difference in business values between the agile mode applying at least 1.5 practices and the plan-driven mode of the Alaska I Configuration. Null Hypothesis H 0,14 : There is no significant difference in business values between the agile mode applying at least 1.5 practices and the plan-driven mode of the California I Configuration. Alternative Hypothesis H 1,14 : There is a significant difference in business values between the agile mode applying at least 1.5 practices and the plan-driven mode of the California I Configuration. Null Hypothesis H 0,15 : There is no significant difference in business values between the agile mode applying at least 2.0 practices and the plan-driven mode of the Alaska I Configuration. Alternative Hypothesis H 1,15 : There is a significant difference in business values between the agile mode applying at least 2.0 practices and the plan-driven mode of the Alaska I Configuration. Null Hypothesis H 0,16 : There is no significant difference in business values between the agile mode applying at least 2.0 practices and the plan-driven mode of the California I Configuration. 49
Alternative Hypothesis H 1,16 : There is a significant difference in business values between the agile mode applying at least 2.0 practices and the plan-driven mode of the California I Configuration. Configuration Practices N KS-test Levene-test T-test Alaska I 1.5 28 0.576 Alaska I 0 34 0.995 0.823 0.016 California I 1.5 23 0.332 California I 0 35 0.205 0.962 0.182 Alaska I 2.0 24 0.461 Alaska I 0 34 0.995 California I 2.0 22 0.293 California I 0 35 0.205 0.796 0.025 0.806 0.132 Table 3.12.: Comparison between agile data sets applying at least 1.5 and 2.0 practices with data of plan-driven mode The results for Configuration Alaska I and California I show that preconditions for conducting the T-test are fulfilled in all test runs (i.e., KS-test shows normally distributed data and the Levene-test confirms variance homogeneity). For Alaska I, the T-test at a level of 1.5 practices results in a statistical significance of 0.016 (< 0.05) and leads to the rejection of hypothesis H 0,13 at a confidence level of 95%. In contrast, the test result for California I shows a p-value of 0.182 (> 0.05) which means no significant differences in business values. Based on that, H 0,14 cannot be rejected at a confidence level of 95%. At a level of 2.0 practices, the T-test shows for the Alaska I Journey a statistical significance of 0.025 (< 0.05) which leads to the rejection of H 0,15 at a confidence level of 95% in this case. However, for the California I Configuration a p-value of 0.132 (> 0.05) can be obtained. Therefore, hypothesis H 0,16 cannot be rejected at a confidence level of 95%. 3.3.4. Discussion of the Results Within the data analysis the application of practices was investigated in combination with several configurations in filtering. The focus of the discussion are the test cases that result in a significant difference regarding business values. The major finding of the data analysis in this chapter is that the adoption of a different number of practices has a statistically significant impact on the total business value of a journey. Comparisons of the adoption of at least 1.5, 2.0 and 2.5 practices with a 50
respectively lower number of applied practices in the context of test 3 and 4 revealed that achieving a higher business value is very much related to the number of applied practices. When examining the adoption of at least 1.5, 2.0 and 2.5 practices, results of the T-test and U-test clearly demonstrate a significant difference in mean business values. In the third test, however, only partial support could be obtained. In comparisons between agile data with an application of at least 1.5 and 2.0 practices and data in plan-driven mode the T-test results, in a significant difference in mean business values (cf. 3 rd Test) for the Alaska test cases, but not for the California I test cases. When interpreting results of the data analysis, a possible argument might be that the consideration of agile principles described in Section 2.5 leads to an optimization of the business process and supports an efficient use of limited resources which results in a higher business value at the termination of the journey. For example, the elimination of waste avoids idle times in process execution. If the stipulated time frame for planning the journey is regarded as scarce time resource, this practice fosters an efficient exploitation of time. In addition, the number of adopted practices can be interpreted as a performance indicator and measurement for agile adoption. The number enables insight into the individual performance of the participants according to their experience and knowledge in modeling. Planning and executing journeys in the agile mode is more challenging. The greatest difficulty lies thereby in the most efficient application of the practices. An example for this would be making right decisions at the right moment in the journey, when bookings of accommodations, guided tours and exhibitions are necessary. Furthermore, elimination of waste, creation of knowledge and the creation of options in order to work against uncertainties are practices which should be applied to fully exploit the benefits of the agile planning mode. Compared to the plan-driven mode, the user is able to influence the planned journey also at execution time. In contrast the plan-driven approach requires the complete planning of the journey path in advance, because there is no possibility of change during execution, except if some uncertainties arise which cause any destruction of the journey plan. The explanation for the obtained test results of California I in Table 3.12 might be a better performance of participants applying the plan-driven approach. According to empirical findings from experiments in [17] the plan-driven approach is easier to handle for inexperienced users. Another reason might be that flexibility while performing the California I Configuration is not as required as during planning and executing the Journey to Alaska I due to different journey characteristics (cf. Section 3.1.2). For the 51
remaining test cases, a statistically significant result was obtained especially if at least 1.5 practices are adopted while planning and executing the journey. Summarizing the findings with reference to the initially defined research question (cf. Section 3.3.1), it can be said that applying a higher number of agile practices will be paid off. Considering the impact of the number of applied practices on the business value, generally it could be observed that a high agile adoption outperform a low agile adoption, except in the two cases of the third test. Test results further show that a higher agile adoption is usually better than the plan-driven approach with reference to the total business value. For sure, this depends on the degree of uncertainty in the business process. Therefore, it may be concluded that, if the business process is characterized by low uncertainty the application of the agile planning approach might be less essential. 52
4. Dealing with Events The goal of evaluations in this chapter is to investigate the impact of events on the process execution, which challenges are tied to the events occurrence, and which common practices and pitfalls in handling can be identified (cf. RQ 3). Section 4.1 describes experimental settings and characteristics of the data sets. The research questions and the following proceeding are defined in Section 4.2. In addition, this section comprises the data analysis of events and a descriptive evaluation of event handling by the user. This section closes with a discussion of the evaluation of events. 4.1. Setup The analysis in this chapter is based on three different journey configurations which have the occurrence of uncertain events in common. Here, the largest proportion of data originates from the Configurations California I and Alaska I, which have already been illustrated in Chapter 3. Further data sets can be obtained from experiments applying Configurations of California II and Alaska II. Finally, evaluations are given for data sets originating from California III and Alaska III. The experimental process, characteristics of configurations and differences between them are described in the following. 4.1.1. Experimental Setting The largest proportion of data originates from data sets I and II (i.e., California I and Alaska I) with 75 data entries each. Data sets I and II are already described in Chapter 3 and will not be explained again. Furthermore, 50 data sets in common can be obtained from experiments applying Configuration California II and Alaska II. The data produced by Configurations California III and Alaska III amounts to 60 entries, of which only half can be evaluated because the journey based on the factor level 1 of this configuration does not show any occurrence of uncertainties during the execution. 53
4.1.1.1. Data Sets III and IV (i.e., California II and Alaska II) Data sets from these configurations are the result of experiments conducted at the University of Technology at Eindhoven in October 2008 [22]. Test persons were 25 students in the Business Process Management course. The goal for conducting experiments on the basis of California II and Alaska II is to investigate effects arising from unforeseen events on the process outcome. The latter is measured in terms of the business value and the number of aborted journeys. For this purpose, two different factor levels were defined: low and high. Variant A of the configuration contains only few constraints which corresponds to the factor level low and variant B shows the high-factor level, with many constraints. In this case, the types of constraints in variant A build a true subset of constraints in variant B. The term, constraints in this regard refers to the selection of activities (e.g., mutual exclusion), the ordering of plan items (e.g., pre- and post-conditions) and resource constraints such as budget limitation. As regards the preparation and the execution of the experiments, the course of actions is the same as for experiments of the data sets I and II (cf. Section 3.1.1). After an introductory lecture, the actual experiment was performed. The experimental design contains a division of test persons into two groups of 12 persons, and one of 13. The experiment was conducted in two subsequent runs, where Group 1 planned and executed Configuration California II with few constraints, and the second group was assigned to the Configuration California II where users have to comply with many constraints. In the second run, requirements are changed. Group 1 is now challenged with many constraints in the Configuration Alaska II. In contrast, Group 2 has to cope with only few constraints in the Configuration California II. For these experiments, only the agile planning approach (cf. Section 2.4.2) was performed. Figure 4.1 gives a graphical summary of the design of the experiments carried out at Eindhoven. Group 1 n/2 Participants Factor Level 1: Few Constraints Configuration CALIFORNIA II Group 1 n/2 Participants Factor Level 2: Many Constraints Configuration ALASKA II Group 2 n/2 Participants Factor Level 2: Many Constraints Configuration CALIFORNIA II Group 2 n/2 Participants Factor Level 1: Few Constraints Configuration ALASKA II First Run Second Run Figure 4.1.: Experimental design of test runs at Eindhoven, based on [23] 54
4.1.1.2. Data Sets V and VI (i.e., California III and Alaska III) The aim of experiments based on the Configurations California III and Alaska III was to investigate the impact of uncertainties on processes, and determine the extent to which process outcomes are affected as measured against the resulting business value and the number of failed journey executions. For this purpose, 20 students attending the student program of Management, Communication and IT at the Management Center in Innsbruck were test persons in experiments conducted in December 2008 [19]. The experimental design and the process of performing the experiments are similar to the ones in the configurations mentioned above (cf. Section 3.1.2). The setup for these experiments is adjusted in order to plan and execute a declarative process based on a journey metaphor applying only the agile approach. Here, components of this configuration include plan items, constraints that are restricting the combination of the items, and weather conditions that influence the activities reliability. There are two different variants in this configuration. The distinction is in the factor level, which refers in this regard to the number of events occurring during run-time. Here, the levels no events and many events can be distinguished. According to this classification, factor level 1 contains no events, while many events is assigned to the second level. A description of the design in the execution of the experiments is provided in Figure 4.2. It can be remarked that participants performing factor level 1 are only challenged with planning and executing the journey plan and the creation of options for activities with low availability and low reliability, in order to mitigate their unavailability during execution. In contrast, factor level 2 requires the treatment of unforeseen events which have an influence on certain activities in addition. Further differences in configuration settings are briefly highlighted in Table 4.2 below. Group 1 n/2 Participants Factor Level 1: No Events Configuration CALIFORNIA III Group 1 n/2 Participants Factor Level 2: Many Events Configuration ALASKA III Group 2 n/2 Participants Factor Level 2: Many Events Configuration CALIFORNIA III Group 2 n/2 Participants Factor Level 1: No Events Configuration ALASKA III First Run Second Run Figure 4.2.: Design of experiments conducted at the Management Center in Innsbruck [20] 55
4.1.2. Characteristics of Journey Configurations In the following, Journey Configurations II and III are compared along the elements of the Meta-Model (cf. Section 2.1.1). 1. Configuration California II and Alaska II Based on the core concepts of the Alaska Meta-Model (cf. Section 2), this section compares the Journey Configurations California II and Alaska II. The journey configurations were performed in order to investigate the outcome of executing a business process by applying the declarative approach. In addition, compliance with a varying number of constraints must be ensured [23]. Table 4.1 provides details related to the configurations. With regards to the available number of plan items, Alaska II has slightly more activities. In California II, users can choose from 15 activities, whereas Alaska II contains 21 activities. Also, the number of available routes is different in the configurations. In contrast, the number of available accommodations is the same in both configurations. Furthermore, the time frame in the AST shows a different number of days for planning the journeys. In California II, three days are available for planning, which is extended to four days in Alaska II. The different time frames result in a different number of available plan items in the configurations. The intention of performing journeys with Configuration I is mainly to investigate the influence of a varying number of constraints on the execution of the business process. Therefore, these journeys exhibit a higher number of constraints compared to the journey configurations in Section 3.1.2. All constraints in California I are contained in California II. In addition, two more constraints have to be considered in California II. The same can be stated when comparing Alaska I and Alaska II. Alaska II has constraints such as mandatory actions, prerequisite conditions and postconditions, in addition. Comparing California II and Alaska II, California II contains fewer constraints. Another difference between the configurations refers to the number of possible events. The execution of California II might be interrupted by the occurrence of three events; whereas in Alaska II, four events might happen. Here, all events have an influencing character. 56
Due to these occurrences of events and the different numbers of constraints, it can be concluded that the two configurations are also different in their degree of difficulty. As already mentioned in the initial descriptions of journey configurations (see Section 3.1.2), planning and executing the Configuration Alaska is a greater challenge for users. Besides, higher amounts in business value can be achieved performing Alaska I. California I Alaska I Number of Plan Items: 22 26 Accommodations 3 3 Activities 15 21 Tours 4 2 Type of Constraints Times of action s execution Endpoint of journey Budget limitation Mandatory actions Minimum delay between activities Times of action s execution Mutual exclusion of actions Endpoint of journey Budget limitation Mandatory actions Prerequisite conditions Postconditions Number of Events 3 4 Name of Events Horrible Physical Conditions Traffic Jam Seals Spotted Table 4.1.: Comparison of California II and Alaska II Exciting Exhibit Traffic Jam Whales Spotted Grizzly Bears 2. Configuration California III and Alaska III In the following, general comparisons between California III and Alaska III are made, which are summarized in Table 4.2. As regards the included number of plan items for these configurations, it can be seen, that California III exhibits considerably fewer items. Differences are in the number of activities, where users can select in California III from 15 available items, and in Alaska III from 22. The number of possible routes in the configurations is slightly different as well. However, the number for accommodations is the same. However, the focus in these two configurations is event handling. Users have to comply with certain constraints in the journey. Here, the number of constraints varies between 57
the configurations. Comparing the types of constraints, the constraints in California I are nearly all found in the Alaska II Configuration. In order to evaluate events, two different factor levels are defined which are integrated in variant A and B. Here, for both California III and Alaska III, variant A exhibits no events at all, whereas in variant B up to five events might occur. Concerning the event type, all events have influencing behavior (cf. Section 4.2.4). Comparing events of California II and California III shows that events of California II are a true subset of California III. Besides, the events Accident and Road Closure can occur. However, Alaska III only contains some events of Alaska II. The events Car got Stuck and Equipment Problems might occur in Alaska III in addition. California II Alaska II Number of Plan Items: 22 27 Accommodations 3 3 Activities 15 22 Tours 4 2 Type of Constraints Times of action s execution Endpoint of journey Budget limitation Mandatory actions Minimum delay between activities Times of action s execution Mutual exclusion of actions Endpoint of journey Budget limitation Mandatory actions Prerequisite conditions Postconditions Number of Events 5 5 Name of Events Accident Road Closure Traffic Jam Seals Spotted Horrible Physical Conditions Table 4.2.: Comparison of Journey Configurations III Car got Stuck Traffic Jam Equipment Problems Exciting Exhibit Grizzly Bears 58
4.2. Evaluation of Events The following sections describe some necessary pre-workings for evaluation, and finally the actual analysis of events. Compared to the previous evaluations in this thesis, analysis in this case is conducted without statistical tool support; rather, it is descriptive in nature. 4.2.1. Research Question Preceding analysis in [19] shows the result that unforeseen events have a statistically significant impact on the outcome of journeys, measured in terms of the response variable Business Value and the number of failed journeys. In particular, this section investigates the impact of the occurrence of exceptional situations on the user s success when planning and executing the process of journey planning. For this reason, the research questions below were formulated: RQ 3: How does the occurrence of events influence planning and executing a journey? 1. What kind of events might occur? 2. How do users handle these events during business process execution? In order to answer these questions, an analysis has to be made with respect to the characteristics of events of the configuration variants in general. Based on that, challenges and appropriate measurements for mitigation can be derived. Further analysis investigates handling of uncertainties by the user. Section 4.2.4 provides analysis in this context. 4.2.2. Proceeding A general classification schema for events is first created here as preparation for the analysis of events. In general, a differentiation between the influencing events and events which are launching new activities is made. Based on these effects, it is possible to derive challenges, causal factors and potential repairs for each event type. In the next step, the events of the journey configurations (i.e., California and Alaska I-III) are classified according to the defined schema. Subsequently, the frequencies of occurrence for each event are evaluated during the replay of each record. An overview of the obtained 59
frequencies is provided in the Appendix A.2. Simultaneously, the handling of events by users is investigated while replaying. The analysis is closed with a discussion where common practices and pitfalls referring to the event handling are summarized. The proceeding is summarized in Figure 4.3. Definition of Classification Criteria Classification of Events Investigation of Event Frequencies Investigation of Event Handling Discussion of Results (Common Practices & Pitfalls) Time Figure 4.3.: Graphical view of process sequence within the evaluation of events The proceeding for analyzing events is significantly different from the proceeding in the preliminary evaluations in Chapter 3. The different proceeding was particularly elected because of the aggregation of multiple journey configurations with different journey settings (e.g., number of events or constraints). Besides, the experimental setups focus on different goals such as the effect of different planning modes (cf. Configurations I), constraints (cf. Configurations II), or events (cf. Configurations III) on the process outcome. Because, all journey configurations exhibit unforeseen events in their execution, data sets can be used for evaluation in this chapter. Besides, statistical analysis cannot be made in this regard because not all data sets are captured in SPSS. Instead, the data analysis is of a descriptive nature, comprising spreadsheets where appropriate investigations are recorded. 4.2.3. Classification Criteria for Events Evaluations in this chapter require a proper specification of events. Basically a classification is made according to the event type. Depending on the type, events have different characteristics. Accordingly, the occurrence of the event causes different effects (e.g., destruction of the plan). Here, created effects signify challenges with which participants are confronted during the execution of their planned journey. Besides, causal factors can be assigned which favor the effect that has arisen. These factors comprise mainly observations referring to the planning attitudes and participants behavior during execution. Depending on the created effect, a certain effort for reparation becomes necessary. Analysis in this regard investigates which methods for reparations are used and the effort required to conduct them, measured as number of steps in the journey trace. The 60
following subsections provide a classification according to the event type. Figure 4.4 gives a graphical summary of associations of general classification criteria. 4.2.3.1. Events Influencing Activities Generally, two types of events could be identified during the evaluation of different configuration variants. One type signifies an influence on a respective activity. In this case, either the activity s business value, duration or availability is influenced. Subsequently, possible effects and assigned characteristics of influencing events are highlighted. Figure 4.4 provides a graphical summary on this. No Effect: The ineffectiveness of events is the most harmless effect. During the analysis, several reasons could be detected why the occurred event might have no effect at all: The main reason is because the corresponding activity is not considered in the journey plan and therefore the event has no relevance. Secondly, it might also happen that the relevant activity is deleted at the same time the event occurred. Participants might have simply luck and the event occurs at a later point in time when the critical activity is already executed. The user might have planned a second activity in parallel to the critical activity. In case the event occurs the affected activity can be deleted and the alternative executed instead. The application of agile planning provides flexibility in order to mitigate for example, the effect of destruction due to unavailability of activities. While planning and executing activities step by step participants can decide spontaneously to not consider the critical activity in case the event occurs. Another causal factor is that participants have considered enough buffer between the influenced and the subsequent activities which compensates the overtime in duration, so that no effect arises. Besides, an favorable phenomenon could be observed in this context: due to the sequential arrangement of the activities planned for one day, mostly there remains some buffer in the time frame at the end. If the 61
duration of the last activity is influenced by an event, this buffer might mitigate caused delays. Increase in Business Value: This effect can be observed for influencing events as well as for events launching new activities. Increasing Business value is the only effect that influences activities in a positive way. Challenge in this regard is the consideration of the activity that is affected by the event. Thereby the relevant activity needs to be considered in the journey either within the planning phase or during execution by agile planning. Decrease in Business Value: This is the opposite of the effect described before which just occurs due to events of influencing character. This event causes a negative impact by reducing activity s business value. Thereby, the user has to overcome a longer duration of the influenced activity. In this case the specific activity is planned at the end of a day in the journey, where no or not enough buffer is between the subsequent accommodation. This causes a conflict at the starting time of the accommodation item. Thus, the respective items are at the end of the journey day there is no possibility for mitigating the temporal bottleneck. Consequently the accommodation will be executed with a decreased expected business value. Unsuccessful Termination: Compared with the effect of decreasing business value, this effect has similar challenge and causality. The occurrence of the event causes a delay in the duration of the activity which needs to be compensated. Thus, the user has planned the affected activity at the very end of the journey where no buffer for possible overtime can be considered, the negative effect of unsuccessful termination arises. In this case no reparation effort is possible anymore. Therefore, the affected activity is executed and the journey execution terminates with an error message in this regard. Destruction of Journey: The main causal factor that the occurrence of an influencing event might destroy the journey plan is the insufficient buffer between single activities in the plan. Most journeys have a tight scheduling what favors the appearance of this effect. If the occurring event causes a delay in the activities duration this might lead to an overlapping between the ending and starting point of the subsequent activity. The caused destruction is the only effect which requires measures for reparation. Shifting of the subsequent activity to a later starting point would solve this situation in the 62
simplest case. In some cases the cancellation of activities is inevitable. If there are any reservations made for the canceled activity, cancellation fees are charged in addition. After cancellation some users try to insert alternative activities in order to avoid large idle times in the journey. Only in a few cases test persons cancel the whole execution due to the occurred event and the created effects. Another challenge that needs to be mitigated in order to avoid the effect of destruction is a caused unavailability of activity. Generally, the concerning activity has to be considered in the journey plan. Furthermore, the relevant activity can be planned in parallel. Planning activities in parallel is in and of itself a method which provides flexibility during execution, especially in order to mitigate unavailability of activities. However not, in case decisions are made too early in this regard and the critical activity will be chosen instead. Reparations are undertaken through canceling booked and not booked activities and shifting of the subsequent ones. Unavailability can also be favored when no or not enough buffer is considered between single activities. If the affected activity shows an overtime in its duration an overlapping with the subsequent activity might be the consequence. Shifting can not always mitigate this problem because there might be restrictions on starting points of activities. This might lead to the unavailability of activities planned sequentially after the affected plan item. Reparations contain similar methods as mentioned before. In most cases the cancellation of the unavailable activity is required. Shifting and inserting alternatives might reduce idle times. 4.2.3.2. Events Launching New Activities The second event type causes the launch of new activities during the execution. Effects that are created are either non-effective or increasing business value. Compared to influencing events, this type creates no significant interruption during journey execution. Therefore, the handling of newly introduced activities is not as sophisticated because their consideration is optionally for a successful journey termination. Challenges and causal factors which are related to the launch will be indicated in the following. Identified characteristics are summarized in the Figure 4.5 for this event type. No Effect: The only and main challenge of launching new activities during execution of the journey is their consideration in the plan within the determined slow-down time. Thus NewActivityEvents create no destruction of the journey plan, the user is not so much concerned to integrate the newly launched activities in the journey. Due to the 63
Effects of Events Challenges Causal Factors Effort for Reparation activity deleted while event arises not relevant luck no effect activity planned in parallel agile planning at run-time enough buffer activity planned at the end increase in business value consideration of activity in journey activity considered in plan agile planning at run-time decrease in business value too long duration of activity no/ not enough buffer activity planned at the end unsuccessful termination too long duration of activity no/ not enough buffer activity planned at the end plan destroyedrepairable unavailability of activity overlapping of activities in journey activity planned in parallel activity considered in plan no/ not enough buffer cancel booked activities cancel activities shift activities insert activities cancel execution Figure 4.4.: Summarizing event characteristics which are set in relation identified for the influencing type of event 64
irrelevance, the occurrence of these events leads to no effect at all in the journey execution. Increase in Business Value: However, some participants are concentrating on the consideration of the new activated items during execution in the journey. While applying the agile planning approach they successfully manage the challenge of consideration. The implementation of the activity will be awarded with an increase in the business value. Effects of Events Challenges Causal Factors no effect consideration of activity in journey not relevant increase in business value consideration of activity in journey agile planning at run-time Figure 4.5.: Summarizing event characteristics which are set in relation identified for the launching type of event 4.2.4. Data Analysis The analytical process starts with a characteristic specification of single events of the different journey configurations in Section 4.2.4.1. Based on this, Section 4.2.5 provides a descriptive evaluation referring the treatment of uncertainties during the journey process. 4.2.4.1. Classification of Events According to the Alaska Simulator Meta-Model (cf. Section 2) either no events or any number of events might be component of the Travel Itinerary. The occurrence of the event can be at any number of Locations in a journey. Further, an event can be distinguished between NewActionEvents and ChangeActionEvents which both refer to an Atomic Activity. The latter event type might change an activity referring to its availability, business value or duration, whereas the NewActionEvent launches an activity 65
for optional selection. In the following for each configuration, events that might interrupt journey execution are described on the basis of the Meta-Model. Additionally, a classification according to the criteria defined in Section 4.2.3 are made. Depending on the event type, caused effects and challenges for the user are identified. Furthermore, causal factors which are favoring the effect of a specific event are investigated and efforts for reparation are indicated. 1. Events in California I During the execution of California I, participants are challenged with handling the effects of four different events. According to the specified characteristics, events of this configuration can be classified as two events are launching new activities whereas the other two show an influencing behavior. Thereby three out of four events create the effect of increasing business value which is not as sophisticated in its treatment comparing to the caused destruction in the journey plan through unavailability of activity. The Table 4.3 provides an overview of event characteristics of the California I Configuration. Due to the effect of launching new activities at the location of occurrence, the events Jet Boat and Flightseeing S.F. Bay Area can be classified as NewActivityEvent. The main challenge presented by this type of event is to consider the new activity in the journey plan during execution. This can be realized by applying agile planning and will be awarded with an increased business value in case of success. Thus, these two events have no expiration time, the user can select the newly launched activities anytime during the stay at the specific location. The remaining two events in California I have an influencing characteristic, but differ in their effects they cause. The event Elephant Seals in Monterey increases the expected business value in case the user has considered the respective activity in the journey plan. The challenge lies in the consideration of the respective activity within the journey, but not before the point of the event s appearance. This event has no slowdown time and holds up until the completion of the execution. On the contrary, the event Tioga Pass Closed influences the route between Lee Vining and Yosemite. Due to heavy snow fall, this pass is not traversable anymore. The caused unavailability is the most challenging situation among the four events in California I, because if the user has planned the respective route, he is forced to find an alternative way to the desired destination which requires replanning during run-time. This event might affect the 66
Event Name Description Expiration Time New Activity Event Change Activity Event Availability Business Value Jet Boat Activity Jet Boat is available at location Lake Tahoe. Elephant Seals The expected business value for the activity Elephant Seals in Monterey increased. Tioga Pass Closed The Route Yosemite - Lee Vining is not available. Flightseeing S.F. Bay Area The Activity Flightseeing S.F. Bay Area is available in San Francisco. never never never never X Location Lake Tahoe Monterey Atomic Activity Jet Boat Trip X X Elephant Seals X X Yosemite - Lee Vining Lee Vining - Yosemite Table 4.3.: Description of Events in California I X San Francisco Flightseeing S.F. Bay Area journey plan seriously and might require a certain amount of reparation effort through canceling the unavailable activity and inserting or shifting alternatives. In contrast, the NewActivityEvents don t compromise the successful termination of the journey at all. 2. Events in Alaska I Compared to the Configuration California I, test persons performing Alaska I might be confronted with the occurrence of up to five events during execution (cf., Table 4.4). The classification related to the event type counts, in this case, two events of influencing character which result in destruction of the journey plan due to unavailability of activities. Another influencing event results in the positive effect of increasing the business value. The two remaining events launch new activities. The event Invitation to Flightseeing has the same characteristic as the event Flightseeing S.F. Bay Area in California I. The second NewActivityEvent in 67
Event Name Description Expiration Time New Activity Event Dogs Sick Activity Long Dog Sledge Ride in Fairbanks is not available anymore. Grizzlies The expected business value of the activity Bus Tour in Denali increased permanently. Invitation to Flightseeing A new action is available in Glenallen : Invitation to Flightseeing. Moose Hunting The Activity Moose Hunting is available in Tok. Camping Equipment Stolen Activity Wilderness Camping in Denali is not available anymore. never never never never never Change Activity X X X Event Availability X X Business Value X Location Atomic Activity Fairbanks Long Dog Sledge Ride Denali National Park Bus Tour X Glenallen Flightseeing to Wrangell St. Elias Mountains X Tok Moose Hunting Denali National Park Wilderness Camping Table 4.4.: Description of events in Alaska I Alaska I is the event Moose Hunting. The challenge for NewActivityEvents refers to the user s flexibility to consider newly launched activities by replanning within the journey during run-time which will be rewarded with an increased business value. Because, both events have no slow-down time in their duration, the consideration is possible from occurrence until the termination of the journey. As regards the ChangeActivityEvents, the events Dogs Sick and Camping Equipment Stolen show an influencing behavior referring to the activity s availability. If the user has planned the critical activities, they can become unavailable due to these events. The ad hoc unavailability of activities challenges participants in immediately finding alternatives while applying agile-planning. Hence, the occurrence of the event might affect the planned journey seriously through a destruction which needs to be repaired in order to terminate the execution successfully. Because, the atomic activity Wilderness Camping which might be influenced by the second mentioned ChangeActivityEvent requires a special treatment anyway due to the attributes of low availability and low reliability, the creation 68
of options would provide additional protection against unavailability in this regard. In contrast, the influencing event Grizzlies leads to an increase in business value of the relevant activity at best. In case the respective activity was not considered during the planning phase in the journey, the user has the possibility to make up for this by agile-planning during run-time, as the event exhibits no slow-down time. 3. Events in California II While applying the Configuration California II, three different events might happen. The classification according to the type of event shows that all three possess influencing character. However, differences are due to the effects which result. Compared to influencing events in the Journey Configurations I, where users are mainly challenged with the unavailability of activities, events in California II affect activities durations. Therefore, the focus is placed on the factor time and the consideration of sufficient buffer between activities. The events Horrible Physical Condition and Traffic Jam listed in Table 4.5 cause a delay in the planned duration of the affected activities. The users are challenged with mitigating the resulting delay and possibly while compensating an overlap with subsequent activities. In order to avoid this negative effect, the consideration of enough before the next activity in the plan, or the application of agile planning during runtime, are essential measures. Otherwise, users are instructed to replan the journey trace through shifting or deleting subsequent activities. Both events possess an expiration time which varies from journey to journey. The third event, Seals Spotted, causes the positive effect of increasing business value in addition to the delay. 4. Events in Alaska II Compared to the Configuration California II, which has the same experimental setup, the same event type could be assigned for events in Alaska I. Further, the event Traffic jam was identified as a common event. A difference is that users performing Alaska II might be confronted with up to four events. Influencing events in Alaska II are summarized in Table 4.6. As regards the individual characteristics, the events Whales Spotted and Grizzly Bears are similar. Both results in an increase in business value if the relevant activity is integrated into the journey plan. The challenge is especially that the activities concerned are considered in the journey plan within the event s expiration time. Besides, the occurrence of the event 69
Event Name Description Horrible Physical conditions It took you longer to do the hike than one could have expected. Traffic Jam You get stuck in a traffic jam and get delayed. Seals Spotted You could see seals in the ocean. This leads to an increase of the business value, but also to delays. Expiration Time varied varied varied Change Activity Event X X X Business Value X Duration X X X Location Atomic Activity San Francisco Muir Wood - Dip Sea Trail San Francisco - Pinnacles San Francisco - Pinnacles Table 4.5.: Description of events in California II San Francisco Ocean Beach Exciting Exhibit shows a positive and simultaneously a negative effect. It can be viewed as positive that the consideration of the relevant activity will be rewarded with an increase in business value. The negative aspect is the resulting delay. Consequently, the user is challenged twofold: on the one hand, the preconditioned activity should be considered in the journey within the expiration time; on the other side, there should be sufficient buffer before the subsequent activity. 5. Events in California III While planning and executing California III, the user might have to handle the occurrence of up to five events. The classification according to the event type leads to the finding that all events have influencing characteristics. All events of California II are components of California III. Additionally, the events Road Closure and Accident might occur, which are described in the following (cf. Table 4.7). If the event Road Closure happens, the preconditioned atomic activity becomes unavailable in the journey. The user is challenged to find alternatives through replanning. This event requires, in most cases, high reparation effort and the application of agile 70
Event Name Description Expiration Time Traffic Jam Due to a traffic jam in Anchorage you arrive at your destination 30 minutes delayed. Exciting Exhibit As the exhibit was very exciting you stayed longer than planned. However, this also resulted in an increase in the business value. Whales Spotted You could spot several humpback whales very close. This led to an increase in the business value. Grizzly Bears Grizzly bears crossed the road just in front of the bus which allowed you to take excellent pictures. This leads to an increase in the business value. varied varied varied varied Change Activity Event X X X X Business Value X X X Duration X X Location Atomic Activity Anchorage - Seward Anchorage - Seward Seward Visitor Center Kenai Fjords Seward Columbia Glacier Cruise Table 4.6.: Description of events in Alaska II Denali National Park Bus Tour To Wonder Lake planning. The measurements for mitigating unavailability cancel the affected activity and insert or shift respective activities. In contrast, the occurrence of the event Accident only results in a delay in the expected duration. The difficulty is to balance a resulting overlap or in the worst case to handle unavailability of the overlapping activity due to the longer duration. In the simplest case shifting would be an essential reparation method in order to compensate insufficient buffer. Otherwise the cancellation of activities might become necessary. Both events have an expiration time which differs in each journey. 6. Events in Alaska III The events in Alaska II, except the event Exciting Exhibit are components of the Configuration Alaska III. Furthermore, the two influencing events Your car got stuck and Equipment Problems might occur during execution (cf. Table 4.8). Both events create the effect of destruction due to a delay in the expected duration of the preconditioned activities. If the user has not considered enough buffer before 71
Event Name Description Expiration Time Horrible Physical conditions It took you longer to do the hike than one could have expected. Traffic Jam You get stuck in a traffic jam and get delayed. Seals Spotted You could see seals in the ocean. This leads to an increase of the business value, but also to delays. Road Closure The road between Napa Valley and Sonoma Valley is closed for 5 hours. Accident You had an accident on the way. It took you some time to call the insurance company. varied varied varied varied varied Change Activity X X X X X Event Availability X Business Value X Duration X X X X Location Atomic Activity San Francisco Muir Wood - Dip Sea Trail San Francisco - Pinnacles San Francisco - Pinnacles San Francisco Ocean Beach Napa Valley - Sonoma Valley Napa Valley - Sonoma Valley San Francisco - Napa Valley San Francisco - Napa Valley Table 4.7.: Description of events in California III the subsequent activity, these events might cause overlapping, or unavailability of the following activity. In this case, reparation requires replanning through shifting, canceling or inserting alternatives. 4.2.5. Descriptive Evaluation of Events The following descriptive report starts with an investigation of the frequency of occurrence of the events for each journey configuration. A graphical summary is provided in Appendix A.2. Furthermore, each event which occurred is analyzed with regard to the resulting effect, and how users overcome challenging situations. The evaluation ends with detection of common practices and pitfalls while handling events. 72
Event Name Description Expiration Time Traffic Jam Due to a traffic jam in Anchorage you arrive at your destination 30 minutes delayed. Exciting Exhibit As the exhibit was very exciting you stayed longer than planned. However, this also resulted in an increase of the business value. Grizzly Bears Grizzly bears crossed the road just in front of the bus which allowed you to take excellent pictures. This lead to an increase in the business value. Your car got stuck You got stuck in mud. It took you some time to get your car back on track. Equipment Problems There were some problems with the equipment and you had to wait till they were resolved. varied varied varied varied varied Change Activity X X X X X Event Business Value X X Duration X X X X Location Atomic Activity Anchorage - Seward Anchorage - Seward Seward Visitor Center Kenai Fjords Denali National Park Bus Tour To Wonder Lake Anchorage - Denali Anchorage - Denali Seward Diving Table 4.8.: Description of events in Alaska III 1. Event handling in the Configurations California I and Alaska I Within the examination of California I, the occurrence of 227 events in total can be counted. Based on a total number of 69 data sets for the California I Configuration, 3 events occur on average per data set. From the total number of events which occurred, 123 events with an influencing effect can be identified - which amounts to about 55 %. The remaining 45 % are represented by events which launch new activities. The total number for this event type is 104 occurrences. Compared to California I for the Configuration Alaska I, 275 events in total can be counted. This results in an average occurrence of 4 events for a total population of 69 data sets. Here, 60 % of the events (165 occurrences in total), are represented by influencing events. The events launching new activities during execution cover - with 110 cases - the remaining 40 %. Because, the influencing events are dominating 73
in both configurations, and evaluations detect similar patterns with respect to effects, challenges, causality and reparation measurements, the subsequent descriptive evaluation is combined for both configurations. Analyzing influencing events in the journey configurations, the majority of occurrences of this event type have no effect (73 occurrences in California I and 149 occurrences in Alaska I). The main causal factor for that observation is that the specific event has no relevance because the preconditioned activities are not included in the planning (57 occurrences in California I and 129 occurrences in Alaska I). Another causal factor is simply luck (10 occurrences in California I and 15 occurrences in Alaska I). The event occurs luckily at a later point in time when the relevant activity was already executed. Other reasons are the application of agile planning during execution and planning activities in parallel. Besides, in some cases the concerning activity was deleted while the event happened. The second most observed effect of influencing events is a destruction of the journey plan, with option for reparation due to unavailability of an activity (32 occurrences in California I and 12 occurrences in Alaska I). This effect is merely caused by the influencing event Tioga pass closed in the California I Journey. In Alaska I, two events influence the activity s availability. The facilitating factor for this effect is the consideration of the preconditioned activities in the journey. Because of the destruction, a certain reparation effort becomes necessary. Essential actions for reparation include the cancellation of booked activities - which incurs additional costs (e.g., cancellation fees) - the cancellation of not-booked activities, shifting and inserting alternative activities. Especially for California I, in some cases the execution of the journey is aborted due to the occurrence of this event. Another reason for unavailability in the California I Configuration is the planning of activities in parallel, where the decision for the influenceable activity is taken too early. The resulting destruction will be repaired by the same measurements such as cancellation and shifting. The last effect of influencing events that could be observed is an increase in business value. This effect is merely initiated by the event Elephant Bulls in Monterey in California I and by the event Grizzlies in Alaska I. The preconditioned activity is already considered in the planning, or integrated during agile planning at run-time. In contrast, NewActivityEvents cause either no effect or an increase in the expected business value. The challenge while launching new activities for the user is generally to manage their consideration in the journey during run time. Therefore, the application of agile-planning is required. However, in about 80 % of the cases in which the launching 74
event appeared (84 occurrences in California I and 94 occurrences in Alaska I), the occurrence caused no effect at all due to irrelevance in both configurations. 2. Event handling in California II Evaluation of California II, which is summarized in Figure A.3, enables the identification of just one type of event. All 39 events which occurred show an influencing effect on activities. About 70 % of these events result in a destruction of the plan, which counts 27 occurrences in total. The second most common effect causes no effect at all (10 occurrences in total). Other effects, which appear at most once during the evaluation of California II, are an unsuccessful termination of the journey and a decrease in the expected business value of the concerned activity. The next paragraphs describe challenges and reasons for favoring certain effects in this configuration. Challenges regarding the destruction of the plan with the option for reparation are either balancing unavailability or handling an overlap of activities. For the latter, the causal factor of considering no or not enough buffer between activities can be identified. Destruction due to unavailability appears because the activity that is effected by the event is considered in the plan. Reparations can be made through cancellation of booked activities, although additional fees are charged. In addition to, canceling activities (not booked), shifting and inserting of alternatives are also conducted for reparation purposes. Overlaps between activities due to the delay in the expected duration of an activity can be mitigated mainly by shifting of the subsequent activity in 15 cases. Secondly, in 14 cases the cancellation of activities couldn t be avoided. Other measures are, in 4 cases, inserting of plan items, and canceling booked activities in 3 cases. In one case the whole execution was canceled due to the occurrence of this event. Another challenge which needs to be mitigated is the longer duration of a certain activity. In 10 cases measured on the total occurrence of influencing events, the happening of the event leads to no effect at all. Participants considered in these cases enough buffer between the subsequent activities which was realized either by agile planning at runtime or planning the concerning activities at the end of a day in the journey. These factors appeared 5 times within the analysis each. However, in 2 cases participants could not overcome this challenge successfully. In one situation the user was not able to equalize the delay because the relevant activity was planned at the very end of the journey. Due to limited buffer after the executed activity, the termination of the journey 75
was not successful in one journey. Attempts for reparations were not possible in this case anymore. In the other case the influenced activity was considered at the end of a day during the journey. Due to the resulting delay a conflict arose with the check-in at the hotel. Reparations cannot be made in this situation which leads to a decrease in business value of the accommodation as consequence. The event Seals Spotted in California II shows a specific characteristic (cf. Section 4.2.4.1) as regards the effect created. In 6 cases of occurrence, this event simultaneously creates an influence on the activity s duration and the positive effect of increasing the business value (cf. dashed branch in Figure A.3). 3. Event handling in Alaska II Basically, for Alaska II the occurrence of 69 events in total can be counted. Although events in California II and Alaska II have the same type, there are slight differences in the caused effects and in their frequency of occurrence. The most common effect in Alaska II shows an increase in the expected business value in 40 cases. Furthermore, no effect takes the second place in this range with 19 incidences. That follows the effect of destroying the plan with the chance for reparation in 18 counted cases. Finally, a decrease in business value occurs just 3 times during observations. An increase in business value of an activity in Alaska II might be achieved if the events Whales Spotted, Grizzly bears or Exiting Exhibit occur. The converse to this effect can be found in decreasing the business value. Challenge and causal factors for these events are the same as already described in California II. In 40 cases of increasing business value, users could benefit from this positive effect. A decrease in business value was inevitable in 3 cases. The second most common effect which can be assigned is the creation of no effect. Besides the causal factors favoring this effect in California II, the factor of irrelevance was additionally subject in Alaska II. Finally, the last effect of destruction might be caused by the events Traffic Jam or Exciting Exhibit where users have to handle overlapping due to the caused delay in an activity s duration. Here, the causal factor is exclusively the consideration of no or not enough buffer. Reparation comprises the same measurements as applied in California II. The occurrence of the event Exciting Exhibit in Alaska II is exception as far as the created effects are concerned. In 11 cases the occurrence of this event creates a destruc- 76
tion of the journey plan due to an overlap of activities - and simultaneously an increase in business value of the influenced activity. 4. Event handling in California III and Alaska III Investigation of California III and Alaska III show that the event types are the same compared to the Configurations California II and Alaska II. Based on the experimental settings in Section 4.1.1, events occur in the Configurations California III and Alaska III performing factor-level 2. Therefore, evaluations in this context refer exclusively to data sets performing this factor-level. For California III a total occurrence of 30 events can be counted which creates a destruction of the journey plan in 17 cases. The second most-created effect is no effect at - all in 9 cases. Furthermore, an increase or a decrease in business value is caused in 3 cases each. Finally, a unsuccessful termination occurred just once. Compared to Alaska III, the same effects can be evaluated. In Alaska III a total occurrence of 50 events can be counted. Of these, a destruction of the journey plan resulted in 18 cases. The second most frequent appearance is no effect at all (15 occurrences). Subsequently, in 11 and 10 cases an increase and decrease in business value was caused. The occurrence of influencing events in Alaska III results in an unsuccessful termination 2 times. Because challenges, causal factors and reparation measurements are almost identical in both configurations, the following descriptions are combined. If the occurrence of the event causes a destruction of the journey plan, the user has to handle either the unavailability of the following activity or an overlap. The latter proves to be most frequent challenge in both configurations (15 times in California III and 17 times in Alaska III). The reason for this is the lack of consideration of buffer in the journey plan between activities. Reparations are made mainly through canceling booked and not booked activities. Other measures are shifting or inserting alternatives. Besides, events which occurred leads to a cancellation of the journey execution. Unavailability, as the second mentioned challenge, requires reparation by replanning. Here, the same measurements are applied as for repairing overlaps. Evaluating the causal factors for ineffectiveness, the same can be detected as already described for California II and Alaska II. Besides, planning activities in parallel and luck additionally favor this effect in California III and Alaska III. Furthermore, considering 77
effects of unsuccessful termination, increasing and decreasing business value, challenges and causality are the same as for California II. The Journey Configurations III include both events that are of a special nature, while influencing activities. The event Seals Spotted in California III and the event Grizzly Bears in Alaska III result in an influence on the activity s duration and simultaneously increase the business value when the preconditioned activities are considered in the plan. Evaluations show 3 occurrences in California III (cf. dashed branch in Figure A.5) and 6 cases for Alaska III (cf. Figure A.6). 4.2.6. Discussion of the Results The manual analysis of the data showed that handling events depends generally on the type of event, which needs to be determined first according to the defined classification criteria in Section 4.2.3. Based on the event type, different effects can be observed which present different challenges for the user. The specification of events showed that events which launch new activities are not as critical for the business process execution as influencing events might be. Launching new activities has no significant influence on the success of journey execution, where the focus is set on a successful termination of the journey. If the success of the business process is defined as achieving a high business value in total, the consideration of newly launched activities is certainly significant. In fact, the occurrence of such events is recognized by the user, but mostly the event creates no effect because the newly launched activity is not considered in the journey. Reasons for that might be that users cannot exploit flexibility provided by the declarative and agile approaches (cf. Section 2) enough to integrate the new activity during journey execution. Furthermore, users might create knowledge concerning the created effect of NewActivityEvents. Consequently, it may appear that they are rather less anxious in implementing newly launched activities in the journey. In contrast, influencing events affect either an activity s business value, it s duration or it s availability. The latter two might require a user s interaction for mitigation in order to terminate the journey execution successfully. In contrast, influencing business value creates a positive effect by increasing value. In order to compensate delays and consequent overlapping or unavailability, replanning during execution is necessary. Besides, some influencing events challenge a user with creating two effects simultaneously - such as increasing business value and creating a delay in duration at the same time. 78
In order to derive common practices and pitfalls while handling events, analysis showed that these vary between journey configurations. In the evaluation of the California I and Alaska I Configurations, it can be observed that participants applying either the agile or plan-driven mode close their planning phase with a rather tightly scheduled journey with hardly any buffer between activities. The reason for that might be that users try to achieve the overall goal of obtaining a high business value in total by avoiding any buffer between activities - especially while planning the journey. This behavior can increasingly be observed for the plan-driven mode, since users can operate during execution only to a certain extent (e.g., in case an event occurs). Because the Configurations I have no events which might influence the duration of activities, tight scheduling cannot be valued as disadvantageous in this case. For the remaining configuration variants, the consideration of buffer between single activities would be essential in order to mitigate overlaps. Consequently, the major effect that appears shows a destruction in journey. The evaluation shows that the most user have considered hardly any buffer between activities in their journey. Although some users create knowledge in this regard, the considered buffer was not sufficient in all cases. When performing journeys in the plan-driven mode, the common practice of using sequences in the journey plan can be observed. Users want to take actions during execution while using these available components. Here, the adoption of sequences differs in the timing of determining their content. However, it can be observed that most sequences determined while planning have to be changed due to events during execution. The majority of users of all configurations are anxious in booking activities, in order to ensure their availability or create options for the treatment of activities with low availability and reliability. The latter motivation can be excluded for California I because this configuration exhibits no activity with such characteristics. However, about one third of the total commitments are made too early and reservations have to be canceled by the deletion of the corresponding activity. Additionally, cancellation causes extra costs due to the charged fees which have to be considered within the limited financial resources to avoid the risk of budget overrun. The second possibility for creating options is planning activities in parallel, which is applied at a rather low frequency in all configurations. A further practice that can be mentioned is the reduction of idle times through the elimination of waste by shifting activities within the journey during execution. Besides, evaluation shows that the practice of agile planning is applied in varying degrees. This can range from ad-hoc planning of single days during the journey to a likewise plan-driven 79
behavior. However, the adoption of agile principles occurs in all configurations commonly where some undetermined fragments in the journey are specified during execution. In summary it can be said that handling events seems to be rather challenging for users. Especially, in case an event occurs, most users have difficulties in ad-hoc mitigating the effects of events (e.g., by agile planning). The mitigation of created effects occurs rather on a trial and error basis where required reparations are sometimes done in a circumstantial and chaotic way. However, the effort for reparation mostly involves rearranging one day or less in the journey for all configuration variants. Here, some effects are already favored by typical errors due to insufficient planning: Planning no or insufficient buffer between activities might cause an overlap, Creating no or no useful options might cause troubles in dealing with activities of low availability and low reliability due to an event, When making too-early commitments it might become necessary to cancel the booked activity later on, due to an occurred event, Making too-late commitments may risk unavailability of activities due to an event, etc. Contrary to expectations, only few users had to cancel their journey execution or terminate their journey execution unsuccessfully due to an event which occurred (cf. Appendix A.2). 80
5. Summary and Outlook The goal of this master thesis was to gain more empirical insight into the use of different planning approaches (e.g., agile, plan-driven). Moreover, the investigation focused on the extent to which the occurrence of uncertain events might influence the user s success in planning and executing a particular business process. The evaluations in this context are focused on different aspects, formulated as research questions, in order to maintain transparency and a better comprehension of the test results. Research Question 1 focuses on a general comparison between the agile and plan-driven planning approach. The effect of the planning approach on the process outcome, measured in terms of the business value, is evaluated in this question. Test results referring to Research Question 1 show that varying the planning approach results in a significant difference in the process outcome of inexperienced users. Except the test case comparing 50 % of best performers of the California I journey shows no significant test results. The explanation for the significant results can be found in the limited flexibility provided by the plan-driven approach. In contrast, in the California I test case mentioned before, the absence of a significant test result might be attributable on the one hand to the configuration characteristic (e.g., no activities of low reliability and low availability) and on the other hand to the ineffective adoption of the agile concepts by the users. The second objective, analyzed in Research Question 2, was to investigate how well inexperienced users can perform process modeling and execution when applying the agile and the plan-driven planning approach. Furthermore, it was examined to what extend the agile principles were adopted by the user. The major finding related to Research Question 2 revealed that achieving a higher business value in total is very much related to the number of the applied agile principles. For this purpose, comparisons were made between different numbers of adopted agile practices, and/or with the plan-driven planning approach. In order to summarize the results of the statistical analysis, it could be stated that a high agile adoption outperforms 81
a low agile adoption, except for two test cases. The absence of a statistical significance in the two cases might be explained with a low uncertainty in the compared business processes, where the application of agile principles might be less essential. Finally, Research Question 3 addresses the influence of the events which occur when planning and executing the business process. Moreover, it was investigated what kind of events might occur and how users handle them during the execution of the process. The manual analysis with reference to Research Question 3 demonstrates that the influence of the actual events as they occurred, and how they are handled, generally depends on the type of the respective event. In this context, it is possible to distinguish between events which launch new activities and events which influence activities considered within the journey. When comparing the two event types, launching new activities is not as challenging as the influencing type. If users manage to consider the newly available activities during the execution, the positive effect of increasing activities business value is created. However, evaluations show that only a small number of users were anxious in implementing the new activities because these events are not as critical for a successful process termination. In contrast, the influencing event might either affect the business value, the duration or the availability of an activity. Consequently, reduction in business values, delays in time, overlaps or unavailability of activities are caused. In order to compensate these challenges, users are forced to replan during execution. The evaluations of events conclude with deriving common practices and pitfalls in handling events, which vary between the different journey configurations. For further research, the evaluation in this thesis might refer to experimental data from users with more experiences in the field of business process modeling. This would enable a more in-depth analysis of the planning approaches addressed in this thesis with reference to their effectiveness. Furthermore, the analysis regarding events might be conducted at an advanced level because some effects created by the events are favored due to insufficient planning. Moreover, the settings of the conducted experiments might be extended by planning in small groups of users, because planning and executing the business process has been done individually up to this point. 82
A. Appendix A.1. Raw Data 83
84
85