University of Groningen. Understanding and Supporting Software Architectural Decisions Tofan, Dan

University of Groningen Understanding and Supporting Software Architectural Decisions Tofan, Dan IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite from it. Please check the document version below. Document Version Publisher's PDF, also known as Version of record Publication date: 2015 Link to publication in University of Groningen/UMCG research database Citation for published version (APA): Tofan, D. (2015). Understanding and Supporting Software Architectural Decisions: for Reducing Architectural Knowledge Vaporization [Groningen]: University of Groningen Copyright Other than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons). Take-down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons the number of authors shown on this cover page is limited to 10 maximum. Download date: 07-11-2018

Chapter 3 Architectural Decisions in Practice Based on: Tofan, D., Galster, M., and Avgeriou, P., Difficulty of Architectural Decisions a Survey with Professional Architects. In Proceedings of the 7th European Conference on Software Architecture. 2013. In this chapter, we investigate characteristics of architectural decisions, as well as factors that contribute to the difficulty of decisions. We compared characteristics of decisions and difficulty factors for junior and senior architects. Furthermore, we studied if decisions with good outcomes (as perceived by the architects) have different characteristics and different difficulty factors than decisions with bad outcomes (i.e. bad outcomes, as perceived by the architects). We performed a survey with 43 architects who described 43 good and 43 bad decisions from their industrial practice. Characteristics of decisions include the time taken to make a decision and the number of decision makers. Also, we learnt that dependencies between decisions and the effort required to analyze decisions are major factors that contribute to the difficulty of architectural decisions. We found that good decisions tend to consider more alternatives than bad decisions.

34 3. Architectural Decisions in Practice 3.1 Introduction Increased understanding of architectural decisions in practice enables researchers to propose approaches that help practitioners improve architectural decision making and therefore reduce overall architectural knowledge vaporization. Although there is much interest in the community on architectural decisions, little work has been done towards a deeper understanding of architectural decisions in practice. This chapter offers a step in this direction, by focusing on understanding two aspects: characteristics of architectural decisions in practice and factors that contribute to the difficulty of making decisions. As presented in Chapter 1, architectural decisions are particularly difficult decisions. Using GQM (Basili and Caldiera, 1994), the goal of this study is to analyze architectural decisions, for the purpose of understanding architectural decisions in practice, from the perspective of software architects, in the context of real-world projects. We refine this goal into four research questions. RQ1. What are the characteristics of architectural decisions? By defining measurable characteristics of architectural decisions, we can perform comparisons between decisions. Based on literature (Kruchten, 2008), our experience and discussions with practitioners, we define the following basic characteristics of decisions as metrics for RQ1. 1. Actual (i.e. while making the decision) and elapsed (i.e. actual plus various interruptions) time spent making a decision. For example, an architect spent five hours (actual time) over three days (elapsed time) to make a decision. 2. Number of people directly (i.e. decision makers) and indirectly (i.e. influenced the decision, but did not make the decision) involved in decision making. 3. Number of alternatives considered in the beginning and later over an extended period of time during the decision making process. 4. Number of quality attributes considered for a decision.

3.2. Related Work 35 Since the difficulty of a decision is a major aspect of decision quality (Yates et al., 2003), we propose RQ2. Answering RQ2 will help researchers and architects understand the factors that increase difficulty of decisions. RQ2. What factors make architectural decisions difficult? To answer RQ2, we defined a list of factors using literature and discussions with experts (Section 3.3.1). The resulting metrics were 22 factors (Table 3.4). These were used as input for our survey and survey participants rated them on a Likert scale. The level of experience of architects influences their decision making (van Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011). We propose RQ3 to investigate how difficulty and characteristics of decisions vary with the level of experience. This helps researchers propose targeted solutions to address the difficulties perceived by either junior architects or experienced architects. RQ3. What are the differences (on difficulty and characteristics of decisions) between junior and senior software architects? Besides difficulty, decision outcome is the other major aspect of decision quality (Yates et al., 2003). We investigate the differences between decisions with a more preferable outcome (i.e. good decisions) and decisions with a less preferable outcome (i.e. bad decisions). Answering RQ4 highlights characteristics and difficulty factors linked to good and bad outcome of architectural decisions. RQ4. What are the differences between good and bad architectural decisions? To answer the research questions, we conducted a survey with software architects in industry. Section 3.2 presents related work. Section 3.3 presents survey design. Section 3.4 contains the analysis of results which are further discussed in Section 3.5. Section 3.6 presents conclusions. 3.2 Related Work Work related to this chapter is discussed for each research question. We start with the research question on characteristics of architectural decisions (RQ1). We could not find any study that focuses on identifying or describing characteristics of architectural decisions. Based on observations of various architecture teams, Kruchten recommends roles, responsibilities and time

36 3. Architectural Decisions in Practice allocation for architects (Kruchten, 2008). Based on (Kruchten, 2008), we consider that involved roles, involved responsibilities (e.g. involved directly or indirectly in the decision making) and the time allocated (e.g. actual, elapsed) for making decisions are characteristics of architectural decisions. Some surveys with practitioners provides insights on architectural decision making (Clerc et al., 2007), and knowledge sharing for architectural decisions (Farenhorst et al., 2009). However, we could not find any study that investigates the characteristics of architectural decisions in RQ1. Our research question on difficulty of architectural decisions (RQ2) draws inspiration from work on the quality of decision making (Yates et al., 2003). Yates et al. (Yates et al., 2003) consider that the difficulty of decisions is similar to a cost of making decisions, so reducing cost increases the quality of decision making. We consider this also applies to architectural decisions because they are a subcategory of decisions. Unfortunately, we could not find related work that focuses on the difficulty of architectural decisions. Related work on junior architects versus senior architects (RQ3) includes two surveys and a case study on the reasoning processes of naïve and professional architects (van Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011; van Heesch et al., 2013). - Results of the first survey indicate that naïve architects (i.e. undergraduate students) do not make trade-off between requirements, fail to validate dependencies between decisions, and do not evaluate critically their decisions (e.g. insufficient risk assessment) (van Heesch and Avgeriou, 2010). - Results of the case study indicate that naïve architects who use decision viewpoints are more systematic in the exploration and evaluation of decision alternatives (van Heesch et al., 2013). - Results of the second survey (van Heesch and Avgeriou, 2011) indicate that professional architects very often search for many design alternatives in their decision making, unless they already have a solution in mind. Also, professional architects prefer familiar alternatives, because unfamiliar alternatives need more analysis effort (van Heesch and Avgeriou, 2011). Additionally, professional

3.3. Survey Design 37 architects do not consider risk assessment as very important (van Heesch and Avgeriou, 2011). In our survey, we ask participants about risks, trade-offs as difficulty factors for their architectural decisions. No related work compares characteristics and difficulty of architectural decisions for professionals with different levels of experience. For our research question on good versus bad decisions (RQ4), we found much research to avoid bad decisions. For example, ATAM and CBAM help discover potentially bad architectural decisions (Kazman and Klein, 2001). They do this by documenting costs, benefits, and uncertainty. Other authors describe architectural smells as bad architectural decisions, which harm quality attributes (Garcia et al., 2009). However, we could not find any work which compares characteristics and difficulty of good and bad architectural decisions. 3.3 Survey Design To design this survey, we used the recommendations and guidelines from (Ciolkowski et al., 2003; Kitchenham and Pfleeger, 2008). Data was collected using an online questionnaire. In this section, we present questionnaire development, evaluation, and data collection. 3.3.1 Questionnaire Development and Evaluation To develop the questionnaire, we took the following steps, based on (Kitchenham and Pfleeger, 2008): 1. Review literature 2. Discuss with experts 3. Pilot questionnaire 4. Publish questionnaire In the first step, we reviewed existing literature on architectural decisions (e.g. (van Heesch and Avgeriou, 2010; van Heesch and Avgeriou, 2011; Zimmermann, 2011)) and decision research (e.g. (Yates et al., 2003)). From the literature, we identified a list of factors that contribute to the difficulty of architectural decisions. Based on these factors, we prepared a first version of the questionnaire that we used as a starting point for interviewing practitioners.

38 3. Architectural Decisions in Practice For the second step, we interviewed four senior architects, each with at least ten years of experience as an architect. We asked each architect to identify two architectural decisions they had been involved in, and discussed the questionnaire items for both decisions. Afterwards, we asked the architects to propose other items that contribute to the difficulty of a decision to be included in the questionnaire. The architects also provided thoughtful feedback on the structure of the questionnaire. For the third step, we piloted the questionnaire with six other persons that included researchers and practitioners. We used their feedback to further improve the questionnaire. For example, we eliminated some items so that the survey would take less than 15 minutes to complete (practitioners are unlikely to finish long surveys (Kitchenham and Pfleeger, 2008)). Additionally, we rephrased some questions to increase their clarity. In the final step, we published the questionnaire online. The final version of the questionnaire showed the following to participants. The full questionnaire is available in the Appendix. 1. Welcome message 2. Control question 3. Background questions 4. Describe good decision 5. Rate 22 factors for the good decision 6. Describe bad decision 7. Rate 22 factors for the bad decision 8. Add other factors 9. Thank you message The first page had a welcome message with an overview of the survey, including targeted audience, required effort and incentive. The next page checked that participants had been involved directly in making architectural decisions during the last two years. The survey continued only after a positive confirmation from the participant. The survey continued with a few questions about the background of the participant. Next, participants were asked to indicate a good architectural decision ( good according to their judgment), and describe its characteristics, in terms of the metrics for RQ1 (e.g. number of alternatives). Next, participants were asked to rate the 22 statements in Table 3.4 about the difficulty of their good architectural decision on a Likert scale. Afterwards,

3.3. Survey Design 39 participants were asked to indicate a bad architectural decision, and describe its characteristics. We asked participants to judge themselves what bad architectural decisions were. Participants were asked to rate the same 22 statements about the difficulty of their bad decision. Finally, participants could optionally add other items that contribute to the difficulty of architectural decisions. Also, participants could send me a short e- mail, to receive a copy of survey results or to offer other feedback. 3.3.2 Data Collection Our target population was software architects who were directly involved in making software architectural decisions during the last two years. This ensured that survey participants could answer reliably to the questions about their architectural decisions. We dropped the requirement of having the job title architect because it might be over-used in the industry, without indicating architecting activities. Also, people often make architectural decisions without having a formal role as architects (e.g. senior software developers). Members of our target population have busy schedules, so they are unlikely to participate in surveys. Thus, we tried to limit the time to complete the survey to around 15 minutes. To reach our target population, we used several approaches. First, we sent invitations to architects in our personal networks, asking them to participate in the survey and to further distribute it. Second, we posted invitations to the survey on the website of a social networking site for professionals. Third, we ran paid ad campaigns on LinkedIn (as detailed in (Galster and Tofan, 2014)). In the advertisement we targeted professionals working for companies in the Computer Software, Internet, or Information Technology and Services industries, from any part of the world, with job functions in Information Technology or Engineering. Also, we selected the following job seniority: CXO, Director, Manager, Owner, Partner, Senior, or VP, with skills in software architecture. These filters resulted in around 91,000 professionals and that could potentially see our ad. Finally, we run an ad campaign with Google AdWords. Here, we did not find options for defining our target audience. Instead, we defined a set of keywords (e.g. software architecture ). People searching for such keywords could see our ad for the survey.

40 3. Architectural Decisions in Practice We ran the survey from the 1 st of October 2012, until 7 th of January 2013, using a third-party, web-based tool. In total, 219 persons started the survey. 163 persons provided partial answers (i.e. respondents answered a few questions, and then abandoned the survey). Twelve participants did not pass the control question about involvement in architectural decisions. Finally, 43 persons provided answers to all questions in the survey. We analyze the 43 answers in the next section. 3.4 Results Analysis We use descriptive statistics to present the background of the survey participants, answers to RQ1 and RQ2. Also, we use statistical tests to answer RQ3 and RQ4. 3.4.1 Participants Background Answers were provided from 23 countries on five continents. Regarding their job title, most respondents described themselves as software architects (twenty), senior software engineers (seven), or enterprise architects (five). The other respondents described themselves as managers, business analysts, system architects, and other roles. Table 3.1 summarizes the years of experience that participants had in their current role and as architects. We notice that most respondents had three to five years of experience in their current roles. Most participants had six to ten years of experience as software developers. The ranges for years of experience for architects and developers were normally distributed.

3.4. Results Analysis 41 Table 3.1. Number of participants with corresponding years of experience and roles. Years of experience In current role As architect As developer 0-2 years 5 12 3 3-5 years 19 10 11 6-10 years 10 13 12 11-15 years 3 7 8 >15 years 6 1 9 The 43 survey participants answered questions about 86 architectural decisions (43 good and 43 bad decisions). Architectural decisions were from domains shown in Figure 3.1. Most decisions (31) belonged to other domains, such as financial trading, insurance, or advertising. Transportation, 3 Healthcare, 6 Banking, 6 Embedded systems, 8 E-government, 10 Telecommunication, 10 Other, 31 E-commerce, 12 Figure 3.1. Domains of decisions. 3.4.2 RQ1 - Characteristics of Architectural Decisions Participants indicated the actual and the elapsed time they spent for making the 86 architectural decisions. For example, an architect can spend three working days over ten working days for a decision, resulting in a ratio of actual versus elapsed time of 30%. In the survey, participants could indicate time in minutes, hours, days, weeks, or months, based on their preference. We converted their responses into working days, by considering that one working day has eight hours, one working week has five days, and that one working months has 22 working days. Outlier values bring a negative influence on the interpretation of averages and standard deviations. To avoid that, we eliminate outliers. We obtain the results

42 3. Architectural Decisions in Practice in Table 3.2. On average, architectural decisions took about eight working days, elapsed over around 35 working days. The average ratio indicates that, overall, a third of the elapsed time is spent on the actual decision making. Table 3.2. Metrics for actual time, elapsed time (in working days), ratio of actual versus elapsed time, and number of directly and indirectly involved persons in the architectural decisions. Metric Actual Elapsed Ratio Direct Indirect Average 7.85 34.74 0.34 3.12 7.05 Standard deviation 9.22 70.59 0.22 1.54 8.91 Minimum 0.50 0.63 0.02 1 0 Maximum 44 600 1 8 50 Mode 1 5 0.5 3 3 Participants indicated how many people were involved directly and indirectly in making the architectural decisions. The number of indirectly involved persons does not include the directly involved persons. We collected this metric because we could not find any related work that describes how many decision makers are usually involved directly in architectural decisions in practice. For example, if most architectural decisions are made in groups, then more work is needed for group decision making, rather than individual decision making. We eliminated outliers for the number of people both for direct and indirect involvement, and we obtain the results in Table 3.3. Table 3.3. Metrics for the number of alternatives considered in the beginning, and for an extended period of time. Last column has the number of quality attributes. Metric Beginning Extended time # Quality attributes Average 2.91 1.96 4.74 Standard deviation 1.43 0.84 4.19 Minimum 1 0 0 Maximum 8 4 30 Mode 3 2 3 When making an architectural decision, more alternatives can be considered. Since little is known about how many alternatives architects usually consider

3.4. Results Analysis 43 in their decision making, we asked participants to indicate the number of alternatives they considered at the beginning of their decision making process, and the number of alternatives they studied for an extended period of time. Architects consider quality attributes in their decisions. However, it is not clear how many quality attributes they consider in practice, so we asked them to indicate this number for their decisions. After eliminating outliers, we obtain the metrics for the number of alternatives and quality attributes in Table 3.3. 3.4.3 RQ2 - Difficulty of Decisions Participants rated 22 statements (Table 3.4) with factors on the difficulty of their decisions, indicating their level of agreement with the statements, using a five-point Likert scale with the following values: strongly disagree, disagree, neutral, agree, strongly agree, and not applicable. ID F1 Table 3.4. Factors to describe the difficulty of a decision. The decision was difficult because you received conflicting recommendations from various sources about which decision alternative to choose F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 there were no previous similar decisions to compare this decision against it was hard to identify a superior decision alternative from the alternatives under consideration the decision required a lot of thinking from you it was hard to convince stakeholders to accept a certain decision alternative stakeholders had strongly diverging perspectives about the decision you needed to influence some stakeholders without having formal authority over them the decision had too many alternatives the decision had too few alternatives analyzing alternatives for this decision took a lot of effort some quality attributes were considered too late in the decision making

44 3. Architectural Decisions in Practice process F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22 too many people were involved in making the decision dependencies with other decisions had to be taken into account the decision had a major business impact you had to respect existing architectural principles serious negative consequences could result from the decision too little time was available to make the decision you had a lot of peer pressure of the trade-offs between quality attributes you lacked experience as an architect you lacked domain-specific knowledge (e.g. new customer) more information was needed to reduce uncertainty when making the decision Results for each factor are summarized in Figure 3.2 (left). From the bar charts, we notice the following. First, participants indicated most agreements (including strong agreements) with statements on dependencies with other decisions (F13 for 69 decisions), major business impact (F14 for 60 decisions) and serious negative consequences (F16 for 59 decisions). Second, participants indicated most disagreements (including strong disagreements) with statements on having too many alternatives (F8 for 57 decisions), too many people involved in decision making (F12 for 49 decisions), lack of domain-specific knowledge (F21 for 46 decisions), and having too few alternatives (F9 for 45 decisions). Third, participants indicated most neutral standpoint with statements on respecting existing architectural principles (F15 for 24 decisions), needing lot of effort for analyzing decision alternatives (F10 for 21 decisions), and having much peer pressure (F18 for 21 decisions). Fourth, participants indicated very few statements were not applicable to their decisions. Most not applicable answers were obtained for influencing some stakeholders without formal authority (F7 for six decisions).

3.4. Results Analysis 45 Figure 3.2 (right) shows average values for all factors, calculated as follows. We assign numerical values to the Likert scale: strongly disagree (1), disagree (2), neutral (3), agree (4), and strongly agree (5). Not applicable values are ignored. We acknowledge the ongoing discussions on treating a Likert scale as either an interval or categorical type of data (e.g. (Jamieson, 2004)). Still, we use averages because they are intuitive and easy to understand for a large audience. From Figure 3.2 (right), we notice that dependencies with other decisions (F13) and major business impact (F14) have highest average agreements across participants. Negative consequences (F16) received second highest average. Effort for analyzing alternatives (F10), lack of similar decisions (F2) and requiring a lot of thinking (F4) received high agreements from participants. We notice that some factors have averages that suggest disagreement that they contribute to the difficulty of architectural decisions, i.e. have averages smaller than three (neutral). For example, we notice that either too many (F8) or too few (F9) alternatives contribute little to difficulty. The same goes for lack of experience (F20) and domain-specific knowledge (F21). However, these two factors need to be considered in the context that many participants were senior architects, who might already have enough experience and knowledge.

46 3. Architectural Decisions in Practice Strongly disagree Disagree Neutral Agree Strongly agree Not applicable ID Avg. 1 0 1 1 0 5 4 4 4 6 4 2 3 1 0 1 2 0 0 2 3 2 3 1 1 F13 3.88 12 26 17 23 17 8 9 6 6 6 5 F14 3.88 12 20 17 20 17 15 23 F16 3.74 21 27 23 F10 3.63 21 15 22 26 F2 3.61 25 F4 3.6 9 29 25 37 F7 3.55 28 17 17 12 F22 3.49 28 25 34 28 20 8 35 F3 3.48 27 19 F6 3.35 30 36 11 49 33 21 F19 3.33 F1 3.3 17 24 F5 3.27 17 50 39 10 16 17 20 F15 3.27 9 17 5 40 21 28 41 F18 3.05 26 10 F11 3.01 34 21 14 20 17 16 29 F17 2.93 20 6 19 20 21 F21 2.82 14 16 21 F20 2.73 14 4 5 2 4 7 7 4 7 9 10 9 10 12 F9 2.71 5 0 2 1 3 1 3 2 5 0 3 F12 2.69 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 F11 F12 F13 F14 F15 F16 F17 F18 F19 F20 F21 F22 F8 2.52 Figure 3.2. Survey results for each factor (left), and sorted average values for factors (right) - a higher average indicates stronger agreement with the difficulty of a factor. Participants could indicate optionally other factors that contribute to the difficulty of architectural decisions. We identified three other difficulty factors: technology evolution (mentioned by six participants), insufficient knowledge (four participants), and organizational politics (two participants). First, technology places unpredictable constraints on decisions, such as uncertain backwards incompatibility of future versions of a platform. Second, insufficient knowledge refers to lack of professional knowledge, lack of peers to discuss non-standard issues, and lack of documentation. Third, participants indicated that organizational politics contribute to the difficulty of decisions, and require awareness from architects. 3.4.4 RQ3 - Differences between Junior and Senior Architects We divide survey participants in junior and senior architects, in an arbitrary manner. We consider that junior architects have up to five years of experience as architects. Senior architects have six or more years of experience as

3.4. Results Analysis 47 architects. Based on this separation, 22 junior and 21 senior architects answered the survey. To compare the answers from junior and senior architects, we use a statistical test. We compare two independent groups and cannot assume that the data is normally distributed. Thus, we use the Mann-Whitney U test (Field, 2009), a non-parametric test. We investigate the differences between junior and senior architects on the answers offered to the 22 factors in Table 3.4, and the eight metrics in Table 3.2 and Table 3.3. We use SPSS 20 to apply the Mann-Whitney U test on the survey data. Not applicable answers are treated as missing values. We obtain significant statistical differences (p-value less than 0.05) between junior and senior architects for five difficulty factors and two characteristics. Table 3.5 presents the Mann-Whitney U test results (U, Z, and p), median and average values for junior and senior architects. Results for difficulty factors without statistical significance are omitted. We notice that junior architects found conflicting recommendations on what to consider for a decision (F1) as more significant to making a decision difficult. Also, in contrast to senior architects, junior architects found that if lots of thinking is required (F4), decisions become more difficult. In turn, senior architects found that decisions become more difficult if they have a major business impact (F14). Finally there are differences between junior and senior architects on experience (F20) and domain-specific knowledge (F21); these differences can be expected to some extent, because senior architects have more experience and domain-specific knowledge.

48 3. Architectural Decisions in Practice Table 3.5. Results of the Mann-Whitney U test are displayed on the first four columns. Median and average values for junior (J) and senior (S) architects are displayed on the last four columns. Lines with p-value > 0.05 have gray background. ID U Z p Mdn. J Mdn. S Avg. J F1 565-2.49.006 4 3 3.61 2.97 F4 722-1.80.036 4 3 3.82 3.38 F14 716-1.71.044 4 4 3.73 4.05 F20 429-4.42.000 3 2 3.25 2.19 F21 699-1.93.027 3 2 3.74 2.59 Avg. S Actual time 553-3.21.001 2.75 9 4.87 19.67 Elapsed time 742-1.57.058 15 21.5 22.87 62.32 Ratio time 621-2.62.004 0.25 0.35 0.29 0.42 Direct people 883 -.36.361 3 3 3.28 3.78 Indirect people Alternatives beginning Alternatives extended 901 -.20.422 4 4 8.79 24.12 780-1.28.100 2.5 3 2.77 3.62 916 -.07.474 2 2 2 2.08 No. QAs 764-1.40.081 3 4.5 4.23 5.07 We notice that senior architects spend significantly more actual time on their decisions than junior architects (four times on average). The ratio of actual on elapsed time is higher for senior architects, with a clear statistical significance. We also notice a tendency of senior architects to consider more alternatives in the beginning of the decision making. Also, senior architects tend to consider more quality attributes than junior architects; however the statistical significance is not sufficient.

3.4. Results Analysis 49 3.4.5 RQ4 - Differences between Good and Bad Decisions Each survey participant answered questions about a good and a bad architectural decision. Comparing participants answers on the two types of decisions increases our understanding on the quality of architectural decisions, by analyzing the link between the two aspects of quality: difficulty (the 22 factors in Table 3.4) and outcome (good and bad decisions). Furthermore, we analyze the link between the characteristics of architectural decisions and their outcome (e.g. are there differences between the time spent on good or bad decisions). We compare differences between the 43 good and 43 bad decisions using the Wilcoxon signed ranks test (Field, 2009), a non-parametric statistical test for comparing groups of two related samples. We treat not applicable answers as missing values. Similar to the analysis in Section 3.4.4, we investigate the differences between good and bad decisions related to the data in Table 3.2 and Table 3.3. The results of the Wilcoxon signed rank test are presented in Table 3.6. Results on difficulty factors with statistically insignificant values are not included.

50 3. Architectural Decisions in Practice Table 3.6. Results of the Wilcoxon signed rank test, medians and averages for variables and metrics that correspnd to good (G) and bad (B) decisions. Lines with p-values > 0.05 have gray background. ID Z p- value Mdn. G Mdn. B Avg. G F9-2.48 0.006 2 3 2.47 2.95 F11-3.07 0.001 2 4 2.61 3.4 F13-2.12 0.018 4 4 4.07 3.7 F17-2.63 0.004 2 3 2.67 3.19 F18-2.03 0.022 2.5 3 2.81 3.29 F20-1.85 0.04 3 3 2.6 2.86 Actual time -0.51.308 4 5 14.44 9.76 Elapsed time -0.73.235 20 20 38.02 46.25 Ratio time -0.09.463 0.33 0.33 0.37 0.34 Avg. B Direct -0.55.294 3 3 3.37 3.63 people Indirect people Alternatives beginning Alternatives extended -1.40.082 4 4 17.14 15.42-2.41.007 3 2 3.65 2.72-1.92.029 2 2 2.21 1.84 No. QAs -2.68.003 4 3 5.02 4.26 We notice differences on having too few alternatives (F9), with a tendency for disagreement with F9 on good decisions, and for neutral with F9 on bad decisions. For bad decisions, participants indicated that some quality attributes were considered too late (F11), in contrast with good decisions. Also, dependencies with other decisions (F13) are more difficult for good than bad decisions. Participants disagreed on too little available time (F17), much peer

3.5. Discussion 51 pressure (F18), and lack of experience (F20) for the good decisions, in contrast with the bad decisions. Regarding the metrics for decisions characteristics, we notice statistically significant differences between the number of alternatives considered at the beginning of the decision making process, number of alternatives studied for an extended period of time, and the number of quality attributes. For all these three parameters, the good decisions had higher numbers. The results for the parameters about time and number of involved people have insufficient statistical significance. 3.5 Discussion Little is known in the literature about the duration of real-world architectural decisions. We found that the time needed for architectural decisions varies much. An architectural decision takes an average actual time of around eight working days, over an average elapsed time of 35 working days. Also, architects spend around one third of the elapsed time on the actual decision making (Table 3.5). The survey results indicate no significant differences between good and bad decisions regarding actual and elapsed time (Table 3.6). However, participants considered they had enough time for the good decisions, and not enough time for the bad decisions. Another insight from this study is that the actual time junior architects spend on making a decision is one quarter of that spent by senior architects (Table 3.5). We expected senior architects to spend less or similar amounts of time to junior architects, because of their extra experience. A possible explanation is that senior architects might deal with higher impact decisions than juniors, so they need the extra time. A future, more precise comparison should use a ratio of time per decision impact, which can be quantified as the estimated cost of reversing the decision. Another insight from this survey concerns the number of people involved in architectural decisions. The importance of stakeholders in architectural decisions is widely recognized in the literature. Stakeholders are always involved indirectly in decision making. However, we could not find any literature on the direct involvement of stakeholders in architectural decisions, as actual decision makers, rather than decision influencers. To improve

52 3. Architectural Decisions in Practice decision support, researchers need to know if architectural decisions are typically made by one person (i.e. the architect) or by groups of persons (i.e. one or more architects, and other stakeholders). For example, researchers can propose group decision making approaches, if a relevant proportion of architectural decisions are made in groups. A surprising result is that only 14% of the decisions in the survey were made by individuals. The typical architectural decision has three decision makers (Table 3.2). Consequently, group architectural decision making is a much needed research direction. Regarding difficulty of decisions, we notice that dependencies with other decisions are a very important factor for the perceived difficulty of decisions. Results from a related survey (van Heesch and Avgeriou, 2011) indicate that architects often come across such dependencies. Moreover, researchers proposed various approaches for handling decisions dependencies (e.g. (Jansen et al., 2009)). Our survey confirms the relevance of the topic, and the need for disseminating research results to practitioners. We also found that analysis effort and lack of similar (or previously made) decisions increase difficulty of decision making. This suggests that practitioners welcome approaches that help them analyze decisions, and appreciate examples of similar decisions as opportunities to reuse architectural knowledge. Regarding differences between junior and senior architects we found that junior architects need help to address the difficulties of analyzing decisions, such as handling conflicting recommendations. This is not relevant for senior architects. We consider that existing documentation approaches help junior architects. However, documentation could be improved by adding capabilities for analyzing decisions (e.g. what-if analysis). Regarding differences between good and bad decisions, the survey results indicate that good decisions have more alternatives than bad decisions. Therefore, as a rule of thumb, we recommend practitioners to identify three or more alternatives. Also, this study confirms that practitioners should pay attention to quality attributes and decisions dependencies while making architectural decisions.

3.6. Conclusions 53 3.5.1 Validity Threats We present potential study limitations, in terms of internal, construct, and external validity (Ciolkowski et al., 2003; Kitchenham and Pfleeger, 2008). To increase internal validity, we piloted the questionnaire and refined it to ensure that participants could understand it. Also, we added explanatory text with small examples to the questions, so that participants could easily interpret the questions. Another threat to internal validity is that our findings were derived from polarized instances of architectural decisions (i.e. good and bad), rather than typical decisions. Not adding typical decisions to the questionnaire was a necessary trade-off for keeping the survey duration within 15 minutes. To make sure we measure the difficulty of decisions, we used existing work (Yates et al., 2003) as a starting point for characterizing decisions and to conceptualize difficulty, as suggested by (Ciolkowski et al., 2003). Moreover, we discussed our conceptualization with experienced architects, who helped us refine it. Furthermore, participants could answer with not applicable to the survey items on difficulty of their decisions. The very low numbers of not applicable answers indicates that the survey items indeed measure difficulty of decisions. We increased the external validity of this survey by recruiting participants through posting survey invitations in venues for professionals, and by using paid ad campaigns. These efforts resulted in a high diversity of respondents, compared to only using respondents from personal networks or only answers from a specific region or company. 3.6 Conclusions In this chapter, 43 participants provided information about 86 architectural decisions. We found that dependencies among decisions, analysis effort and lack of similar and previously made decisions make architectural decisions difficult. Moreover, junior architects need decision analysis support more than senior architects. Also, we found that considering more alternatives may lead to better decisions. Additionally, this survey confirmed the impact of quality attributes and decisions dependencies on the quality of architectural decisions. In addition, this survey confirms the importance of reducing architectural

54 3. Architectural Decisions in Practice knowledge vaporization (a challenge also identified in the previous chapter), so that architects can access similar and previously made decisions. This survey provided insights on the duration and number of people, alternatives and quality attributes in architectural decision making, which suggests the need of further research on group architectural decision making. To offer decision analysis support, Chapters 6 and 7 present approaches for making architectural decisions (including group decisions). Chapter 8 presents an open-source tool that includes support for group decision making, analyzing alternatives and dependencies with other decisions. After studying practical architectural knowledge management and architectural decisions in Chapter 2 and Chapter 3, the next chapter presents a systematic mapping study of literature on architectural decisions, in which we identify and synthesize existing research on architectural decisions. Acknowledgment We thank the study participants for their help.