page 1 HaT Technical Report 2016.01 March, 2016 Elements to a Theory of Software Development Dr. Alistair Cockburn Humans and Technology, inc. A theory of software development should explain why successful projects succeed and failing projects fail, predict important issues in running projects, lead a person on a project to derive sensible advice as to how to proceed, and ideally, provide a sound pedagogical basis for teaching. This paper outlines elements of such a theory. The key elements are (a) viewing software development as a cooperative game of invention and communication, (b) recognizing the importance of individual people s personalities, (c) considering the specialties involved as craft professions, (d) accounting for unvalidated decisions as internal inventory, and (e) viewing learning as a deliberate exercise. These elements of the theory have been derived and stress-tested on industrial projects for between seven and twenty years each with good results, and have been used to train undergraduate students and professional workers. (1) Software development consists of people inventing, communicating, deciding; solving a problem they don t yet understand and which keeps changing underneath them, creating a solution they don t yet understand and which keeps changing underneath them, working with people they don t yet understand and who keep changing underneath them, and expressing their ideas in limited languages they don t yet understand and which keep changing underneath them; to an interpreter unforgiving of error; in a context where everyone is making decisions, each decision has ripple consequences across other people; and resources are limited. We would like a theory of software development to (T1) explain why successful projects succeed and failing projects fail, (T2) predict important issues in running projects, (T3) lead a person on a project to derive sensible advice as to how to proceed, and ideally (T4) ) provide a sound pedagogical basis for teaching. Figure for 1: Capsule description of the software development activity. Sentence (1) is a statement of the problem to be addressed, a description of software development without analogies or metaphors. The first part of the proposed theory is a declaration that sentence
page 2 HaT Technical Report 2016.01 March, 2016 (1) is accurate enough to lead us toward a suitable theory, that accounting for each part of this description should strengthen the proposed theory. The theory has five major sections, which will be expanded in the subsequent sections: A cooperative game. Personalities Craft Decisions as inventory Learning as an activity Taken separately, each of these meets the desiderata. Each has been tested and taught. They are not orthogonal, that is, mutually independent, not is it claimed that they are complete. Hence their description as elements of a theory. What follows are graphs and charts that identify the elements of the theory. The value of such charts is that they don t make recommendations by themselves, but rather, as good theory should, point to consequences of decisions made. 1. A Cooperative Game (2) A software development project is a cooperative game of invention and communication. More exactly, it is a finite, goal-directed cooperative game whose moves consist of inventing, communicating and deciding. The cooperative part highlights that anything that affects cooperation affects the team. This includes trust, distance, frequency and ease of communication, amicability, personal safety, and mood. The game part of highlights the importance of collecting tactics and strategies to produce good outcomes, of paying attention to positions, moves, and patterns of movements. Inside of the game aspect, we will find the mathematics of queuing theory as well as collected and conflicting heuristics from experienced project managers. Figure for 2: Cooperative, competitive, finite and infinite games. The chart highlights that as we learn the properties of cooperative, finite, goal-directed games, we can apply them to otherwise seemingly different activities, such as journalism, business, theater, even rock-climbing and exploration.
page 3 HaT Technical Report 2016.01 March, 2016 1a. Distance and Culture (3) Anything that slows the movement of ideas between minds slows the projects. The speed of the project is limited by the speed at which ideas, especially uncomfortable news, move between minds. Distance, physical obstacles, emotional and cultural barriers matter: (4) Communication frequency and richness of communication events affect trust. (6) Warmth, quantity, immediacy of communications channels affect communication efficiency. Figure for 6: Communication efficiency relies on number, warmth of communication channels used (7) Cost to develop software rises rapidly with physical distance. (This is a consequence of 4-6) Figure for 4: Trust depends on frequency and richness of communication (5) Frequency drops with distance. Figure for 7: Cost rises with distance (8) Cultural factors such as power distance, individualism, masculinity, uncertainty avoidance [Hofstede] link to ease of moving information. Low power distance (as one of several) makes it easier to surface uncomfortable information, high power distance makes it harder. Figure for 5: Frequency of communication drops with distance [Allen] Note: the curve is shown as continuous, but it is not. Discontinuities occur when a wall, a staircase, or drive time is injected. Figure for 8: National and organizational culture characteristics such as power distance affect inclination to surface bad news
page 4 HaT Technical Report 2016.01 March, 2016 (9) Organization tolerance to ambiguity, uncertainty and flux affects the possibility of using more productive techniques. The most productive techniques rely on parallel development, with embedded ambiguity, uncertainty and flux. However, not all people or organizations can tolerate high levels of those. To the extent they cannot handle them, they cannot use those techniques. This presents an upper limit to an organization s productivity. Figure for 9: Organization tolerance to ambiguity, uncertainty and flux affects the possibility of using more productive techniques 1c. Finite Cooperative Games Software development differs from some cooperative games: (10) Whereas the first goal of the game is to deliver the software system, there is a second goal present: to set the team in an advantageous position for the next game. The second goal comes about because the system will be extended or a neighboring one will be attached to it. The strategies for developing the system will be different if the system will never be extended. The second goal includes such topics as system documentation, quality and simplicity of the architecture and of the code, extensiveness of the tests, and training of the junior people to become senior people. (11) The game is rich, such that optimal moves are not obvious, and positions rarely repeat. Number 11 is implies there is no closed-form formula for winning the game, creativity is continually needed. Collecting strategies is an important professional activity. Figure for 11: Collecting strategies is important
page 5 HaT Technical Report 2016.01 March, 2016 (12) Opposing strategies can each be optimal, in different situations [PM Strategies]. 1d. On Methodologies (14) More critical projects need heavier or more predictive methodologies. [Boehm] Figure for 12: Opposing strategies can be useful, separately and together (13) Different projects need different sorts of strategies, processes and methodologies. Figure for 14: More critical projects need heavier or more predictive methodologies (15) Effectiveness per person drops as people are added to a project. Figure for 15: Effectiveness per person drops as people are added Figure for 13: Different projects need different strategies, processes and methodologies Figure for 13 shows projects organized by number of people needing to be coordinated, and criticality (damage is caused by an undetected defect). Other dimensions affect the choice, so that the space of appropriate methodologies or strategies is very large. Number 13 indicates that there is no possibility of finding one, ultimate process or methodology for all projects, but that, rather, each project team should find its own, localized methodology in order to get optimal results. Number 15 is a straightforward consequence of the communications and coordination load increasing as people are added to a project.
page 6 HaT Technical Report 2016.01 March, 2016 (16) A little methodology goes a long way. At the beginning, the rules used to coordinate people s work helps; as the rules and ceremonies increase, effectiveness per person drops. 1e. Strategies for handoffs (18) Concurrent development can be faster but more expensive than serial development. In comparing waterfall, pipelined, or serial development with concurrent development, (17) shows that well-executed serial development, although taking longer, may represent a cost optimization. Conversely, well-executed concurrent development may speed development and come in with a higher cost, due to the rework needed. Figure for 16: Methodology improves a team, with diminishing returns (17) As problems grow in size, more people might be needed, and consequently, heavier methodologies. Figure for 18: Concurrent development can be faster but more expensive than serial development ` Figure for 17: Bigger problems need more people and hence heavier methodologies (19) The optimal moment for handover to a downstream group depends on the stability and completeness of the upstream group. As a group is limited in ability to keep up with work and rework, it is better to delay their start. Conversely, as a group has capacity for work and rework, it becomes optimal to advance their start. Figure for 19: The optimal start moment varies by work / rework capacity
page 7 HaT Technical Report 2016.01 March, 2016 Number 19 will be applied in the section on decisions as inventory and deriving processes by examining bottleneck stations. (20) Not every organization can use every methodology, as the most productive techniques make use of ambiguity, uncertainty and flux. Not every organization can tolerate the amount needed for the technique. Number 20 shows a limiting factor on methodology or technique adoption by organizational culture. 2. Decisions as Inventory (21) Design looks like manufacturing if the unvalidated decision is considered the unit of internal inventory. Figure for 21: Design resembles manufacturing if unvalidated decisions are the unit of internal inventory. Number 21 allows all of manufacturing theory, including lean manufacturing, to be adopted in design circumstances. Figure 20: Not every organization can use the most productive techniques (22) Concepts of work-in-progress limits, queues, continuous or single-piece flow, and bottleneck stations are applicable to design. Figure for 22: Queuing theory applies to the movement of decisions in the network
page 8 HaT Technical Report 2016.01 March, 2016 (23) The optimal process depends on the location of the bottleneck stations. 3. Strategic Learning (24) Batching decisions increases costs. Figure for 23: Queuing theory applies to the movement of decisions in the network Number 23 is an extension of number 13, coming from the examination of the bottleneck stations. From 23 and 19, one can derive handoff points and rework strategies to optimize the global efficiency of the network (see [Efficiency]). Figure for 24: Delaying integration and deployment increases the number of decisions needing to be validated at one time, increasing cost. The network of decision-making includes markets, technologies, requirements, technical design, testing and scheduling. The figure illustrates that if integration or deployment is batched, many interlocked decisions need to be (in)validated at the same time, raised cost of discovery and repair. Reducing that number reduces the cost of detecting and correcting mistakes. (25) Small decision batches allows for varied strategies to reduce risks and learn faster, with an option to trim the tail on feature requests to shorten the project time. Figure for 25: Small decision batches allow strategies to reach high value sooner
page 9 HaT Technical Report 2016.01 March, 2016 Number 25 opens strategies to more quickly learn what consumers want, how teammates work, what technical problems lie ahead, the cost of development, what features should be enhanced or what left bare, how best to trim the tail on the feature set and implementation quality to fit timing needs [Disciplined Learning]. (26) Features are mandatory, optional, or attractive / delighters [Kano]. Customer Satisfaction Very Satisfied Attractive One-Dimensional 4. Craft (28) Every role in the cooperative game operates in a craft profession. Being in a craft profession implies lifelong learning and deepening the mastery of the profession, learning to work with the craft medium. In the case of managers, the medium is people; in the case of user interface designers, the characteristics of the input and output devices; in the case of programmers, it is the properties of the languages and components they use, and so on. Number 28 implies that software development must include practice and development. Not at All Indifferent Fully Degree of Achievement Must-Be (29) People develop skills in four stages (shu, ha, ri, kokoro) Reverse Very Dissatisfied Figure for 26: Subjective quality separates mandatory, linear, delighter features (27) Delighters introduced even before the end of the linear or value-development period can give the development team extra time to develop the value features. Figure for 27: It can be useful to introduce delighters before the end of the value section Figure for 29: Stages of craft mastery: shu, ha, ri, kokoro. Shu, Japanese for follow, is the stage of learning an initial technique for a skill. Ha, or break, is the stage of collecting techniques. Ri, or leave, is the stage for improvising, inventing and combining, often without conscious thought about it. Kokoro, or heart/spirit, is the radical simplification, particular to teaching the skill. Ri and shu are particularly conflicting. While beginners start by learning one technique, advanced practitioners know that no one technique can serve for all situations. Thus, both are needed, shu for newcomers, and ri for operating at the level needed for international competition.
page 10 HaT Technical Report 2016.01 March, 2016 5. Personalities (30) The personalities of the actual people override any theory about them. Figure for 30: People s personalities may not match the stereotype for their job title A person who occupies a role may not have the personality characteristics presupposed in the job role. A non-communicative team lead or an indecisive project manager can have a large negative effect on a project. Two people who have had a bad history together, such as a recently divorced couple, can poison an entire project atmosphere. In general, the effects on a project of individual people are unpredictable. Number 30 limits any theory of software development. Conclusion Taken individually, each of the elements of this theory adds understanding of the trajectories that projects follow. They fit the desiderata for a theory of software development or software engineering management, in that they explain why successful projects succeed and failing projects fail, predict important issues in running projects, lead a person on a project to derive sensible advice as to how to proceed. Taken together, they cover a very large part of what we observe before and after intervening in projects using the theory. The elements have been published between 1998-2015, have been used on real projects since 1995, have been taught to professionals since 1998, to undergraduate students since 2006, as they have been uncovered. Thus, the theory elements at least partially meet the final desideratum, to: provide a sound pedagogical basis for teaching.
page 11 HaT Technical Report 2016.01 March, 2016 References Most of the elements of the theory are described in Agile Software Development, 2 nd ed: The Cooperative Game (Cockburn, A., Pearson Education, 2006). Additionally: [Efficiency] Cockburn, A., Spending Efficiency to Go Faster, available online at http://alistair.cockburn.us/%22spending%22+efficien cy+to+go+faster [Hofstede] Hofstede, Geert. National Culture. http://geert-hofstede.com/national-culture.html. [Allen] Allen, T., Managing the Flow of Technology, The MIT Press, 1984. [Boehm] Boehm, B., Turner, R., Balancing Agility and Discipline, Addison-Wesley, 2003. [Disciplined Learning] Cockburn, A., Disciplined Learning, CrossTalk magazine, 2015. [Kano] Lofgren, M.; Witell, L., Kano s Theory of Attractive Quality and Packaging, Quality Management Journal, vol. 12, no. 3,,July 2005, pp. 7-20. [PM Strategies] Cockburn, A., online at http://alistair.cockburn.us/project+management+strategies.p pt.