Aligned with your needs. Improving Performance with Enhanced Measures and Causal Analysis in an Agile Organization Paul E. McMahon CMMI/Agile Consultant, PEM Systems Dr. Bela Joshi Sr. Principal, Alion Science and Technology
Slide 2 Topics Overview Background Motivation for 2013 Improvement Effort How Improvement Effort was Conducted Defect Analysis Results Achieved Validation Implementation of Improvements Plan Forward/Questions
Slide 3 Overview The Advanced Modeling and Simulation Technology Operation (AMSTO) of Alion Science and Technology is a CMMI-DEV Maturity Level 3 organization In 2013 AMSTO undertook an initiative to improve performance through enhanced measures and causal analysis In this presentation we discuss the analysis conducted and the resulting improvements
Slide 4 Background AMSTO provides technical solutions to the Department of Defense mainly in modeling and simulation arena Majority software projects use a tailored version of Scrum AMSTO s process journey began in 2003 CMMI-DEV Maturity Level (ML) 3 in 2007 Maintained ML 3 with re-appraisals in 2010 and 2013 Lightweight processes, minimal essential must-dos supporting agile approach
Slide 5 Motivation for 2013 Improvement Effort AMSTO successful with happy customers providing repeat business Committed to CMMI ML 4/5 since 2010 In preparation for 2013 re-appraisal CMMI/Agile consultant encouraged maturing Measurement and Analysis (MA) process area While MA was institutionalized, significant potential benefits believed to exist
Slide 6 How the Improvement Effort was Conducted Initial brainstorming (EPG, Project personnel, consultant) considered: Enhancing retrospectives Task estimation improvements Defect analysis improvements Multiple improvements ensued This report focuses on defect analysis and improvements focused on reducing escapes
Slide 7 Defect Analysis Defect analysis performed on successful on-going project with well established processes and mix of personnel experience level Software product undergoes joint verification event with government customer before release in end-user environment
Slide 8 Defect Analysis (continued) Step 1: Extract key information about each escape by interviewing developers Goal: Determine root cause using 5 Why Technique Step 2: Categorize escapes according to root cause Step 3: Use Pareto principal to apply improvements to category with most escapes Crucial step for management buy-in Initial results: Pareto principal did not hold Close analysis: Requirements most often underlying cause
Slide 9 Defect Analysis (continued) Step 4: Search for assignable causes, and ask developers for suggestions to keep escapes from occurring in future Step 5: Report result to Senior Management with time effort took Crucial to selling approach to management as part of AMSTO MA process
Slide 10 Data Analysis Defects count by source prior to 5 Why analysis Defects count by source after 5 Why analysis
Slide 11 Results Achieved Two most common patterns: Inadequate requirements Inadequate regression testing Root cause: Weakness in Requirements Development process Poor requirements lead to inferior product and inferior tests
Slide 12 Validation Needed to validate findings before making significant changes to process that had worked successfully for many years Compared study results with observations from periodic gap analyses and training workshops conducted by CMMI/Agile consultant AMSTO teams frequently work on cutting edge solutions in research and development environment Incomplete, ambiguous, vague requirements often surfaced One reason why iterative agile/scrum cycle worked well
Slide 13 Validation (cont.) While good collaborative relationship with customers existed, due to busy schedules and priorities, customers not always available for clarifications Thus project teams often had to make assumptions when producing test scripts The results were thus validated in this manner
Slide 14 Implementation of Improvements No assignable causes or quick fixes Practitioners would need to be trained and mentored on how to develop complete, unambiguous, and testable requirements Improved requirements development guidelines were developed Included concrete and practical methods to develop requirements under the common conditions teams often faced
Slide 15 Plan Forward Improved requirements development guidelines being deployed and trained Plan to continue monitoring escapes to determine if changes lead to measurable improvements, or if further improvements needed
Slide 16 Questions and contact information Paul E. McMahon Ph: 607-798-7740 Email: pemcmahon@acm.org Web: WWW.PEMSystems.com Dr. Bela Joshi Ph: 757-793-4372 Email: bjoshi@alionscience.com Web: WWW.Alionscience.com