White Paper Integrating Six Sigma and Software Testing Process for Removal of Wastage & Optimizing Resource Utilization 24 October 2013 With resources working for extended hours and in a pressurized environment, there comes a situation where knowingly or unknowingly we have to compromise with the quality. The reason behind this is that everyday resources have to spend time on doing repetitive tasks which consumes a lot of time and effort thus impacting the project timelines, budget and increasing associated with the quality. Author Ankit Tuteja ankit.tuteja@crestechglobal.com
1. Introduction In the past few years we have found that every IT organization whether it is a Service based industry or a product based industry, are emphasizing a lot on QA & Testing in their projects. Either they are hiring testers from outside to build there in house testing teams or they are exponentially increasing the size of their existing teams or they are outsourcing it to other testing companies. But in all the cases there is one major problem that can be seen, even with the increased size of these teams often there comes a situation in which we are not able to meet the deadlines. With resources working for extended hours and in a pressurized environment, there comes a situation where knowingly or unknowingly we have to compromise with the quality. The reason behind this is that everyday resources have to spend time on doing repetitive tasks which consumes a lot of time and effort thus impacting the project timelines, budget and increasing associated with the quality. Current Scenario Taking an example of the QA team of a reputed travel product based company, on an average each resource is expected to work for 48.5 hours weekly (9.5 x 5 hours (weekdays) + 1 hour (average working time of 1 resource every Sunday)). This is the ideal condition that should happen but this is not the case in reality. As per the data collected from the timesheets of the team members, the working hours came out to be 63 hours weekly (12 x 5 hours (weekdays) + 3 hours (average working time of 1 resource every Sunday)) which is 13.5 hours overhead of the actual working time of single resource every week. But even working for these extended hours the team is not able to meet its deliverables on time & in the budget allocated to it with the risk of compromising with the quality. This is an example of one organization but this is the story of almost every other IT industry which makes this topic a good study to implement Six Sigma techniques so as to improve the overall process thus resulting in the effective Resource Utilization and Optimizing the costs related to the project delays. With the help of Six Sigma Concepts and using the DMAIC approach in our projects we can actually overcome this problem to a great extent by minimizing wastage of time in doing repetitive tasks and by automating various manual activities which can actually increase the daily productivity of a resource. Problem Statement The purpose of this paper is to establish Process Improvement Techniques to rapidly address concerns that sometimes why even working for extended hours in a project the testing teams are not able to deliver modules on time and have to risk the quality. We will be closely analyzing the existing performance reports, examining source data, identifying inconsistencies in existing processes, developing standard control procedures, and implementing improvements to ensure optimum utilization of all the resources and delivering modules with quality. Goal To design and develop a complete set of solutions to address the root causes behind the wastage of time on daily repetitive tasks which are not required that the QA team have to work on. We will now determine the full extent of the problem through data analysis, interviews, and other tests and develop solutions for improving the processes and monitor the results of the implemented solutions. Business Impact No matter how small the project is, its timelines & quality matters a lot. On the basis of present projects, the timelines, budgets and pipelines of the future projects are decided. If the current process is broken then there are material distortions to the number of hours that should be spent on a project than the actual number of hours that are
currently being spent. This study will attempt to fix this broken process by suggesting various improvement techniques through which we can identify major critical problems that impacts our deliveries in terms of resource utilization and then carefully designing and implementing solutions to overcome these problems hence optimizing overall resource utilization. This study will also give management the correct picture of the total man hours actually spent on a project so that they can accordingly plan the upcoming projects in terms of Cost, Time & Quality. 2. What is Six Sigma? The term Six Sigma originated from terminology associated with manufacturing, specific terms associated with statistical modeling of manufacturing processes. The maturity of a manufacturing process can be described by a sigma rating indicating its yield or the percentage of defect-free products it creates. A six sigma process is one in which 99.99966% of the products manufactured are statistically expected to be free of defects i.e. 3.4 defects per million, although, this defect level corresponds to only a 4.5 sigma level. Motorola set a goal of "six sigma" for all of its manufacturing operations, and this goal became a byword for the management and engineering practices used to achieve it. Six Sigma Tools & Techniques Figure 1 As shown below, Six Sigma is based on a DMAIC approach that comprises of 5 sequential phases which are Define, Measure, Analyze, Improve & Control which leads to an improved process at the end. Figure 2 The term "Six Sigma Process" comes from the notion that if one has six standard deviations between the process mean and the nearest specification limit, as shown in the graph below, then practically no items will fail to meet the specifications. This is based on the calculation method employed in process capability studies.
Capability studies measure the number of standard deviations between the process mean and the nearest specification limit in sigma units, represented by the Greek letter σ (sigma). As the process standard deviation goes up, or the mean of the process moves away from the center of the tolerance, fewer standard deviations will fit between the mean and the nearest specification limit, thus decreasing the sigma number and increasing the likelihood of items outside specification. Figure 3 There are various methods or the tools & the techniques that are used in the DMAIC approach for e.g. SIPOC diagram, CTQ characteristics, Pareto analysis, etc. We will be covering them in details in the coming phases. 3. Integrating Six Sigma with Software Testing process In this study, we are using a word Defect. This term defect is not related to the one in testing. It is a term used in Six Sigma which means cause of errors which affects our process. Below is the diagram which shows us that how can we incorporate the 5 phases of Six Sigma in our testing projects so as to identify the Critical problems that impact our projects, propose their solutions and can continuously monitor them.
3.1 Define Phase Figure 4 Consider below SIPOC diagram applicable to all software testing lifecycles. This is a very high level diagram to explain the characteristic flow of STLC. Figure 5 To go into further details, below is a detailed functional flow diagram to understand the system deeply.
Figure 6 This diagram describes the standard software testing process in any organization i.e. how a developer gets requirements from the user, how the QA gets the files for testing from the developer, how developer & QA interacts in a bug lifecycle, how UAT is done and finally how the product is released in the market. Critical to Quality Characteristics CTQs are specific factors or attributes that are associated with the products, processes, and services that customers consider extremely important. This is often identified by listening to the customers through surveys, interviews, complaints, and other ways for fully understanding the requirements of the customers. Six Sigma professionals refer to this as Voice of the Customer or VOC. Once you have captured VOC, you can then translate this information into a CTQ tree diagram. Considering the e.g. of a travel product based company again, we had a discussion with the entire QA team (who are our end users in this case) to capture there problem areas. On further research it is found that these problems are not limited to this organization but almost to every other organization who is in the process of software development & testing. On the basis of that a standard CTQ tree is derived as shown below.
3.2 Measure Phase Figure 7 Pareto Analysis Few projects from different domains like travel, CMS and telecom are sampled to perform some analysis. After doing brainstorming sessions with QA teams of these projects, we have shortlisted 5 repetitive tasks which if minimized will reduce the overall time spent on a project. These tasks are: T1 T2 T3 T4 T5 Manual sanity of the build after every release Deployment on testing servers Inadequate product knowledge Rework due to requirement not clear Test Case authoring/scripting Table 1 Based on the data collected from the sample projects which is the average number of hours spent by each resource on the above tasks, we have prepared a Pareto Chart as shown below: 40.0 35.0 35.1 30.0 25.0 21.6 20.0 15.0 16.2 13.5 13.5 10.0 5.0 0.0 ual sanity of the build Deployment after every release on testing Test servers Case authoring/scripting Inadequate product Rework knowledge due to requirement Contributor Figure 8 # of Hours (per day) % Cum
Manual sanity of the build after every release 6.5 35.1 Deployment on testing servers 4 21.6 Test Case authoring/scripting 3 16.2 Inadequate product knowledge 2.5 13.5 Rework due to requirement not clear 2.5 13.5 Table 2 The results of the Pareto analysis help us to prioritize our problems (P1, P2, P3, P4, P5) and if the time spent on these problems can be minimized then this will result in overall process improvement in our testing projects. Data Collection Procedures Data is the most important part of this study and we have used various tools specific to our sampled projects like Bugzilla, Clarizen, etc. to generate various reports so as to collect data at resource level and then using various techniques like attribute MSA, gauge R&R, etc. for analysis of this data. For our study, we have taken the data of 3 months (Nove 12-Jan 13) of travel product company that we have discussed above. This data shows that how much time a team of 12 QA resources have spent on the problems P1-P5 for 3 months working on 179 different projects. Below is the pivot chart representing that data: Figure 9 Also, below is a pie chart that shows us how much % of time each QA resource have spent out of his/her total productive time on working on problems P1-P5.
Figure 10 Calculations The calculation of a Sigma level, is based on the number of defects per million opportunities (DPMO). In order to calculate the DPMO, three distinct pieces of information are required: a) The number of units produced b) The number of defect opportunities per unit c) The number of defects Using the formula: DPMO = (Number of defects X 1,000,000) (Number of Defects Opportunities per Unit X Number of Units) Where, Total number of projects examined or Number of Units 179 Number of projects in which working in extended hours required or Number of Defects 97 Total number of resources or Number of defects opportunities per unit 12 Hence, DPMO is 45158.28678. Using the Sigma table below, the current Six Sigma value will come out to be 3.19. Six Sigma Table: Value DPMO 1 690,000 2 308,000 3 66,800 4 6,210 5 320 6 3.4 Table 3 After examining other sampled projects we found that there sigma rating is also around this value itself. Summarize Test Results
To find the critical defects that need our immediate attention, we again carry out Pareto Analysis. Below are the results for the same. Type of Defect or Error Estimated Impact of Defect (hours) % of Total P1 1512 32% < Critical Defect P2 1116 23% P3 900 19% P4 630 13% P5 630 13% Total 4788 Table 4 2000 Impact of Number of Defects 1500 1000 500 0 Figure 11 The above Pareto analysis shows that if we work on our critical problem P1 then this will overall increase our performance to a great extent (based on 80-20 principle). Histogram P1 P2 P3 P4 P5 To go into further details of problem P1, we prepared a histogram based on the data collected from travel product based organization. Twelve observations were made over 12 QA resources regarding the time spent by each QA resource on the task P1 (Value) which is considered as the Critical Defect. Observation Value 1 179 2 51 3 93 4 173 5 140 6 122 7 52 8 85
Frequency 9 185 Maximum 185.4 10 173 Minimum 51.2 11 152 Range 134.2 12 119 Table 5 Number of Intervals based on the Square Root of Observations = 3.464101615 rounded off to 3 Minimum intervals required = 5 Width of each interval based on Range / (No. of Intervals - 1) = 33.56 rounding off to 34 Interval Description Frequency 1 Less than or Equal to 51 1 2 Between 52 and 85 2 3 Between 86 and 119 2 4 Between 120 and 153 3 5 Between 154 and 187 4 Table 6 Frequency Distribution 4.5 4 3.5 3 2.5 2 1.5 1 0.5 0 1 2 3 4 5 Intervals Freq Figure 12 In the above Histogram we can see that we have a defined peak which further proves that working on problem P1 will definitely enhance our overall process. 3.3 Analyze Phase In this phase we will analyze the root cause of our problem P1 i.e. time spent on doing manual sanity of build after every release. Fish Bone Diagram Discussing our Critical Defect with experienced QA team members and using various brainstorming techniques, below are major causes that we ve listed responsible for spending more than the required number of hours on doing sanity of the build after every release: 1. Manually executing test Cases which can be automated 2. Not enough knowledge of new features that impact our sanity
3. Improper KTs in case of new resources 4. Server Slow 5. Ambiguity in requirements from Dev end Doing a fish bone analysis of these problems we get a diagram as mentioned below: Figure 13 From this diagram we can explore the causes of our problem P1 and can provide a solution to it by discussing with SMEs or doing some brainstorming sessions. Other analytical techniques like FMEA, 5 Why analysis, etc. can also be used to perform root cause analysis of our problem P1. Scatter Plot A scatter plot is used when a variable exists that is under the control of the experimenter. In our scatter plot, we will use time spent on problem P1 as a variable (y-axis) and total time spent on project as a constant (x-axis) and will show a relationship between these two. Step 1 - Enter the X and Y data that we want to plot. This is a simple regression analysis i.e. we want to see if a change in the X value correlates in some way to a change in the Y value. X Values Y Values 19 6 140 57 60 21 57 18 20 6 For total number of hours spent on a project, 22 6 how does it correlate with the 14 3 number of hours spent on P! 12 3 22 6 30 9
125 45 Step 2 - Insert a Scatter Chart with Trendline: Table 7 Figure 14 From the scatter plot above we can clearly see that the points are equally scattered around the line. This shows a strong dependency of our variable which is the number of hours spent on problem P1 on our constant which is total number of hours on manual sanity spent for a project. 3.4 Improve Phase After all the measurement and analysis, now it is time to actually implement our observations in the practical scenarios, measure the changes and monitor the progress that occurs by implementing 6 Sigma methodologies. Solutions Rating Matrix It is a 5 step process in which we jot down all the possible solutions, rate them on various characteristics and then on the basis of these ratings we decide which solution is best to implement and will give effective results. Step 1: Establish Criteria for Rating Your Solutions Step 2: Describe each rating from lowest to highest for each of the criteria in Step 1 Table 8 Step 3: Describe each solution and assign a rating as defined in Steps 1 & 2
Table 9 Step 4: If you want to weight the criteria so that some criteria are more heavily considered than others, then assign weights and recalculate the ratings Table 10 Step 5: Place each solution into the following Cost Benefit Matrix to quickly assess its return Cost - All resources to implement, such as funding, personnel, time, etc. Benefit - Both tangible and intangible benefits such as fewer errors, reduces time, improves customer satisfaction, lowers costs, etc. Figure 15 Benchmarking Report We need to benchmark the points for the existing process which we will use for comparison after the process has been improved. Again considering our travel company example, taking 4 major projects, we will identify the number of hours spent on the tasks P1-P5 and will take the minimum time as Best in Class. Project ID P1 P2 P3 P4 P5 64864 25% 21% 21% 16% 10% 64880 9% 7% 7% 5% 3% 66414 21% 16% 16% 12% 8% 68782 20% 15% 15% 11% 7% Best in Class 9% 7% 7% 5% 3%
Average Performance 19% 15% 15% 11% 7% Table 11 Figure 16 To ensure that there is no leakage in the improvement methods that we have suggested, we can perform error proofing of our Critical Process. This can be a simple Q&A process in which we can have some questions like What is a defect, What are sources of documents, What positions are responsible for running this process, etc. and can have a discussion on these questions with SMEs, QA team and other relevant stakeholders. After this is required a corrective action plan stating our immediate action items and long term plans. 3.5 Control Phase This phase will suggest various control methods & techniques to make sure that the solution that we have provided will be implemented more effectively. Control Plan for Critical Process Below is a sample control plan which can be used to monitor the entire process. It will be continuously updated based on the coming updates and changes related to this project.
Reaction Plan Table 12 To avoid all the breach in the process, monitor defects and take appropriate actions wherever leakage occurs, a reaction plan is designed. This reaction plan is shared with all the relevant stakeholders. Below is a reaction plan that we have designed to cater to our solution proposed for the problem P1.
Before After Trend Chart Figure 17 According to our analysis, if this solution is implemented as described then we can see some good changes in the overall testing process of a project. Below is a trend chart that we plotted to show the difference between the performance of the current process and the expected process for the travel product based org. For current performance we used the data available for the time period of November, 2012 to January, 2013 and predicted the result for consecutive 3 months of the new solution implementation. Table 14 Conclusion Figure 18 This is a study which is carried on a sample of standard testing projects of various domains like Travel, CMS, etc. In some testing projects, the defects may be different but the overall solution to the problem using the methods of DMAIC as suggested in this paper will remain the same. Considering again our travel product based organization s project, as per our predicted results, we can see the total number of defects coming down from 45% to 21% as per the above trend chart. This means that our critical defect that is the number of hours spent on sanity will reduce from 42 hours per QA resource per month to 28 hours per QA resource per month. Also, if everything goes as expected then we can achieve new sigma value of 3.54 from the current sigma value of 3.19 which means a reduction of approx. 24,209 defects per million. References Six Sigma Overview: http://en.wikipedia.org/wiki/six_sigma About the Author Ankit Tuteja, a technology & quality enthusiast is currently working as an Associate Team Lead Testing at Crestech Software Systems Pvt. Ltd. He is an ISTQB certified software tester and has achieved Green Belt in Six Sigma from Indian Statistical Institute, New Delhi. He is having more than 3 years of experience in Testing & Quality Analysis.