Expert Judgment in Groups Chapter 13 Group judgments when estimating early in the project or for estimating LARGE unknowns. 3 Simple Rules: 1. Each team member estimates pieces of the project individually then meets with others to compare estimates 2. Don t just average our estimates discuss the differences 3. Arrive at a consensus if not, discuss differences and get buy-in from all
Figure 13-1 Reviewed Estimates Individual Estimates Individual Estimates: MRE 55% Group Reviewed Estimates: MRE 30% 92% of Group Estimates better than Individual Estimates How many experts are enough? 3 to 5 with different backgrounds
Wideband Delphi Technique 1. Each estimator gets a specification and an estimation form 2. Estimators prepare initial estimates 3. Group meets to discuss estimation issues related to the project. If there is early agreement, a devil s advocate is chosen 4. Estimates are submitted anonymously 5. Summary is represented with an iteration form 6. Estimators meet to discuss variations in their estimates 7. Estimators vote anonymously on the average estimate If there are no votes, go to Step 3 Average Estimate 0 5 10 15 20 Staff Months 8. The final estimate is the single point estimate or a range agreed to in the discussion (with the single point estimate being the expected value)
Effectiveness of Wideband Delphi Author s experience For the first 25 groups I worked with Wideband Delphi reduces estimation error by an average of roughly 40% Two-thirds of his groups produced more accurate estimates using Wideband Delphi 10 Groups that produced the worst estimates, 8 of 10 improved accuracy by 60% Author is a believer especially for avoiding erroneous results. Law of Large Numbers? The correct answer lies somewhere between the lowest and highest estimate But 20% of groups initial estimation ranges did not include the actual number
The Truth is Out There Correct Answer Range of Initial Individual estimates In none of the observed cases Did the final Wideband Delphi Estimate move in this direction (out of the initial range, away From the correct answer) In one third of cases, the final Wideband Delphi estimate moved Into this range (out of the original range, toward the correct answer)
When to Use Wideband Delphi Where in the wide part of the Cone of Uncertainty Author s experience Reduced estimation error from 290% to 170% (a 41% reduction) 170% error is high! A reduction from 50% to 30% is a 40% reduction NOTE The studies give this method mixed reviews. The approach is costly burning hours of staff time But again early in the Cone there may be no better approach. (project is complex, new, etc.)
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Planning Defining and Scoping Large problems decomposed Estimated separately, possibly by different team Provide all the information available Participants (experienced and trained) Moderator Project Manager 2 to 4 other estimators
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Kickoff Up to an hour Estimators given problem specs and whatever assumptions or project constraints apply Review estimation objectives Discuss problem and whatever estimation issues exist Moderator decides if the team is ready to roll At this point: Appropriate team members selected Kickoff meeting complete Team agrees to estimation goal and units PM can participate Estimators have the information needed to participate effectively
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Individual Preparations Participants develop initial list of tasks Each estimates the effort needed for each task Break each activity into tasks that are more easily estimated Rule: no task should take more then 20 hours Record any tasks for related or supporting activities Record your assumptions Guidelines: Assume one you will perform the tasks Assume all tasks performed sequentially Assume you can devote uninterrupted time to each task In calendar time, list any known waiting times you expect
Estimation Form Task Estimate #1 Change #1 Change #2 Change #3 Change
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Estimation Moderator collects estimates & creates chart Round 4 Round 3 Round 2 Round 1 0 500 1000 1500 2000 2500 Labor Hours (tasks only) Estimates are anonymous Estimators each read their task list (include assumptions and whatever questions or issues they want addressed)
Estimation Combining task lists leads to a more complete list than any one estimator produces limit maybe several dozen Discuss assumptions, estimation issues and questions about the problem Converging on a shared set Then each estimator modifies their estimates concurrently Revising their task list revising estimates based on new info Moderator collects estimates and plots the next Round Additional rounds with same type of discussions will narrow the distribution
Estimation Continue, until: Completed at least 4 rounds Estimates have converged to an acceptable range (defined in advance) Allotted estimation meeting time is over (typically 2 hours), or All participants are unwilling to alter their latest estimates Moderator roll: Controls meeting Time-boxed discussions (15 to 20 minutes) Facilitate particpation Preserves anonymity of estimators
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Assemble Tasks Defining and Scoping Large problems decomposed Estimated separately, possibly by different team Provide all the information available Participants (experienced and trained) Moderator Project Manager 2 to 4 other estimators
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Assemble Tasks Create single Master Task List Merge individual assumptions, quality- and process-related activities, overhead tasks and wait times. Note. All estimators should define their tasks clearly, so that merging is consistent (no same task name, different specs)
Wideband Delphi Process Flow Planning Kickoff Individual Preparation Estimation Assemble Tasks Review Results Review Results Team reviews summarized results and reaches agreement (buy-in) on final result 30- to 60-minute review meeting Retrospective on the process identifying possible improvements
Completing the Estimation Exit criteria: The overall task list has been assembled Summarized list of estimating assumptions is complete Estimators have reached consensus on how their estimates were used and synthesized into a single set with an acceptable range.