Delphi Estimation Process Document Page 1 of 9

ABSTRACT

This document is the Generic Delphi Estimation Process document for the generic maintenance process for the
University of West Florida. This document describes the estimation process, the activities required to be performed,
and the associated responsibilities of each team member for estimating project duration for the generic maintenance
process architecture. The method described herein employs the Delphi estimation technique.

TABLE OF CONTENTS
ABSTRACT
LIST OF FIGURES
LIST OF TABLES
1.0 INTRODUCTION
1.1 BACKGROUND INFORMATION
1.1.1 History of the Delphi Estimation Process
1.1.2 Employment of Delphi in Other Organizations
1.1.3 Results of Other Estimation Methods
1.1.3.1. GUMP Estimation Methodology
1.1.3.2. Implementation of GUMP Estimates
1.1.3.3. GUMP Historical Data
1.1.3.4. Accuracy of GUMP Estimates
1.2 Advantages of the Delphi Estimation Process
1.3 Disadvantages of the Delphi Estimation Process
2.0 DELPHI ESTIMATION PROCESS
2.1 ROLES AND RESPONSIBILITIES
2.1.1 The Delphi Estimation Team
2.1.2 Responsibilities
2.2 PROCEDURES
3.0 DOCUMENTATION
REVISION HISTORY
APPENDIX A
BIBLIOGRAPHY
GLOSSARY

LIST OF FIGURES
Figure 1. Delphi Estimation Form

Figure 2. Sample First Round Estimation Form

Figure 3. Sample Second (or subsequent) Round Estimation Form

LIST OF TABLES
http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 2 of 9
Table 1. GUMP Estimation Accuracy

1.0 INTRODUCTION
1.1 BACKGROUND INFORMATION

1.1.1 History of the Delphi Estimation Process

The Rand Corporation developed the Wideband Delphi Technique in 1948. Its goal was to provide a process for
achieving expert consensus without group conflict. Its initial purpose was the estimating of probable effects of a
massive atomic bombing attack on the United States. It has subsequently been applied in the technological forecasting
area and its use has proliferated in the United States and abroad.

The Wideband Delphi Technique has been applied to normative forecasting, ascertainment of values and preferences,
quality of life issues, simulated and real decision making. Delphi has been applied to a process called "inventive
planning" by which is meant the identification including invention of potential measures. These measures might be
taken to deal with a given problem situation and the assessment of such proposed measures with regard to their
feasibility, desirability, and effectiveness.

This procedure states that a group of experts is formed and asked not to discuss their work with one another. They give
estimates of the product individually and anonymously. These estimates are collected and a coordinator assesses them.
This assessment has to fall within an acceptable range, otherwise the coordinator asks for revised estimates, this
process continues until a consensus is reached.

Overall, this process usually works well, but may be expensive and time consuming.

1.1.2 Employment of Delphi in Other Organizations

The Wideband Delphi technique is widely taught in universities and software organizations. After researching
estimation techniques used by software organizations, we found a very interesting tenet. Several software organizations
utilize the technique, without claiming the inheritance. Thus, even though there are few organizations who claim
utilization of the Delphi technique, the methodology is widely prevalent. The Wideband Delphi Technique is also used
by many organizations outside the software industry. Once again, it seems that the software industry is behind the
power curve in adopting useful methodologies resident in other engineering organizations. Thus, our initiative to make
the Wideband Delphi technique an integral part of GUMP.

1.1.3 Results of Other Estimation Methods

1.1.3.1. GUMP Estimation Methodology

GUMP estimates are presented in the form of man-hours for each tisk. The estimated man-hours are compared to the
actual man-hours documented for tisk completion. Estimates are arrived at based on prior experience and best guess.

1.1.3.2. Implementation of GUMP Estimates

The GUMP software estimates are accomplished in process block 300 (Analyze) and are documented in the tisk
analysis report. The estimates in the tisk analysis report typically cover detailed design, implementation, unit testing,
and independent verification and validation. The actual man-hours are based on time card data and are recorded in the
metrics summaries. The metrics summaries list man-hours for process blocks 500 (Implementation), 600 (IV&V), and
700 (Wrap Up).
http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 3 of 9

1.1.3.3. GUMP Historical Data

The historical accuracy of GUMP estimates is shown in Table 1. The estimated hours were obtained from tisk analysis
reports. The actual hours came from metrics summaries. The variance was calculated using the formula:

Variance = ((Estimate / Actual) - 1) x 100%

A variance of zero indicates the estimate exactly matches the actual. A positive variance indicates an overestimate. A
negative variance indicates an underestimate. As the magnitude of the variance increases, the accuracy of estimate
decreases. A positive variance represents the percentage of actual hours that were included in the estimate, but were not
used. A negative variance represents the percentage of actual hours that were not accounted for in the estimate. In tisk
T005, for example, 29 hours were estimated and 71.5 hours were actually expended. Therefore, 59.4 percent of the
actual hours were not included in the estimate.

ESTIMATE ACTUAL VARIANCE

TISK
( HOURS ) ( HOURS ) (%)
T003 185.00 111.8 +65.5
T005 29.00 71.5 -59.4
T006 82.00 173.5 -52.7
T007 20.00 22.5 -11.1
T008 48.00 229.0 -79.0
T009 65.00 202.0 -67.8
T010 56.00 112.0 -50.0
T011 23.25 125.7 -81.5
T015 37.00 30.8 +20.3
T016 120.00 74.5 +61.1
T017 366.00 526.5 -30.5
T018 28.00 87.0 -67.8
T021 27.00 36.8 -26.5
T022 7.50 30.5 -75.4
T023 2.00 44.0 -95.5
T024 15.50 91.0 -83.0
T025 55.00 71.0 -22.5

Table 1. GUMP Estimation Accuracy.

1.1.3.4. Accuracy of GUMP Estimates

Based on the data in Table 1, GUMP estimates were off by an average of 55.9 percent. The estimates also tended to
underrate the actual work required. In 82 percent of the cases the effort was underestimated. The historically high level
of inaccuracy in the GUMP estimates indicates that adoption of the Delphi estimation method may be beneficial.

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 4 of 9
1.2. Advantages of the Delphi Estimation Process

z Free of social pressure, personality influence, and individual dominance

z Allows sharing of information and reasoning among participants
z Conducive to independent thinking and gradual formulation
z Respondent panel provides broad analytical perspective on problems and issues
z Can be used to reach consensus among groups hostile towards each other

1.3. Disadvantages of the Delphi Estimation Process

z Judgments are those of a selected group, and may not represent prevailing opinion
z Tendency to eliminate extreme positions and force middle-of-the-road consensus
z More time-consuming than nominal group process
z Requires skill in written communication
z Requires adequate time and participant commitment (may require 30 to 45 days to complete entire process)

2.0 DELPHI ESTIMATION PROCESS

2.1 ROLES AND RESPONSIBILITIES

2.1.1 The Delphi Estimation Team

The Delphi Estimation Team shall consist of at least four estimators. The estimators will be, as a minimum:

a. The Project Coordinator (PC)

b. At least one representative from IV&V
c. Optimally four Software Engineers, but as a minimum two Software Engineers

EXCEPTION: When Delphi is being used to estimate an IV&V process, the estimation team shall include the PC and
all members of the IV&V team.

2.1.2 Responsibilities

PC: The PC shall be the single point of contact for the Delphi estimation documents dissemination and collection. The
PC shall conduct the balloting process for each round of estimation. The PC will then forward the ballots to the Metrics
section for data collection, maintenance, and analysis. The PC will also act as the tiebreaker, if the need arises.

IV&V: IV&V uses the estimation data to monitor the progress of the software engineering groups(s) in preparation for
testing new changes. IV&V also uses the data to monitor their own progress and effectiveness at estimating their own
testing requirements.

SQA: The SQA will use these metrics to ensure that the software engineers are maintaining a proper course and
heading. If the engineers begin to vastly deviate from the estimation metrics, the SQA members will promptly bring
this matter to everyone’s attention. This will permit the initiation of corrective action, or in extreme cases, a call for

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 5 of 9
another Delphi Estimation.

SEPG: The SEPG will use these metrics to monitor the software engineering activities of the engineering groups. The
SEPG will also maintain these metrics in the SEPG database. The SEPG will control and assist any changes
necessitated by multiple rounds of the estimation process.

Software Engineers: The Software Engineers are responsible for using the resultant metrics as a guideline for timeline
production, and work scheduling. They will also use these metrics to track their performance, and to inform the PC of
any forthcoming problems.

2.2 PROCEDURES

Many variations of the Delphi technique can be created. The following steps outline our approach for using the Delphi
technique:

a. After sufficient familiarization with the tisk requirements, each estimator is given an estimation form as shown in
Figure 1 of Appendix A.
b. The estimators meet to discuss the tisk and any estimation issues.
c. Each independently completes the estimation form.
d. The estimates are given to the Project Coordinator, who tabulates the results and returns them to the estimators
on an estimation form with the previous round's data filled in on the table as shown in Figure 3 of Appendix A.
e. Only the estimator’s personal estimates are identified, all others are anonymous.
f. The estimators may meet to discuss the results and revise their estimates as they feel appropriate.
g. This cycle is repeated until the estimates converge to an acceptable range or until the completion of three cycles,
whichever occurs first.

3.0 DOCUMENTATION
a. The Delphi estimation process requires the use of an estimation form (shown in Figures 1 through 3, Appendix
A).
b. The tisk number, the estimator's name, and the estimation date should be entered in the appropriate blocks on the
form. Each requirement is identified by number from the Analysis Report.
c. Estimators provide estimates for each requirement on a separate estimation form.
d. Estimates are based on the lines of code (LOC) and lines of documentation (LOD) to be inserted and deleted
(Ins+Del) to implement the various requirements. Estimates for each requirement are entered in the LOC and
LOD columns on the form(s).
e. Person-hour estimates are made for each of the process blocks/cells in which development and testing work is to
be done (cells 502, 503, 504, 505, and 601). Estimates may be given in whole hours or hours and tenths of hours.
f. For each subsequent estimation cycle, the same estimation form is used. Statistics from the previous estimation
cycle are provided for each requirement. These include:
1. the estimator's previous estimate,
2. other highest and lowest estimates of the other estimators, and
3. the average estimate.

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 6 of 9
Additional space is provided for re-estimates of any or all of the estimator's previous estimates. If any re-estimates are
made, the estimator may explain the rationale for each re-estimate at the bottom of the form.

REVISION HISTORY
February 2001
Bonnie Currey
Made changes to reflect new process cell numbers

April 1999
Joel Cook
Made Changes to Reduce Process Overhead

August 1998
Robert Penley
Converted Document to HTML Format

June 1998
Joel Cook, Shelly Wherley
Preliminary Document

APPENDIX A
Delphi Estimation Form

Name: _____________________________________ Date: _____________________

Tisk #: ___________________ Requirement #: _______________________

Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection): 1 2 3

LOC LOD 502 503 504 505 601

Ins+Del Ins+Del (Hours) (Hours) (Hours) (Hours) (Hours)
Highest
Estimate
Average
Estimate
Lowest
Estimate
Your
Previous
Estimate
Your
New
Estimate

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 7 of 9
Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:

Figure 1: Delphi Estimation Form

Delphi Estimation Form

Name: __Joe Programmer_____________ Date: _____________________

Tisk #: __T038______ Requirement #: _2.1________________

Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection): (1) 2 3

LOC LOD 502 503 504 505 601

Ins+Del Ins+Del (Hours) (Hours) (Hours) (Hours) (Hours)
Highest
Estimate
Average
Estimate
Lowest
Estimate
Your
Previous
Estimate
Your
New 85 15 65 10 2 30 5
Estimate

Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:

Figure 2: Sample First Round Estimation Form

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 8 of 9

Delphi Estimation Form

Name: __Joe Programmer_____________ Date: _____________________

Tisk #: __T038______ Requirement #: _2.1________________

Note: Use a separate ballot for each tisk and/or requirement being estimated.

Estimation Round (circle selection): 1 (2) 3

LOC LOD 502 503 504 505 601

Ins+Del Ins+Del (Hours) (Hours) (Hours) (Hours) (Hours)
Highest
100 20 95 25 10 50 10
Estimate
Average
80 15 72.5 20 5 32 8
Estimate
Lowest
60 10 50 10 2 20 2
Estimate
Your
Previous 85 15 65 10 2 30 5
Estimate
Your
New
Estimate

Note: First round estimates will have no data in the table above.
Estimators complete only the bottom row of the table in any round.

Estimator's Comments:

Figure 3: Sample Second (or subsequent) Round Estimation Form

BIBLIOGRAPHY
Boehm, Barry W., Software Engineering Economics, Prentice-Hall, New Jersey, 1981

Helmer-Hirschberg, O., Social Technology, Basic Books, New York, 1966

Hiltz, Starr Roxanne, Group-Based Estimation Methods: DELPHI

Humphrey, Watts S., Managing the Software Process, SEI Series in Software Engineering, Addison-Wesley, 1989

Schach, Stephen R., Classical and Object-Oriented Programming, Richard D. Irwin, Chicago, 1996

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005
Delphi Estimation Process Document Page 9 of 9

Tomayko, Dr. James E., Sizing Software http://www.cs.cmu.edu/~SW_Managemnt/html/mod_1/mod_1_3.html

Turoff, Murray; Hiltz, Starr Roxanne, Computer Based Delphi Processes

http://eies.njit.edu/~turoff/Papers/delphi3.html

GLOSSARY
Ins+Del
Inserted and Deleted

IV&V
Independent Verification and Validation

LOC
Lines of Code

LOD
Lines of Documentation

PC
Project Coordinator

SEPG
Software Engineering Process Group

SQA
Software Quality Assurance

http://www.cs.uwf.edu/~wilde/gump/delphi.htm 07/02/2005