Scribd
Upload
236 views20 pages

Cost Benefits Analysis of Test Automation

Presentation from Star West 99.

Uploaded by

wnecka
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
236 views20 pages

Cost Benefits Analysis of Test Automation

Presentation from Star West 99.

Uploaded by

wnecka
Copyright
© Attribution Non-Commercial (BY-NC)
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 20

Cost Benefits Analysis of

Test Automation
STAR West ’99

Douglas Hoffman
Software Quality Methods, LLC.
24646 Heather Heights Place
Saratoga, California 95070-9710
Phone 408-741-4830
Fax 408-867-4550

Copyright © 1999, Software Quality Methods, LLC. No part of these graphic overhead slides may
be reproduced, or used in any form by any electronic or mechanical duplication, or stored in a
computer system, without written permission of the author.

Douglas Hoffman Copyright © 1999, SQM, LLC. 1


Why Are We Doing This?

• Management needs numbers

• Management needs justification

• We want to know where to invest

Douglas Hoffman Copyright © 1999, SQM, LLC. 2


Intangibles
• Not all members of the test team will want to change
• Immediate reduction in perceived productivity of the
test organization
• Quality of tests
• Professionalism of test organization
• Expansion into advanced test issues
• Hands-off testing
• Number of product rolls (test cycles) before release
• Test coverage

Douglas Hoffman Copyright © 1999, SQM, LLC. 3


Falsely Expected Benefits
• All tests will be automated
• Immediate payback from automation
• Automation of existing manual tests
• Zero ramp up time
• Automated comprehensive test planning
• Capture/Playback for regression testing
• One tool that fits perfectly
• Automatic defect reporting (without human
intervention)
Douglas Hoffman Copyright © 1999, SQM, LLC. 4
Background to Costing
• Many changes would equally benefit manual testing
• Most automation benefits come from discipline in
analysis and planning
• Payback from automation is usually in the next
project or thereafter
• Automating usually causes significant negative
schedule and performance impacts at introduction
• Automated tests require more programming and
design skills from testers
• Automated tests frequently require maintenance
• Software metrics aren’t unbiased statistics
Douglas Hoffman Copyright © 1999, SQM, LLC. 5
ROI Factors
• Dependent upon factors besides costs and savings:
• Fixed costs need to be amortized over useful life
• Investments last only as long as they remain in service
• Variable benefits depend on the number of tests and test
runs
• Expected number of runs of tests is most important factor
in relative ROI computations
• Computed for a given time period (t)
• Incremental return on incremental investment vs.
manual testing

Douglas Hoffman Copyright © 1999, SQM, LLC. 6


ROI Computations
En = Aa/Am = (Va + n*Da) / (Vm + n*Dm) †
En = Aa/Am = (Va + n1*Da) / (Vm + n2*Dm)
ROIautomation(in time t) = (Savings from
automation) / (Costs of automation)
ROIautomation(in time t) = ∆(Savings from
automation) / ∆(Costs of automation)
† Linz, T, Daigl, M. “GUI Testing Made Painless. Implementation and results of the ESSI
Project Number 24306”, 1998. Analysis in Case Study: Value of Test Automation
Measurement, p. 52+ of Dustin, et. al., Automated Software Testing, Addison-Wesley, 1999.

Douglas Hoffman Copyright © 1999, SQM, LLC. 7


Relative ROI

• Changes in benefits and costs


• Tangible savings and costs
• Fixed and variable elements

Douglas Hoffman Copyright © 1999, SQM, LLC. 8


Common Cost Examples
• SUT analysis
• Test planning
• Base test design
• Defect reporting
• Management reporting (results)

Douglas Hoffman Copyright © 1999, SQM, LLC. 9


Fixed Cost Examples
• Hardware (additional or upgrades to existing)
• Testware software licenses
• Testware software support
• Per seat licenses
• Automation environment design, implementation
• Scripting tools
• Tool and environment licenses
• Tool training
• Tool introduction and ramp up

Douglas Hoffman Copyright © 1999, SQM, LLC. 10


Variable Cost Examples

• Test case design for automation


• Test case implementation
• Test maintenance
• Oracle creation
• Test running and analysis

Douglas Hoffman Copyright © 1999, SQM, LLC. 11


Usual Benefits

• Test execution savings


• After-hours testing by systems (not
people)

Douglas Hoffman Copyright © 1999, SQM, LLC. 12


Changed Cost Examples
• Automation environment maintenance
• Test case execution
• Test results analysis
• Defect reporting
• Test results reporting
• Test data generation

Douglas Hoffman Copyright © 1999, SQM, LLC. 13


ROI Computation

• Computing costs

• Computing savings

• ROI

Douglas Hoffman Copyright © 1999, SQM, LLC. 14


Cost Computation
• Σ (fixed costs of automated testing) times (t/Useful Life)

• Plus Σ (variable costs of creating automated tests)

• Less Σ (variable costs of creating manual tests)

• Plus Σ (variable costs of maintaining automated tests)


times (n/N)

• Plus increased Changed Costs values

Douglas Hoffman Copyright © 1999, SQM, LLC. 15


Alternate Cost Computation
• Σ (fixed costs of automated testing) times (t/Useful Life)

• Plus Σ (costs of people creating automated tests)

• Less Σ (costs of people creating manual tests)

• Plus Σ (costs of people maintaining automated tests)

• Plus increased Changed Costs values

Douglas Hoffman Copyright © 1999, SQM, LLC. 16


Benefit Computation

• Σ (variable costs of running manual tests n2


times during time t)
• Less Σ (variable costs of running automated
tests n1 times during time t)
• Plus Changed Costs savings values

Douglas Hoffman Copyright © 1999, SQM, LLC. 17


Conclusions
• Figures don’t lie, but …
• Understand why you need ROI

• Planning can increase ROI

• Identify costs and savings

Douglas Hoffman Copyright © 1999, SQM, LLC. 18


ROI Computation Example
• Daily builds and test runs (5 times a week)
• Manual tests take 5 days to design, 2 hours to run
• Only half the manual tests would be run on any given day (1 hour) with
the other half run the following day
• Automated tests take 15 days to design and implement, automatically
run (zero cost)
• Automation is done with batch scripts and integrated into the build
process, requiring $1,000 in added hardware, with a useful life of 3
years
• Automated tests need to be maintained every 25 runs, one day of work
required
• Periods of time (t) selected: 6 months (125 days) and 18 months (375
days)
• People cost $100,000 per year = $400 per day = $50 per hour

Douglas Hoffman Copyright © 1999, SQM, LLC. 19


ROI Computation Example
• A new product with all new tests
• 5 people-years developing manual tests, 15 for automated tests
• 1 person maintenance after 1st year for automated tests
• 10 people full time running manual tests, 1 person for
automated
• Fixed costs for automated tests of $90,000 with useful life of 3
years
• Periods of time (t) selected: 12 months (250 days) and 24
months (500 days)
• People cost $100,000 per year = $400 per day = $50 per hour

Douglas Hoffman Copyright © 1999, SQM, LLC. 20

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy