BCS, THE CHARTERED INSTITUTE FOR IT

BCS, The Chartered Institute for IT, is committed to making IT good for society. We use the power of our network to bring about positive, tangible change. We champion the global IT profession and the interests of individuals, engaged in that profession, for the benefit of all.

Exchanging IT expertise and knowledge

The Institute fosters links between experts from industry, academia and business to promote new thinking, education and knowledge sharing.

Supporting practitioners

Through continuing professional development and a series of respected IT qualifications, the Institute seeks to promote professional practice tuned to the demands of business. It provides practical support and information services to its members and volunteer communities around the world.

Setting standards and frameworks

The Institute collaborates with government, industry and relevant bodies to establish good working practices, codes of conduct, skills frameworks and common standards. It also offers a range of consultancy services to employers to help them adopt best practice.

Become a member

Over 70,000 people including students, teachers, professionals and practitioners enjoy the benefits of BCS membership. These include access to an international community, invitations to a roster of local and national events, career development tools and a quarterly thought-leadership magazine. Visit www.bcs.org/membership to find out more.

Further information

BCS, The Chartered Institute for IT,

3 Newbridge Square,

Swindon, SN1 1BY, United Kingdom.

T +44 (0) 1793 417 417

(Monday to Friday, 09:00 to 17:00 UK time)

www.bcs.org/contact

http://shop.bcs.org/

© BCS Learning and Development Ltd 2021

The right of Boby Jose to be identified as author of this work has been asserted by him in accordance with sections 77 and 78 of the Copyright, Designs and Patents Act 1988.

All rights reserved. Apart from any fair dealing for the purposes of research or private study, or criticism or review, as permitted by the Copyright Designs and Patents Act 1988, no part of this publication may be reproduced, stored or transmitted in any form or by any means, except with the prior permission in writing of the publisher, or in the case of reprographic reproduction, in accordance with the terms of the licences issued by the Copyright Licensing Agency. Enquiries for permission to reproduce material outside those terms should be directed to the publisher.

All trade marks, registered names etc. acknowledged in this publication are the property of their respective owners. BCS and the BCS logo are the registered trade marks of the British Computer Society charity number 292786 (BCS).

Published by BCS Learning and Development Ltd, a wholly owned subsidiary of BCS, The Chartered Institute for IT, 3 Newbridge Square, Swindon, SN1 1BY, UK.

www.bcs.org

Paperback ISBN: 978-1-78017-5454

PDF ISBN: 978-1-78017-5461

ePUB ISBN: 978-1-78017-5478

British Cataloguing in Publication Data.

A CIP catalogue record for this book is available at the British Library.

Disclaimer:

The views expressed in this book are of the author and do not necessarily reflect the views of the Institute or BCS Learning and Development Ltd except where explicitly stated as such. Although every care has been taken by the authors and BCS Learning and Development Ltd in the preparation of the publication, no warranty is given by the authors or BCS Learning and Development Ltd as publisher as to the accuracy or completeness of the information contained within it and neither the authors nor BCS Learning and Development Ltd shall be responsible or liable for any loss or damage whatsoever arising by virtue of such information or any instructions or advice contained within this publication or by any of the aforementioned.

All URLs were correct at the time of publication.

Publisher’s acknowledgements

Reviewers: Kari Kakkonen, Francisca Cano Ortiz and Matthew Riddiough Boylan

Publisher: Ian Borthwick

Commissioning editor: Rebecca Youé

Production manager: Florence Leroy

Project manager: Sunrise Setting Ltd

Copy-editor: Gillian Bourn

Proofreader: Barbara Eastman

Indexer: John Silvester

Cover design: Alex Wright

Cover image: iStock - Leonid Andronov

Typeset by Lapiz Digital Services, Chennai, India

To all my mentors who have inspired and encouraged me

CONTENTS

List of figures and tables

Author

Abbreviations

Useful websites

Preface

PART ONE: THE ‘WHATS’ AND ‘WHYS’ OF TEST AUTOMATION

1. TEST AUTOMATION: A STRATEGIC VIEW

1.1 Introduction

1.2 Manual testing and automated testing

1.3 The ‘whats’ and the ‘whys’ of test automation

1.4 Managing stakeholders in test automation

1.5 Test automation policy, strategy and plan

1.6 CBA and ROI for test automation

1.7 Automation feasibility

1.8 Overselling automation

1.9 Summary

2. DOMAIN-FOCUSED TEST AUTOMATION

2.1 Software development approaches

2.2 Test automation and business models

2.3 Test automation in functional and non-functional testing

2.4 Trends

2.5 Summary

3. TEST AUTOMATION TOOL SELECTION

3.1 Test automation tool selection

3.2 Tool selection assessment

3.3 Coding or scripting

3.4 Scriptless automation

3.5 Summary

4. PEOPLE AND TEAM

4.1 Test automation team

4.2 Skills

4.3 Hiring process

4.4 Summary

5. AUTOMATION FRAMEWORKS

5.1 Linear test automation framework

5.2 Modular testing framework

5.3 Library architecture testing framework

5.4 Data-driven testing framework

5.5 Keyword-driven testing framework

5.6 Hybrid testing framework

5.7 Test-driven development testing framework

5.8 Behaviour-driven development testing framework

5.9 Summary

6. ENVIRONMENTS

6.1 Test automation environments

6.2 Summary

7. CANDIDATES FOR AUTOMATION

7.1 What should be automated?

7.2 What should not be automated?

7.3 Summary

8. TEST AUTOMATION AND TEST COVERAGE

8.1 Test coverage

8.2 Test automation metrics

8.3 Summary

PART TWO: HOW AND WHEN TO DO AUTOMATION

9. TEST AUTOMATION SUBJECT MATTER EXPERT

9.1 Essential skills for a test automation engineer

9.2 Building a successful career in test automation

9.3 Summary

10. TEST AUTOMATION TOOLS

10.1 Understanding licence models

10.2 Leading test automation tools

10.3 Cost factors

10.4 Summary

11. TEST AUTOMATION AND PROGRAMMING LANGUAGES

11.1 Programming languages for test automation

11.2 Coding or scripting methods and techniques

11.3 Candidates for test automation scripting

11.4 Developing your first script

11.5 Summary

12. AUTOMATION FRAMEWORK DESIGN AND DEVELOPMENT

12.1 Test automation components’ definitions

12.2 Building a test automation framework

12.3 SUT architecture

12.4 Test data management

12.5 Summary

13. MEASURING TEST AUTOMATION

13.1 Test automation metrics

13.2 Requirement and coverage metrics

13.3 Defect metrics

13.4 Management reporting

13.5 Automated test metrics

13.6 Summary

14. CONCLUSION

APPENDICES

APPENDIX A: CODING STANDARDS AND COMMENTS

APPENDIX B: SAMPLE TEST AUTOMATION FRAMEWORK

APPENDIX C: SAMPLE INTERVIEW QUESTIONS

APPENDIX D: SAMPLE SKILL SET OF TEST AUTOMATION ENGINEERS

APPENDIX E: TEMPLATES

Template 1: Sample test policy

Template 2: Sample test automation plan

Template 3: Cost-benefit analysis

Template 4: Automation suitability checklist

Template 5: Tool evaluation

Template 6: ROI trend

Template 7: ROI

APPENDIX F: TEST AUTOMATION – INDUSTRY EXAMPLES

APPENDIX G: ISTQB TEST AUTOMATION ARCHITECTURE

References

Further reading

Glossary

Index

LIST OF FIGURES AND TABLES

FIGURES

Figure 1.1 Triple constraints

Figure 1.2 Manual testing versus test automation

Figure 1.3 Why test automation?

Figure 1.4 Stakeholder involvement matrix

Figure 1.5 Stakeholders

Figure 1.6 Document hierarchy

Figure 1.7 Automation test approach

Figure 1.8 Cost-benefit analysis framework

Figure 1.9 ROI versus releases

Figure 2.1 Waterfall approach

Figure 2.2 Agile approach

Figure 2.3 Waterfall testing

Figure 2.4 Test automation in Waterfall

Figure 2.5 Agile testing

Figure 2.6 Test automation in Agile at an enterprise level

Figure 2.7 DevOps

Figure 2.8 DevOps CI/CD workflow

Figure 2.9 DevOps test automation

Figure 2.10 MTS pillars

Figure 2.11 Functional test automation types

Figure 2.12 Non-functional test automation types

Figure 3.1 Automation tools versus cost matrix

Figure 3.2 Tool selection process

Figure 5.1 Test automation frameworks

Figure 5.2 Linear test automation framework

Figure 5.3 Modular test automation framework

Figure 5.4 Library architecture test automation framework

Figure 5.5 Data-driven test automation framework

Figure 5.6 Keyword-driven test automation framework

Figure 5.7 Hybrid test automation framework

Figure 5.8 Test-driven development

Figure 5.9 Behaviour-driven development

Figure 6.1 Test environment sequential diagram

Figure 7.1 Test automation: initial days

Figure 7.2 Test automation growth

Figure 9.1 Test automation engineer essential skills

Figure 10.1 Software licence models

Figure 10.2 Testing tools

Figure 11.1 Generations of programming languages

Figure 11.2 Structured scripting

Figure 12.1 Test automation road map

Figure 12.2 Framework folder structure

Figure 12.3 BDD SpecFlow for an ecommerce project (example)

Figure 12.4 Application data sheet

Figure 12.5 Application areas/modules data sheet (regression suite)

Figure 12.6 Data sheet (regression suite)

Figure 12.7 Test report

Figure 12.8 Test execution summary

Figure 12.9 Detailed result

Figure 12.10 Test data creation

Figure 13.1 Product automation metrics

Figure 13.2 Defect trends

Figure 13.3 Test automation trend based on coverage and cost

Figure 13.4 Test automation trend based on effort and quality

Figure 13.5 Test automation trend based on test execution rate

Figure 13.6 Test automation defect removal efficiency

Figure 13.7 Test automation defect root cause analysis

Figure 13.8 Test automation defect density

Figure 13.9 Test automation environment down metrics

Figure A.1 Test automation framework folder structure

Figure A.2 Test automation framework for a real-life project

Figure A.3 Test automation framework test result 1

Figure A.4 Test automation framework test result 2

Figure A.5 Tentative high-level plan and schedule

Figure A.6 ROI chart for automation

Figure A.7 The Generic Test Automation Framework

TABLES

Table 2.1 Test automation and domains

Table 2.2 Comparison between Waterfall and Agile testing

Table 2.3 Functional requirements specifications

Table 2.4 Non-functional requirements specifications

Table 2.5 Testing stages and automation

Table 2.6 Testing types and automation

Table 4.1 Competency matrix for test automation

Table 4.2 Skill assessment matrix

Table 5.1 Example keyword table

Table 6.1 Environments and their features

Table 7.1 Candidates for automated testing

Table 10.1 API/web service testing tools

Table 10.2 Unit testing tools

Table 10.3 Functional and web tools

Table 10.4 Mobile testing tools

Table 10.5 Test and defect management tools

Table 10.6 Cross-browser testing tools

Table 10.7 Performance and load testing tools

Table 10.8 Security and penetration testing tools

Table 10.9 AL/ML powered testing tools

Table 10.10 Test framework tools

Table 12.1 Common definitions

Table 12.2 Framework platform requirements

Table 12.3 Test data management tools

Table A.1 Test automation framework folder description

Table A.2 Test automation framework folder description for a real-life project

AUTHOR

Boby Jose holds a BSc in Physics and Masters in Business Administration (MBA), and has over two decades of work experience in software testing within the business, technology, IT, infrastructure, outsourcing and consulting industries. He started his professional career as a business analyst and ecommerce consultant. Professionally trained and certified in test automation, Boby’s career progressed through offshore IT service companies, cutting edge product development and testing in informatics, and test consulting with a leading North American technology consultancy, and later with a leading European IT consultancy. He has worked in companies ranging from 500 to 250,000 employees and has led testing engagements with more than 100 members.

Boby has successfully managed the testing of many large IT transformation programmes and globally distributed software engagements. He is based in London, United Kingdom, and has software testing experience in the public, private, secure and product sectors in the US, Europe, Middle East and Asia. Boby has extensive experience in the fields of test strategy, test management, test planning, Test automation, test governance, test transformation, test environment management, infrastructure test management, test consulting and product test management.

Boby also has substantial testing and test automation experience in agro, bioinformatics, client/server, custom and bespoke solutions, digital, ecommerce, education, enterprise applications, enterprise resource planning (ERP), finance, healthcare, legacy, life science, marketplace, material science, mobile, multichannel, retail, scientific, tax, telecom, travel and transport, and web technologies. His skills spread across Agile, continuous integration (CI) and continuous delivery (CD), functional testing, infrastructure testing, iterative, IT service continuity (ITSC) testing, non-functional testing, operational acceptance testing (cloud, on-premises and hybrid), performance testing, portfolio, project and programme test management, security and penetration testing, test strategies and plans, user acceptance testing (for over 100 countries), V-model and Waterfall.

He is a certified ISTQB® Advanced Test Manager, as well as having qualifications that include TPI NEXT® Foundation, ISTQB-ISEB Foundation Level, PRINCE2® Foundation and Practitioner, Mobile App Testing, Certified Agile Scrum Master and Certified Scrum Product Owner.

He has published many papers, articles and points of view on topics such as the intricacies of mobile application testing, managing up in testing, process improvement through goal problem approach, investing in test automation (cost-benefit analysis and return on investment (ROI)) and software testing of product versus application-based approaches. In addition to this, Boby has previously managed testing for an ‘award-winning mobile application’ (Mobile Application of the Year by the Real IT forum).

You can find Boby on LinkedIn at ‘Boby Jose’ (https://www.linkedin.com/in/boby-jose-893b513/).

ABBREVIATIONS

AI Artificial intelligence

ALM Application life cycle management

API Application programming interface

APO Advanced planning and optimisation

ATDD Acceptance test-driven development

ATF Automated test framework

ATT Automated testing tool

AUT Application under test

AWS Amazon Web Services

BAU Business as usual

BDD Behaviour-driven development

CapEx Capital expenditure

CBA Cost-benefit analysis

CC Change control

CI Continuous integration

CI/CD Continuous integration and continuous delivery

CIT Component integration testing

CMDB Configuration management database

COTS Commercial off-the-shelf

CRM Customer relationship management

CSV Comma separated value

CT Component testing

CTO Chief technology officer

DDD Detailed design document

DDT Defect detection trends

DL Deep learning

DMT Data migration testing

DOM Document Object Model

DRY Don’t repeat yourself

E2E End-to-end

ELS Early life support

ERP Enterprise resource planning

ESB Enterprise service bus

ETL Extract, transform, load

FAT Factory acceptance testing

FOSS Free and open-source software

FT/FP Fixed time and fixed price

GDPR General Data Protection Regulation

GIS Geographic information system

GPL General Public Licence

GPS Global Positioning System

gTAA Generic Test Automation Architecture

GUI Graphical user interface

HIPAA Health Insurance Portability and Accountability Act

HLD High-level design

HR Human resources

HTML HyperText Markup Language

HTTP HyperText Transfer Protocol

HTTPS HyperText Transfer Protocol Secure

I18N Internationalisation

IaaS Infrastructure as a service

IAM Identity and access management

IE Internet Explorer

IP Internet Protocol

ISCP International supply chain planning

ISEB Information System Examination Board

ISO International Organization for Standardization

ISTQB International Software Testing Qualifications Board

IT Information technology

ITSC IT service continuity

JAWS Job Access With Speech

JD Job description

JS JavaScript

KPI Key performance indicator

L10N Localisation

LLD Low-level design

LOE Level of effort

ML Machine learning

MRD Manufacturing, retail and distribution

MS Multi-supplier

MS Microsoft

MTS Managed testing services

MV Multi-vendor

MVP Minimum viable/valuable product

OAT Operational acceptance testing

OEM Original equipment manufacturer

OOP Object-oriented programming

OpEx Operational expenditure

OS Operating system

PaaS Platform as a service

Pen test Penetration testing

PHP Hypertext Preprocessor

PII Personal identifiable information

PoC Proof of concept

PoT Proof of technology

PPR Payroll parallel run

PVT Performance and volume testing

Q&A Question and answer

QA Quality assurance

QTP QuickTest Professional

RACI Responsible, accountable, consulted and informed

RAG Red, amber and green

RAID Risks, assumptions, issues and dependencies

RBT Risk-based testing

REST Representational state transfer

ROI Return on investment

RPA Robotic process automation

RPO Recovery point objective

RTM Requirement traceability matrix

RTO Recovery time objective

RUP Rational Unified Process

SaaS Software as a service

SDLC Software development life cycle

SI System integrator

SIT System integration testing

SLA Service level agreement

SME Subject matter expert

SOA Service-oriented architecture

SOAP Simple Object Access Protocol

SQL Structured Query Language

SSO Single sign on

ST System testing

STAF Software test automation framework

STLC Software test life cycle

SUT System under test

T&M Time and materials

TaaS Testing as a service

TAS Test automation solution

TDD Test-driven development

TFS Team Foundation Server

TMMi Test Maturity Model integration

UAT User acceptance testing

UFT Unified Functional Testing

UI User interface

UT Unit testing

UX User experience

VSTS Visual Studio Team Services

VUs Virtual users

W3C World Wide Web Consortium

WCAG Web Content Accessibility Guidelines

WMI Windows Management Instrumentation

WORA Write Once, Run Anywhere

WORE Write Once, Run Everywhere

WSH Windows Scripting Host

XML Extensible Markup Language

XP Extreme programming

XSS Cross-site scripting

USEFUL WEBSITES

Accessibility testing https://dynomapper.com/blog/27-accessibility-testing

Agile https://blog.planview.com

https://www.agilealliance.org/agile101/agile-glossary

Artificial intelligence https://bernardmarr.com

https://www.parasoft.com

Automation framework https://bellatrix.solutions

https://www.browserstack.com

https://www.softwaretestinghelp.com

https://www.toolsqa.com

https://devops.com

Automation metrics https://www.a1qa.com

https://www.getzephyr.com

https://blog.qatestlab.com

CI/CD https://martinfowler.com/articles/continuous

Integration.html

https://martinfowler.com/tags/extreme%

20programming.html

https://www.atlassian.com/continuous-delivery/continuous-

integration/how-to-get-to-continuous-integration

https://continuousdelivery.com

https://stackify.com/continuous-delivery-vs-continuous-

deployment-vs-continuous-integration

https://bitbar.com/blog/top-continuous-integration-

tools-for-devops

https://apiumhub.com/tech-blog-barcelona/

benefits-of-continuous-integration

https://www.simplilearn.com/tutorials/devops-

tutorial/continuous-integration

https://aws.amazon.com/devops/continuous-integration

https://martinfowler.com/articles/continuousIntegration.html

https://www.agilealliance.org/agile101/agile-glossary

https://dl.acm.org/doi/abs/10.1145/3236024.3275528

Certification https://www.bcs.org

https://techcanvass.com

Cloud solutions https://phoenixnap.com

Coding standards http://themoderndeveloper.com/code-standards

Development approaches https://blog.planview.com

DevOps https://devops.com

Encyclopedia https://www.pcmag.com/encyclopedia/term/mts

Glossary https://glossary.istqb.org/app/en/search

Mobile testing https://www.mobindustry.net

Object-oriented https://docs.oracle.com/javase/tutorial/

java/concepts

programming https://www.w3schools.com/cpp/cpp_oop.asp

Programming languages https://www.codingninjas.com

https://www.w3schools.com/cpp/cpp_oop.asp

Software testing https://blog.qatestlab.com

Test automation https://dzone.com

https://www.capgemini.com

https://www.codeproject.com

https://www.katalon.com

https://www.leapwork.com/blog/why-good-test-environments-

are-crucial-for-successful-automation

https://www.techwell.com

https://www.testim.io/home-with-form

Test automation

architecture https://www.istqb.org/downloads/category/48-advanced-

level-test-automation-engineer-documents.html

https://labs.sogeti.com/benefits-of-generic-test-

automation-architecture

https://huddle.eurostarsoftwaretesting.com/test-automation-

framework-architecture-types

https://medium.com/koderlabs/introduction-to-monolithic-

architecture-and-microservices-architecture-b211a5955c63

Test-driven design https://www.madetech.com

Testing resources http://www.softwareqatest.com

https://www.istqb.org

https://www.stickyminds.com

https://bernardmarr.com

Testing skills https://testsigma.com

https://uk.indeed.com/?r=us

Testing standards https://www.bcs.org/membership/member-communities/software-

testing-specialist-group

https://softwaretestingstandard.org

Testing tools https://briananderson2209.medium.com/best-automation-

testing-tools-for-2018-top-10-reviews-8a4a19f664d2

https://docs.katalon.com/katalon-studio/

docs/index.html

https://reqtest.com

https://www.altexsoft.com

https://www.edureka.co

https://www.knowledgehut.com

https://www.methodsandtools.com

https://www.saviantconsulting.com

Testing trends https://techbeacon.com/app-dev-testing/5-ways-ai-will-

change-software-testing

Testing types https://geteasyqa.com/qa/software-testing-types

Training https://testinginstitute.com/Company.php

Tutorial https://www.guru99.com

PREFACE

Software testing today faces numerous challenges, including ever-shrinking release schedules, a lack of knowledge on business priorities, the misunderstanding of development methodologies, complex technologies, changing expectations, delays within implementation, a lack of awareness of non-functional requirements, insufficient resourcing, time and cost pressure versus quality, and the unfamiliarity of testing conventions. Most of these challenges become a risk and later a reality soon after the software solution development commences. Software products or projects try to mitigate the above issues in various ways, such as through planning and replanning, adding more people to the team or additional budget; however, testing is one of the key areas to be viewed as a mitigation to address the delay in delivery due to the above issues.

The testing team is often asked to reduce the test effort by following a risk-based approach, optimising testing stages, descoping testing types, merging testing phases and so on to compensate for the delay in the overall delivery schedule. Experienced and seasoned test managers are aware of these challenges and often plan at the beginning to address them. Test managers are often pushed against the wall to reduce the testing timeline in order to absorb delays in the planning, design and implementation phases. Test automation is one of the preferred ways of addressing these challenges by minimising the impact on overall quality.

This book is primarily written for test managers, Scrum masters, product and project managers on how to implement test automation and automated testing in their organisation.

Test automation is to automate the end-to-end testing process and activities, including tracking and managing the different tests. Automated testing is performing specific tests, such as performance testing and functional testing, via automation rather than manual means. (Refer to Chapter 1 for more details on test automation and automated testing.)

This book is a manager’s guide to building and leading successful test automation at an organisational level.

Software testing, test automation and automated testing require more people who have good experience in the full project life cycle from the project concept to implementation, and early life support (ELS) to closure and decommission. Software testing demands good subject matter experts (SMEs) in end-to-end (E2E) testing, test automation, functional and non-functional testing, infrastructure testing, performance and volume testing, security testing, accessibility testing, recovery point objective (RPO) and recovery time objective (RTO) validation etc. Test organisations face challenges on the level of testing required, test coverage and expectations to complete testing as quickly and thoroughly as possible under the constraints of limited budget and time. Manual testing is labour-intensive, requires a good understanding of the application, knowledge of testing methodologies and skilful testers with good attitude and aptitude. While test automation complements manual testing, it requires detailed planning and support from the right stakeholders in the organisation. Testing tools and resources come at a high price, and test automation is effort-intensive with significant capital expenditure (CapEx). Due to the disparate capital expenditure involved, Test automation, if not implemented properly, will not provide the expected outcome while also increasing operational expenditure (OpEx). The goal of test automation