SOFTWARE TESTING OVERVIEW

Agenda
• Famous Software Failures
• Why Software Testing is necessary?
• Software Testing Definition
• Software Testing Life Cycle
• Levels of Testing
• Testing process, Activities & Planning
• Testing Methodologies
• Various types of testing
• Industry accepted Certifications in Software Testing
• Summary
2
 Some Software Failure Examples
1. Microsoft’s Anti-Unix Site Crashes; Apr 3, 2002 :
A marketing Web site, part of a multi-million dollar campaign by Microsoft and Unisys to get customers to switch
from Unix, was turning into a major embarrassment. www.wehavethewayout.com which was powered by Unix kit
and that, when Microsoft switched it to Internet Information Server software, the site was crashed completely
Observation : Was not tested on different OS/Environments before switching from one environment to another
2. Yahoo Glitch strikes again; Mar 20, 2002:
Parts of Yahoo were shut down on Tuesday following software problems encountered in the integrating of Yahoo
Groups and Yahoo Clubs.
Observation: The system was not properly tested to see whether all its components successfully integrated before
going live
3.Computer Glitch Causes £22m Tax Error; Mar 22, 2002:
The Inland Revenue has said that computer problems were responsible for an estimated 134,000 basic rate taxpayers
being overcharged by a total of £22m. Individual taxpayers paid an average of £148 over the odds because of data
transfer problems between the national insurance computer system and the PAYE
system.
Observation: Not properly tested the system as a whole. Lack of sufficient system testing to take care of data
transfer aspects
4. Heathrow Terminal 5 Computer Glitch Causes Chaos As Passengers were Left Stranded On the day of opening in
2008 and recently in September 2013
Passengers at Heathrow Airport faced chaos after a computer glitch left them helplessly unable to check-in any
luggage

3
Source : Google & other sites
 Why Software Testing?

To meet the
Organizational
Goals like meeting If testing is done
To get adequate
requirements, Poorly, it leads to
trust and
satisfied high maintenance
confidence on the
customers, cost and user
product
improved market dissatisfaction
share, zero
defects, etc

4
 Software Testing Definition

The process of detecting & identifying defects,

where a defect is any variance between actual and
expected results

-Reporting the above and

-Taking necessary corrective measures

5
 What is Software Testing?

• Process of executing a program with the intent of finding errors

• Confirming that a system performs its intended functions correctly
• Establishing confidence that a system does what it is supposed to do
• The process of analyzing a system to detect the
difference between existing and required conditions
and to evaluate the feature of the system
• Numerous software development and testing methodologies, tools, and
techniques have emerged over the last few decades promising to enhance
software quality
• Software testing provides a means to reduce errors, cut maintenance and
overall software costs
• Testing demonstrates that software functions meet specifications and
performance requirements

6
 Functional vs. Non-Functional Testing

Functional testing refers to activities that verify a specific action or function

of the code
These are usually found in the code requirements documentation, although
some development methodologies work from use cases or user stories.
Functional tests tend to answer the question of "can the user do this" or "does
this particular feature work

Non-functional testing refers to aspects of the software that may not be

related to a specific function or user action, such as scalability or other
performance, behavior under certain constraints, or security.

Testing will determine the flake point, the point at which extremes of scalability
or performance leads to unstable execution.

Non-functional requirements tend to be those that reflect the quality of the

product, particularly in the context of the suitability perspective of its users 7
Testing Life Cycle

Requirements
Analysis
Test Closure
Test Planning

Regression Test Case

Testing Development

Defects Test Case

Retesting Execution
Test results
Reporting

8
 Software Testing Lifecycle

Requirements Study
• Testing cycle starts with the study of client’s requirements
• Understanding of requirements is very essential to test the product
Test Case Design and Development
• Component Identification
• Test Specification Design
• Test Specification Review
Test Execution
• Code Review
• Test Execution and Evaluation
• Performance and Simulation
Test Closure
• Test Summary Report
• Project debrief
• Project Documentation
Test Process Analysis
• Implementing new technologies and additional features
9
 Levels of Testing

Testing is applied at different levels in the software development life cycle and
has different objectives at each level. Though the focus of all testing is to find
errors, different type of errors are looked for at each level.

The levels of testing in a Development project could be:

Unit Testing: To test individual Software items

Module/Feature testing: Module testing shall be carried out if there are
multiple modules in the project
Integration Testing: To test the interfaces between the Software items
System Testing: To validate the entire product
Operational Testing: To test the product for the non-functional
requirements in target environment.
Acceptance Testing: To test the product for acceptance

10
Levels of Testing

11
 Common Software Problems : Key
Terminologies
Mistake – A human action that produces an incorrect result
Fault [or Defect] – An incorrect step, process, or data definition in a program
Failure – The inability of a system or component to perform its Required function
within the specified performance requirement
Error – The difference between a computed, observed, or Measured value or
condition and the true, specified, or theoretically correct value or condition.
Specification – A document that specifies in a complete, precise, Verifiable
manner, the requirements, design, behavior, or other Characteristic of a system or
component

12
 Objectives of Testing
A good test case is one that has a probability of finding an as yet undiscovered
error
A good test is not redundant

A successful test is one that uncovers a yet undiscovered error

A good test should be “best of breed”

A good test should neither be too simple nor too complex.

To check if the system does what it is expected to do

To check if the system is “Fit for purpose”

To check if the system meets the requirements and be executed successfully in
the Intended environment
Executing a program with the intent of finding an error
13
Testing Process

Define
Test Plan
Test Cases

Identify Area
for Automation

Create Test
Track Data
Defects

Analyse Execute
results Tests

14
 Testing Activities

Test Strategy

• Define following key things

• Test Design
• Test Data
• Test Environment
• Automation
• Regression Test

Test Development

• Establishing a test approach through test design and test planning based on the nature and life cycle of the project
• Developing test cases based on the test approach
• Test Scripts/ Code development for execution of test cases

Test Planning

• Unit testing, integration testing , system and acceptance testing planning

Product Integration Strategy

15
 Test Planning
The major elements of test planning are:

Establish objectives for each test phase

Establish schedules and responsibilities for each test activity

Determine the availability of tools, facilities, and test libraries

Establish procedures and standards for planning and conducting

the tests

Set the criteria for test completion and success

16
 Preparing Test Plan

Purpose
• Validate the acceptability of a software product
• Help the people outside the test group to understand “why” and “how” of product
validation
• A Test Plan should be Thorough enough (Overall coverage of test to be conducted)
• Useful and understandable by the people inside and outside the test group
Scope
• The areas to be tested by the QA team
• Specify the areas which are out of scope (screens, database, mainframe processes
etc)
Test Approach
• Details on how the testing is to be performed
• Any specific strategy is to be followed for testing (including configuration
management)
17
 Testing Methodologies

BLACK BOX TESTING

Black Box Functional
Definition Techniques
Testing
• No knowledge of internal • Incorrect or missing • Based on requirements
design or code required. functions and functionality
• Tests are based on • Interface errors • Not based on any
requirements and • Errors in data structures knowledge of internal
functionality or external database design or code Covers all
access combined parts of a
• Performance errors system
• Initialization and • Tests are data driven
termination errors

The software tester only knows what the system is supposed to do, he does not
know how it works internally. He has to test on the basis of expected outputs
e.g. System testing
18
 Testing Methodologies

WHITE BOX TESTING

Definition Techniques White Box Structural Testing

• Knowledge of the internal • All independent paths within a

• Based on knowledge of
program design and code module have been exercised
internal logic of an
required at least once
application's code
• Tests are based on coverage of • Exercise all logical decisions
• Based on coverage of code
code statements, branches, on their true and false sides
statements, branches, paths,
paths, conditions • Execute all loops at their conditions
boundaries and within their
• Tests are logic driven
operational bounds
• Exercise internal data
structures to ensure their
validity

The software tester has access to the program/ code and examine it for errors.
Also called Clear Box testing or Structural Testing e.g. Unit testing

19
 Positive and Negative Testing

Positive Testing
• Ensures that system behaves as expected when a positive data is entered
• Example: if we try to enter 29th Feb for a leap year we should be allowed to
proceed. If the error is encountered in positive testing then the test case fails
• In positive testing we check the behavior of application for valid data

Negative Testing
• Ensures that the system is able to handle invalid inputs appropriately and
expected user behavior.
• Example: if we try to enter 29th Feb for any year except a leap year we should
be not allowed to proceed
• Negative testing helps in preventing the system crash when unexpected user
input is entered
• It also helps us to find the weak points of our application and helps us to
improve the quality of our application
• If the error is encountered in negative testing then the test case passes
• During negative testing we check the behavior of application for Invalid data
20
 Positive and Negative Testing : Why
Important?
Usually stability and
reliability of application
improves by using both
negative and positive
testing

It is advisable to go for both

these testing approaches to
keep a track of both
expected and unexpected
behavior of the application
21
 Positive Testing
In Positive testing we check the behavior of application for valid data. This
technique focuses on validating a predefined behavior of an application.

In positive testing user tries to follow how the system behaves for a valid input and
if what was supposed to happen is happening or not. If does not happen then we
can call it as bug, failure or defect which is deviation from expected result.

Positive Testing may Involves the below testing types :

Acceptance Testing Smoke Testing

Compatibility Testing Unit Testing
Functional Testing Conformance Testing
Integration Testing Load Testing
Regression Testing Performance

22
 Negative Testing
Negative testing helps in identifying application crashes, caused by improper user
inputs. Below are some examples to illustrate negative testing:
 Mandatory fields
 For fields requiring specific inputs
 For fields with character specific limit
 For field that accepts range of values
 For authentication testing

Negative Testing may Involves the below testing types.

 Acceptance Testing
 Ad Hoc Testing/Exploratory Testing
 Boundary Testing
 Smoke Testing
 Stress Testing
 Unit Testing 23
 Test Bed Testing
• A testbed (also commonly spelled as test bed in research publications) is a
platform for experimentation of large development projects.
• Testbeds allow for rigorous, transparent, and replicable testing of scientific
theories, computational tools, and new technologies.
• A typical testbed could include software, hardware, and networking
components
• In software development, the specified hardware and software environment
can be set up as a testbed for the application under test. In this context, a
testbed is also known as the test environment
• Testbeds are also pages on the Internet where the public are given the
opportunity to test CSS or HTML they have created and want to preview the
results

An execution environment configured for testing. May consist of specific

hardware, OS, network topology, configuration of the product under test, other
application or system software, etc
24
 Boundary Value Testing
• A greater number of errors occur at the boundaries of the input domain rather
than in the "center"
• Boundary value analysis is a test case design method that complements
equivalence partitioning
– It selects test cases at the edges of a class
– It derives test cases from both the input domain and output domain

• Boundary value analysis refers to selecting test values near the extremes of the
input value domain e.g., if a variable can have values between 0 and 100 then
domain boundaries (0 and 100) and just outside boundary ( -1, 101) is required
to be checked

25
 Acceptance & Compatibility Testing

Acceptance Testing
• Testing to verify a product meets customer specified requirements
• A customer usually does this type of testing on a product that is
developed externally
• Done at the customer’s premises on the production similar environment
• Customers undertake typical tasks to check their requirement

Compatibility Testing
• Testing to ensure compatibility of an application or Web site with different
browsers, Operating systems, and hardware platforms
• Compatibility testing can be performed manually or can be driven by an
automated functional or regression test suite

26
 Functional & Integration Testing

Functional Testing
• Validating an application or Web site conforms to its specifications and
correctly performs all its required functions
• This entails a series of tests which perform a feature by feature validation of
behavior, using a wide range of normal and erroneous input data
• This can involve testing of the product's user interface, APIs, database
management, security, installation, networking etc
• Functional testing can be performed on an automated or manual basis using
black box or white box methodologies
Integration Testing
• Testing in which modules are combined and tested as a group
• Modules are typically code modules, individual applications, client and server
applications on a network, etc
• Integration Testing follows unit testing and precedes system testing

27
 Regression, Smoke & Conformance
Regression Testing
• Regression testing is any type of software testing that seeks to uncover new software bugs, or
regressions, in existing functional and non-functional areas of a system after changes, such as
enhancements, patches or configuration changes, have been made to them.
• The intent of regression testing is to ensure that a change such as those mentioned above has not
introduced new faults
• One of the main reasons for regression testing is to determine whether a change in one part of the
software affects other parts of the software

Smoke Testing
• smoke testing is preliminary testing to reveal simple failures severe enough to reject a prospective
software release
• A subset of test cases that cover the most important functionality of a component or system are
selected and run, to ascertain if the most crucial functions of a program work correctly

Conformance Testing
• conformance testing verifies that a product performs according to its specified standards
• Verifying implementation conformance to industry standards.
• Producing tests for the behavior of an implementation to be sure it provides the portability,
interoperability, and/or compatibility a standard defines. 28
 Structural and Static Testing

Structural Testing
• The structural testing is the testing of the structure of the system or component
• Structural testing is often referred to as ‘white box’ because in structural testing we are interested
in what is happening ‘inside the system/application
• In structural testing the testers are required to have the knowledge of the internal
implementations of the code. Here the testers require knowledge of how the software is
implemented, how it works
• Structural testing can be used at all levels of testing. Developers use structural testing in
component testing and component integration testing, especially where there is good tool support
for code coverage. Structural testing is also used in system and acceptance testing

Static Testing
• Static testing is a form of software testing where the software isn't actually used
• This is in contrast to dynamic testing
• It is generally not detailed testing, but checks mainly for the sanity of the code, algorithm, or
document
• It is primarily checking of the code and/or manually reviewing the code or document to find errors
• This type of testing can be used by the developer who wrote the code, in isolation. Code reviews,
inspections and Software walkthroughs are also used
• From the black box testing point of view, static testing involves reviewing requirements
29
 Load and Performance Testing
• Load Testing
 Load testing is the simplest form of performance testing.
 A load test is usually conducted to understand the behavior of the system under a
specific expected load.
 This load can be the expected concurrent number of users on the application
performing a specific number of transactions within the set duration.
 This test will give out the response times of all the important business critical
transactions.
 If the database, application server, etc. are also monitored, then this simple test can
itself point towards bottlenecks in the application software

• Performance Testing –
 In software engineering, performance testing is in general testing performed to
determine how a system performs in terms of responsiveness and stability under a
particular workload
 It can also serve to investigate, measure, validate or verify other quality attributes of
the system, such as scalability, reliability and resource usage
30
 Unit Testing

Objective is to test
• The function of a program or unit of code such as a program
or module
• Internal logic
• Verify internal design
• Test path & conditions coverage
• Test exception conditions & error handling
• Test the function of a program or unit of code such as a
program or module
• Test internal logic
• Test exception conditions & error handling

31
 Domain, RBT & Security Testing

Objective
• Domain testing is one of the most widely practiced software testing
techniques. It is a method of selecting a small number of test cases from a
nearly infinite group of candidate test cases.
• Domain knowledge plays a very critical role while testing domain-specific work
Requirement Based Testing
• Requirements based testing involves verifying that the software meets the
application's requirements
• Each requirement has a set of tests demonstrating the successful
implementation of that requiremen
Security Testing
• Verify how easily a system is subject to security violations under different
conditions and environments

32
 Software Testability
Software testability is a key aspect to allow the detection of difficult to uncover
defects in software. Software testability supports the testing process and
facilitates the creation of better quality software.

Software testability can be described as the probability that a piece of software

will fail on its next execution during testing if the software includes a fault.

A traceability matrix is a table that correlates requirements or design documents

to test documents. It is used to change tests when related source documents are
changed, to select test cases for execution when planning for regression tests by
considering requirement coverage

33
 User Documentation Testing
User Documentation covers all the manuals, user guides, installation guides, setup guides,
read me files,software release notes, and online help that are provided along with the
software to help the end user to understand the software system

This testing plays a vital role as the users will refer this document when they start using the
software

A badly written document can put off a user and bias them against the product even the
product offers rich functionality

Defects found in the user documentation need to be tracked to closure like any regular
software defect because these documents are the first interactions the users have with the
product

A good User Documentation aids in reducing customer support calls. The effort and money
spend on this effort would form a valuable investment in the long run for the organization
34
 Certifications
Several certification programs exist to support the professional aspirations of
software testers and quality assurance specialists

 Certified Associate in Software Testing (CAST) offered by the QAI

 CATe offered by the International Institute for Software Testing
 Certified Manager in Software Testing (CMST) offered by the QAI
 Certified Test Manager (CTM) offered by International Institute for Software Testing
 Certified Software Tester (CSTE) offered by the Quality Assurance Institute (QAI)
 Certified Software Test Professional (CSTP) offered by the International Institute for Software
Testing
 CSTP (TM) (Australian Version) offered by K. J. Ross & Associates

 ISTQB Certified Tester, Foundation Level (CTFL) offered by the International Software Testing
Qualification Board
 ISTQB Certified Tester, Advanced Level (CTAL) offered by the International Software Testing
Qualification Board
 TMPF TMap Next Foundation offered by the Examination Institute for Information Science
 TMPA TMap Next Advanced offered by the Examination Institute for Information Science
35
 Software Testing Summary

Testing can show the presence of faults in a system; it cannot prove there are no
remaining faults

Component developers are responsible for component testing; system testing is

the responsibility of a separate team

Integration testing is testing increments of the system; release testing involves

testing a system to be released to a customer

Use experience and guidelines to design test cases in defect testing

Interface testing is designed to discover defects in the interfaces of composite

components

Structural analysis relies on analyzing a program and deriving tests from this
analysis

Test automation reduces testing costs by supporting the test process with a range
of software tools
36