SOFTWARE ENGINEERING UNIT - 3

Software Testing | Basics

Software testing can be stated as the process of verifying and validating that a
software or application is bug free, meets the technical requirements as
guided by it’s design and development and meets the user requirements
effectively and efficiently with handling all the exceptional and boundary
cases.

The process of software testing aims not only at finding faults in the existing
software but also at finding measures to improve the software in terms of
efficiency, accuracy and usability. It mainly aims at measuring specification,
functionality and performance of a software program or application.

Software testing can be divided into two steps:

1. ​
Verification:​it refers to the set of tasks that ensure that software correctly
implements a specific function.

2. ​
Validation:​it refers to a different set of tasks that ensure that the software
that has been built is traceable to customer requirements.

Verification:​“Are we building the product right?”

Validation: ​
“Are we building the right product?”
What are different types of software testing?

Software Testing can be broadly classified into two types:

1. ​
Manual Testing:​Manual testing includes testing a software manually, i.e.,
without using any automated tool or any script. In this type, the tester takes
over the role of an end-user and tests the software to identify any unexpected
behavior or bug. There are different stages for manual testing such as unit
testing, integration testing, system testing, and user acceptance testing.

Testers use test plans, test cases, or test scenarios to test a software to
ensure the completeness of testing. Manual testing also includes exploratory
testing, as testers explore the software to identify errors in it.
2. ​
Automation Testing:​Automation testing, which is also known as Test
Automation, is when the tester writes scripts and uses another software to test
the product. This process involves automation of a manual process.
Automation Testing is used to re-run the test scenarios that were performed
manually, quickly, and repeatedly.

Apart from regression testing, automation testing is also used to test the
application from load, performance, and stress point of view. It increases the
test coverage, improves accuracy, and saves time and money in comparison
to manual testing.

What are different techniques of Software Testing?

Software techniques can be majorly classified into two categories:

1.​Black Box Testing:​The technique of testing in which the tester doesn’t

have access to the source code of the software and is conducted at the
software interface without concerning with the internal logical structure of the
software is known as black box testing.

2. ​
White-Box Testing:​The technique of testing in which the tester is aware of
the internal workings of the product, has access to it’s source code and is
conducted by making sure that all internal operations are performed according
to the specifications is known as white box testing.​white box testing which is
also known as glass box testing​, structural testing, clear box testing, open box

testing and transparent box testing​.

Advantages of White box testing

● White box testing optimizes code so hidden errors can be identified.

● Test cases of white box testing can be easily automated.

● This testing is more thorough than other testing approaches as it covers all

code paths.

● It can be started in the SDLC phase even without GUI.

Disadvantages of White box testing

● White box testing is too much time consuming when it comes to large-scale

programming applications.

● White box testing is much more expensive and complex.

● It can lead to production error because it is not detailed by the developers.

● White box testing needs professional programmers who have a detailed

knowledge and understanding of programming language and implementation.

Black Box Testing White Box Testing

 Internal workings of an application Knowledge of the internal workings is
are not required. must.

Also known as closed box/data Also known as clear box/structural

driven testing. testing.

Normally done by testers and

End users, testers and developers.
developers.

THis can only be done by trial and Data domains and internal boundaries
error method. can be better tested.

GreyBox Testing
Gray Box testing is a software testing method to test the software application with

partial knowledge of the internal working structure. It is a combination of black box

and white box testing because it involves access to internal coding to design test

cases as white box testing and testing practices are done at functionality level as

black box testing.

GreyBox testing commonly identifies context-specific errors that belong to web

systems. For example; while testing, if the tester encounters any defect then he

makes changes in code to resolve the defect and then tests it again in real time. It

concentrates on all the layers of any complex software system to increase testing

coverage. It gives the ability to test both the presentation layer as well as internal

coding structure. It is primarily used in integration testing and penetration testing.

What are different levels of software testing?

Software level testing can be majorly classified into 4 levels:

1. ​
Unit Testing:​A level of the software testing process where individual
units/components of a software/system are tested. The purpose is to validate
that each unit of the software performs as designed.
2. ​
Integration Testing:​A level of the software testing process where
individual units are combined and tested as a group. The purpose of this level
of testing is to expose faults in the interaction between integrated units.

3. ​
System Testing:​A level of the software testing process where a complete,
integrated system/software is tested. The purpose of this test is to evaluate
the system’s compliance with the specified requirements.

4. ​
Acceptance Testing:​A level of the software testing process where a
system is tested for acceptability. The purpose of this test is to evaluate the
system’s compliance with the business requirements and assess whether it is
acceptable for delivery.

Types of Software Testing

Introduction:-

Testing is the process of executing a program with the aim of finding errors.
To make our software perform well it should be error-free. If testing is done
successfully it will remove all the errors from the software.

Principles of Testing:-

(i) All the test should meet the customer requirements

(ii) To make our software testing should be performed by a third party

(iii) Exhaustive testing is not possible. As we need the optimal amount of

testing based on the risk assessment of the application.

(iv) All the test to be conducted should be planned before implementing it

(v) It follows the Pareto rule(80/20 rule) which states that 80% of errors come
from 20% of program components.

(vi) Start testing with small parts and extend it to large parts.
Types of Testing:-

1. Unit Testing: ​
It focuses on the smallest unit of software design. In this, we
test an individual unit or group of interrelated units. It is often done by the
programmer by using sample input and observing its corresponding outputs.

Example:

a) In a program we are checking if loop, method or

function is working fine
b) Misunderstood or incorrect, arithmetic precedence.

c) Incorrect initialization

2. Integration Testing

The objective is to take unit tested components and build a program structure
that has been dictated by design. Integration testing is testing in which a group
of components is combined to produce output.

Integration testing is of four types: (i) Top-down (ii) Bottom-up (iii) Sandwich
(iv) Big-Bang

Example

(a) Black Box testing:- It is used for validation.

In this we ignore internal working mechanism and
focus on ​what is the output?​.
(b) White Box testing:- It is used for verification.
In this we focus on internal mechanism i.e.

how the output is achieved?

3. Regression Testing

Every time a new module is added leads to changes in the program. This type
of testing makes sure that the whole component works properly even after
adding components to the complete program.

Example

In school record suppose we have module staff, students

and finance combining these modules and checking if on

integration these module works fine is regression testing

4. Smoke Testing

This test is done to make sure that software under testing is ready or stable
for further testing
It is called a smoke test as the testing of an initial pass is done to check if it
did not catch the fire or smoke in the initial switch on.

Example:

If project has 2 modules so before going to module

make sure that module 1 works properly

5. Alpha Testing

acceptance testing ​which is

This is a type of validation testing. It is a type of ​
done before the product is released to customers. It is typically done by QA
people.

Example:

When software testing is performed internally within

the organization

6. Beta Testing

The beta test is conducted at one or more customer sites by the end-user of
the software. This version is released for a limited number of users for testing
in a real-time environment
Example:

When software testing is performed for the limited

number of people

7. System Testing

This software is tested such that it works fine for the different operating
systems. It is covered under the black box testing technique. In this, we just
focus on the required input and output without focusing on internal working.

In this, we have security testing, recovery testing, stress testing, and

performance testing

Example:

This include functional as well as non functional

testing

8. Stress Testing
In this, we give unfavorable conditions to the system and check how they
perform in those conditions.

Example:

(a) Test cases that require maximum memory or other

resources are executed
(b) Test cases that may cause thrashing in a virtual
operating system

(c) Test cases that may cause excessive disk requirement

9. Performance Testing

It is designed to test the run-time performance of software within the context of

an integrated system. It is used to test the speed and effectiveness of the
program. It is also called load testing. In it we check, what is the performance
of the system in the given load.

Example:

Checking number of processor cycles.

10. Object-Oriented Testing

This testing is a combination of various testing techniques that help to verify
and validate object-oriented software. This testing is done in the following
manner:

● Testing of Requirements,
● Design and Analysis of Testing,
● Testing of Code,
● Integration testing,
● System testing,
● User Testing.
 Next →​← Prev

Test Case
The test case is defined as a group of conditions under which a tester determines

whether a software application is working as per the customer's requirements or

not. Test case designing includes preconditions, case name, input conditions, and

expected result. A test case is a first level action and derived from test scenarios.

It is an in-details document that contains all possible inputs (positive as well as

negative) and the navigation steps, which are used for the test execution process.

Writing of test cases is a one-time attempt that can be used in the future at the

time of regression testing.

Test case gives detailed information about testing strategy, testing process,

preconditions, and expected output. These are executed during the testing process

to check whether the software application is performing the task for that it was

developed or not.

Test case helps the tester in defect reporting by linking defect with test case ID.

Detailed test case documentation works as a full proof guard for the testing team
because if developer missed something, then it can be caught during execution of

these full-proof test cases.

To write the test case, we must have the requirements to derive the inputs, and

the test scenarios must be written so that we do not miss out on any features for

testing. Then we should have the test case template to maintain the uniformity, or

every test engineer follows the same approach to prepare the test document.

Generally, we will write the test case whenever the developer is busy in writing the

code.

When do we write a test case?

We will write the test case when we get the following:

● When the customer gives the business needs then, the developer starts

developing and says that they need 3.5 months to build this product.

● And In the meantime, the testing team will ​start writing the test cases​.

● Once it is done, it will send it to the Test Lead for the review process.

● And when the developers finish developing the product, it is handed over to

the testing team.

● The test engineers never look at the requirement while testing the product

document because testing is constant and does not depends on the mood of

the person rather than the quality of the test engineer.

Note: When writing the test case, the actual result should never be written as the

product is still being in the development process. That?s why the actual result should

be written only after the execution of the test cases.

Why we write the test cases?

We will write the test for the following reasons:

● To require consistency in the test case execution

● To make sure a better test coverage

● It depends on the process rather than on a person

● To avoid training for every new test engineer on the product

To require consistency in the test case execution:​ we will see the test case

and start testing the application.

To make sure a better test coverage:​ for this, we should cover all possible

scenarios and document it, so that we need not remember all the scenarios again

and again.

It depends on the process rather than on a person:​ A test engineer has

tested an application during the first release, second release, and left the company

at the time of third release. As the test engineer understood a module and tested

the application thoroughly by deriving many values. If the person is not there for

the third release, it becomes difficult for the new person. Hence all the derived

values are documented so that it can be used in the future.

To avoid giving training for every new test engineer on the product:​ When

the test engineer leaves, he/she leaves with a lot of knowledge and scenarios.

Those scenarios should be documented so that the new test engineer can test with

the given scenarios and also can write the new scenarios.
What is Mutation Testing?
Mutation testing is a structural testing technique, which uses the structure of the
code to guide the testing process. On a very high level, it is the process of rewriting
the source code in small ways in order to remove the redundancies in the source
code
These ambiguities might cause failures in the software if not fixed and can easily
pass through testing phase undetected.

Mutation Testing​is a type of ​ Software Testing​that is performed to design

new software tests and also evaluate the quality of already existing software
tests. Mutation testing is related to modification a program in small ways. It
focuses to help the tester develop effective tests or locate weaknesses in the
test data used for the program.
History of Mutation Testing:
Richard Lipton​ proposed the mutation testing in 1971 for the first time.
Although high cost reduced the use of mutation testing but now it is widely
used for languages such as Java and XML.
Mutation Testing is a ​
White Box Testing​
.
Mutation testing can be applied to design models, specifications, databases,
tests, and XML. It is a structural testing technique, which uses the structure of
the code to guide the testing process. It can be described as the process of
rewriting the source code in small ways in order to remove the redundancies
in the source code.

Objective of Mutation Testing:

The objective of mutation testing is:
● To identify pieces of code that are not tested properly.
● To identify hidden defects that can’t be detected using other testing
methods.
● To discover new kinds of errors or bugs.
● To calculate the mutation score.
● To study error propagation and state infection in the program.
● To assess the quality of the test cases.

Types of Mutation Testing:

Mutation testing is basically of 3 types:
Value Mutations:
In this type of testing the values are changed to detect errors in the program.
Basically a small value is changed to a larger value or a larger value is
changed to a smaller value. In this testing basically constants are changed.
Example:
Initial Code:

int mod = 1000000007;

int a = 12345678;
int b = 98765432;
int c = (a + b) % mod;

Changed Code:

int mod = 1007;

int a = 12345678;
int b = 98765432;
1. int c = (a + b) % mod;

Decision Mutations:
In decisions mutations are logical or arithmetic operators are changed to
detect errors in the program.
Example:
Initial Code:

if(a < b)
c = 10;
else
c = 20;

Changed Code:

if(a > b)
c = 10;
else
2. c = 20;

Statement Mutations:
In statement mutations a statement is deleted or it is replaces by some other
statement.
Example:
Initial Code:

if(a < b)
c = 10;
else
c = 20;

Changed Code:

if(a < b)
d = 10;
else
3. d = 20;
Advantages of Mutation Testing:
● It brings a good level of error detection in the program.
● It discovers ambiguities in the source code.

Disadvantages of Mutation Testing:

● It is highly costly and time-consuming.
● It is not able for Black Box Testing.

Static Code Analysis

Static code analysis​, or simply Static Analysis, is an application testing method in

which an application’s source code is examined to detect potential security
vulnerabilities. It is usually accomplished by testing the code against a set of
standards and best practices that identify vulnerabilities within the application.

What Does it Cover?

A static code analysis often addresses code vulnerabilities and other code
weaknesses. It often uses data tracing tools that find many vulnerabilities that often
escape most human eyes.

Static Code Analysis Techniques

● Automated tools​- Static code analysis involves many automated tools that
help detect potential vulnerabilities in the source code.
○ Simple grep searches​- Grep is a search utility that allows you to search
for specific terms. Grep style searches can help discover information
related to encryption, cryptography, SQL queries, URLs and sockets,
 and file read and writes. While powerful, these simple grep searches
can lead to many false positives.
○ Tools with data flow analysis​- Automated tools like PumaScan, Fortify
and Checkmarx actually trace data through the application and so it
can detect when untrusted data is used in a trusted manner, e.g., when
untrusted user input is sent to the browser unencoded (XSS
vulnerability) or inserted directly into a SQL query (SQL Injection). This
data flow analysis capability is essential for any serious assessment of
the security of an application.
● Static Code Review​ - A static code review is also known as white-box testing.
It includes checking the overall security of the application to ​identify potential
vulnerabilities​ and risks associated with it.

Where in the SDLC can we use Static Code Analysis?

Static code analysis should be performed at all stages of the SDLC once the
development phase has begun. It should be incorporated into the SDLC before the
unit/component/integration testing phases begin.

In most cases, the static code analysis results are incorporated as a quality check
for code promotion in continuous integration (CI) and continuous delivery (CD)
pipelines.

Benefits

Static code analysis has plenty of benefits, and its ability to quickly discover
weaknesses in the code and to comply with security standards helps reduce
potential vulnerabilities.

What are some of the key benefits of using static code analysis?

● Early detection of code vulnerabilities reduces the cost of debugging and

fixing at a later stage in the SDLC.
● It is conducted by trained security engineers who have strong knowledge
about secure coding practices.
● It is a highly scalable method which means it can run on multiple code bases
and can be run repeatedly.
● It is a fast and efficient way to identify vulnerabilities in the code such as SQL
Injection, buffer overflow, etc.
 ● Automated tools can scan the entire application source code with minimal
effort.
● The use of automated tools helps provide mitigation recommendations,
thereby reducing research time.

Limitations

Despite its benefits, static code analysis has a few limitations that can impact your
software development lifecycle process.

One of the major limitations of static code analysis is that the use of automation
tools often gives a false assurance that everything is being validated. Often there are
a large number of business rules, standards, and expectations involved that
automated tools are not very efficient at analyzing in a code.

What are some of the key limitations of static code analysis?

● It requires a large amount of time if done manually.

● Automated tools often work with only a few programming languages.
● Automated tools will provide false negatives and many false positives.
● Automated tools can not scan the source code for compliance with business
goals.
● Security vulnerabilities like authentication problems, insecure use of
cryptography, access control issues, etc. are very difficult to find
automatically.

Dynamic Testing

Dynamic testing is the method of debugging an application’s source code in a

run-time environment, i.e when the application is running. It is used to identify
security vulnerabilities while the program is running.

Since the program can be run with multiple inputs, the scope of a dynamic
assessment can get quite large.

What Does it Cover?

Dynamic testing can help discover a wide range of vulnerabilities such as
input/output validation issues that could expose security vulnerabilities and make it
vulnerable to SQL Injection (SQLi), Cross-Site Scripting (XSS), etc.

In addition to this, dynamic testing also helps identify security misconfiguration and
other common issues that might impact the overall security of the application.

Automated dynamic testing allows users to continually scan applications as they

evolve and facilitates the automatic detection of vulnerabilities. Automated alerts
can be set in these tools that will send alerts and notifications once a vulnerability is
detected.

Dynamic testing also includes manual test cases to detect vulnerabilities that are
otherwise not easily discovered by automated tools such as session management
issues, information leakage, authentication issues, access control, and more.

Where in the SDLC can We use Dynamic Testing?

Dynamic testing can be incorporated during multiple stages. In the pre-production

stage, dynamic testing prevents error-prone code from going into the production
phase. This can be used as a quality test with continuous integration (CI)/continuous
delivery (CD) tools for code promotion. During the production stage, dynamic testing
can help troubleshoot production issues quickly.

Benefits

Dynamic testing is a great method to identify vulnerabilities in a run-time

environment. It helps identify issues in the application which thereby reduces the
time to identify production issues.

What are some of the key benefits of using dynamic testing?

● Identifies vulnerabilities in a run-time environment.

● Automated tools offer flexibility on what to scan for.
● Helps identify vulnerabilities that might not adhere to business context or
standards.
● Permits you to validate the findings from static code analysis.
 ● Allows for the testing of the application without having access to the source
code.
● Can be applied to any application.

Limitations

Although dynamic testing helps validate the reports by static code analysis, which
gives a better security assurance, there are still some limitations to dynamic testing.

What are some of the key limitations of dynamic testing?

● Automated tools may produce false negatives and false positives.

● There is a lack of trained professionals who can thoroughly conduct dynamic
testing.

Software Reliability
Software Reliability means ​Operational reliability​. It is described as the ability of a
system or component to perform its required functions under static conditions for a
specific period.

Software reliability is also defined as the probability that a software system fulfills its
assigned task in a given environment for a predefined number of input cases,
assuming that the hardware and the input are free of error.

Software Reliability is an essential connect of software quality, composed with

functionality, usability, performance, serviceability, capability, installability,
maintainability, and documentation. Software Reliability is hard to achieve because
the complexity of software turn to be high. While any system with a high degree of
complexity, containing software, will be hard to reach a certain level of reliability,
system developers tend to push complexity into the software layer, with the speedy
growth of system size and ease of doing so by upgrading the software.

For example​, large next-generation aircraft will have over 1 million source lines of
software on-board; next-generation air traffic control systems will contain between
one and two million lines; the upcoming International Space Station will have over
two million lines on-board and over 10 million lines of ground support software;
several significant life-critical defense systems will have over 5 million source lines
of software. While the complexity of software is inversely associated with software
reliability, it is directly related to other vital factors in software quality, especially
functionality, capability, etc.
Reliability Metrics
Reliability metrics are used to quantitatively express the reliability of the software
product. The option of which metric is to be used depends upon the type of system
to which it applies & the requirements of the application domain.

Some reliability metrics which can be used to quantify the reliability of the software
product are as follows:

IN DETAIL -- ​Software Engineering Software Reliability Metrics - javatpoint

Reliability Growth Models

The reliability growth group of models measures and predicts the

improvement of reliability programs through the testing process. The growth
model represents the reliability or failure rate of a system as a function of time
or the number of test cases. Models included in this group are as following
below.

1. Coutinho Model –
Coutinho adapted the Duane growth model to represent the software
testing process. Coutinho plotted the cumulative number of
deficiencies discovered and the number of correction actions made
vs the cumulative testing weeks on log-log paper. Let N(t) denote the
cumulative number of failures and let t be the total testing time. The
failure rate, (t), the model can be expressed as

where are the model parameters. The least squares method

can be used to estimate the parameters of this model.
2. Wall and Ferguson Model –
Wall and Ferguson proposed a model similar to the Weibull growth
model for predicting the failure rate of software during testing. The
cumulative number of failures at time t, m(t), can be expressed as

where are the unknown parameters. The function b(t)

can be obtained as the number of test cases or total testing time.
Similarly, the failure rate function at time t is given by
Wall and Ferguson tested this model using several software failure
data and observed that failure data correlate well with the model.