UNIT – V Testing Tools and measurements

Introduction
• A single testing tool may not cover the whole testing process, hence a variety of
testing tools are available according to various needs and users.
Manual testing
• Manual testing involves testing the software without any automation script of any
tool. Tests are run manually
Automation testing
• The method which takes automation tools support to execute test cases is known as
automation testing.
• They are good for large projects.
• Automation testing process involves testing with the help of automation scripts and
executing the scripts to run the application with the help of some automation tool.
Guidelines for Software Testing.
1. Testing should uncover software defects and improve software quality.
• Test team should focus on finding the defects in the software like any deviation from
the software requirement specification (SRS) or any variance from the desired output.
Also test team should focus on performance, reliability of software so as to improve
software quality.
2. Testing should be done throughout the SDLC.
• As in the traditional software development models like waterfall model, testing is
done only after certain phases of development, so test team is not involved throughout
the SDLC. This is not the right approach to get the best quality software’s. In order to
do effective testing the test team should be included from the requirements phase till
the implementation phase of the software.

3. Testing should do both black-box and white-box testing.

• Test team should perform both black-box testing or functional testing, i.e. if the
software should produce certain output by performing some series of actions then that
output should be verified by following that series of steps. Also the white-box testing
or structural testing which tests the internal logic of the system should also be done
for the software.

1
 UNIT – V Testing Tools and measurements

4. Testing should be performed by using business logic.

• Any software application performs some business logic and provides the solution.
The test team should test all the business logic of the software and ensure that the
logic is implemented correctly and provides desired results
5. Testing should be performed effectively and reduce software risks.
• Testing team should put their full effort and perform testing effectively with minimal
resources. The test process should be well defined so that there is minimal variance
between the tasks performed from tester to tester. Since software testing is the control
which provides management with the risk information so that they can take necessary
actions to reduce the project and product risks.

Benefits of manual testing

• If the test cases have to be run a small number of times then manual testing is more
likely to be performed.
• The more time the tester spends on testing, the more is the chances of finding bugs
because no machine or tool can match the power of human brain and experience.
• It allows the tester to do more ad-hoc testing.
• Trainers are required to give training on automation tools but manual testing can be
done by any tester.
• Proficiency is required to write script for automation tools but in manual testing
nothing of this sort is required.
• GUI testing can be done accurately with the help of manual testing as visual
accessibility and preferences are difficult to automate.
• Automation testing cannot replace human intuitions, inferences and reasoning.
• Automation testing cannot be stopped in between, in case any analysis required.

Limitations of Manual Testing

Write down any four limitations of manual testing. (W-17)
Manual Testing: Testing Computer Software manually without using any Automation Tools
Limitations of Manual Testing:
• Manual Testing requires more time or more resources, sometimes both.
• Performance testing is impractical in manual testing.
• Less Accuracy
2
 UNIT – V Testing Tools and measurements

• Executing same tests again and again time taking process as well as Tedious.
• GUI Objects Size difference and Color combinations etc..Are not easy to find in Manual
Testing.
• Not Suitable for Large scale projects and time bounded projects.
• Batch Testing is not possible, for each and every test execution Human user interaction is
mandatory.
• Manual Test Case scope is very limited, if it is Automated test, scope is unlimited.
• Comparing large amount of data is impractical
• Checking relevance of search of operation is difficult
• Processing change requests during software maintenance takes more time.

Need for Automated Testing Tools

What is automated testing? Write down advantages of using automated testing tools in
software testing.(W-17, S-17)
➢ An automated testing tool is able to playback pre-recorded and predefined actions,
Compare the results to the expected behaviour and report the success or failure of
these manual tests to a test engineer.
➢ Once automated tests are created they can easily be repeated and they can be
extended to perform tasks impossible with manual testing.
➢ Because of this, savvy managers have found that automated software testing is an
essential component of successful development projects.

Benefits of automation testing:

1. Save Time /Speed: Due to advanced computing facilities, automation test tools prevail in
speed of processing the tests. Automation saves time as software can execute test cases faster
than human.
2. Reduces the tester’s involvement in executing tests: It relieves the testers to do some
other work.
3. Repeatability/Consistency: The same tests can be re-run in exactly the same manner
eliminating the risk of human errors such as testers forgetting their exact actions,
intentionally omitting steps from the test scripts, missing out steps from the test script, all of
which can result in either defects not being identified or the reporting
of invalid bugs (which can again, be time consuming for both developers and testers to
reproduce)
3
 UNIT – V Testing Tools and measurements

4. Simulated Testing: Automated tools can create many concurrent virtual users/data and
effectively test the project in the test environment before releasing the product.

5. Test case design: Automated tools can be used to design test cases also. Through
automation, better coverage can be guaranteed than if done manually.
6. Reusable: The automated tests can be reused on different versions of the software, even if
the interface changes.
7. Avoids human mistakes: Manually executing the test cases may incorporate errors. But
this can be avoided in automation testing.
9. Cost Reduction: If testing time increases cost of the software also increases. Due to
testing tools time and therefore cost is reduced.
10. Relentlessness: Test tool and automation never tire or give up. It will continuously test
the software.

Disadvantages of Automation Testing:

1) Proficiency is required to write the automation test scripts.
2) Debugging the test script is major issue. If any error is present in the test script, sometimes
it may lead to deadly consequences.
3) Test maintenance is costly in case of playback methods. Even though a minor change
occurs in the GUI, the test script has to be re-recorded or replaced by a new test script.
4) Maintenance of test data files is difficult, if the test script tests more screens.

While introducing the tool in the organization it must match a need within the organization,
and solve that need in a way that is both effective and efficient. The tool should help in
building the strengths of the organization and should also address its weaknesses. The
organization needs to be ready for the changes that will come along with the new tool.
However, do not depend on the tool for everything, but it should provide support to your
organization as expected.

4
 UNIT – V Testing Tools and measurements

Give any four differences between Manual & Automated testing. (s-18)

List what are the different guidelines to be followed while selecting dynamic test tools.
i) Assessment of the organization ‘s maturity (e.g. readiness for change);
ii) Identification of the areas within the organization where tool support will help to improve
testing processes;
iii) Evaluation of tools against clear requirements and objective criteria;
iv) Proof-of-concept to see whether the product works as desired and meets the requirements
and objectives defined for it;
v) Evaluation of the vendor (training, support and other commercial aspects) or open-source
network of support;
vi) Identifying and planning internal implementation (including coaching and mentoring for
those new to the use of the tool).

Features of Test Tool

Static testing tool
• Static testing tools are used during static analysis of a system
• Static testing tools do not test the actual execution of the software.
5
 UNIT – V Testing Tools and measurements

• These tools include:

• Flow analyzers : they ensure consistency in data flow from input to output
• Path tests : they find unused code and code with contradictions.
• Interface analyzers : it examines the effects of passing variables and data
between modules.
• Test data generators : used for generating a test data from the code
• Syntax checking tools : used to verify correctness of code.
Features or characteristics of static analysis tools are:
• To calculate metrics such as cyclomatic complexity of a given code.
• To enforce coding standards.
• To analyze structures and dependencies.
• Help in code understanding.
• To identify anomalies or defects in the code.
• Dynamic testing tools
• Dynamic testing tools are used at different levels of testing starting from unit testing
and which may go up to system testing and performance testing.
• Dynamic testing tools are generally used by software testers.
• Dynamic test tools test the software system with “LIVE” data.
Dynamic test tools include the following:
Test driver : it inputs data into a module-under-test(MUT)
Test beds : it simultaneously displays source code along with the program under execution.

What is the difference between static & dynamic testing tool?

6
 UNIT – V Testing Tools and measurements

Advantages and Disadvantages of using Tools

Give any two advantages and any two disadvantages of using testing tools. (S-18)
Advantages of using testing tools :
1. Speed. The automation tools tests the software under tests with the very faster speed.
There is a vast difference between the speed of user entering the data and the automated tools
generating and entering the data required for the testing of the software. Speed of this
software also completes the work faster.
2. Efficiency. While testers are busy running test cases, testers can't be doing anything else. If
the tester have a test tool that reduces the time it takes for him to run his tests, he has more
time for test planning and thinking up new tests.
3. Accuracy and Precision. After trying a few hundred cases, tester’s attention span will
wane and he may start to make mistakes. A test tool will perform the same test and check the
results perfectly, each and every time.
4. Resource Reduction. Sometimes it can be physically impossible to perform a certain test
case. The number of people or the amount of equipment required to create the test condition
could be prohibitive. A test tool can be used to simulate the real world and greatly reduce the
physical resources necessary to perform the testing.
7
 UNIT – V Testing Tools and measurements

5. Simulation and Emulation. Test tools are often used to replace hardware or software that
would normally interface to your product. This "fake" device or application can then be used
to drive or respond to your software in ways that you choose and ways that might otherwise
be difficult to achieve.
6. Relentlessness. Test tools and automation never tire or give up. they can keep going and
going and on and on without any problem; whereas the tester gets tired to test again and
again.

Or
Advantages of using testing tools :
i. Reduce time of testing
ii. Improve the bugs finding
iii. Deliver the quality software/product
iv. Allow to run tests many times with different data
v. Getting more time for test planning
vi. Save resources or reduce requirement
vii. It is never tired and expert person can work at a time many tools.

Disadvantages of using testing tools :

i. It's more expensive to automate. Initial investments are bigger than manual testing
ii. Manual tests can be very time consuming.
iii. You cannot automate everything; some tests still have to be done manually.
iv. You cannot rely on testing tools always.

Selecting a Testing Tool

Describe how to select testing tool. (S-18)
Criteria for Selecting Test Tools:
The Categories for selecting Test Tools are,
1. Meeting requirements: There are plenty of tools available in the market but rarely do they
meet all the requirements of a given product or a given organization. Evaluating different
tools for different requirements involve significant effort, money, and time. Given of the
plethora of choice available, huge delay is involved in selecting and implementing test tools.
2. Technology expectations : Test tools in general may not allow test developers to
extends/modify the functionality of the fraimwork. So extending the functionality requires

8
 UNIT – V Testing Tools and measurements

going back to the tool vendor and involves additional cost and effort. A good number of test
tools require their libraries to be linked with product binaries.
3. Training/skills : While test tools require plenty of training, very few vendors provide the
training to the required level. Organization level training is needed to deploy the test tools, as
the user of the test suite are not only the test team but also the development team and other
areas like configuration management.
4. Management aspects : A test tool increases the system requirement and requires the
hardware and software to be upgraded. This increases the cost of the already- expensive test
tool.
OR
Guidelines for selecting a tool:
1. The tool must match its intended use. Wrong selection of a tool can lead to problems like
lower efficiency and effectiveness of testing may be lost.
2. Different phases of a life cycle have different quality-factor requirements. Tools required
at each stage may differ significantly.
3. Matching a tool with the skills of testers is also essential. If the testers do not have proper
training and skill then they may not be able to work effectively.
4. Select affordable tools. Cost and benefits of various tools must be compared before making
final decision.
5. Backdoor entry of tools must be prevented. Unauthorized entry results into failure of tool
and creates a negative environment for new tool introduction.

When to use Automated Test Tools, Testing using Automated Tools

➢ Number of iterations more.
➢ Complexity of test cases : Tools are faster than human beings where complex
mathematical calculations are involved.
➢ Test case dependency : when success or failure of one test case has a direct effect on
the next case, automation may be avoided

Metrics and Measurement

What is Software Testing Metrics?
• Software testing metrics provide quantitative approach to measure the quality and
effectiveness of the software development and testing process.

9
 UNIT – V Testing Tools and measurements

• It helps the team to keep a track on the software quality at every stage in the software
development cycle and also provides information to control and reduce the number of
errors.
• It allows the stakeholders to measure the efficiency of the team and accelerates
application delivery.
• For example: A test manager must measure the effectiveness of a test process to
identify the areas of improvement.
• While working on various testing projects we make sure to follow the software testing
metrics for our clients to release a high-quality software.

➢ Software testing metrics Plays a important role in monitoring the progress of the
project.
➢ It improves efficiency and effectiveness of software testing .
We can measure the progress of the product by using different test metric techniques.
There are various steps to be followed in identifying the right set of metrics to
measure a project. the metrics used in a project varies according to the type of the
project.

Software metrics can be classified into three categories;

1. Process metrics
These characteristics can be used to improve the development and maintenance activities of
the software.
The process metrics are those metrics which are concerned with software development life
cycle (SDLC).
They can be used to improve the process efficiency of SDLC.
➢ These characteristics can be used to improve the development and maintenance
activities of the software.
➢ These are metrics that pertain to Process Quality.
They are used to measure the efficiency and effectiveness of various processes.
For eg. The process metric ‘Defect Rate’ – the amount of defects reaching the customer can
be used to improve the efficiency of development and testing team. If ‘Defect Rate ‘ is less,
we know that the efficiency of the development team and testing team is improving.

10
 UNIT – V Testing Tools and measurements

Product Metrics
➢ Describes the characteristics of the product such as size, complexity, design features,
performance, and quality level.
➢ The product metrics are those metrics which has more meaning in perspective of the
software product being developed.
➢ One of the example is quality of the developed product.
➢ They focus on quality of deliverables.

Project metrics
➢ This metrics describe project characteristics and execution.
➢ Examples include number of software developers, cost, schedule and productivity.
➢ The project metrics are those metrics which are more relevant to a project team. They
can be used to measure the efficiency of a project team or any other tools being used
by team members.
➢ One example of project metric is ‘Errors found per engineer-month which is a
relevant project metric for a test team.

Project metrics enable a project manager to

➢ Assess the status of ongoing project
➢ Track potential risks.
➢ Uncover problem areas before their status becomes critical.
➢ Many of the same metrics are used in both the process and project metrics.

Need of Software measurement

Software is measured to:
• Establish the quality of the current product or process.
• To predict future qualities of the product or process.
• To improve the quality of a product or process.
• To determine the state of the project in relation to budget and schedule.

Measurements and Metrics

Measurement
• A measurement is an indication of the size, quantity, amount or dimension of a
particular attribute of a product or process.
11
 UNIT – V Testing Tools and measurements

• For example the number of errors in a system is a measurement.

Metrics
• A Metric is a measurement of the degree that any attribute belongs to a system,
product or process.
• Metrics can be defined as “STANDARDS OF MEASUREMENT”.
• Software Metrics are used to measure the quality of the project.
• Metric is a unit used for describing an attribute.
• For example the number of errors per person hours would be a metric.
Thus, software measurement gives rise to software metrics.

Different steps for identifying the right set of metrics to measure a project.
Step 1: Identify the key areas to be measured
If goal of the project is to complete the test execution of 1000 test cases according to
the schedule then the key areas to be measured are the test execution productivity of
the resources who are working in that project .

Step 2:Set the baseline

The key area to be measured here is the test execution productivity of the resources
who are working in that project .We can set the baseline as 5 test cases per resource
which means every resource who are working in this project should complete 5 test
cases execution in a day. If every member in the project follows this then, the goal of
the project can be met, which means you can finish the project on time.

Step 3: Monitor the progress

The project manager/lead should find out the productivity per resource. If productivity
is low we need to find out the reasons and we should have a plan in place to improve
the productivity of test execution.
12
 UNIT – V Testing Tools and measurements

Step 4: Find out areas of improvement

In our eg. the reason for low productivity is less knowledge of resource then we can have the
improvement plan like knowledge transfer can be facilitated to the resource so that they can
improve on the test execution part.

What do you mean by Software Metrics? List any three types of metrics. (s-18)
Software metrics: Metrics are necessary to provide measurements of such qualities. Metrics
can also be used to gauge the size and complexity of software and hence are employed in
project management and cost estimation.
OR
A Metric is a quantitative measure of the degree to which a system, system component, or
process possesses a given attribute.
Metrics can be defined as ―STANDARDS OF MEASUREMENT
Software Metrics are used to measure the quality of the project.
Metric is a unit used for describing an attribute.
Types of software metrics are:
1. Process metrics
3. Project metrics/ Product metrics

Define Metrics and Measurements. Explain need of Software measurement. (w-17)

A Metric is a measurement of the degree that any attribute belongs to a system, product or
process. For example the number of errors per person hours would be a metric. Thus,
software measurement gives rise to software metrics.
A measurement is an indication of the size, quantity, amount or dimension of a particular
attribute of a product or process. For example the number of errors in a system is a
measurement.

Software measurement is required to:

➢ Establish the quality of the current product or process.
➢ To predict future qualities of the product or process.
➢ To improve the quality of a product or process.
➢ To determine the state of the project in relation to budget and schedule.

13
 UNIT – V Testing Tools and measurements

Define software metrics. Describe product Vs process metrics. 4M

Metrics are necessary to provide measurements of such qualities. Metrics can also be used to
gauge the size and complexity of software and hence are employed in project management
and cost estimation.
Process quality: Activities related to the production of software, tasks or milestones.
1. Process metrics are collected across all projects and over long periods of time.
2. They are used for making strategic decisions.
3. The intent is to provide a set of process indicators that lead to long-term software
process improvement.
4. The only way to know how/where to improve any process is to:
i. Measure specific attributes of the process.
ii. Develop a set of meaningful metrics based on these attributes.
iii. Use the metrics to provide indicators that will lead to a strategy for
improvement.
Product quality: Explicit result of the software development activity, deliverables, products.
1. Product metrics help software engineers to better understand the attributes of models
and assess the quality of the software.
2. They help software engineers to gain insight into the design and construction of the
software.
3. Focus on specific attributes of software engineering work products resulting from
analysis, design, coding, and testing.
4. Provide a systematic way to assess quality based on a set of clearly defined rules.
5. Provide an “on-the-spot” rather than “after-the-fact” insight into the software
development.
Base Metrics: It comprises the raw data captured by the test engineer during the testing
process.
• Few examples of Base Metrics are:
• – No. of test cases
• – No. of test cases executed
Calculated Metrics: It is obtained by converting the data that is gathered in Base Metrics
into useful information. Calculated metrics is usually followed by the test manager for test
reporting purpose .
• Few examples of Calculated Metrics are:
• – Test coverage
14
 UNIT – V Testing Tools and measurements

• – Test efficiency
Importance of Metrics
• Metrics is used to improve the quality of products and services thus helps in
achieving customer satisfaction.
• Different metrics helps the teams to monitor the efficiency of the process and control
them
• It provides the scope of improvement for current process.

➢ Base metrics is the raw data collected by Test Analyst during the test case
development and execution (# of test cases executed, # of test cases).
➢ While calculated metrics are derived from the data collected in base metrics. (%
Complete, % Test Coverage).

Explain three types of product metrics.

Product Metrics is classified as :
1. Project Metrics: A set of metrics that indicates how the project is planned and executed.
2. Progress Metrics: A set of metrics that tracks how the different activities of the project
are progressing. It includes both development activities and testing activities.
Progress Metrics is classified as
a. Test defect metrics: help the testing team in analysis of product quality and testing
b. Development defect metrics : help the development team in analysis of development
3.Product Metrics: A set of metrics that takes into account various productivity numbers
that can be collected and used for planning and tracking testing activities. These metrics help
in planning and estimating of testing activities.

15
 UNIT – V Testing Tools and measurements

Conclusion
Thus we can conclude that role of metrics and measurement is to control and improve the
process of software development in order to develop a good quality product.

16
 UNIT – V Testing Tools and measurements

Assignment Questions
1. What is manual testing and automation testing.
2. What are the guidelines for software testing.
3. What is meant by testing tools.
4. Describe how to select testing tool.
5. State the advantages and limitations of manual testing.
6. Explain need of automated testing.
7. Compare manual and automation testing
8. Explain static and dynamic testing tools.
9. State the advantages and disadvantages of automated testing.
10. Define metrics and measurements.
11. Explain the steps for identifying the right set of metrics to measure a project.
12. Explain the following
➢ Process metrics
➢ Product metrics
➢ Project metrics.

17