STQA NOTES SYMCA

UNIT 6
TEST AUTOMATION
1.Design and architecture for automation

Designing a practical test automation architecture provides a solid foundation for a successful
automation effort. This paper describes key elements of automated testing that need to be
considered, models for testing that can be used for designing a test automation architecture,
and considerations for successfully combining the elements to form an automated test
environment. The paper first develops a general framework for discussion of software testing
and test automation. This includes a definition of test automation, a model for software tests,
and a discussion of test oracles. The remainder of the paper focuses on using the framework
to plan for a test automation architecture that addresses the requirements for the specific
software under test (SUT).

Definition of Automated Software Tests

Manual testing can be described as a situation where a person initiates each test, interacts
with it, and interprets, analyzes, and reports the results. Software testing is automated when
there is a mechanism for tester-free running of test cases. I generally call test cases automated
when all of the following elements are present. If one or more elements are absent I consider
the tests semiautomated. (Which is often the most cost-effective.)

· Ability to run two or more specified test cases

· Ability to run a subset of all the automated test cases

· No intervention is needed after launching the tests

· Automatically sets-up and/or records the relevant test environment parameters

· Runs the test cases

· Captures the relevant results

· Compares actual with expected results and flags differences

· Analyzes and reports pass/fail for each test case and for the test run.

A Model of Testing

Software testing involves more than feeding inputs to a program and observing results.
Software today also has states and interacts with stored data and the computer environment.
Figure 1 models the inputs and results for some software. Such a model is important in test
automation because it provides categories to identify the inputs and results that must be
monitored and manipulated during automated testing. For even a simple automated test that

Page 59 of 88
STQA NOTES SYMCA

feeds inputs to the SUT, the automated test should verify the expected direct results and the
postcondition program state to be sure the SUT did the right thing and ended in the correct
program state. If the program isn’t supposed to change the system environment or any data
sets, then some verification should be performed to confirm the correct environment and data
values after the test. (These are particularly difficult to do in many systems and
extraordinarily important. Manual testers see these types of problems as “program hangs after
completion,” “user no longer has correct permissions,” or “data corruption from unknown
source” errors. When we automate tests we must ensure that such problems are detected of
we risk incorrectly passing software that obviously doesn’t work.

Architecting Test Automation

To this point in this paper the factors and models have been generic; they apply for any SUT.
But, it is not true that any single automation mechanism today fits all situations for SUT. The
SUT itself plays a major role in the architecture of the test engine. Assuming that good test
cases exist, automation of SUT testing involves specific input values being fed and
corresponding results checked through the SUT’s interfaces. The process I use for deciding
upon the architecture for test automation is derived from models and analysis from the key
factors. Models for the specific SUT are used to identify interfaces with the environment and
to split the testing into smaller components. Once the SUT and environments are understood,
the best places and mechanisms for automation can be selected. From this information an
architecture can be articulated that provides the best solution to the automation requirements.
Understanding the SUT for automation requires more than evaluating the inputs, processes,
and outputs. Each of the eight input and output sections in the Testing Model must be
identified in the SUT. Software can effect data values in memory, stored in databases, and
program internal states. There are also side effects to program execution such as resource
utilization (disk space, printers, system variables, system environment). Programs can
interfere with one another, consume all of a resource, change the system state, etc. Manual

Page 60 of 88
STQA NOTES SYMCA

testing factors in most of these effects because a human observes more than just the SUT.
Test automation needs to account for these results from running the SUT. This is most easily
done by modeling the SUT and identifying the various interface points. Figure 2 is an
illustration of a simple model for some SUT. A model such as this for the specific SUT for
which the automated test environment is intended is used for several things.

There are two parts to the description of the automation architecture. The first is a structural
description that shows the elements and connections between them. Like a data flow diagram,
information can be displayed and its control and movement depicted. An example of a
structure diagram is shown in Figure 4. The second part of the description is the sequence or
flow of events. What event starts what process, and when. This second part explains the
process for the automated test sequence. A typical sequence is shown below. Not all steps
need to be automated. There is always a cost tradeoff to be considered. Each step must be
done, but the cost of automating them may be prohibitive for some of the activities

Page 61 of 88
STQA NOTES SYMCA

1. Testware version control and configuration management 2. Selecting the

subset of test cases to run 3. Set-up and/or record environmental variables 4.
Run the test exercises 5. Monitor test activities 6. Capture relevant results 7.
Compare actual with expected results 8. Report analysis of pass/fail Note that
the two descriptive elements may be combined into a powerful description of
the automation environment. By labeling the structure diagram with the event
numbers we can depict which elements perform what tasks in time sequence.

2.Testing of object oriented System

The shift from traditional to object-oriented environment involves looking at and

reconsidering old strategies and methods for testing the software. The traditional
programming consists of procedures operating on data, while the object-oriented paradigm
focuses on objects that are instances of classes. In object-oriented (OO) paradigm, software
engineers identify and specify the objects and services provided by each object. In addition,
interaction of any two objects and constraints on each identified object are also determined.
The main advantages of OO paradigm include increased reusability, reliability,
interoperability, and extendibility.

Object-oriented Testing Methods

As many organizations are currently using or targeting to switch to the OO paradigm, the
importance of OO software testing is increasing. The methods used for performing object-
oriented testing are discussed in this section.

State-based testing is used to verify whether the methods (a procedure that is executed by an
object) of a class are interacting properly with each other. This testing seeks to exercise the
transitions among the states of objects based upon the identified inputs.
For this testing, finite-state machine (FSM) or state-transition diagram representing the
possible states of the object and how state transition occurs is built. In addition, state-based
testing generates test cases, which check whether the method is able to change the state of

Page 62 of 88
 STQA NOTES SYMCA

object as expected. If any method of the class does not change the object state as expected, the
method is said to contain errors.
To perform state-based testing, a number of steps are followed, which are listed below.
1. Derive a new class from an existing class with some additional features, which are used to
examine and set the state of the object.

2. Next, the test driver is written. This test driver contains a main program to create an object,
send messages to set the state of the object, send messages to invoke methods of the class that
is being tested and send messages to check the final state of the object.

3. Finally, stubs are written. These stubs call the untested methods.

Fault-based Testing
Fault-based testing is used to determine or uncover a set of plausible faults. In other words, the
focus of tester in this testing is to detect the presence of possible faults. Fault-based testing
starts by examining the analysis and design models of OO software as these models may
provide an idea of problems in the implementation of software. With the knowledge of system
under test and experience in the application domain, tester designs test cases where each test
case targets to uncover some particular faults.
The effectiveness of this testing depends highly on tester experience in application domain and
the system under test. This is because if he fails to perceive real faults in the system to be
plausible, testing may leave many faults undetected. However, examining analysis and design
models may enable tester to detect large number of errors with less effort. As testing only
proves the existence and not the absence of errors, this testing approach is considered to be an
effective method and hence is often used when security or safety of a system is to be tested.
Integration testing applied for OO software targets to uncover the possible faults in both
operation calls and various types of messages (like a message sent to invoke an object). These
faults may be unexpected outputs, incorrect messages or operations, and incorrect invocation.
The faults can be recognized by determining the behavior of all operations performed to invoke
the methods of a class.
Scenario-based Testing
Scenario-based testing is used to detect errors that are caused due to incorrect specifications
and improper interactions among various segments of the software. Incorrect interactions often
lead to incorrect outputs that can cause malfunctioning of some segments of the software. The
use of scenarios in testing is a common way of describing how a user might accomplish a task
or achieve a goal within a specific context or environment. Note that these scenarios are more
context- and user specific instead of being product-specific. Generally, the structure of a
scenario includes the following points.
1. A condition under which the scenario runs.

2. A goal to achieve, which can also be a name of the scenario.

Page 63 of 88
 STQA NOTES SYMCA

3. A set of steps of actions.

4. An end condition at which the goal is achieved.

5. A possible set of extensions written as scenario fragments.

Scenario- based testing combines all the classes that support a use-case (scenarios are subset
of use-cases) and executes a test case to test them. Execution of all the test cases ensures that
all methods in all the classes are executed at least once during testing. However, testing all the
objects (present in the classes combined together) collectively is difficult. Thus, rather than
testing all objects collectively, they are tested using either top-down or bottom-up integration
approach.
This testing is considered to be the most effective method as scenarios can be organized in such
a manner that the most likely scenarios are tested first with unusual or exceptional scenarios
considered later in the testing process. This satisfies a fundamental principle of testing that
most testing effort should be devoted to those paths of the system that are mostly used.
Challenges in Testing Object-oriented Programs
Traditional testing methods are not directly applicable to OO programs as they involve OO
concepts including encapsulation, inheritance, and polymorphism. These concepts lead to
issues, which are yet to be resolved. Some of these issues are listed below.
1. Encapsulation of attributes and methods in class may create obstacles while testing. As
methods are invoked through the object of corresponding class, testing cannot be accomplished
without object. In addition, the state of object at the time of invocation of method affects its
behavior. Hence, testing depends not only on the object but on the state of object also, which
is very difficult to acquire.

Inheritance and polymorphism also introduce problems that are not found in
traditional software. Test cases designed for base class are not applicable to
derived class always (especially, when derived class is used in different
context). Thus, most testing methods require some kind of adaptation in order
to function properly in an OO environment

3.Criteria for selecting the test tool

Today, the entire software development process is going through a cultural shift from the
traditional Waterfall model to the modern agile methodology. With this paradigm change, the
entire development group, including testing, is facing significant changes in workplace culture,
philosophy, tool usage, and delivery patterns. Different important factors are leading to this
paradigm shift. Some of them are:

 Tremendous pressure of reducing time-to-market cycles in a competitive market.

 Different IT platforms, languages, and systems are getting used for development.
 Usage of IT systems have increased manyfold among regular people.

Page 64 of 88
STQA NOTES SYMCA

On one side, organizations working on “System of Engagement” types of projects are adopting
DevOps. On the other side, organizations working on “System of Records” types of projects
are going for Continuous Delivery. Both these philosophies demand an integrated testing
platform for ensuring quality at every lifecycle stage, right from the beginning of execution to
delivery.

But, for all this to become a reality, organizations need a centralized tool, from where all the
test activities can be managed and tracked efficiently with a clear visibility of the release
progress. There are many Test Management tools on the market, offering various testing
capabilities. Organizations need to be extremely careful about choosing the right one that fits
their business needs.

In this blog, we have discussed some of the key features to look for in a Test Management
tool before making a buying decision.

Flexibility and Ease of Use

In the absence of a proper tool, MS Excel is the most popular offline medium for managing
test artifacts. One of the major reasons for this is the ease of use and flexibility of the tool. So,
while moving from Excel to a standard test tool, it has to be made sure that the tool is very easy
to use and its training and user adaptation time is very less. Every organization, and in some
cases every project follow their own model of testing. The testing tool should be configurable
enough to support these model variances.

Support for End-to-End Traceability

It is very important for the testers to able to trace back all their work in a centralized test
management system. A bi-directional traceability between test artifacts and the associated
requirements and defects increase the efficiency of measuring quality of a project. It also allows
organizations to track the coverage of both Requirements and Test Cases, failing which may
lead to missing information, loss of productivity and fall in quality.

Real-time Reports and Dashboards

Most software projects fail due to the lack of proper visualization of analytical data related to
a project’s progress. In the absence of a centralized tool, the entire process of reporting is
dependent on manual interactions, making it error-prone. So, the tool to be procured should
have the facility of providing real-time reports and dashboards, keeping stakeholders updated
with the latest status of progress and assess quality at every step.

Support for Test Automation

Today, due to fierce market competition, Test Automation is no more a choice but has become
a mandate for organizations. A testing tool must have the support for managing Test
Automation scripts from a single repository. However, that is not enough for a Test automation
project. Test automation needs to be an integral part of the entire execution process. Features

Page 65 of 88
STQA NOTES SYMCA

like the central execution of test automation scripts, automatic capturing of test results and
making them visible from a central platform are necessary. Viewing Test Automation results
needs to be a part of the end-to-end traceability chain.

Integration With Other Phases of Application Lifecycle

Today, in the world of Agility, testing is no longer an isolated phase or a security gate to final
delivery, but an integral part of the entire lifecycle. To achieve this and ensure quality right
from the beginning, testing should get involved at every stage of the lifecycle. Therefore, a
testing tool should have the capability to integrate with tools from other phases of the lifecycle,
so that a centralized status update on the project’s progress and quality can be achieved.
Seamless integration between testing and other lifecycle tools paves the way for an
organization to achieve Continuous Integration (CI) and Continuous Delivery (CD), the
milestones to implement DevOps.

If you find any more criteria to be imperative to procure a test management tool in your
organization, please share your comments. We would appreciate your participation.

Page 66 of 88