Integration Testing

By:
Dr. Akshay Jadhav

1
Outline

• Case Study - Mars Climate Orbiter

• Integration Testing
• Integration Test Strategies
• System, Acceptance, and Regression Testing
• Acceptance Testing
• System Testing
• Regression Testing
• Summary

2
 Case Study −
Mars Climate Orbiter

3
Case Study − Mars Climate Orbiter
• NASA’s Mars Climate Orbiter
• Launched on December 11, 1998
• Intended to enter an orbit at 140 –150 km
above Mars.
• On September 23, 1999
• It smashed into the planet's atmosphere
and was destroyed.
• Cost: $328M

4
Case Study − Mars Climate Orbiter
• Cause of failure
• The software controlling the thrusters on the spacecraft used different
units.
• Software modules were developed by teams in US and Europe
• Engineers failed to convert the measure of rocket thrusts
• English unit: Pound-Force
• Metric unit: Newton, kg  m / s2
• Difference: a factor of ≈ 4.45

5
 Integration Testing
• Integration testing (sometimes called integration and testing, abbreviated
I&T) is the phase in software testing in which individual software modules
are combined and tested as a group. It occurs after unit testing and before
validation testing.

6
Objectives
• Understand the purpose of integration testing
• Distinguish typical integration faults from faults that should be eliminated in unit
testing
• Understand the nature of integration faults and how to prevent as well as detect
them
• Understand strategies for ordering construction and testing
• Approaches to incremental assembly and testing to reduce effort and control risk

7
Integration vs. Unit Testing
• Unit (module) testing is a necessary foundation
• Unit level has maximum controllability and visibility
• Integration testing can never compensate for inadequate unit testing
• Integration testing may serve as a process check
• If module faults are revealed in integration testing, they signal inadequate unit
testing
• If integration faults occur in interfaces between correctly implemented modules,
the errors can be traced to module breakdown and interface specifications

8
Integration Testing
• The entire system is viewed as a collection of subsystems (sets of classes)
determined during the system and object design
• Goal: Test all interfaces between subsystems and the interaction of
subsystems
• The Integration testing strategy determines the order in which the
subsystems are selected for testing and integration.

9
Why do we do integration testing?
• Unit tests only test the unit in isolation
• Many failures result from faults in the interaction of subsystems
• Often many Off-the-shelf components are used that cannot be unit tested
• Without integration testing the system test will be very time consuming
• Failures that are not discovered in integration testing will be discovered
after the system is deployed and can be very expensive.

10
Types of Testing
• Unit Testing:
• Individual subsystem
• Carried out by developers (of components)
• Goal: Confirm that subsystems is correctly coded and carries out the intended
functionality
• Integration Testing:
• Groups of subsystems (collection of classes) and eventually the entire system
• Carried out by developers
• Goal: Test the interface and the interplay among the subsystems

11
Types of Testing
• System Testing:
• The entire system
• Carried out by developers (testers!)
• Goal: Determine if the system meets the requirements (functional and global)
• Functional Testing: Test of functional requirements
• Performance Testing: Test of non-functional requirements
• Acceptance and Installation Testing:
• Evaluates the system delivered by developers
• Carried out by the client.
• Goal: Demonstrate that the system meets customer requirements and is ready to
use

12
What is Integration Testing?
Unit/module test Integration test System test

Specification: Module interface Interface specs, module Requirements

breakdown specification

Visible structure: Coding details Modular structure (software — none —

architecture)

Scaffolding required: Some Often extensive Some

Looking for faults in: Modules Interactions, compatibility System functionality

13
Testing Level Assumptions and Objectives
Unit Assumptions Unit Goals

• All other units are correct • Correct unit function

• Compiles correctly • Coverage metrics satisfied

Integration Assumptions Integration Goals

• Interfaces correct
Unit testing complete • Correct function across units
• Fault isolation support

System Assumptions System Goals

• Integration testing complete • Correct system functions

• Tests occur at port boundary • Non-functional requirements tested
• Customer satisfaction

14
What is Software Integration Testing?
• Testing activities that integrate software components together to form a complete system. To
perform a cost-effective software integration, integration test strategy, integration test set are
needed.
• Integration testing focuses on:
• Interfaces between modules (or components)
• Integrated functional features
• Interacting protocols and messages
• System architectures
• Who performs software integration:
• Developers and test engineers
• What do you need?:
• Integration strategy
• Integration test environment and test suite
• Module (or component) specifications
• Interface and design documents

15
Integration Faults
• Inconsistent interpretation of parameters or values
• Example: Mixed units (Pound/Newton) in Martian Lander
• Violations of value domains, capacity, or size limits
• Example: Buffer overflow
• Side effects on parameters or resources
• Example: Conflict on (unspecified) temporary file

16
Integration Faults
• Omitted or misunderstood functionality
• Example: Inconsistent interpretation of web requests
• Nonfunctional properties
• Example: Unanticipated performance issues
• Dynamic mismatches
• Example: Incompatible polymorphic method calls

17
Example: A Memory Leak
Apache web server, version 2.0.48
Response to normal page request on secure (https) port

static void ssl_io_filter_disable(ap filter t *f) {

bio_filter_in_ctx_t *inctx = f->ctx;
No obvious error, but
inctx->ssl = NULL; Apache leaked memory
slowly (in normal use) or
inctx->filter_ctx->pssl = NULL; quickly (if exploited for a
} DOS attack)

18
Example: A Memory Leak
Apache web server, version 2.0.48
Response to normal page request on secure (https) port

static void ssl_io_filter_disable(ap filter t *f) {

bio_filter_in_ctx_t *inctx = f->ctx;
The missing code is for a
SSL_free(inctx -> ssl); structure defined and
inctx->ssl = NULL; created elsewhere,
inctx->filter_ctx->pssl = NULL; accessed through an
opaque pointer.
}

19
Example: A Memory Leak
Apache web server, version 2.0.48
Response to normal page request on secure (https) port

static void ssl_io_filter_disable(ap filter t *f) {

bio_filter_in_ctx_t *inctx = f->ctx;
SSL_free(inctx -> ssl); Almost impossible to find
with unit testing.
inctx->ssl = NULL;
(Inspection and some
inctx->filter_ctx->pssl = NULL; dynamic techniques could
} have found it.)

20
What is a software integration strategy?
• Software test strategy provides the basic strategy and guidelines to test
engineers to perform software testing activities in a rational way.
• Software integration strategy usually refers to
• an integration sequence (or order) to integrate different parts (or components)
together

21
Integration Test Strategies

22
Maybe You’ve Heard ...
• Yes, I implemented module A, but I didn’t test it thoroughly yet.
• It will be tested along with module B when that’s ready.

23
Translation ...

• Yes, I implemented module • I didn’t think at all about

A, but I didn’t test it the strategy for testing.
thoroughly yet. • I didn’t design module A
• It will be tested along with for testability and I didn’t
module B when that’s ready. think about the best order
to build and test modules
A and B

24
Integration Plan & Test Plan

• Integration test plan

... drives and is driven by
... the project “build plan”
• A key feature of the
Build Plan Test Plan system architecture
and project plan
...
System Architecture

25
Drivers and Stubs
• Driver: A program that calls the interface procedures of the module being
tested and reports the results
• A driver simulates a module that calls the module currently being tested
• Stub: A program that has the same interface as a module that is being used
by the module being tested, but is simpler.
• A stub simulates a module called by the module currently being tested

26
Drivers and Stubs

Module
Driver Stub
procedure Under Test procedure
call call

access to global
variables

▪ Driver and Stub should have the same interface as the modules they replace

▪ Driver and Stub should be simpler than the modules they replace

27
Stubs and drivers

• Driver: Driver
• A component, that calls the TestedUnit
• Controls the test cases
Tested
Unit
• Stub:
• A component, the TestedUnit
depends on Stub
• Partial implementation
• Returns fake values.

28
Example: A 3-Layer-Design (Spreadsheet)

A
Spread
A
SheetView Layer I

B C D
Entity
Data Currency
B Calculator
C D Layer II
Model Converter

E F G
BinaryFile XMLFile Currency Layer III
E F G
Storage Storage DataBase

29
Approaches to Integration Testing
(“source” of test cases)

• Functional Decomposition (most commonly described in the literature)

– Top-down
– Bottom-up
– Sandwich
– “Big bang”
• Call graph
– Pairwise integration
– Neighborhood integration

30
Basis of Integration Testing Strategies

• Functional Decomposition applies best to procedural code

• Call Graph
• applies to both procedural and object-oriented code

31
Example—Calendar Program
• Date in the form mm, dd, yyyy
• Calendar functions
– the date of the next day (NextDate)
– the day of the week corresponding to the date
– the zodiac sign of the date
– the most recent year in which Memorial Day was celebrated on May 27
– the most recent Friday the Thirteenth

32
Calendar Program Units
Main Calendar
Function isLeap
Procedure weekDay
Procedure getDate
Function isValidDate
Function lastDayOfMonth
Procedure getDigits
Procedure memorialDay
Function isMonday
Procedure friday13th
Function isFriday
Procedure nextDate
Procedure dayNumToDate
Procedure zodiac
33
Functional Decomposition of Calendar

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

isValidDate getDigits DaynumToDate isFriday isMonday

lastDayOfMonth

dateToDaynum

34
First Step in Top-Down Integration

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

“Grey” units are stubs that return the correct values when referenced.
This level checks the main program logic.

35
weekDayStub

Procedure weekDayStub(mm, dd, yyyy, dayName) If

((mm = 10) AND (dd = 28) AND (yyyy = 2013))
Then dayName = “Monday”
EndIf
.
.
.
If ((mm = 10) AND (dd = 30) AND (yyyy = 2013))
Then dayName = “Wednesday”
EndIf

36
Next Three Steps
(replace one stub at a time with the actual code.)

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

37
Top-Down Integration Mechanism
• Breadth-first traversal of the functional decomposition tree.
• First step: Check main program logic, with all called units replaced by stubs that always
return correct values.
• Move down one level
– replace one stub at a time with actual code.
– any fault must be in the newly integrated unit

38
Bottom-Up Integration Mechanism
• Reverse of top-down integration
• Start at leaves of the functional decomposition tree.
• Driver units...
– call next level unit
– serve as a small test bed
– “drive” the unit with inputs
– drivers know expected outputs
• As with top-down integration, one driver unit at a time is replaced with actual code.
• Any fault is (most likely) in the newly integrated code.

39
Top-Down and Bottom-Up Integration
• Both depend on throwaway code.
• drivers are usually more complex than stubs
• Both test just the interface between two units at a time.
• In Bottom-Up integration, a driver might simply reuse unit level tests
for the “lower” unit.
• Fan-in and fan-out in the decomposition tree results in some
redundancy.

40
Starting Point of Bottom-Up Integration

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

isValidDate getDigits DaynumToDate isFriday isMonday

lastDayOfMonth

dateToDaynum

41
Bottom-Up Integration of Zodiac

Calendar
(driver)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

42
Sandwich Integration
• Avoids some of the repetition on both top-down and bottom-up
integration.
• Nicely understood as a depth-first traversal of the functional
decomposition tree.
• A “sandwich” is one path from the root to a leaf of the functional
decomposition tree.
• Avoids stub and driver development.
• More complex fault isolation.

43
A Sample Sandwich

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

isValidDate getDigits DaynumToDate isFriday isMonday

lastDayOfMonth

dateToDaynum

44
“Big Bang” Integration

Calendar
(Main)

isLeap weekDay getDate zodiac nextDate Friday13th MemorialDay

isValidDate getDigits DaynumToDate isFriday isMonday

lastDayOfMonth

dateToDaynum

45
“Big Bang” Integration
• No...
– stubs
– drivers
– strategy
• And very difficult fault isolation
• This is the practice in an agile environment with a daily run of the project to that point.

46
Top-down Approach Vs Bottom-up Approach
Basis for Comparison Top-down Approach Bottom-up Approach
Basic Breaks the massive problem into smaller sub- Solves the fundamental low-level problem
problems and integrates them into a larger one
Process Submodules are solitarily analyzed Examine what data is to be encapsulated, and
implies the concept of information hiding
Redundancy Contain redundant information Redundancy can be eliminated
Programming Structure/procedural oriented programming Object-oriented programming languages
languages languages
Mainly used in Module documentation, test case creation, Testing
code implementation and debugging

47
Pros and Cons of Decomposition-Based Integration

• Pros
– intuitively clear
– “build” with proven components
– fault isolation varies with the number of units being integrated
• Cons
– based on lexicographic inclusion (a purely structural consideration)
– some branches in a functional decomposition may not correspond with actual interfaces.
– stub and driver development can be extensive

48
Call Graph-Based Integration
• Definition: The Call Graph of a program is a directed graph in which
– nodes are unit
– edges correspond to actual program calls (or messages)
• Call Graph Integration avoids the possibility of impossible edges in decomposition-
based integration.
• Can still use the notions of stubs and drivers.
• Can still traverse the Call Graph in a top-down or bottom-up strategy.

49
Call Graph of the Calendar Program

Calendar
(Main)

getDate zodiac nextDate memorialDay weekDay Friday13th

getDigits dateToDaynum dayNumToDate isMonday isFriday

isValidDate

lastDayOfMonth isLeap

50
Call Graph-Based Integration (continued)
• Two strategies
– Pair-wise integration
– Neighborhood integration
• Degrees of nodes in the Call Graph indicate integration sessions
– isLeap and weekDay are each used by three units
• Possible strategies
– test high in degree nodes first, or at least,
– pay special attention to “popular” nodes

51
Pair-Wise Integration
• By definition, an edge in the Call Graph refers to an interface between the
units that are the endpoints of the edge.
• Every edge represents a pair of units to test.
• Still might need stubs and drivers
• Fault isolation is localized to the pair being integrated

52
Three Pairs for Pair-Wise Integration

53
Neighborhood Integration
• The neighborhood (or radius 1) of a node in a graph is the set of nodes
that are one edge away from the given node.
• This can be extended to larger sets by choosing larger values for the
radius.
• Stub and driver effort is reduced.

54
Two Neighborhoods (radius = 1)

55
Calendar Call Graph with Numbered Nodes

56
 Neighborhoods in the Calendar Program Call Graph

Node Unit name Predecessors Successors

1 Calendar (Main) (none) 2, 3, 4, 5, 6, 7
2 getDate 1 8, 9
3 zodiac 1 9
4 nextDate 1 10
5 memorialDay 1 11
6 weekday 1, 11, 12 (none)
7 Friday13th 1 12
8 getDigits 2 13
9 dateToDayNum 3 15
10 dayNumToDate 4 15
11 isMonday 5 6
12 isFriday 7 6
13 isValidDate 8 14
14 lastDayOfMonth 13 15
15 isLeap 9, 10, 14 (none)
57
Comparison of Integration Testing Strategies

Strategy Basis Ability to test interfaces Ability to test co- Fault isolation and
functionality resolution

Functional acceptable, but can be limited to pairs of good to a faulty unit

Decomposition deceptive (phantom units
edges)

Call Graph acceptable limited to pairs of good to a faulty unit

units

58
Continuous Testing
• Continuous Integration is a software development practice where
members of a team integrate their work frequently, usually each person
integrates at least daily - leading to multiple integrations per day.

• Each integration is verified by an automated build (including test) to detect

integration errors as quickly as possible.

59
Continuous Testing
• Continuous build:
• Build from day one
• Test from day one
• Integrate from day one
• System is always runnable
• Requires integrated tool support:
• Continuous build server
• Automated tests with high coverage
• Tool supported refactoring
• Software configuration management
• Issue tracking

60
Continuous Testing Strategy
A
Spread
SheetView Layer I

B C D
Data Currency
Calculator Layer II
Model Converter

E F G
BinaryFile XMLFile Currency Layer III
Storage Storage DataBase

+ Cells
Sheet View + File Storage
+ Addition
61
Which Integration Strategy should you use?

• Factors to consider
• Location of critical parts in the system
• Availability of hardware
• Availability of components
• Scheduling concerns

62
Which Integration Strategy should you use?

• Bottom up approach
• Good for object oriented design methodologies
• Test driver interfaces must match component interfaces
• Top-level components are usually important and cannot be neglected up to the end
of testing
• Detection of design errors postponed until end of testing
• Top down approach
• Test cases can be defined in terms of functions examined
• Need to maintain correctness of test stubs
• Writing stubs can be difficult

63
Steps in Integration Testing

1. Based on the integration strategy, 4. Test subsystem decomposition:

select a component to be tested. Define test cases that exercise all
Unit test all the classes in the dependencies
component. 5. Test non-functional requirements:
Execute performance tests
2. Put selected component together;
do any preliminary fix-up necessary 6. Keep records of the test cases and
to make the integration test testing activities.
operational (drivers, stubs) 7. Repeat steps 1 to 7 until the full
system is tested.
3. Test functional requirements: Define
test cases that exercise all use cases
with the selected component The primary goal of integration testing
is to identify failures with the
(current) component configuration.

64
Summary

• Integration testing focuses on interactions

• Must be built on foundation of thorough unit testing
• Integration faults often traceable to incomplete or misunderstood
interface specifications
• Prefer prevention to detection, and make detection easier by imposing
design constraints
• Strategies tied to project build order
• Order construction, integration, and testing to reduce cost or risk

65