ATS 12-13 Security Testing 1

Software Testing
&
Security Testing
Bugs a.k.a. …
• Defect • Failure
• Fault • Inconsistency
• Problem • Product
• Error Anomaly
• Incident • Product
• Anomaly Incidence
• Variance • Feature :-)
Defective Software
• We develop programs that contain defects
– How many? What kind?
• Hard to predict the future, however…
it is highly likely, that the software we
(including you!) will develop in the future
will not be significantly better.
Sources of Problems
• Requirements Definition: Erroneous, incomplete,
inconsistent requirements.
• Design: Fundamental design flaws in the software.
• Implementation: Mistakes in chip fabrication, wiring,
programming faults, malicious code.
• Support Systems: Poor programming languages,
faulty compilers and debuggers, misleading
development tools.
Sources of Problems (Cont’d)
• Inadequate Testing of Software:
Incomplete testing, poor verification,
mistakes in debugging.
• Evolution: Sloppy redevelopment or
maintenance, introduction of new flaws in
attempts to fix old flaws.
Adverse Effects of Faulty
Software
• Communications: Loss or corruption of
communication media, non delivery of
data.
• Space Applications: Lost lives, launch
delays.
• Defense and Warfare: Misidentification of
friend or enemy.
Software (Cont’d)
• Transportation: Deaths, delays, sudden
acceleration, inability to brake.
• Safety-critical Applications: Death,
injuries.
• Electric Power: Death, injuries, power
outages, long-term health hazards
(radiation).
Software (Cont’d)
• Money Management: Fraud, violation of
privacy, shutdown of stock exchanges and
banks, negative interest rates.
• Control of Elections: Wrong results
(intentional or non-intentional).
• Control of Jails: Technology-aided escape
attempts and successes, accidental release
of inmates, failures in software controlled
locks.
• Law Enforcement: False arrests and
imprisonments.
Discussion …
• Have you heard of other software bugs?
– In the media?
– From personal experience?
• Does this embarrass you as a future
software engineer?
Specification
“if you can’t say it, you can’t do it”
• You have to know what your product is before

you can say if it has a bug.
• A specification defines the product being
created and includes:
– Functional requirements that describes the
features the product will support. e.g., on a word
processor
• Save, print, check spelling, change font, …
– Non-functional requirements are constraints on
the product. e.g,
• Security, reliability, user friendliness, platform, …
A software bug occurs when at
least one of these rules is true
• The software does not do something that the
specification says it should do.
• The software does something that the specification
says it should not do.
• The software does something that the specification
does not mention.
• The software does not do something that the
product specification does not mention but should.
• The software is difficult to understand, hard to use,
slow …
Most bugs are not because of
mistakes in the code …
• Specification (~= 55%)

• Design (~= 25%)
• Code (~= 15%)
• Other (~= 5%)
Bug Free Software
• Software is in the news for the wrong reason
– Security breach, Mars Lander lost, hackers getting
credit card information, etc.
• Why can’t software engineers develop software
that just works?
– As software gets more features and supports more
platforms it becomes increasingly difficult to make it
create bug-free.
Discussion …
• Do you think bug free software is
unattainable?
– Are their technical barriers that make this
impossible?
– Is it just a question of time before we can do
this?
– Are we missing technology or processes?
Goal of a software tester
• … to find bugs
• … as early in the software development
processes as possible
• … and make sure they get fixed.
• demonstrating that there are no faults in the
software (for the test cases that has been used
during testing)
• It is not possible to prove that there are no
faults in the software using testing
• Testing should help locate errors, not just
detect their presence
– a “yes/no” answer to the question “is the
program correct?” is not very helpful
• Testing should be repeatable

– could be difficult for distributed or concurrent
software
– effect of the environment, uninitialized
variables
• Advice: Be careful not to get caught in the

dangerous spiral of unattainable
perfection.
Defect testing
• Testing programs to
establish the presence of
system defects
Test documents
• Test plan
– Quality objectives, resource needs, schedules,
assignments, methods, etc.
• Test cases
– Inputs and expected outputs.
• Bug reports
– E.g., the Bugzilla web-based bug tracker.
• Test tools and automation
• Metrics, statistics, and summaries
– Number of unresolved bugs, mean time to repair a
bug, etc.
The testing process
• Component testing
– Testing of individual program components
– Usually the responsibility of the component
developer (except sometimes for critical systems)
– Tests are derived from the developer’s
experience
• Integration testing
– Testing of groups of components integrated to
create a system or sub-system
– The responsibility of an independent testing team
– Tests are based on a system specification
Testing phases
Defect testing
• The goal of defect testing is to discover
defects in programs
• A successful defect test is a test which
causes a program to behave in an
anomalous way
• Tests show the presence not the absence
of defects
Test data and test cases
• Test data Inputs which have been devised
to test the system
• Test cases Inputs to test the system and
the predicted outputs from these inputs if
the system operates according to its
specification
– Scenario (Inputs, preconditions, post-
conditions etc.)
– Test data is required to create the test case
• Test Data Generation
The defect testing process
Black-box testing
(functional Testing)
• An approach to testing where the program
is considered as a ‘black-box’
• The program test cases are based on the
system specification
• Test planning can begin early in the
software process
Black-box testing
Structural testing
(White Box Testing)
• Sometime called white-box testing
• Derivation of test cases according to
program structure. Knowledge of the
program is used to identify additional test
cases
• Objective is to exercise all program
statements
(not all path combinations)
White-box testing
Coverage Metrics
• Coverage metrics
– Statement coverage: all statements in the programs should be
executed at least once
– Branch coverage: all branches in the program should be
– Path coverage: all execution paths in the program should be
• The best case would be to execute all paths through the code, but
there are some problems with this:
– the number of paths increases fast with the number of branches in
the program
– the number of executions of a loop may depend on the input
variables and hence may not be possible to determine
– most of the paths can be infeasible
Statement Coverage
• Choose a test set T such

that by executing program bool isEqual(int x, int y)
P for each test case in T, {
each basic statement of P if (x = y)
z := false;
is executed at least once
else
• Executing a statement once z := false;
and observing that it return z;
behaves correctly is not a }
guarantee for correctness,
but it is an heuristic int max(int x, int y)
{
– this goes for all testing if (x > y)
efforts since in general return x;
checking correctness is else
undecidable return x;
}
Statement Coverage
areTheyPositive(int x, int y) • Following test set will give us statement

{ coverage:
if (x >= 0) T1 = {(x=12,y=5), (x= -1,y=35),
print(“x is positive”); (x=115,y=-13),(x=-91,y= -2)}
else
print(“x is negative”); • There are smaller test cases which will
if (y >= 0) give us statement coverage too:
print(“y is positive”); T2 = {(x=12,y= - 5), (x= -1,y=35)}
else
print(“y is negative”); • There is a difference between these two
} test sets though
Statement vs. Branch Coverage
assignAbsolute(int x)
{ Consider this program segment, the test set
if (x < 0) T = {x=-1} will give statement coverage,
x := -x; however not branch coverage
z := x;
}
B0
Control Flow Graph: (x < 0)
true false
B1 Test set {x=-1} does not
x := -x execute this edge, hence, it
does not give branch coverage
B2
z := x
Control Flow Graphs (CFGs)
• Nodes in the control flow graph are basic blocks
– A basic block is a sequence of statements always entered at the
beginning of the block and exited at the end
• Edges in the control flow graph represent the control flow
if (x < y) { B0 (x < y)
x = 5 * y; Y N
x = x + 3;
} x = 5 * y B1 B2 y = 5
else x = x + 3
y = 5;
x = x+y;
x = x+y B3
• Each block has a sequence of statements

• No jump from or to the middle of the block
• Once a block starts executing, it will execute till the end
Branch Coverage
• Construct the control flow graph
• Select a test set T such that by executing program P for each test
case d in T, each edge of P’s control flow graph is traversed at least
once
B0
(x < 0)
true false
B1 Test set {x=-1} does not
x := -x execute this edge, hence, it
does not give branch coverage
Test set {x= -1, x=2}gives

B2 both statement and branch
z := x coverage
Path Coverage
• Select a test set T such that by executing program P for each test
case d in T, all paths leading from the initial to the final node of P’s
control flow graph are traversed
Path Coverage
B0
areTheyPositive(int x, int y)
{ (x >= 0)
if (x >= 0) true false
print(“x is positive”); B1 B2
else print(“x is p”) print(“x is n”)
print(“x is negative”);
if (y >= 0)
print(“y is positive”); B3
else (y >= 0)
print(“y is negative”); true false
} B4 B5
Test set: print(“y is p”) print(“y is n”)
T2 = {(x=12,y= - 5), (x= -1,y=35)}
gives both branch and statement B6
coverage but it does not give path
return
coverage
Set of all execution paths: {(B0,B1,B3,B4,B6), (B0,B1,B3,B5,B6), (B0,B2,B3,B4,B6),
(B0,B2,B3,B5,B6)}
Test set T2 executes only paths: (B0,B1,B3,B5,B6) and (B0,B2,B3,B4,B6)
Path Coverage
B0
areTheyPositive(int x, int y)
{ (x >= 0)
if (x >= 0) true false
print(“x is positive”); B1 B2
else print(“x is p”) print(“x is n”)
print(“x is negative”);
if (y >= 0)
B3
print(“y is positive”);
else (y >= 0)
print(“y is negative”); true false
} B4 B5
print(“y is p”) print(“y is n”)
Test set:
T1 = {(x=12,y=5), (x= -1,y=35),
(x=115,y=-13),(x=-91,y= -2)} B6
gives both branch, statement and path return
coverage

ATS 12-13 Security Testing 1

Uploaded by

Copyright:

Available Formats

ATS 12-13 Security Testing 1

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

ATS 12-13 Security Testing 1

Uploaded by

Copyright:

Available Formats

Software Testing

• You have to know what your product is before

• Specification (~= 55%)

• Testing should be repeatable

• Advice: Be careful not to get caught in the

• Choose a test set T such

areTheyPositive(int x, int y) • Following test set will give us statement

• Each block has a sequence of statements

Test set {x= -1, x=2}gives

You might also like

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.