Fuzzing

Suman Jana

*Acknowledgements: Dawn Song, Kostya Serebryany,

Peter Collingbourne
 Techniques for bug finding

Automa'c test StaFc analysis Program verificaFon

case genera'on

Fuzzing Dynamic
symbolic execuFon
Lower coverage Higher coverage
Lower false posi0ves Higher false posi0ves
Higher false nega0ves Lower false nega0ves
Blackbox fuzzing

Random
input

Test program

Miller et al. ‘89

 Blackbox fuzzing
• Given a program simply feed random inputs and see
whether it exhibits incorrect behavior (e.g., crashes)
• Advantage: easy, low programmer cost
• Disadvantage: inefficient
– Inputs oUen require structures, random inputs are likely to
be malformed
– Inputs that trigger an incorrect behavior is a a very small
fracFon, probably of geVng lucky is very low
 Fuzzing
• AutomaFcally generate test cases
• Many slightly anomalous test cases are input into a
target
• ApplicaFon is monitored for errors
• Inputs are generally either file based
(.pdf, .png, .wav, etc.) or network based (hWp, SNMP,
etc.)
Monitor

Input generator Test applicaFon

 Problem detecFon
• See if program crashed
– Type of crash can tell a lot (SEGV vs. assert fail)
• Run program under dynamic memory error detector
(valgrind/purify/AddressSaniFzer)
– Catch more bugs, but more expensive per run.
• See if program locks up
• Roll your own dynamic checker e.g. valgrind skins
 Regression vs. Fuzzing
Regrssion Fuzzing
DefiniFon Run program on many Run program on many
normal inputs, look for abnormal inputs, look
badness for badness
Goals Prevent normal users Prevent aWackers from
from encountering encountering
errors (e.g., asserFon exploitable errors (e.g.,
failures are bad) asserFon failures are
oUen ok)
 Enhancement 1:
MutaFon-Based fuzzing
• Take a well-formed input, randomly perturb
(flipping bit, etc.)
• LiWle or no knowledge of the structure of the
inputs is assumed
• Anomalies are added to exisFng valid inputs
– Anomalies may be completely random or follow some
heurisFcs (e.g., remove NULL, shiU character forward)
• Examples: ZZUF, Taof, GPF, ProxyFuzz, FileFuzz,
Filep, etc.

?
Seed input Mutated input Run test program
 Example: fuzzing a PDF viewer
• Google for .pdf (about 1 billion results)
• Crawl pages to build a corpus
• Use fuzzing tool (or script)
– Collect seed PDF files
– Mutate that file
– Feed it to the program
– Record if it crashed (and input that crashed it)
 MutaFon-based fuzzing
• Super easy to setup and automate
• LiWle or no file format knowledge is required
• Limited by iniFal corpus
• May fail for protocols with checksums, those
which depend on challenge
 Enhancement II:
GeneraFon-Based Fuzzing
• Test cases are generated from some descripFon of
the input format: RFC, documentaFon, etc.
– Using specified protocols/file format info
– E.g., SPIKE by Immunity
• Anomalies are added to each possible spot in the
inputs
• Knowledge of protocol should give beWer results
than random fuzzing

RFC ?
Input spec Generated inputs Run test program
 Enhancement II:
GeneraFon-Based Fuzzing

Sample PNG spec

MutaFon-based vs. GeneraFon-based
• MutaFon-based fuzzer
– Pros: Easy to set up and automate, liWle to no
knowledge of input format required
– Cons: Limited by iniFal corpus, may fall for
protocols with checksums and other hard checks
• GeneraFon-based fuzzers
– Pros: Completeness, can deal with complex
dependncies (e.g, checksum)
– Cons: wriFng generators is hard, performance
depends on the quality of the spec
 How much fuzzing is enough?
• MutaFon-based-fuzzers may generate an
infinite number of test cases. When has the
fuzzer run long enough?
• GeneraFon-based fuzzers may generate a
finite number of test cases. What happens
when they’re all run and no bugs are found?
 Code coverage
• Some of the answers to these quesFons lie in
code coverage
• Code coverage is a metric that can be used to
determine how much code has been
executed.
• Data can be obtained using a variety of
profiling tools. e.g. gcov, lcov
 Line coverage
• Line/block coverage: Measures
how many lines of source code if( a > 2 )
have been executed. a = 2;
if( b >2 )
• For the code on the right, how b = 2;
many test cases (values of pair
(a,b)) needed for full(100%) line
coverage?
 Branch coverage
• Branch coverage: Measures
how many branches in code if( a > 2 )
have been taken a = 2;
(condiFonal jmps) if( b >2 )
b = 2;
• For the code on the right,
how many test cases
needed for full branch
coverage?
 Path coverage
• Path coverage: Measures how
many paths have been taken if( a > 2 )
a = 2;
if( b >2 )
• For the code on the right, how b = 2;
many test cases needed for
full path coverage?
 Benefits of Code coverage
• Can answer the following quesFons
– How good is an iniFal file?
– Am I geVng stuck somewhere?
if (packet[0x10] < 7) { //hot path
} else { //cold path }
– How good is fuzzerX vs. fuzzerY
– Am I geVng benefits by running mulFple
fuzzers?
 Problems of code coverage
• For:
mySafeCopy(char *dst, char* src) {
if(dst && src)
strcpy(dst, src); }

• Does full line coverage guarantee finding the

bug?
• Does full branch coverage guarantee finding
the bug?
 Enhancement III:
Coverage-guided gray-box fuzzing
• Special type of mutaFon-based fuzzing
– Run mutated inputs on instrumented program
and measure code coverage
– Search for mutants that result in coverage
increase
– OUen use geneFc algorithms, i.e., try random
mutaFons on test corpus and only add mutants
to the corpus if coverage increases
– Examples: AFL, libfuzzer
 American Fuzzy Lop (AFL)

Execute
Seed against
inputs MutaFon instrumented
target
Input Next input
queue

branch/
edge
Add mutant coverage
to the queue increased?
Periodically culls the
queue without
affecFng total coverage
 AFL
• Instrument the binary at compile-Fme
• Regular mode: instrument assembly
• Recent addiFon: LLVM compiler instrumentaFon mode
• Provide 64K counters represenFng all edges in the app
• Hashtable keeps track of # of execuFon of edges
– 8 bits per edge (# of execuFons: 1, 2, 3, 4-7, 8-15, 16-31,
32-127, 128+)
– Imprecise (edges may collide) but very efficient
• AFL-fuzz is the driver process, the target app runs as
separate process(es)
 Data-flow-guided fuzzing
• Intercept the data flow, analyze the inputs of
comparisons
– Incurs extra overhead
• Modify the test inputs, observe the effect on
comparisons
• Prototype implementaFons in libFuzzer and
go-fuzz
 Fuzzing challenges
• How to seed a fuzzer?
– Seed inputs must cover different branches
– Remove duplicate seeds covering the same
branches
– Small seeds are beWer (Why?)

• Some branches might be very hard to get past as the

# of inputs staFsfying the condiFons are very small
– Manually/automaFcally transform/remove those branches
Hard to fuzz code

void test (int n) {

if (n==0x12345678)
crash();
}

needs 2^32 or 4 billion aWempts

In the worst case
Make it easier to fuzz
void test (int n) {
int dummy = 0;
char *p = (char *)&n;
if (p[3]==0x12) dummy++;
if (p[2]==0x34) dummy++;
if (p[1]==0x56) dummy++;
if (p[0]==0x56) dummy++;
if (dummy==4)
crash();
}

needs around 2^10 aWempts

 Fuzzing rules of thumb
• Input-format knowledge is very helpful
• GeneraFonal tends to beat random, beWer specs make
beWer fuzzers
• Each implementaFon will vary, different fuzzers find
different bugs
– More fuzzing with is beWer
• The longer you run, the more bugs you may find
– But it reaches a plateau and saturates aUer a while
• Best results come from guiding the process
• NoFce where you are geVng stuck, use profiling (gcov,
lcov)!