QuickFuzz: An Automatic Random

Fuzzer for Common File Formats

Gustavo Grieco Martı́n Ceresa Pablo Buiras
CIFASIS-CONICET, Argentina Harvard University, United States
{ gg, ceresa }@cifasis-conicet.gov.ar pbuiras@seas.harvard.edu

Abstract tool for finding bugs in software with complex inputs, and con-
Fuzzing is a technique that involves testing programs using invalid sists in random testing of programs using potentially invalid or er-
or erroneous inputs. Most fuzzers require a set of valid inputs as a roneous inputs. There are two ways of producing invalid inputs:
starting point, in which mutations are then introduced. QuickFuzz mutational fuzzing involves taking valid inputs and altering them
is a fuzzer that leverages QuickCheck-style random test-case gen- through randomization, producing erroneous or invalid inputs that
eration to automatically test programs that manipulate common file are fed into the program; generational fuzzing (sometimes also
formats by fuzzing. We rely on existing Haskell implementations of known as grammar-based fuzzing) involves generating invalid in-
file-format-handling libraries found on Hackage, the community- puts from a specification or model of a file format. A program that
driven Haskell code repository. We have tried QuickFuzz in the performs fuzzing to test a target program is known as a fuzzer.
wild and found that the approach is effective in discovering vul- While fuzzers are powerful tools with impressive bug-finding
nerabilities in real-world implementations of browsers, image pro- ability [8, 13, 7], they are not without disadvantages. Mutational
cessing utilities and file compressors among others. In addition, we fuzzers usually rely on an external set of input files which they use
introduce a mechanism to automatically derive random generators as a starting point. The fuzzer then takes each file and introduces
for the types representing these formats. QuickFuzz handles most small bit-level mutations in them before using them as test cases
well-known image and media formats, and can be used to test pro- for the program in question. The user has to collect/generate and
grams and libraries written in any language. maintain this set of input files manually for each file format they
might want to test. By contrast, generational fuzzers avoid this
Categories and Subject Descriptors D.1.1 [Programming Tech- problem, but the user must then develop and maintain models of
niques]: Applicative (Functional) Programming; D.2.5 [Software the file format types they want to generate. As expected, creating
Engineering]: Testing and Debugging—Testing tools such models requires a deep domain knowledge of the desired file
format and can be very expensive to formulate.
Keywords Fuzzing, Haskell, QuickCheck, Hackage In this paper, we introduce QuickFuzz, a tool that leverages
Haskell’s QuickCheck [4] (the well-known property-based ran-
1. Introduction dom testing library) and Hackage (the community Haskell soft-
ware repository) in conjunction with off-the-shelf bit-level muta-
Modern software is able to manipulate complex file formats that tional fuzzers to provide automatic fuzzing for several common file
encode richly-structured data such as images, audio, video, HTML formats, without the need of an external set of input files and with-
documents, PDF documents or archive files. These entities are out having to develop models for the file types involved. QuickFuzz
usually represented either as binary files or as text files with a generates invalid inputs using a mix of generational and mutational
specific structure that must be correctly interpreted by programs fuzzing to try to discover unexpected behavior in a target applica-
and libraries that work with such data. Dealing with the low-level tion.
nature of such formats involves complex, error-prone artifacts such Hackage already contains Haskell libraries that handle well-
as parsers and decoders that must check invariants and handle a known image, document, archive and media formats. These li-
significant number of corner cases. At the same time, bugs and braries have two important features: (a) they provide a data type
vulnerabilities in programs that handle complex file formats often T that serves as a lightweight specification and can be used to
have serious consequences that pave the way for security exploits represent individual files of these formats, and (b) they provide
[3]. a function to serialize elements of T to write into files. In gen-
How can we test this software? As a complement to the usual eral we call this function encode and model it as having type
testing process, and considering that the space of possible inputs T → ByteString. Using ready-made Hackage libraries as mod-
is quite large, we might want to test how these programs handle els saves the programmers from having to write these by hand.
unexpected input. Fuzzing [9, 6, 17] has emerged as a promising The key insight behind QuickFuzz is that we can make random
Permission to make digital or hard copies of all or part of this work for personal or values of T using QuickCheck’s generators, then serialize them us-
classroom use is granted without fee provided that copies are not made or distributed ing encode and pass them to an off-the-shelf fuzzer to randomize.
for profit or commercial advantage and that copies bear this notice and the full citation Such mutation is likely to produce a corrupted version of the file.
on the first page. Copyrights for components of this work owned by others than ACM
must be honored.
Permission Abstracting
to make digital or hard copieswith
of partcredit
or all ofis permitted.
this To copy
work for personal otherwise,
or classroom or republish,
use is granted without fee Then, the target application is executed with the corrupted file as
to post that
provided
the full
on copies
servers
citation on the
or made
are not
first
to redistribute
page. Copyrights for
to profit
or distributed for lists,orrequires
components
commercialprior
of this work owned
specific
advantage
by
and thatpermission
others
copies bear thisand/or
than ACM must be honored.
a
notice and input.
fee. Request permissions from Permissions@acm.org.
Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, contact The missing piece of the puzzle is a mechanism to automati-
Copyright is heldRequest
the Owner/Author(s). by the owner/author(s).
permissions Publicationor rights
from permissions@acm.org licensed
Publications to ACM.
Dept., ACM, Inc., fax +1 (212)
869-0481. cally derive the QuickCheck generators from the definitions of the
Haskell’16, September 22-23, 2016, Nara, Japan data types in the libraries, which we do in Template Haskell and
ACM.
Copyright978-1-4503-4434-0/16/09...$15.00
© held by owner/author(s). Publication rights licensed to ACM.
also provide in the form of a library, which we call MegaDeTH.
ACM [to be supplied]. . . $15.00
http://dx.doi.org/10.1145/2976002.2976017

13
QuickCheck generates random tests cases by means of the Gen black box for now, and defer an in-depth look at MegaDeTH to
monad. A generator of type Gen T for type T is given as an in- Section 3.
stance of the Arbitrary type class. MegaDeTH inspects the struc- To launch a fuzzing campaign on giffix, we simply execute:
ture of a data type and automatically constructs an appropriate in-
stance of Arbitrary for it. Unlike existing tools for deriving such $ Q u i c k F u z z G i f ' g i f f i x @' −a r a d a m s a −s 10
instances, MegaDeTH can correctly handle mutually recursive data With these command-line parameters, our tool generates GIF
types as well as types with nested nonprimitive types. files using Radamsa to perform bit-level mutations.
Finally, if an abnormal termination is detected (for instance, After a few seconds, QuickFuzz stops since it found an execu-
a segmentation fault), the tool will try to minimize the size of tion that fails with a segmentation fault. At this point we can ex-
the corrupted file in order to output the smallest test case. Such amine the output directory (outdir by default) to see the GIF file
information can be very useful for the developers of the faulty produced by our tool that caused giffix to fail.
program to fix the issue. Figure 1 shows the QuickFuzz pipeline and architecture. An ex-
Thanks to Haskell implementations of file-format-handling li- ecution of QuickFuzz consists of three phases: high-level fuzzing,
braries found on Hackage, QuickFuzz currently generates and mu- low-level fuzzing and execution. The diagram also shows the inter-
tates a large set of different file types out of the box. However, it action with MegaDeTH, which just provides Arbitrary instances
is also possible for the user to add file types by providing a data for the high-level fuzzing phase and can be run offline. Let us take
type T and encode function. Our framework can derive Arbitrary a look at what happens in each phase in the giffix example.
instances fully automatically, to be used by QuickFuzz to discover
bugs in new applications. 2.1 High-Level Fuzzing
Although QuickFuzz is written in Haskell, we remark that
it treats its target program as a black box, giving it randomly- During this phase, QuickFuzz generates values of the data type T
generated, invalid files as arguments. Therefore, QuickFuzz can be that represents the file format of the input to the target program. It
used to test programs written in any language. relies on the Gen monad defined in QuickCheck, which provides
Our contributions can be summarized as follows: convenient access to a random-number generator that can be used
to construct randomized structured data compositionally. In our
• We present QuickFuzz, a tool for automatically generating in- example this representation type T (borrowed from JuicyPixels) is
called GifFile. A GifFile contains a header (of type GifHeader )
puts and fuzzing programs parsing several common types of
and the raw bitmap data specified as a list, among other things.
files. QuickFuzz uses QuickCheck behind the scenes to gen-
Note that randomly generated elements of type GifFile might
erate test cases, and is integrated with fuzzers like Radamsa,
not be valid GIF files, since the type system is unable to encode
Honggfuzz and other bug-finding tools such as Valgrind and Ad-
all invariants that should hold among the parts of the value. For
dress Sanitizer.
example, the header might specify a width and height that doesn’t
• We released QuickFuzz as open-source and free of charge. match the bitmap data. For this reason, we consider that this step
As far as we know, QuickFuzz is the first fuzzer to offer the corresponds to generational fuzzing, where the data type definition
generation and mutation of more than a dozen complex file serves as a lightweight approximate model of the GIF file format
types without requiring the user to develop the models: just which generates potentially invalid instances of the file format.
install, select a target program and wait for crashes. The tool After running QuickFuzz, the output directory contains, for each
is available at http://quickfuzz.org/. test case, a text file that shows the value generated by this step. For
• We introduce MegaDeTH, a library to derive Arbitrary in- instance, for the running example we get
stances for Haskell data types. MegaDeTH is fully automatic GifFile {
and capable of handling mutually recursive types and deriving gifHeader = GifHeader {
instances from external modules. This library can be used to gifVersion = GIF87a,
extend QuickFuzz with new data types. gifScreenDescriptor =
• We evaluate the practical feasibility of QuickFuzz and show LogicalScreenDescriptor {
a list of security-related bugs discovered using QuickFuzz in screenWidth = 0,
complex real-world applications like browsers, image-processing screenHeight = 0,
utilities and file archivers among others. backgroundIndex = 1,
hasGlobalMap = False,
The rest of the paper is organized as follows. Section 2 provides colorResolution = 0,
an overview of how QuickFuzz works using an example. Section 3 isColorTableSorted = False,
discusses how to generate Arbitrary instances and the implemen- colorTableSize = 1
tation of MegaDeTH. In Section 4 we highlight some of the key },
principles in the design and implementation of our tool using the
...
QuickCheck framework. Later, in Section 5, we perform an evalu-
ation of its applicability. Section 6 presents related work and Sec- After generating a value of type GifFile with QuickCheck’s
tion 7 concludes. Gen monad, we use the Hackage library’s encode function for
this file type (in this case GifFile has an instance of Binary) to
serialize the GifFile into a ByteString, which is also written as-is
2. A Quick Tour of QuickFuzz to the output directory for further inspection by the user.
In this section, we show QuickFuzz in action with a simple exam-
ple. More specifically, we will see how to discover bugs in giffix, a 2.2 Low-Level Fuzzing
small command line utility from giflib that attempts to fix broken Usually the use of high-level fuzzing produced by the values gener-
GIF images. Our tool has built-in support for the generation of GIF ated by QuickCheck is not enough to trigger some interesting bugs.
files using the JuicyPixels library [18] and the Arbitrary instances Therefore, this phase relies on an off-the-shelf mutation fuzzer to
automatically derived by MegaDeTH . We treat MegaDeTH as a introduce errors/mutations at the bit level on the ByteString pro-

14
 Figure 1: Summary of high-level and low-level fuzzing of QuickFuzz where grey nodes represent inputs provided by a user.

duced by the previous step. In particular, the current version sup- Test Case Minimization Immediately after a failed execution is
ports the following fuzzers: found, QuickFuzz uses QuickCheck’s shrinking feature to start
the test case minimization. This procedure is very important for
• Zzuf: a transparent application input fuzzer by Caca Labs [2]. the developers looking to fix the issue, since the minimized test
• Radamsa: a general purpose fuzzer developed by the Oulu Uni- case should only trigger the code that is required to reproduce
versity Secure Programming Group [13]. the unexpected behavior. It is also called input simplification [20].
QuickFuzz uses a simple strategy to reduce the number of bytes in
• Honggfuzz: a general purpose fuzzer developed by Google [7]. the test cases causing the failed execution, defining the following
function:
One of the key principles of the design of QuickFuzz was to
require no parameter tuning in the use of 3rd party fuzzers and shrink :: ByteString → [ByteString ]
bug-detection tools. Usually, the use of mutational fuzzers requires shrink bs = (tail $ tails bs) +
+ (init $ inits bs)
fine tuning of some critical parameters. Instead, we decided to The shrinking procedure will check every reduced input re-
incorporate sane default values to perform an effective fuzzing turned by our shrink function in order to detect which are still
campaign even without fine-tuning values like mutation rates. causing an execution fail. Since the use of headers, blocks and
Additionally, QuickFuzz can use Valgrind [12] and Address trailer sections in files is common practice, we decided to simplify
Sanitizer [15] to detect more subtle bugs like a read out-of-bounds the shrink function to prune either the beggining or the end of a
that would not cause a segmentation fault or the use of uninitialized file.
memory. Given a shrink function that returns a list of smaller files to
check, the minimization procedure is performed automatically by
2.3 Execution QuickCheck. It is important to note that in our implementation
The final phase involves running the target program with the mu- there is no guarantee that a failed execution in the minimization
tated file as input. As we have seen, for each test case file producing process will trigger the same bug as the original one. Nevertheless,
runtime failure, we can also find in the output directory the inter- in the worst case the test case minimization process discovers a
mediate values for each step of the process: different issue.

• A text file with the printed value generated by QuickCheck. 3. Automatically Deriving Type Class Instances
• The original encoded value, before the mutation by the muta- As explained in Section 2, our first step is the generation of complex
tional fuzzer. data. We obtain this by selecting a type that represents the data we
• The actual mutated file which was passed as input to the target want to generate, and giving a good Arbitrary instance for that
program and resulted in failure. type. Then it is up to QuickCheck to generate test cases and test
properties about them.
Developers can examine how the file was corrupted in order to In order to define an instance of Arbitrary for a particular
understand why their program failed and how it can be fixed. type, we have to provide a definition of a monadic computation
QuickFuzz found a test case to reproduce a heap-based overflow arbitrary :: Gen a, known as a generator, that generates arbitrary
in giffix (CVE-2015-7555). This issue is caused by the lack of val- elements of type a.
idation of the size of the logical screen and the size of the actual gif In this section we present MegaDeTH, a tool to automatically
frames. In fact, if we run the tool during no more than 5 minutes in derive suitable Arbitrary instances for a given type.
a single core, we will obtain dozens of test cases triggering failed
executions (crashes and aborts). Crash de-duplication is currently 3.1 MegaDeTH
outside the scope of our tool, so we manually checked the back- MegaDeTH is a tool implemented in Template Haskell that gives
traces using a debugger and determined that giffix was failing in 3 the user the ability to provide instances to a type and all of its nested
distinctive ways. types. Given a type A MegaDeTH generates a list of all types that
The root cause of such crashes can be the same, for instance are needed in order to instantiate A and instantiates each one until
if the program is performing a read out-of-bounds. Nevertheless, A can be instantiated.
QuickFuzz can still obtain valuable information finding different Let us consider the situation when a Haskell programmer wants
crashes associated with the same issue: they can be very useful to to use a library and needs to print out on the screen certain results
determine if the original issue is exploitable or not. (of a library-specific data type) produced by it. If the library does

15
not provide a Show instance for it, writing such an instance can Now we can use MegaDeTH to derive the Show instance for
be tortuous task. The cause of this is that in general the top-level Bin. In order to use Template Haskell we need to activate the
type refers to a number of other nonprimitive types, which we shall extension TemplateHaskell , which provides us with a reification
refer to as nested types. These types, in turn, might refer to further function to get the corresponding element of type Name for a given
nested types that the user does not need to know about, and usually type (′′ ).
those nested types are not even exported. For example, consider the
{-# Language TemplateHaskell #-}
following data type definitions for a binary tree and a type for the
contents of a node: devShow ′′ Bin
Which will generate the following code and insert it in compi-
data Bin a = L a | B (Bin a) Node (Bin a)
lation time:
data Node = Node {name :: String, l :: Int, d :: Int}
instance Show Node where
In order to define a Show instance for type Bin, our top-level showsPrec (Node x1 x2 x3 )
type in this toy example, we need to provide a Show instance for = ((showString "Node {name = ")
type Node, a nested type for Bin. ◦ ((showsPrec 0 x1 )
Mega Derivation TH (MegaDeTH) offers a solution to this ◦ ((showString ", l = ")
problem: it gives the user a way to thoroughly derive instances for ◦ ((showsPrec 0 x2 )
all the intermediate nested types that are needed to make the top- ◦ ((showString ", d = ") ◦ ((showsPrec 0 x3 )
level instance work.
◦ (showChar ’}’)))))))
MegaDeTH was implemented using Template Haskell [16], a
instance Show a → Show (Bin a) where
metaprogramming mechanism built into GHC that is extremely
useful to process the AST of Haskell programs and insert new showsPrec p (L x1 )
declarations at compilation time. We use the power of Template = ((showParen (p > 10)) $((showString "L ")
Haskell to extract all the nested types for a given type and derive ◦ (showsPrec 11 x1 )))
a class instance for each of them, finally instantiating the top- showsPrec p (B x1 x2 x3 )
level type. Since Haskell gives the user the possibility of writing = ((showParen (p > 10))
mutually recursive types, MegaDeTH implements a topological $((showString "B ")
sort to find a suitable order in which to instantiate each type. ◦ ((showsPrec 11 x1 )
In Template Haskell, type names are reified into type Name, ◦ ((showChar ’ ’)
and declarations into type Dec. Considering that type class in- ◦ ((showsPrec 11 x2 ) ◦ ((showChar ’ ’)
stances are a kind of declaration, we use a Template Haskell func- ◦ (showsPrec 11 x3 )))))))
tion of type Name → Q [Dec ] to model a compile-time meta-
Given that the implementation of MegaDeTH was driven by
function that derives an instance of a class for a given type, which
the needs of QuickFuzz, we design a new instantiation method for
we refer to as a derivation function. The function returns in the Q
the Arbitrary class. While Derive provides a derivation method
monad, which gives us access to Template Haskell’s internal state
for Arbitrary instances, in practice it proved to be not a good
and allows us to inspect the structure of types.
choice in the presence of mutually recursive types. Given a type
The main function exported by MegaDeTH is megaderive:
A with constructor declarations C1 a11 a21 and C2 a12 a22 ,
megaderive :: (Name → Q [Dec ]) Derive’s instantiation for Arbitrary A will select with the same
→ (Name → Q Bool ) probability one of the constructors, C1 or C2 , and compute each
→ Name → Q [Dec ] of the arguments using arbitrary again, as the following example
code:
We can interpret this type as a way of lifting an existing deriva- instance Arbitrary A where
tion function (of type Name → Q [Dec ]) into a new deriva- arbitrary = do
tion function that works on all nested types of the argument type. x ← choose (0 :: Int, 2)
The first argument of megaderive is a function that takes care of case x of
providing an instance of the wanted class for a single type. As 0 → C1 h$i arbitrary h⋆i arbitrary
mentioned before, megaderive’s traversal is deep: it explores all
1 → C2 h$i arbitrary h⋆i arbitrary
the dependencies and tries to derive instances for all the nested
types. Hence in order to exclude some types we can indicate those The main problem with this derivation method is that in the
types by passing a filter function. Given that megaderive does presence of mutually recursive types, say T and R, it is possi-
not know the name of the class, we can use the filter function ble to always select the constructors of T that contain an R and
to stop MegaDeTH from instantiating types that were already de- vice versa, leading to non-termination. To avoid this problem, the
fined. In order to give a clear user interface we provide a function Gen monad is equipped with a size parameter and a function
isinsName (of type Name → Name → Q Bool ) for that very sized :: (Int → Gen a) → Gen a that is normally used to
purpose. This filter is extremely useful, it gives the user the ability write generators that produce values of finite depth. So MegaDeTH
to provide instances to a particular nested type without having to implements all its arbitrary generators using sized to have more
define all the other ones. However we keep the filter function as control over which constructor to choose. These generators sim-
general as we can to give more control to the user. ply decrease the size parameter each time they are called, and upon
For example, we can use the makeShow method given by the reaching the value 0 they limit the constructor selection to only
package Derive [11] to easily instantiate all the required nested nonrecursive constructors.
types with MegaDeTH as follows: However if Derive’s derivation function for Arbitrary instances
were to be improved, we would gladly integrate it into our library
devShow :: Name → Q [Dec ] as we did with Show . Again, thanks to the flexibility of Tem-
devShow = megaderive (derive makeShow ) plate Haskell we define a function that instantiates just one type,
(isinsName ′′ Show) given its name, called deriveArbitrary and in composition with

16
megaderive we can generate a mega tactic devArbitrary to in- A program execution fails if we detect an abnormal termina-
stantiate all the intermediate types: tion. In the POSIX.1-1990 standard, a program can be abnormally
terminated after receiving the following signals:
deriveArbitrary :: Name → Q [Dec ]
devArbitrary :: Name → Q [Dec ] • A SIGILL when it tries to execute an illegal instruction.
devArbitrary = megaderive deriveArbitrary • A SIGABRT when it called abort.
(isinsName ′′ Arbitrary)
• A SIGFPE when it raised a floating point exception.
Example. For the sake of the argument, we are going to simplify • A SIGSEGV when it accessed an invalid memory reference.
some types. Take for example the following type GifFile found in
JuicyPixels’s Juicy.Pixels.Gif module: • A SIGKILL at any time (usually when the operating system
detects it is consuming too many resources).
data GifFile = GifFile {
gifHeader :: GifHeader , After a process finished, it is possible to detect signals associ-
gifImages :: [(Maybe GraphicControlExtension, ated with failed executions by examining its exit status code. Tra-
GifImage)], ditionally in GNU/Linux systems a process which exits with a zero
gifLoopingBehaviour :: GifLooping exit status has succeeded, while a non-zero exit status indicates fail-
} ure. When a process is terminated by a signal with number n, a shell
sets the exit status to a value greater than 128. Most of the shells
We can simply derive all the required instances with will use 128+N . We capture such condition in the Haskell function
′′
$(devArbitrary Gif F ile), and MegaDeTH will generate the has failed , in order to catch when a program finished abnormally:
following instances (among others):
has failed :: ExitCode → Bool
instance Arbitrary GifLooping where has failed (ExitFailure n) =
arbitrary = sized go (n < 0 ∨ n > 128) ∧ n 6≡ 143
where go n = oneof has failed ExitSuccess = False
[return LoopingNever
We only excluded SIGTERM (with exit status of 143) since we
, return LoopingForever
want to be able to use a timeout in order to catch long executions
, LoopingRepeat h$i resize (n − 1) arbitrary ]
without considering them failed.
instance Arbitrary GifFile where
arbitrary = sized go 4.2 High-Level Fuzzing Properties
where go n =
In order to use QuickCheck to uncover failed executions in pro-
GifFile h$i resize (n − 1) arbitrary grams, we need to define a property to check. Given an executable
h⋆i(listOf $(resize (n ‘div ‘ 10) arbitrary)) program and some arguments, QuickFuzz tries to verify that there
h⋆i resize (n − 1) arbitrary is no failed execution as we defined above for arbitrary inputs. We
Here we can observe that the go functions always decrease their call this property prop NoFail . It serializes inputs to files and ex-
argument n and call the QuickCheck function resize before calling ecutes a given program, so it should be defined using monadicIO.
arbitrary. Whenever we use a constructor, we reduce the size by Its definition is very straightforward:
one, and whenever we generate a list we reduce the size by an order prop NoFail :: Cmd → (a → ByteString)
of magnitude. Given that the instances are automatically generated, → FilePath → a → Property
they seem very artificial, but it is crucial to reduce the size and prop NoFail pcmd encode filename x = monadicIO $
insert the new size in the Gen monad in order to generate a good
do
mix of elements with relatively low sizes.
run $ write filename (encode x )
In the implementation of deriveArbitrary we make a lot of as-
sumptions, the most important one is how we reduce the size pa- ret ← run $ execute pcmd
rameter inside the Gen monad, as we explained before. Implement- assert (¬ (has failed ret))
ing a good strategy to reduce the size parameter is crucial in order After that, we can quickCheck the property of no failed execu-
to finish execution in some reasonable time, but it is also impor- tions instantiating prop NoFail with suitable values. For instance,
tant to decide how values are generated, and what those values are. let us assume we want to test the conversion from gif to png images
Because if the size is reduced too abruptly some values are never using ImageMagick. The usual command to achieve this would be:
going to be generated or the probability for some elements will be
very high, while for others very low. However, given the expressive $ c o n v e r t i n . g i f o u t . png
power of Haskell’s data types, we delegate all the responsability to In terms of prop NoFail , to test the command above we should
guide the generation (the how and what) to them and our sole goal quickCheck the following property:
is to terminate in reasonable time.
prop NoFail "/usr/bin/convert in.gif out.png"
encodeGif
4. Design and Implementation "in.gif"
This section details how we defined suitable properties in QuickCheck
to perform the different phases of the fuzzing process. where encodeGif is a function to serialize GifFiles.
4.3 Low-Level Fuzzing Properties
4.1 Detecting Unexpected Termination in Programs
In the next phase of the fuzzing process, we enhanced the value
One of the key concepts in fuzzing is the repeated execution of
generation of QuickCheck with the systematic file corruption
a target program. In Haskell, a program execution using certain
produced by off-the-shelf fuzzers. Intuitively, we augmented
arguments can be summarized using this type:
prop NoFail with a low-level fuzzing procedure abstracted as a
type Cmd = (FilePath, [String ]) call to the fuzz function.

17
 other components of the tool. Additionally, as expected, there is a
noticeable overhead in the execution. It is possible that most of the
extra time executing is used for calling fork and exec primitives:
this why is one the reasons some fuzzers implement a fork server.
Interestingly enough, the overhead introduced by the use of a
fuzzer is not always consistent. On one hand, in the case of zzuf,
which only XORs bits from the input files without reading them, it
should be a constant overhead. But on the other hand, Radamsa is
a fuzzer which looks at the structure of the data and performs some
mutations according to it. In fact, it was specially designed to detect
and fuzz markup languages: this can explain the higher overhead in
the mutation of Svg files using it.
5.2 Real-World Vulnerabilities Detection
Thanks to Haskell implementations of file-format-handling li-
braries found on Hackage, QuickFuzz currently generates and mu-
tates a large set of different file types out of the box. Table 1a
shows a list of supported file types to generate and corrupt using
our tool.
We tested QuickFuzz using complex real-world applications
like browsers, image processing utilities and file archivers among
Figure 2: Overhead of QuickFuzz performing the fuzzing process. others. All the security vulnerabilities presented in this work were
previously unknown (also known as zero-days). The results are
summarized in Table 1b. An exhaustive list is available at the
fuzz :: Cmd → FilePath → IO () official website of QuickFuzz, including frequent updates on the
After calling fuzz , the content of a file will be changed some- latest bugs discovered using the tool.
how. Using this function, we defined a new property,
prop noFailFuzzed to perform a mutation of the serialized file be- 5.3 Limitations
fore the execution takes place: QuickFuzz shares some of the limitations of QuickCheck. In par-
ticular, we observed that our Arbitrary instances are not always
prop noFailFuzzed :: Cmd → Cmd → (a → ByteString)
effective in the generation of source code, since it requires to care-
→ FilePath → a → Property fully define variable names and functions before trying to use them.
prop NoFailFuzzed pcmd fcmd encode filename x = Then the fuzzed generated source code will be very likely rejected
monadicIO $ do in the first steps of the parsing of interpreters or compilers. This is
run $ write filename (encode x ) a well-known issue that has been studied extensively by Pałka et
run $ fuzz fcmd filename al. [14] and Yang et al. [19] in the context of testing a compiler.
ret ← run $ execute pcmd The use of third-party modules from Hackage is associated with
assert (¬ (has failed ret)) some limitations. Some of the modules we used to serialize com-
plex file types do not implement all the features. For instance, the
bmp support in Juicy.Pixels cannot handle or serialize com-
5. Evaluation pressed files. Therefore this feature will not be effectively tested
5.1 Generation, Mutation and Execution Overhead in the bmp parsers. Also, the encode function used in the serial-
For the overhead evaluation of QuickFuzz in the different stages of ization includes its own bugs. Unsurprisingly some of them can be
the fuzzing process, we measured the time required for high-level triggered by the generation of QuickCheck values. In this case, we
fuzzing with and without execution (noted as gen and gen+exec have a simple workaround: if the encode function throws an un-
respectively) as well as high and low-level fuzzing using zzuf and handled exception, we ignore it and continue the fuzzing process
radamsa (noted as gen+exec+zzuf and gen+exec+rad respec- using the next generated value to serialize.
tively).
In order to strictly quantify the overhead in execution, we used 6. Related Work
/bin/echo which does not read any file. Therefore, it should Generational fuzzing The idea of a generational fuzzer is not
always take the same amount of time to execute. Since this simple new at all. One of the most mature and commercially supported
program will not crash, the shrinking process was not measured in generational fuzzers is Peach. This fuzzer was originally written in
this overhead evaluation. Each experiment was repeated 10 times Python in 2007, and later re-written in C# for the latest release. It
in a dedicated core of an Intel i7 running at 3.40GHz. It is worth provides a wide set of features for generation and mutation of data,
noting that every fuzzer performs one round of mutations and as well as monitoring remote processes. In order to start, it requires
the parameters are defined in the tool itself, so these experiments the specification of two main components to generate and mutate
should be easy to reproduce. program inputs:
Figure 2 shows a comparison of the time that QuickFuzz took
to perform each step of the fuzzing process for three different file • Data Models: a formal description of how data is composed in
types: Zip, Png and Svg. We selected these file types because Zip order to be able to generate fuzzed data.
and Png are binary formats while Svg is a complex human-readable • Target: a formal description of how data can be mutated and
markup language, and we wanted to observe how overhead varied how to detect unexpected behavior in monitored software.
among those.
Our experiments suggest that the performance of the code gen- As expected, the main issue with Peach is that the user has
erated by MegaDeTH for Arbitrary instances is not limiting the to write these configuration files, which requires very specific do-

18
 Program File-Type Reference
Firefox Gif CVE-2016-1933
Images Code Archives Media Firefox Zip CVE-2015-7194
Bmp Css Tar Ogg VLC Wav CVE-2016-3941
Gif Javascript Zip Wav GraphicsMagick Svg CVE-2016-2317
Jpeg Python Gzip ID3 GraphicsMagick Svg CVE-2016-2318
GDK-pixbuf Bmp CVE-2015-7552
Png Html CPIO MIDI
GDK-pixbuf Gif CVE-2015-7674
Pnm Xml GDK-pixbuf Tga CVE-2015-7673
Svg Dot Jasper Jpeg CVE-2015-5203
Tga GLSL libTIFF Tiff CVE-2015-7313
Tiff Json libXML Xml CVE-2016-3627
Ico Regex Jq Json CVE-2016-4074
Jasson Json CVE-2016-4425
(a) List of the file-types supported for fuzzing
cpio CPIO CVE-2016-2037
(b) Some of the security issues found by QuickFuzz

Table 1: Implementation and results

main knowledge. Another option is Sulley [1], a fuzzing engine and burden of writing specifications for file formats on the program-
framework in Python. It is frequently presented as a simpler alter- mer. Our tool combines both generational and mutational fuzzing
native to Peach since the model specification can be written using techniques by bringing together Haskell’s QuickCheck library and
Python code. A more recent alternative open-sourced by Mozilla off-the-shelf, robust mutational fuzzers. In addition, we introduce
in 2015 is Dharma [10], a generation-based, context-free grammar MegaDeTH, a library that can be used to generate instances of the
fuzzer also in Python. It also requires the specification of the data to Arbitrary type classes. MegaDeTH works in tandem with Quick-
generate, but it uses a context-free grammar in a simple plain text Fuzz, allowing us to crowdsource the specifications for well-known
format. file formats that are already present in Hackage libraries. We have
To make a fair comparison between fuzzers is always a chal- tried QuickFuzz in the wild and found that the approach is ef-
lenge. First, it only makes sense to compare between fuzzers us- fective in discovering interesting bugs in real-world implementa-
ing similar techniques. Second, in the case of generative ones, the tions. Moreover, to the best of our knowledge QuickFuzz is the
model to create data in all the compared fuzzers should be similar; only fuzzing tool that provides out-of-the-box generation and muta-
otherwise, generating a complex input will most likely take vary- tion of dozens of complex, common file formats, without requiring
ing amounts of time and could result in some fuzzers being unfairly users to write models or configuration files.
flagged as inefficient. As future work, we intend to introduce mutations at different
Moreover, some fuzzers like Peach are not useful to start dis- levels of the QuickFuzz pipeline, rather than just at the level of the
covering bugs immediately after installing them since they include serialized ByteString. In particular, we aim to explore code anal-
almost no models to start the input generation process. Usually, if ysis of the serializations functions to detect and selectively break
you want to have a wide support of file-types or protocols to fuzz, invariants, and to perform mutations on such functions to corrupt
you need to pay to access them, or hire someone to create them. files. Finally, we would like to extend our approach to the gener-
In other cases like Sulley, fuzzers are developed to be more like ation and fuzzing of network protocols, as well as adding support
a framework in which you can define models, mutate and monitor for automatic derivation of formats with a monadic structure.
process. As a result, no file-type specifications are provided out of
the box.
At first look, Dharma seems to be a good fuzzer to compare with
Acknowledgments
QuickFuzz. Unfortunately, it only includes very specific grammars We would like to thank Alejandro Russo and Daniel Schoepe for
like Canvas2D used by the Mozilla Security team to stress a very interesting discussions, as well as the anonymous reviewers for
specific API of Firefox. QuickFuzz currently does not support their useful feedback and comments. This work was supported in
generation of these types of files. part by The Sloan Foundation.
Automatic algebraic data type test generation Claessen et al. [5]
propose a technique for automatically deriving test data generators References
from a predicate expressed as a Boolean function. The derived [1] Bitflip. Sulley: a pure-python fully automated and unattended
generators are both efficient and guaranteed to produce a uniform fuzzing framework. https://github.com/OpenRCE/
distribution over values of a given size. sulley, 2011.
While MegaDeTH currently produces generators with ad-hoc
distributions, it would be feasible to integrate this technique to [2] CACA Labs. zzuf - multi-purpose fuzzer. http://caca.
the existing machinery to achieve more control over the test case zoy.org/wiki/zzuf, 2010.
generation process. [3] S. K. Cha, T. Avgerinos, A. Rebert, and D. Brumley. Un-
leashing Mayhem on Binary Code. In Proceedings of the
7. Conclusions and Future Work 2012 IEEE Symposium on Security and Privacy, SP ’12. IEEE
We have presented QuickFuzz, a tool for automatically generat- Computer Society, 2012.
ing inputs and fuzzing programs that work on common file for- [4] K. Claessen and J. Hughes. QuickCheck: a lightweight tool
mats. Unlike other fuzzers, QuickFuzz does not require the user to for random testing of Haskell programs. Acm sigplan notices,
provide a set of valid inputs to mutate, and it does not place the 46(4):53–64, 2011.

19
 [5] K. Claessen, J. Duregård, and M. H. Pałka. Generating Con- [14] M. H. Pałka, K. Claessen, A. Russo, and J. Hughes. Testing an
strained RandomData withUniformDistribution, pages 18– Optimising Compiler by Generating Random Lambda Terms.
34. Springer International Publishing, Cham, 2014. ISBN In Proceedings of the 6th International Workshop on Automa-
978-3-319-07151-0. tion of Software Test, AST ’11, pages 91–97, New York, NY,
[6] P. Godefroid, A. Kiezun, and M. Y. Levin. Grammar-based USA, 2011. ACM. ISBN 978-1-4503-0592-1.
Whitebox Fuzzing. SIGPLAN Not., 2008. [15] K. Serebryany, D. Bruening, A. Potapenko, and D. Vyukov.
[7] Google. honggfuzz: a general-purpose, easy-to-use fuzzer Addresssanitizer: A fast address sanity checker. USENIX
with interesting analysis options. https://github.com/ ATC’12, pages 28–28, 2012.
aoh/radamsa, 2010. [16] T. Sheard and S. P. Jones. Template Meta-programming for
[8] Michal Zalewski. American Fuzzy Lop: a security-oriented Haskell. SIGPLAN Not., 37(12):60–75, Dec. 2002. ISSN
fuzzer. http://lcamtuf.coredump.cx/afl/, 2010. 0362-1340. doi: 10.1145/636517.636528. URL http://
doi.acm.org/10.1145/636517.636528.
[9] B. P. Miller, L. Fredriksen, and B. So. An Empirical Study
of the Reliability of UNIX Utilities. Commun. ACM, 33(12): [17] M. Sutton, A. Greene, and P. Amini. Fuzzing: Brute Force
32–44, Dec. 1990. ISSN 0001-0782. Vulnerability Discovery. Addison-Wesley Professional, 2007.
[10] Mozilla. Dharma: a generation-based, context-free gram- [18] Vincent Berthoux. Juicy.Pixels: Haskell library to load &
mar fuzzer. https://github.com/MozillaSecurity/ save pictures. https://hackage.haskell.org/package/
dharma, 2015. JuicyPixels, 2012.
[11] Neil Mitchell . Data.Derive is a library and a tool for de- [19] X. Yang, Y. Chen, E. Eide, and J. Regehr. Finding and
riving instances for Haskell programs. http://hackage. Understanding Bugs in C Compilers. In Proceedings of the
haskell.org/package/derive, 2006. 32Nd ACM SIGPLAN Conference on Programming Language
Design and Implementation, PLDI ’11, pages 283–294, New
[12] N. Nethercote and J. Seward. Valgrind: A Framework for York, NY, USA, 2011. ACM. ISBN 978-1-4503-0663-8.
Heavyweight Dynamic Binary Instrumentation. SIGPLAN
Not., 42(6):89–100, 2007. [20] A. Zeller and R. Hildebrandt. Simplifying and Isolating
Failure-Inducing Input. IEEE Trans. Softw. Eng., 28(2):183–
[13] Oulu University Secure Programming Group. A Crash Course 200, 2002.
to Radamsa. https://github.com/aoh/radamsa, 2010.

20