Fuzzing Error Handling Code using Context-Sensitive Software Fault Injection

Zu-Ming Jiang, Jia-Ju Bai Kangjie Lu Shi-Min Hu

Tsinghua University University of Minnesota Tsinghua University

Abstract because such code is infrequently executed and often receives

insufficient attention. For these reasons, many bugs may exist
Error handling code is often critical but difficult to test
in error handling code, and they are often difficult to find in
in reality. As a result, many hard-to-find bugs exist in error
real execution. Some recent works [8, 32, 37, 68] have shown
handling code and may cause serious security problems once
that many bugs in error handling code can cause serious secu-
triggered. Fuzzing has become a widely used technique for
rity problems, such as denial of service (DoS) and information
finding software bugs nowadays. Fuzzing approaches mutate
disclosure. In fact, many CVE-assigned vulnerabilities (such
and/or generate various inputs to cover infrequently-executed
as CVE-2019-7846 [19], CVE-2019-2240 [20], CVE-2019-
code. However, existing fuzzing approaches are very limited
1750 [21] and CVE-2019-1785 [22]) stem from bugs in error
in testing error handling code, because some of this code can
handling code.
be only triggered by occasional errors (such as insufficient
memory and network-connection failures), but not specific in- Considering that error handling code is critical but buggy,
puts. Therefore, existing fuzzing approaches in general cannot various tools have been proposed to detect bugs in error han-
effectively test such error handling code. dling code. Some approaches [28, 32, 33, 37, 53] use static
analysis, but they often introduce many false positives, due to
In this paper, we propose a new fuzzing framework named
the lack of runtime information and inherent limitations with
FIFUZZ, to effectively test error handling code and detect
static analysis. To reduce false positives, recent approaches [1,
bugs. The core of FIFUZZ is a context-sensitive software
6,7,13,15,26,27,29,30,38,45,50,50,51,59,60,64,65] instead
fault injection (SFI) approach, which can effectively cover
use fuzzing to test infrequently executed code. They gener-
error handling code in different calling contexts to find deep
ate effective program inputs to cover infrequently executed
bugs hidden in error handling code with complicated contexts.
code, according to the input specification or the feedback of
We have implemented FIFUZZ and evaluated it on 9 widely-
program execution. However, the input-driven fuzzing cannot
used C programs. It reports 317 alerts which are caused by
effectively cover error handling code, as some of this code
50 unique bugs in terms of the root causes. 32 of these bugs
can be only triggered by non-input occasional errors, such
have been confirmed by related developers. We also compare
as insufficient memory and network-connection failures. As
FIFUZZ to existing fuzzing tools (including AFL, AFLFast,
a result, existing fuzzing approaches cannot effectively test
AFLSmart and FairFuzz), and find that FIFUZZ finds many
error handling code.
bugs missed by these tools. We believe that FIFUZZ can
effectively augment existing fuzzing approaches to find many Testing error handing code is challenging by nature, as er-
real bugs that have been otherwise missed. rors are often hard to deterministically produce. An intuitive
solution to triggering error handling code is to use software
fault injection (SFI) [52]. SFI intentionally injects faults or
1 Introduction errors into the code of the tested program, and then executes
the program to test whether it can correctly handle the in-
A program may encounter various errors and needs to handle jected faults or errors at runtime. Specifically, the faults are
these errors at runtime. Otherwise, the program may suffer injected into the sites that can fail and trigger error handling
from security or reliability issues. While error handing is code, and we call each such site an error site. In this way,
critical, itself is error-prone. Firstly, error handling code is SFI can intentionally cover error handling code at runtime.
difficult to correctly implement [14, 23, 34, 54] because it Existing SFI-based approaches [9–11, 18, 25, 39, 40, 55, 67]
often involves special and complicated semantics. Secondly, have shown encouraging results in testing error handling code
error handling code is also challenging to test [25, 28, 53, 61], and detecting hard-to-find bugs.
 However, existing SFI-based approaches suffer from a crit- program, mutating each created error sequence to generate
ical limitation: to our knowledge, they perform only context- new sequences; 5) running the tested program and injecting
insensitive fault injection, which often stops testing from go- faults according to the mutated error sequences; 6) collect-
ing deep. Specifically, they inject faults according to the lo- ing runtime information, creating new error sequences and
cations of error sites in source code, without considering the performing mutation of these error sequences again, which
execution contexts of these error sites, i.e., the execution paths constructs a fuzzing loop.
reaching to the error sites. Thus, if a fault is constantly in- Based on our approach, we propose a new fuzzing frame-
jected into an error site, this error site will always fail when work named FIFUZZ. At compile time, to reduce manual
being executed at runtime. However, an error site is typically work of identifying error sites, FIFUZZ performs a static anal-
executed in different calling contexts, and real bugs can be ysis of the source code of tested programs, to identify possible
only triggered when this error site fails in a specific calling error sites. The user can select realistic error sites that can ac-
context but succeeds in other calling contexts. In this case, tually fail and trigger error handling code. Then, FIFUZZ uses
existing SFI-based approaches may miss these real bugs. our context-sensitive SFI-based fuzzing approach in runtime
Figure 1 shows a simple example of this case. In the func- testing. To be compatible with traditional fuzzing process
tion main, the objects x and y are allocated, and then the func- for program inputs, FIFUZZ mutates the error sequences and
tions FuncA and FuncB are called. FuncA and FuncB both call program inputs together by analyzing runtime information of
FuncP, but FuncB frees the argument object before calling the tested program.
FuncP. In FuncP, the object z is allocated by calling malloc; Overall, we make the following technical contributions:
if this function call fails, the argument object is freed, and
the program exits abnormally by calling exit. If we perform • We perform two studies of error handling code in widely-
context-insensitive fault injection by just statically injecting used applications and vulnerabilities found by existing
a fault into malloc in FuncP, the program will always exit fuzzing tools, and find that: nearly 42% of sites that
when FuncA is executed, without finding any bug. If we con- can trigger error handling code are related to occasional
sider calling context, and inject a fault into malloc in FuncP errors, but only few vulnerabilities found by existing
only when FuncB calls FuncP, a double-free bug of the ob- fuzzing tools are related to error handling code triggered
ject y can be triggered at runtime. Since such a case is fairly by occasional errors. Thus, it is important to improve
common, it may incur a significant impact on detecting bugs fuzzing to support the testing of error handling code.
in error handling code. • We propose a novel context-sensitive SFI-based fuzzing
approach, which can dynamically inject faults based on
int main() { void FuncA(x) { void FuncP(arg) {
both locations of error sites and their calling contexts, to
x = malloc(...); FuncP(x); z = malloc(...)
cover hard-to-trigger error handling code.
y = malloc(...); ...... if (!z) {
...... } free(arg); • Based on this approach, we develop a new fuzzing frame-
FuncA(x); void FuncB(y) { exit(-1); work named FIFUZZ, to effectively test error handling
FuncB(y); free(y); } code. To our knowledge, FIFUZZ is the first systematic
...... FuncP(y); ......
fuzzing framework that can test error handling code in
} ...... }
different calling contexts.
}

Fault 1: main -> FuncA -> FuncP -> malloc exit abnormally...
• We evaluate FIFUZZ on 9 well-tested and widely-used
Fault 2: main -> FuncB -> FuncP -> malloc double free! C applications of the latest versions as of our evalua-
tion. It reports 317 alerts which are caused by 50 unique
Figure 1: Examples of function calls that can fail. bugs in terms of the root causes. 32 of these bugs have
been confirmed by related developers. We also com-
In this paper, to effectively detect bugs in error handling pare FIFUZZ to existing fuzzing tools (including AFL,
code, we design a novel context-sensitive SFI-based fuzzing AFLFast, AFLSmart and FairFuzz) on 5 common pro-
approach. The approach takes execution contexts into ac- grams in the Binutils toolset, and find that FIFUZZ
count to effectively guide SFI to maximize bug finding. It finds many bugs missed by these tools.
consists of six steps: 1) statically identifying the error sites in
the source code of the tested program; 2) running the tested The rest of this paper is organized as follows. Section 2
program and collecting runtime information about calling introduces background and our two studies. Section 3 intro-
contexts of each executed error site and code coverage; 3) duces basic idea and our context-sensitive SFI-based fuzzing
creating error sequences about executed error sites according approach. Section 4 introduces FIFUZZ in detail. Section 5
to runtime information, and each element of such a sequence shows our evaluation. Section 6 makes a discussion about
is differentiated by the location of the executed error site and FIFUZZ and its found bugs. Section 7 presents related work,
the information about its calling context; 4) after running the and Section 8 concludes this paper.
2 Background --- a/libavcodec/aacsbr.c
+++ b/libavcodec/aacsbr.c
@@ -334,6 +334,9
In this section, we first introduce error handling code with static int sbr_make_f_master(AACContext *ac,
related bug examples, and then show our studies of error SpectralBandReplication *sbr, ...) {
...
handling code in widely-used applications and CVEs found + if (!sbr->sample_rate)
by existing fuzzing tools. + return -1;
// BUG: sbr->sample_rate may be zero
start_min = ... / sbr->sample_rate;
...
2.1 Error Handling Code }

A program may encounter exceptional situations at runtime, Figure 2: Patch A: fixing a divide-by-zero bug.
due to special execution conditions such as invalid inputs from
--- a/libavutil/frame.c
users, insufficient memory and network-connection failures. +++ b/libavutil/frame.c
We refer to such exceptional situations as errors, and the code @@ -383,8 +383,8
int av_frame_copy_props(...) {
used to handle an error is called error handling code. ...
In fact, errors can be classified into two categories: input- AVFrameSideData *sd_dst = av_frame_new_side_data(...);
if (!sd_dst) {
related errors and occasional errors. An input-related error for (i = 0; i < dst->nb_side_data; i++) {
is caused by invalid inputs, such as abnormal commands and av_freep(&dst->side_data[i]->data);
- av_freep(&dst->side_data[i]);
bad data. Such an error can be triggered by providing spe- av_dict_free(&dst->side_data[i]->metadata);
cific inputs. An occasional error is caused by an exceptional + av_freep(&dst->side_data[i]);
}
event that occasionally occurs, such as insufficient memory }
or network-connection failure. Such an error is related to the }
state of execution environment and system resources (such Figure 3: Patch B: fixing a use-after-free bug.
as memory and network connection), but unrelated to inputs,
so it typically cannot be triggered by existing fuzzing that fo-
cuses on inputs. While this error occurs occasionally, they can The bug in Figure 3 is instead caused by incorrect handling of
be reliably triggered in an adversarial setting. For example, an occasional error, because av_frame_new_side_data fails
by exhaustively consuming memory, an attacker can reliably only when memory is insufficient, which occasionally occurs
result a function call to malloc() in returning a null pointer. at runtime.
As such, bugs in error handing code can be as critical as the
ones in normal code.
2.3 Study of Error Handling Code
2.2 Bug Examples in Error Handling Code To understand the proportion of input-related errors and occa-
Figures 2 and 3 show two patches fixing bugs in error handling sional errors that can trigger error handling code in software,
code of the libav library in ffmpeg [24]. In Figure 2, the vari- we perform a manual study of the source files (.c and .h) of 9
able sbr->sample_rate could be zero, but it is divided in the widely-used applications (vim, bison, ffmpeg, nasm, catdoc,
code, causing a divide-by-zero bug. This bug is also reported clamav, cflow, gif2png+libpng, and openssl). Due to time con-
as CVE-2016-7499 [48]. To fix this bug, Patch A [46] checks straints, if an application contains over 100 source files, we
whether sbr->sample_rate is zero before this variable is di- randomly select 100 source files of this application to study.
vided, and returns abnormally if so. The report of this bug [47] Otherwise, we study all the source files of this application.
mentions that this bug was found by AFL. On the other Specifically, we first manually identify the sites that can fail
hand, in Figure 3, the function av_frame_new_side_data and trigger error handling code by looking for if or goto state-
is used to allocate memory for new data, and it can fail ments, which are often used as entries of error handling code
and return a null pointer when memory is insufficient. In in C applications [33]. Then, we manually check whether the
this case, the variable dst->side_data[i]->metadata is identified sites are related to input-related errors or occasional
freed after dst->side_data[i] is freed, which causes a errors. Table 1 shows the study results.
use-after-free bug. To fix this bug, PatchB [49] frees the We find that 42% of the sites that can fail and trigger error
variable dst->side_data[i]->metadata before freeing handling code are related to occasional errors. Besides, in the
dst->side_data[i]. Because the report of this bug or the study, we also observe that about 70% of the identified error
patch does not mention any tool, the bug might be found by sites are related to checking error-indicating return values of
manual inspection or real execution. function calls (such as the example in Figure 3). This observa-
The bug in Figure 2 is caused by missing handling of an tion indicates that manipulating the return values of specific
input-related error, because the variable sbr->sample_rate function calls can cover most error handling code, which has
is related to the function argument sbr affected by inputs. been adopted by some existing SFI-based approaches [10,18].
 Application Studied file Error site Input-related Occasional
can occur and trigger error handling code. When performing
vim 100 1163 530 (46%) 633 (54%) fault injection, each error point in an error sequence can nor-
bison 100 184 96 (52%) 88 (48%) mally run (indicated as 0) or fail by injecting a fault (indicated
ffmpeg 100 881 518 (59%) 363 (41%)
nasm 100 673 564 (84%) 109 (16%) as 1). Thus, an error sequence is actually as 0-1 sequence that
catdoc 29 91 43 (47%) 48 (53%) describes the failure situation of error points at runtime:
clamav 100 1089 522 (48%) 567 (52%)
cflow 100 286 170 (59%) 116 (41%)
gif2png+libpng 95 830 556 (67%) 274 (33%) ErrSeq = [ErrPt1 , ErrPt2 , ..., ErrPtx ], ErrPti = {0, 1} (1)
openssl 100 989 571 (58%) 418 (42%)
Total 824 6,168 3,570 (58%) 2,616 (42%)
Similar to program inputs, an error sequence also affects
Table 1: Study results of error handling code. program execution. This sequence can be regarded as the
“input” of possibly triggered errors. A key problem here is
Tool CVE Error handling Occasional error which error points in an error sequence should be injected
AFL 218 85 3 with faults to cover as much error handling code as possible.
Honggfuzz 57 17 3
AFLFast 8 2 0 Inspired by existing fuzzing that fuzz program inputs using
CollAFL 93 15 4 the feedback of program execution, our basic idea is to fuzz
QSYM 6 0 0
REDQUEEN 11 2 1
error sequence for fault injection to test error handling code.
Total 393 121 11

Table 2: Study results of existing fuzzing tools. 3.2 Error Sequence Model
Existing SFI-based approaches often use context-insensitive
2.4 Study of CVEs Found by Existing Fuzzing fault injection. Specifically, they only use the location of each
error site in source code to describe an error point, namely
To understand how existing fuzzing tools perform in detecting ErrPt = <ErrLoc>, without considering the execution context
bugs in error handling code, we further study the CVEs found of this error site. In this way, if an fault is injected into an
by some start-of-the-art fuzzing tools, including AFL [1], error site, this error site will always fail when being executed
Honggfuzz [30], AFLFast [13], CollAFL [26], QSYM [65] at runtime. However, an error site can be executed in different
and REDQUEEN [7]. We select these fuzzing tools because calling contexts, and some real bugs (such as the double-free
CVEs found by them are publicly available. Specifically, for bug shown in Figure 1) can be triggered only when this error
AFL, a website [2] collects its found CVEs; for Honggfuzz, site only fails in specific calling context and succeeds in other
the found CVEs are listed in its homepage; for AFLFast, Col- calling contexts. Thus, existing SFI-based approaches may
lAFL, QSYM and REDQUEEN, the found CVEs are listed in miss these real bugs.
their papers as well. We manually read these CVEs and iden- To solve this problem, we propose a context-sensitive soft-
tify the ones related to error handling code, and also check ware fault injection (SFI) method. Besides the location of
whether the identified CVEs are related to occasional errors. each error site, our method also considers the calling context
Table 2 shows the study results. of the error site to describe error points, namely:
We find that 31% of CVEs found by these fuzzing tools
are caused by incorrect error handling code, such as the bug ErrPt =< ErrLoc,CallCtx > (2)
shown in Figure 2. Only 9% of these CVEs are related to oc-
casional errors. This proportion is far less than the proportion To describe calling context of an error site, we consider
(42%) of occasional error sites among all error sites (found in the runtime call stack when the error site is executed. This
Section 2.3). The results indicate that existing fuzzing tools runtime call stack includes the information of each function
may have missed many real bugs in error handling code trig- call at the call stack (in order from caller to callee), including
gered by occasional errors. Thus, it is important to improve the locations of this function call and called function. In this
fuzzing to support the testing of error handling code. way, a calling context is described as:

CallCtx = [CallIn f o1 ,CallIn f o2 , ...,CallIn f ox ] (3)

3 Basic Idea and Approach CallIn f o =< CallLoc, FuncLoc > (4)

3.1 Basic Idea Based on the above description, the information about each
error point can be hashed as a key, and whether this error point
To effectively test error handling code, we introduce SFI in should fail can be represented as a 0-1 value. Thus, an error
fuzz testing by “fuzzing” injected faults according to the run- sequence can be stored as a key-value pair in a hash table:
time information of the tested program. To achieve this idea,
we build an error sequence that contains multiple error points. KEY Hash(ErrPt1) Hash(ErrPt2) ...... Hash(ErrPtx)
An error point represents an execution point where an error VALUE 0 or 1 0 or 1 ...... 0 or 1
 Note that the runtime call stack of an executed error site is executions, it performs the subsequent generation and mu-
related to program execution. Thus, error points cannot be stat- tation of error sequences. All the generated error sequences
ically determined, and they should be dynamically identified that increase code coverage are stored in a pool, and they
during program execution. Accordingly, when performing are ranked by contribution to code coverage. Our approach
fault injection using error sequences, the faults should be preferentially selects error sequences for mutation.
injected into error points during program execution. Initial mutation. Our approach first executes the tested pro-
According to our method, when an error site is executed gram normally, and creates an initial error sequence according
in N different calling contexts, there will be N different error to runtime information. This error sequence contains executed
points for fault injection, instead of just one error point iden- error points, and it is all-zero and used for the initial mutation.
tified by context-insensitive fault injection. Thus, our method The mutation generates each new error sequence by making
can perform finer-grained fault injection. just one executed error point fail (0→1), as each error point
may trigger uncovered error handling code in related calling
3.3 Context-Sensitive SFI-based Fuzzing context. Figure 5 shows an example of the initial mutation for
an error sequence, which generates four new error sequences.
To effectively cover as much error handling code as possi-
ble, based on our context-sensitive SFI method, we propose ErrPta ErrPtb ErrPtc ErrPtd
1 0 0 0
a novel context-sensitive SFI-based fuzzing approach to per- ErrPta ErrPtb ErrPtc ErrPtd
Tested First execution ErrPta ErrPtb ErrPtc ErrPtd Initial mutation 0 1 0 0
form fault injection using the feedback of program execution. Program 0 0 0 0 ErrPta ErrPtb ErrPtc ErrPtd
As shown in Figure 4, our approach has six steps: 1) stati- Initial error sequence 0 0 1 0
ErrPta ErrPtb ErrPtc ErrPtd
cally identifying the error sites in the source code of the tested 0 0 0 1
Generated error sequences
program; 2) running the tested program and collecting run-
time information about calling contexts of each executed error Figure 5: Example of the initial mutation.
site and code coverage; 3) creating error sequences about ex-
ecuted error sites according to runtime information; 4) after Subsequent generation and mutation. After executing the
running the program, mutating each created error sequence to tested program by injecting faults according to an original er-
generate new sequences; 5) running the tested program and ror sequence, some new error points may be executed, making
injecting faults into error sites in specific calling contexts ac- a new error sequence created. Our approach checks whether
cording to the mutated error sequences; 6) collecting runtime the code coverage is increased (namely new basic blocks or
information, creating new error sequences and performing code branches are covered) during this execution. If not, the
mutation of these error sequences again, which constructs a original error sequence and the created error sequence (if it
fuzzing loop. When no new error sequences are generated or exists) are dropped; if so, our approach separately mutates
the time limit is reached, the fuzzing loop ends. the original error sequence and the created error sequence (if
it exists) to generate each new error sequence by changing
Identify error sites
the value of just one error point (0→1 or 1→0). Then, our
approach compares these generated error sequences with ex-
Run the tested program
isting error sequences, to drop repeated ones. Figure 6 shows
an example of this procedure for two error sequences, For
Collect runtime information Perform fault injection the first error sequence ErrSeq1 , a new error point ErrPtx is
executed, and thus our approach creates an error sequence
Create error sequences Run the tested program containing ErrPtx . As the code coverage is increased, our
Y approach mutates the two error sequences and generates nine
Generate new new error sequences. However, one of them is the same with
Mutate error sequences error sequences and
within time limit? existing error sequence ErrSeq2 , thus this new error sequence
Fuzzing Loop N is dropped. For the second error sequence ErrSeq2 , a new
End error point ErrPty is executed, and thus our approach creates
an error sequence containing ErrPty . As the code coverage is
Figure 4: Procedure of our SFI-based fuzzing approach. not increased, our approach drops the two error sequences.
Note that each error point in an error sequence is related to
In our approach, mutating and generating error sequences runtime calling context, thus when injecting faults into this
are important operations. Given a program input, our approach error point during program execution, our approach needs
considers code coverage in these operations and drops re- to dynamically check whether the current runtime calling
peated error sequences. Initially such information is unavail- context and error sites match the target error point. If this
able, and thus our approach performs a special initial mutation error point is not executed during program execution, our
for the first execution of the tested program. For subsequent approach will ignore this error point.
 ErrPta ErrPtb ErrPtx ErrPtc ErrPtd
0 1 0 0 0 Bug Checkers Bug Reports
ErrPta ErrPtb ErrPtx ErrPtc ErrPtd
1 0 0 0 0 Runtime Information
Executable
ErrPta ErrPtb ErrPtx ErrPtc ErrPtd Mutation ErrPta ErrPtb ErrPtx ErrPtc ErrPtd Original Program Runtime
1 1 0 0 0 1 1 1 0 0 Generator Program Monitor
Program Inputs
Created error sequence ErrPta ErrPtb ErrPtx ErrPtc ErrPtd
1 1 0 1 0 Recommended Runtime Information
ErrPta ErrPtb ErrPtx ErrPtc ErrPtd Error Sites
1 1 0 0 1
A new error point is executed Error-Site Error-Sequence Input
ErrPta ErrPtb ErrPtc ErrPtd Extractor Generator Generator
0 1 0 0
ErrSeq1 ErrPta ErrPtb ErrPtc ErrPtd Source Files of the Error Sequences
ErrPta ErrPtb ErrPtc ErrPtd Execution Tested Mutation 1 0 0 0
tested program
1 1 0 0 Fault injection Program Inputs
Program ErrPta ErrPtb ErrPtc ErrPtd
Original error sequence 1 1 1 0
Code coverage is increased! ErrPta ErrPtb ErrPtc ErrPtd
1 1 0 1
Generated error sequences
Figure 7: Overall architecture of FIFUZZ.
ErrPta ErrPtb ErrPty ErrPtc ErrPtd
0 1 0 0 0
Created error sequence
A new error point is executed
ErrSeq2
4.1 Compile-Time Analysis
ErrPta ErrPtb ErrPtc ErrPtd Execution Tested
0 1 0 0 Fault injection Drop error sequences
Program In this phase, FIFUZZ performs two main tasks:
Original error sequence
Code coverage is not increased!
Drop repeated error sequences Error-site extraction. For SFI-based approaches, the in-
jected errors should be realistic. Otherwise, the found bugs
Figure 6: Example of the normal mutation. might be false positives. To ensure that injected errors are real-
istic, many SFI-based approaches [18, 40, 55] require the user
4 FIFUZZ Framework to manually provide error sites, which requires much manual
work and cannot scale to large programs. To reduce manual
Based on our context-sensitive SFI-based fuzzing approach, work, the error-site extractor uses a static analysis against the
we design a new fuzzing framework named FIFUZZ, to effec- source code of the tested program, to identify possible error
tively test error handling code. We have implemented FIFUZZ sites, from which the user can select realistic ones.
using Clang [16]. FIFUZZ performs code analysis and code Our analysis focuses on extracting specific function calls as
instrumentation on the LLVM bytecode of the tested program. error sites, because our study in Section 2.3 reveals that most
To be compatible with traditional fuzzing process, FIFUZZ of error sites are related to checking error-indicating return
mutates the error sequences and program inputs together. Fig- values of function calls. Our analysis has three steps:
ure 7 shows its architecture, consisting of six parts: S1: Identifying candidate error sites. In many cases, a func-
tion call returns a null pointer or negative integer to indicate a
• Error-site extractor. It performs an automated static failure. Thus, our analysis identifies a function call as a candi-
analysis of the source code of the tested program, to date error site if: 1) it returns a pointer or integer; and 2) the
idenfity possible error sites. return value is checked by an if statement with NULL or zero.
• Program generator. It performs code instrumention on The function call to av_frame_new_side_data in Figure 3 is
the program code, including identified error sites, func- an example that satisfies the two requirements.
tion calls, function entries and exits, code branches, etc. S2: Selecting library functions. A called function can be
It generates an executable tested program. defined in the tested program or an external library. In most
cases, a function defined in the tested program can fail, as it
• Runtime monitor. It runs the tested program with gen- calls specific library functions that can fail. If this function
erated inputs, collects runtime information of the tested and its called library functions are both considered for fault
program, and performs fault injection according to gen- injection, repeated faults may be injected. To avoid repetition,
erated error sequences. from all the identified function calls, our analysis only selects
• Error-sequence generator. It creates error sequences, those whose called functions are library functions.
and mutates error sequences to generate new error se- S3: Performing statistical analysis. In some cases, a func-
quences, according to collected runtime information. tion can actually fail and trigger error handling, but the return
values of several calls to this function are not checked by if
• Input generator. It performs traditional fuzzing process statements. To handle such cases, our analysis use a statistical
to mutate and generate new inputs, according to collected method to extract functions that can fail from the identified
runtime information. function calls, and we refer to such a function as an error
function. At first, this method classifies the selected func-
• Bug checkers. They check the collected runtime infor-
tion calls by called function, and collects all function calls
mation to detect bugs and generate bug reports.
to each called function in the tested program code. Then, for
Based on the above architecture, FIFUZZ consists of two the function calls to a given function, this method calculates
phases, which are introduced as follows. the percent of them whose return values are checked by if
statements. If this percent is larger than a threshold R, this
Program Description Version LOC
method identifies this function as an error function. Finally, vim Text editor v8.1.1764 349K
this method extracts all function calls to this function are bison Parser generator v3.4 82K
ffmpeg Solution for media processing n4.3-dev 1.1M
identified error sites. For accuracy and generality, if there are nasm 80x86 and x86-64 assembler v2.14.02 94K
multiple tested programs, this method analyzes the source catdoc MS-Word-file viewer v0.95 4K
clamav Antivirus engine v0.101.2 844K
code of all the tested programs together. cflow Code analyzer of C source files v1.6 37K
In our analysis, the value of the threshold R in the third step gif2png+libpng File converter for pictures v2.5.14+v1.6.37 59K
openssl Cryptography library v1.1.1d 415K
heavily affects the identified error functions and identified
error sites (function calls). For example, less error functions Table 3: Basic information of the tested applications.
and error sites can be identified, as R becomes larger. In this
case, more unrealistic error functions and error sites can be
dropped, but more realistic ones may be also missed. We
4.2 Runtime Fuzzing
study the impact of the value of R in Section 5.2. In this phase, with the identified error sites and instrumented
code, FIFUZZ performs our context-sensitive SFI-based
int *FuncA() { fuzzing approach, and uses traditional fuzzing process of
+ FuncEntry(FuncA);
int *p; program inputs referring to AFL [1].
+ CallEntry(FuncB); The runtime fuzzer executes the tested program using the
p = FuncB();
+ CallExit(FuncB); program inputs generated by traditional fuzzing process, and
int *FuncA() { (*p)++;
int *p; + FuncExit(FuncA); injects faults into the program using the error sequences gen-
p = FuncB(); return p;
(*p)++; } erated by our SFI-based fuzzing approach. It also collects run-
return p; time information about executed error points, code branches,
} int *FuncB() {
Instrument + FuncEntry(FuncB); etc. According to the collected runtime information, the error-
int *FuncB() { int *q;
int *q; + ErrorPointCollect(...); sequence generator creates error sequences and performs mu-
q = malloc(...); + if (ErrorPointFail(...) == TRUE)
if (!q) { + q = NULL; tation to generate new error sequences; the input generator
return NULL; + else performs coverage-guided mutation to generate new inputs.
} q = malloc(...); // error site
*q = 100; if (!q) { Then, FIFUZZ combines these generated error sequences and
return q; + FuncExit(FuncB);
} return NULL; inputs together, and use them in runtime fuzzer to execute the
}
*q = 100; tested program again. To detect bugs, the bug checkers ana-
+ FuncExit(FuncB);
return q; lyze the collected runtime information. These bug checkers
} can be third-party sanitizers, such as ASan [4] and MSan [41].
Figure 8: Example of code instrumentation.
5 Evaluation
Code instrumentation. The code instrumentation serves
5.1 Experimental Setup
for two purposes: collecting runtime information about error
sites and injecting faults. To collect the information about run- To validate the effectiveness of FIFUZZ, we evaluate it on
time calling context of each error site, the program generator 9 extensively-tested and widely-used C applications of the
instruments code before and after each function call to each latest versions as of our evaluation. These applications are
function defined in the tested program code, and at the entry used for different purposes, such as text editor (vim), media
and exit in each function definition. Besides, on the other processing (ffmpeg), virus scan (clamav) and so on. The in-
hand, to monitor the execution of error sites and perform fault formation of these applications are listed in Table 3 (the lines
injection into them, the program generator instruments code of source code are counted by CLOC [17]). The experiment
before each error site. During program execution, the runtime runs on a regular desktop with eight Intel i7-3770@3.40G
calling context of this error site and its location are collected processors and 16GB physical memory. The used compiler is
to create an error point. Then, if this error point can be found Clang 6.0 [16], and the operating system is Ubuntu 18.04.
in the current error sequence, and its value is 1 (indicating this
error point should fail for fault injection) a fault is injected
5.2 Error-Site Extraction
into the error point. In this case, the function call of related
error site is not executed, and its return values is assigned to Before testing programs, FIFUZZ first performs a static anal-
a null pointer or a random negative integer. If the value of ysis of their source code to first identify error functions that
this error point in the error sequence is 0, the function call can fail, and then to identify error sites. We set R = 0.6 in
of related error site is normally executed. Figure 8 shows an this analysis, and perform the third step of this analysis for
example of instrumented code in the C code. Note that code the source code of all the tested programs. After FIFUZZ
instrumentation is actually performed on the LLVM bytecode. produces identified error sites, we manually select realistic
 Program Function call Identified Realistic
step. Figure 9 shows the results. We find that the number
vim 67,768 1,589 (2.3%) 283 (17.8%) of identified error functions and realistic error functions are
bison 11,861 966 (8.1%) 145 (15.0%) both decreased when R becomes larger. In this case, more
ffmpeg 459,986 2,157 (0.5%) 190 (8.8%)
nasm 10,246 429 (4.2%) 44 (10.3%) unrealistic error functions are dropped, but more realistic ones
catdoc 1,293 103 (8.0%) 45 (43.7%) are also missed. Thus, if R is too small, many unrealistic error
clamav 52,830 2,183 (4.1%) 816 (37.4%)
cflow 4,049 149 (3.7%) 88 (59.1%) functions will be identified, which may introduce many false
gif2png+libpng 11,209 303 (2.7%) 54 (17.8%) positives in bug detection; if R is too large, many realistic
openssl 158,625 1916 (1.2%) 157 (8.2%)
Total 777,867 9,795 (1.3%) 1,822 (18.6%)
error functions will be missed, which may introduce many
false negatives in bug detection.
Table 4: Results of error-site extraction.

ones that can actually fail and trigger error handling code, by 5.3 Runtime Testing
reading related source code. Table 4 shows the results. The Using the 1,822 realistic error sites identified with R = 0.6, we
first column presents the application name; the second column test the 9 target applications. We fuzz each application with a
presents the number of all function calls in the application; well-know sanitizer ASan [4] and then without ASan (because
the third column presents the number of error sites identified it often introduces much runtime overhead), for three times.
by FIFUZZ; the last column presents the number of realistic The time limit of each fuzzing is 24 hours. For the alerts
error sites that we manually select. found by fault injection, we count them by trigger location
In total, FIFUZZ identifies 287 error functions, and identi- and error point (not error site). Table 5 shows the fuzzing
fies 9,795 function calls to these error functions as possible results with ASan and without ASan. The columns “Error
error sites. Among them, we manually select 150 error func- sequence” and “Input” show the results about generated error
tions as realistic ones, and 1,822 function calls to these error sequences and inputs; in these columns, the columns “Gen”
functions that are considered as realistic error sites are auto- show the number of generated ones, and the columns “Useful”
matically extracted from the source code. Thus, the accuracy show the number of ones that increase code coverage. From
rates of FIFUZZ for identifying realistic error functions and the results, we find that:
error sites are 52.3% and 18.6%. The manual confirmation is Error sequence. FIFUZZ generates many useful error se-
easily manageable and not hard. The user only needs to scan quences for fault injection to cover error handling code. In
the definition of each error function, to check whether it can total, 3% and 2% of generated error sequences increase code
trigger an error by returning an error number or a null pointer. coverage by covering new code branches, with and without
One master student spent only 2 hours on the manual selection ASan, respectively. These proportions are larger than those
of error functions for the 9 tested applications. Considering (0.02% with ASan and 0.007% without ASan) for generated
there are over 600K function calls in the tested programs, FI- program inputs. To know about the variation of useful error
FUZZ is able to drop 99% of them, as they are considered not sequences and program inputs increasing code coverage, we
to be fail and trigger error handling code according to their select vim as an example to study. Figure 10 shows the results.
contexts in source code. We find that many of the selected er- We find that the number of useful error sequences increases
ror functions and error sites are related to memory allocation quickly during earlier tests, and then tends to be stable in the
that can indeed fail at runtime, and nearly half of the selected later tests. This trend is quite similar to program inputs.
error functions and error sites are related to occasional errors.
The results show that FIFUZZ can dramatically help reduce
the manual work of identifying realistic error sites.
Number of error functions

350 Identified error functions

300 Realistic error functions
250
200
150
100 (a) With ASan
50
0
0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1
Value of the threshold R

Figure 9: Variation of results affected by the value of R.

As described in Section 4.1, the value of R = 0.6 in the

static analysis heavily affects the identified error functions. (b) Without ASan
The above results are obtained with R = 0.6. To understand
the variation caused by R, we test R from 0.5 to 1 with 0.05 Figure 10: Variation of error sequences and inputs for vim.
 With ASan Without ASan
Program Error sequence Input Reported alert Error sequence Input Reported alert
Gen Useful Gen Useful Null MemErr Assert All Gen Useful Gen Useful Null MemErr Assert All
vim 9,199 772 504,736 338 27 5 0 32 44,322 1,664 2,355,965 451 55 3 0 58
bison 1,450 221 1,995,831 1,168 11 0 0 11 8,692 289 14,602,760 1,207 11 0 0 11
ffmpeg 591 311 139,543 758 13 13 3 29 3,060 516 4,817,284 1,766 14 18 3 35
nasm 5,316 65 2,571,182 2,748 8 0 0 8 38,667 78 17,326,673 4,203 8 0 0 8
catdoc 84 34 4,721,501 158 1 1 0 2 798 38 40,357,609 234 2 0 0 2
clamav 482 339 98,352 26 7 106 0 113 482 325 331,930 34 7 96 0 103
cflow 1,623 159 4,551,244 724 0 0 0 0 12,209 217 29,909,026 1,235 1 0 0 1
gif2png+libpng 781 9 15,019,720 320 0 1 0 1 1,498 6 29,717,956 409 0 0 0 0
openssl 73,200 626 369,613 13 59 0 0 59 82,214 671 428,447 15 80 0 0 80
Total 92,726 2,536 29,971,722 6,253 126 126 3 255 191,942 3,804 139,847,650 9,554 178 117 3 298

Table 5: Fuzzing results.

FILE: bison/src/uniqstr.c
FILE: bison/lib/hash.c
159. void uniqstrs_new(void) {
Type Null MemErr Assert All / Error handling 160. uniqstrs_table = hash_initialize(...); Bug1 597. Hash_table *hash_initialize(...) {
...... ......
Unique alert 182 132 3 317 / 313 Bug1 626. table->bucket = calloc(...);
165. }
Found bug 36 13 1 50 / 46 FILE: bison/src/main.c 627. if (table->bucket == NULL)
FILE: bison/src/muscle-tab.c 628. goto fail;
61. int main(...) {
Confirmed bug 26 6 0 32 / 32 ...... 125. void muscle_init(void) {
......
646. }
86. uniqstrs_new(); ...... Bug2
87. muscle_init(); 129. muscle_table = hash_initialize(...); Bug3
Table 6: Summary of reported alerts and found bugs. ......
104. reader();
Bug2
134. }
......
FILE: bison/src/symtab.c
......
254. } FILE: bison/src/reader.c Bug3 775. void symbol_new(void) {
776. symbol_table = hash_initialize(...);
Bug3 + Bug4 711. void reader(void) { ......
...... 781. type_table = hash_initialize(...); Bug4
Reported alerts. With ASan, FIFUZZ reports 255 alerts, 714. symbols_new();
......
Bug4
786. }
......
745. }
including 126 null-pointer dereferences, 126 memory errors
Bug1: main -> uniqstrs_new(86) -> hash_initialize(160) -> calloc(626)
(such as use-after-free and buffer-overflow alerts) and 3 asser- Bug2: main -> muscle_init(87) -> hash_initialize(129) -> calloc(626)
tion failures. Among these alerts, 114 are reported by ASan, Bug3: main -> reader(104) -> symbols_new(714) -> hash_initialize(776) -> calloc(626)
Bug4: main -> reader(104) -> symbols_new(714) -> hash_initialize(781) -> calloc(626)
and 82 are found due to causing crashes. Without ASan,
FIFUZZ reports 298 alerts, including 178 null-pointer deref- Figure 11: Example bugs caused by the same error site.
erences, 117 memory errors and 3 assertion failures. All these
alerts are found due to causing crashes. Indeed, ASan can find adding separate error handling code. The text in each line
memory errors that do not cause crashes. Thus, with ASan, presents the call stack of error site, including the function
FIFUZZ finds more memory errors. However, due to monitor- name and code line number of function call. The four bugs
ing memory accesses, ASan often introduces over 2x runtime are all caused by the failures of the function call to calloc in
overhead [5]. Thus, with ASan, FIFUZZ executes less test hash_initialize, but the failures occur in different calling
cases within given time and some null-pointer dereferences contexts. Besides, the call stacks of Bug 3 and Bug 4 are the
causing crashes are missed. same except for the different calls to hash_initialize in
Alert summary. In Table 6, we summarize the alerts found symbols_new. If a fault is injected into the call to calloc
by FIFUZZ with and without ASan, and identify 317 unique without considering calling context, Bug 3 can be found, but
alerts, including 182 null-pointer dereferences, 132 memory Bug 4 will be missed. The results confirm the advantages of
errors and 3 assertion failures. 313 of them are related to context-sensitive SFI over traditional context-insensitive one.
incorrect error handling caused by occasional errors, and only Bug features. Reviewing the bugs found by FIFUZZ, we
4 alerts are caused by program inputs. Section Appendix find two interesting features. Firstly, among the 46 found bugs
shows 50 randomly-selected alerts. related to incorrect error handling, only 4 are triggered by two
Found bugs. In Table 6, we check the root causes of the or more error points’ failures, and the remaining 42 bugs are
317 reported alerts, and identify 50 new and unique bugs in triggered by only one error point’s failure. The results indicate
terms of their root causes. Specifically, 313 alerts are related that error-handling bugs are often triggered by just one error.
to incorrect error handling, which are caused by 46 bugs. The Secondly, most of found bugs are caused by the case that an
remaining 4 alerts are caused by four bugs that are not in error error is correctly handled in the function containing related er-
handling code. We have reported all these bugs to related ror site but incorrectly handled in this function’s ancestors in
developers. 32 of them have been confirmed, and we are still the call stack. For example in Figure 11, the failure of the func-
waiting for the response of remaining ones. tion call to calloc is correctly handled in hash_initialize,
Error handling bugs. The 46 found bugs related to incor- and hash_initialize returns a null pointer. In this case,
rect error handling are caused by only 18 error sites but in the functions calling hash_initialize make some global
different calling contexts. Most of the error sites are related variables become NULL, but these global variables are still
to occasional errors of memory allocation. Figure 11 shows dereferenced in subsequent execution. Indeed, developers can
such examples of four bugs found in bison, and these bugs often implement correct error handling code in current func-
have different root causes according to our manual check- tions, but often make mistakes in error propagation due to
ing. Additionally, the developer fixes each of these bugs by complex calling contexts of error sites.
 Memory Arbitrary Memory FIFUZZ_insensitive FIFUZZ
Bug type Crash/DoS
corruption read overread Program Useful error Useful error
Alert Bug Alert Bug
Null pointer dereference 36 0 0 0 sequence sequence
Double free 0 5 0 0 vim 689 1 1 1,664 58 12
Use after free 0 1 2 2 bison 108 3 3 289 11 6
Buffer overflow 0 0 0 1 ffmpeg 5 0 0 516 35 12
Free invalid pointer 2 0 0 0 nasm 7 2 1 78 8 1
Assertion failure 1 0 0 0 catdoc 29 2 2 38 2 3
Total 39 6 2 3 clamav 29 1 1 325 103 6
cflow 105 1 1 217 1 1
gif2png+libpng 4 0 0 6 0 1
Table 7: Security impact classified by bug type. openssl 18 0 0 671 80 8
Total 994 10 9 3,804 298 50
FILE: clamav/libclamav/matcher-ac.c
572. void cli_ac_free(struct cli_matcher *root) {
......
Table 8: Results of sensitivity analysis.
577. for (i = 0; i < root->ac_patterns; i++) {
// ''patt'' can be ''new'' given specific ''i''
578. patt = root->ac_pattable[i];

580.
......
mpool_free(root->mempool, patt->virname) // use after free
without considering their calling contexts. We evaluate the
581. if (patt->special) // use after free resulting tool on the 9 tested applications in Table 3, without
......
620. } using any sanitizer. Each application is also tested for three
2413. int cli_ac_addsig(struct cli_matcher *root, ...) { times, and the time limit of each testing is 24 hours. Table 8
......
2835. if ((ret = cli_ac_addpatt(root, new))) { shows the results of the resulting tool (FIFUZZ_insensitive)
......
2839. mpool_free(root->mempool, new); // free ''new'' and FIFUZZ.
2840. return ret;
2841. } Compared to FIFUZZ, the resulting tool generates less
......
2857. } useful error sequences that increase code coverage. Indeed,
some error handling code is only triggered when related error
Figure 12: Two use-after-free bugs found in clamav. sites fail in specific calling contexts and succeed in other
calling contexts, but the resulting tool always makes these
error sites fail and cannot cover such code. The results indicate
5.4 Security Impact of Found Bugs that our context-sensitive SFI method is effective in covering
We manually review the 50 found bugs to estimate their secu- hard-to-trigger error handling code.
rity impact. The results are shown in Table 7, classified by bug Besides, the resulting tool finds 9 bugs (including 8 null-
type, including double-free, use-after-free, buffer-overflow pointer dereferences and 1 memory error). All these bugs
and free-invalid-pointer bugs. The results show that many are also reported by FIFUZZ, but 41 bugs found by FIFUZZ
found bugs can cause serious security problems, such as mem- are missed by this tool, because it does not consider calling
ory corruption and arbitrary read. contexts of error sites. The results indicate that our context-
Figure 12 shows two use-after-free bugs reported in clamav. sensitive SFI method is effective in finding deep bugs in
When the program starts, the function cli_ac_addsig is ex- different calling contexts.
ecuted, and it calls cli_ac_addpatt that can fail and trig-
ger error handling code. In this code, mpool_free is called 5.6 Comparison to Existing Fuzzing Tools
to free the pointer new. When the program exits, the func-
tion cli_ac_free is called, and it executes a loop to handle Many fuzzing approaches have proposed to test infrequently
each element patt in the pointer array root->ac_pattable. executed code and shown promising results in bug detection.
When i is a specific value, patt is an alias of new which Among them, we select four state-of-the-art and open-source
has been freed in cli_ac_addsig, and then patt is used to fuzzing tools to make detailed comparison, including AFL [1],
access patt->virname (a pointer) and patt->special (a AFLFast [13], AFLSmart [50] and FairFuzz [38]. Meanwhile,
condition variable), causing two use-after-free bugs. Once to validate the generality of FIFUZZ, we select 5 common
these bugs are triggered, the attacker can exploit them to con- programs (including nm, objdump, size, ar and readelf) in the
trol the values of patt->virname and patt->special, and Binutils toolset [12] of an old version 2.26 (release in Jan-
thus to corrupt memory and switch the control flow between uary 2016) as tested programs, instead of the 8 applications of
the branches of the if statement in line 581. the lasted versions in the above experiments. We use FIFUZZ
and the four fuzzing tools to fuzz each program without us-
ing any sanitizer for three times, and the time limit of each
5.5 Comparison to Context-Insensitive SFI
fuzzing is 24 hours. For the alerts or crashes reported by these
In FIFUZZ, our context-sensitive SFI method is an impor- tools, we also check their root causes to count unique bugs.
tant technique of covering error handling code in different Figure 13 plots the covered code branches of each tested
calling contexts. To show the value of this technique, we program during fuzzing. Compared to AFL and AFLFast,
modify FIFUZZ by replacing it with a context-insensitive FIFUZZ covers more code branches in all the tested programs,
SFI method, which builds error sequences using error sites, by covering much more error handling code. Compared to
 nm objdump size ar readelf
5000 8000 5000 5000 10000
Covered branches

Covered branches

Covered branches

Covered branches

Covered branches
4000 6400 4000 4000 8000

3000 4800 3000 3000 6000

2000 3200 2000 2000 4000

1000 1600 1000 1000 2000

0 0 0 0 0
0 4 8 12 16 20 24 0 4 8 12 16 20 24 0 4 8 12 16 20 24 0 4 8 12 16 20 24 0 4 8 12 16 20 24
Time (hour) Time (hour) Time (hour) Time (hour) Time (hour)

FIFUZZ AFL AFLFast AFLSmart FairFuzz

Figure 13: Code coverage of FIFUZZ and the four fuzzing tools.

AFL AFLFast AFLSmart FairFuzz FIFUZZ

Program
Null MemErr All Null MemErr All Null MemErr All Null MemErr All Null MemErr All
nm 0 1 1 0 1 1 0 1 1 0 1 1 4 1 5
objdump 0 1 1 0 1 1 1 1 2 0 1 1 2 1 3
size 0 0 0 0 0 0 0 0 0 0 1 1 2 0 2
ar 0 0 0 0 0 0 0 0 0 0 0 0 4 0 4
readelf 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
Total 0 2 2 0 2 2 1 2 3 0 3 3 12 2 14

Table 9: Results of bug detection for comparison.

AFLSmart and FairFuzz, FIFUZZ covers more code branches in the code. The functions strcmp and strstr are examples.
in nm, size and ar, but covers less code branches in objdump However, our static analysis still considers that such func-
and readelf. The main reason is that the fuzzing process of tions can cause error, and identifies the function calls to them
program inputs in FIFUZZ is implemented by referring to as possible error sites, causing false positives. To solve this
AFL, while AFLSmart and FairFuzz use some techniques problem, we plan to analyze the definition and call graph of
to improve mutation and seed selection of fuzzing program each such function, to check whether it can indeed return an
inputs compared to AFL. For this reason, AFLSmart and erroneous value that represents an error.
FairFuzz can cover more infrequently executed code related Secondly, a function can indeed fail and trigger error han-
to inputs than FIFUZZ, though they still miss much error dling code, but some function calls to this function never fail
handling code covered by FIFUZZ. We believe that if we considering their code contexts. This case can occur for some
implement their fuzzing process of program inputs in FIFUZZ, function calls that can cause input-related errors, when all
it can cover more code branches than AFLSmart and FairFuzz possible inputs may have been changed into valid data before
in all the tested programs. these function calls are used. However, our static analysis still
Table 9 shows the results of bug detection. Firstly, the two identifies these function calls as possible error sites, causing
bugs found by AFL and AFLFast are also found by AFLSmart, false positives. To solve this problem, we plan to use sym-
FairFuzz and FIFUZZ. Secondly, AFLSmart and FairFuzz re- bolic execution [36] to analyze code context and calculate the
spectively find one bug missed by AFL, AFLFast and FIFUZZ. constraints for each identified function call.
The one extra bug found by AFLSmart is different from that
found by FairFuzz, as they improve mutation and seed selec-
tion for program inputs in different ways. Finally, FIFUZZ 6.2 False Negatives of Bug Detection
finds 14 bugs, and 12 of them related to error handling code FIFUZZ may miss real bugs in error handling code due to
are missed by AFL, AFLFast, AFLSmart and FairFuzz. three possible reasons:
Firstly, as described in Section 4.1, to avoid injecting re-
6 Discussion peated faults, we only consider library functions for fault
injection. However, some functions defined in the tested pro-
6.1 False Positives of Error-Site Extraction gram can also fail, and they do not call any library function.
Thus, FIFUZZ does not cover the error handling code caused
Our static analysis in Section 4.1 describes how to identify by the failures of the calls to such functions.
possible error sites from the tested program code. However, Secondly, some error sites are executed only when specific
as shown in Section 5.2, our static analysis still has some false program inputs and configuration are provided. In the evalua-
positives in identifying error sites, due to two main reasons: tion, FIFUZZ cannot provide all possible program inputs and
Firstly, some functions that return pointers or integers never configuration. As a result, some error sites may not be exe-
cause errors, even though their return values are often checked cuted, and thus their error handling code cannot be covered.
 Thirdly, we only detect the bugs causing crashes and those 6.5 Exploitability of Error Handling Bugs
reported by ASan. We can use other checkers to detect other
kinds of bugs, such as MSan [41] which detects uninitialized To detect bugs in error handling code, FIFUZZ injects errors
uses, UBSan [57] which detects undefined behaviors, and in specific orders according to calling context. Thus, to ac-
TSan [56] which detects concurrency bugs. tually reproduce and exploit a bug found by FIFUZZ, two
requirements should be satisfied: (1) being able to actually
produce related errors: (2) controlling the occurrence order
6.3 Manual Analyses and time of related errors.
For the first requirement, different kinds of errors can be
FIFUZZ requires two manual analyses in this paper. Firstly,
produced in different ways. We have to manually look into the
we perform a manual study in Section 2.3. This manual study
error-site function to understand its semantics. However, most
is required for gaining the insights into building the automated
of the bugs found in our experiments are related to failures of
static analysis, and we believe that the manual study provides
heap-memory allocations. Thus, an intuitive exploitation way
the most representative and comprehensive results that help
is to exhaustively consume the heap memory, which has been
estimate the causes of errors. Secondly, in Section 5.2, we
used in some approaches [58, 66] to perform attacks.
manually select realistic error sites from the possible error
sites identified by FIFUZZ. This manual selection is required, For the second requirement, as we have the error sequence
as the static analysis of identifying possible error sites still of the bug, we can know when to and when not to produce the
has many false positives. For example, as shown in Table 4, errors. A key challenge here is, when errors are dependent to
we manually check the possible error sites identified by the each other, we must timely produce an error in a specific time
static analysis, and find that only 18.6% of them are real. We window. Similar to exploiting use-after-free bugs [62, 63], if
believe that improving the accuracy of this static analysis can the window is too small, the exploitation may not be feasible.
help reduce such manual work.
7 Related Work
6.4 Performance Improvement
7.1 Fuzzing
The performance of FIFUZZ can be improved in several ways:
Dropping useless error sequences. As shown in Table 5, Fuzzing is a promising technique of runtime testing to detect
FIFUZZ generates many useless error sequences that fail to bugs and discover vulnerabilities. It generates lots of program
increase code coverage. However, they are still used in fault inputs in a specific way to cover infrequently executed code.
injection to execute the tested program, reducing the fuzzing A typical fuzzing approach can be generation-based, mutation-
efficiency. We believe that static analysis can be helpful to based, or the hybrid of them.
dropping these useless error sequences. For example, after Generation-based fuzzing approaches [15,27,59,64] gener-
an original error sequence mutates and generates new error ate inputs according to the specific input format or grammer.
sequences, a static analysis can be used to analyze the code of Csmith [64] is a randomized test-case generator to fuzz C-
the tested program, and infer whether each new error sequence language compilers. According to C99 standard, Csmith ran-
can increase code coverage compared to the original error domly generates a large number of C programs as inputs for
sequence. If not, this error sequence will be dropped, before the tested compiler. These generated programs contain com-
being used in fault injection to execute the tested program. plex code using different kinds of C-language features free of
Lightweight runtime monitoring. As shown in Figure 8, undefined behaviors. LangFuzz [29] is a black-box fuzzing
to collect runtime calling context, FIFUZZ instruments each framework for programming-language (PL) interpreters based
function call to the function defined in the tested program on a context-free grammar. Given a specific language gram-
code and each function definition. Thus, obvious runtime mer, LangFuzz generates many programs in this language as
overhead may be introduced. To reduce runtime overhead, inputs for the tested language interpreter. To improve possi-
FIFUZZ can use some existing techniques of lightweight bility of finding bugs, LangFuzz uses the language grammer
runtime monitoring, such as hardware-based tracing [3, 31] to learn code fragments from a given code base.
and call-path inferring [42]. Mutation-based fuzzing approaches [1, 7, 13, 26, 30, 38,
Multi-threading. At present, FIFUZZ works on simple 51, 65] start from some original seeds, and perform muta-
thread. Referring to AFL, to improve efficiency, FIFUZZ tion of the selected seeds, to generate new inputs, without
can work on multiple threads. Specifically, after an original requirement of specific format or grammer. To improve code
error sequence mutates and generates new error sequences, coverage, these approaches often mutate existing inputs ac-
FIFUZZ can use each new error sequence for fault injection cording to the feedback of program execution, such as code
and execute the tested program on a separate thread. When coverage and bug-detection results. AFL [1] is a well-known
synchronization is required, all the execution results and gen- coverage-guided fuzzing framework, which has been widely-
erated error sequences can be managed in a specific thread. used in industry and research. It uses many effective fuzzing
strategies and technical tricks to reduce runtime overhead fault injection introduces much uncertainty, causing that the
and improve fuzzing efficiency. To improve mutation for in- code coverage is low and many detected bugs are false. To
puts, FairFuzz [38] first identifies the code branches that are solve this problem, some approaches [11, 18, 67] analyze pro-
rarely hit by previously-generated inputs, and then uses a new gram information to guide fault injection, which can achieve
lightweight mutation method to increase the probability of higher code coverage and detect more bugs. ADFI [18] uses
hitting the identified branches. Specifically, this method ana- a bounded trace-based iterative generation strategy to reduce
lyzes the input hitting a rarely hit branch, to identify the parts fault scenario searching, and uses a permutation-based replay
of this input that are crucial to satisfy the conditions of hitting mechanism to ensure the fidelity of runtime fault injection.
that branch; this method never changes the identified parts of To our knowledge, existing SFI-based approaches perform
the input during mutation. only context-insensitive fault injection. Specifically, they in-
Some approaches [6, 45, 50, 60] combine generation-based ject faults based on the locations of error sites in source code,
and mutation-based fuzzing to efficiently find deep bugs. without considering the execution contexts of these error sites.
AFLSmart [50] uses a high-level structural representation Thus, if an fault is constantly injected into an error site, this
of the seed file to generate new files. It mutates on the file- error site will always fail when being executed at runtime.
structure level instead of on the bit level, which can com- However, some error handling code is only triggered when
pletely explores new input domains without breaking file va- related error site fails in a specific calling context but succeeds
lidity. Superion [60] is a grammar-aware and coverage-based in other calling contexts. In this case, existing SFI-based ap-
fuzzing approach to test programs that process structured in- proaches cannot effectively cover such error handling code,
puts. Given the grammar of inputs, it uses a grammar-aware and thus often miss related bugs.
trimming strategy to trim test inputs using the abstract syn-
tax trees of parsed inputs. It also uses two grammar-aware 7.3 Static Analysis of Error Handling Code
mutation strategies to quickly carry the fuzzing exploration.
Existing fuzzing approaches focus on generating inputs to Static analysis can conveniently analyze the source code of the
cover infrequently executed code. However, this way cannot target program without actually executing the program. Thus,
effectively cover error handling code triggered by non-input some existing approaches [28, 32, 33, 37, 53] use static analy-
occasional errors. To solve this problem, FIFUZZ introduces sis to detect bugs in error handling code. EDP [28] statically
software fault injection in fuzzing, and fuzzes injected faults validates the error propagation through file systems and stor-
according to the feedback of program execution. In this way, age device drivers. It builds a function-call graph that shows
it can effectively cover error handling code. how error codes propagate through return values and function
parameters. By analyzing this call graph, EDP detects bugs
about incorrect operations on error codes. APEx [33] infers
7.2 Software Fault Injection API error specifications from their usage patterns, based on a
Software fault injection (SFI) [52] is a classical and widely- key insight that error paths tend to have fewer code branches
used technique of runtime testing. SFI intentionally injects and program statements than regular code.
faults or errors into the code of the tested program, and then Due to lacking exact runtime information, static analysis
executes the program to test whether it can correctly handle often reports many false positives (for example, the false
the injected faults or errors during execution. Many existing positive rate of EPEx is 22%). However, static analysis could
SFI-based approaches [9–11,18,25,39,40,55,67] have shown be introduced in FIFUZZ to drop useless error sequences,
promising results in testing error handling code. which can improve its fuzzing efficiency.
Some approaches [9, 10, 55] inject single fault in each test
case to efficiently cover error handling code triggered by just 8 Conclusion
one error. PairCheck [9] first injects single fault by corrupt-
ing the return values of specific function calls that can fail Error handling code is error-prone and hard-to-test, and ex-
and trigger error handling code, to collect runtime informa- isting fuzzing approaches cannot effectively test such code
tion about error handling code. Then, it performs a statistical especially triggered by occasional errors. To solve this prob-
analysis of the collected runtime information to mine pairs lem, we propose a new fuzzing framework named FIFUZZ, to
of resource-acquire and resouce-release functions. Finally, effectively test error handling code and detect bugs. The core
based on the mined function pairs, it detects resource-release of FIFUZZ is a context-sensitive software fault injection (SFI)
omissions in error handling code. approach, which can effectively cover error handling code in
To cover more error handling code, some approaches [11, different calling contexts to find deep bugs hidden in error
18, 25, 39, 40, 67] inject multiple faults in each test case. Some handling code with complicated contexts. We have evaluated
of them [25, 39, 40] inject random faults, namely they inject FIFUZZ on 9 widely-used C applications. It reports 317 alerts,
faults on random sites or randomly change program data. which are caused by 50 new and unique bugs in terms of their
However, some studies [35, 43, 44] have shown that random root causes. 32 of these bugs have been confirmed by related
developers. The comparison to existing fuzzing tools shows Languages and Analysis for Security (PLAS) (2009),
that, FIFUZZ can find many bugs missed by these tools. pp. 45–57.
FIFUZZ can be still improved in some aspects. Firstly, the
static analysis of identifying possible error sites still has many [9] BAI , J.-J., WANG , Y.-P., L IU , H.-Q., AND H U , S.-M.
false positives. We plan to reduce these false positives us- Mining and checking paired functions in device drivers
ing the ways mentioned in Section 6.1. Secondly, we plan using characteristic fault injection. Information and
to improve FIFUZZ’s performance in some ways, such as Software Technology (IST) 73 (2016), 122–133.
dropping useless error sequences, performing lightweight run-
[10] BAI , J.-J., WANG , Y.-P., Y IN , J., AND H U , S.-M. Test-
time monitoring and exploiting multi-threading mentioned
ing error handling code in device drivers using character-
in Section 6.4. Finally, we only use FIFUZZ to test C pro-
istic fault injection. In Proceedings of the 2016 USENIX
grams at present, and we plan to test the program in other
Annual Technical Conference (2016), pp. 635–647.
programming languages (such as C++ and Java).
[11] BANABIC , R., AND C ANDEA , G. Fast black-box testing
Acknowledgment of system recovery code. In Proceedings of the 7th
European Conference on Computer Systems (EuroSys)
We thank our shepherd, Deian Stefan, and anonymous review- (2012), pp. 281–294.
ers for their helpful advice on the paper. This work was mainly
supported by the China Postdoctoral Science Foundation un- [12] GNU Binutils. http://www.gnu.org/software/binutils/.
der Project 2019T120093. Kangjie Lu was supported in part [13] B ÖHME , M., P HAM , V.-T., AND ROYCHOUDHURY, A.
by the NSF award CNS-1931208. Any opinions, findings, Coverage-based greybox fuzzing as markov chain. In
conclusions or recommendations expressed in this material Proceedings of the 23rd International Conference on
are those of the authors and do not necessarily reflect the Computer and Communications Security (CCS) (2016),
views of NSF. Jia-Ju Bai is the corresponding author. pp. 1032–1043.

References [14] C ABRAL , B., AND M ARQUES , P. Exception handling:

A field study in java and. net. In Proceedings of the 2007
[1] American Fuzzy Lop. http://lcamtuf.coredump.cx/afl/. European Conference on Object-Oriented Programming
(ECOOP) (2007), pp. 151–175.
[2] A collection of vulnerabilities discovered by the AFL
fuzzer. https://github.com/mrash/afl-cve. [15] C HEN , Y., G ROCE , A., Z HANG , C., W ONG , W.-K.,
F ERN , X., E IDE , E., AND R EGEHR , J. Taming com-
[3] ARM System Trace Macrocell (STM). https://commu- piler fuzzers. In Proceedings of the 34th International
nity.arm.com/tools/b/blog/posts/introduction-to-arm-s- Conference on Programming Language Design and Im-
system-trace-macrocell. plementation (PLDI) (2013), pp. 197–208.
[4] ASan: address sanitizer. https://github.com/google/san- [16] Clang: a LLVM-based compiler for C/C++ program.
itizers/wiki/AddressSanitizer. https://clang.llvm.org/.
[5] ASan performance. https://github.com/google/sanitizers/ [17] CLOC: count lines of code. https://cloc.sourceforge.net.
wiki/AddressSanitizerPerformanceNumbers.
[18] C ONG , K., L EI , L., YANG , Z., AND X IE , F. Automatic
[6] A SCHERMANN , C., F RASSETTO , T., H OLZ , T., fault injection for driver robustness testing. In Proceed-
JAUERNIG , P., S ADEGHI , A.-R., AND T EUCHERT, ings of the 2015 International Symposium on Software
D. NAUTILUS: fishing for deep bugs with grammars. Testing and Analysis (ISSTA) (2015), pp. 361–372.
In Proceedings of the 26th Network and Distributed
Systems Security Symposium (NDSS) (2019). [19] CVE-2019-7846. https://nvd.nist.gov/vuln/detail/CVE-
2019-7846.
[7] A SCHERMANN , C., S CHUMILO , S., B LAZYTKO , T.,
G AWLIK , R., AND H OLZ , T. REDQUEEN: fuzzing [20] CVE-2019-2240. https://nvd.nist.gov/vuln/detail/CVE-
with input-to-state correspondence. In Proceedings of 2019-2240.
the 26th Network and Distributed Systems Security Sym-
posium (NDSS) (2019). [21] CVE-2019-1750. https://nvd.nist.gov/vuln/detail/CVE-
2019-1750.
[8] A SKAROV, A., AND S ABELFELD , A. Catch me if you
can: permissive yet secure error handling. In Proceed- [22] CVE-2019-1785. https://nvd.nist.gov/vuln/detail/CVE-
ings of the 4th International Workshop on Programming 2019-1785.
[23] E BERT, F., AND C ASTOR , F. A study on developers’ [35] K IKUCHI , N., YOSHIMURA , T., S AKUMA , R., AND
perceptions about exception handling bugs. In Proceed- KONO , K. Do injected faults cause real failures? a case
ings of the 2013 International Conference on Software study of Linux. In Proceedings of the 25th Interna-
Maintenance (ICSM) (2013), pp. 448–451. tional Symposium on Software Reliability Engineering
Workshops (ISSRE-W) (2014), pp. 174–179.
[24] FFmpeg: a complete, cross-platform solution to record,
convert and stream audio and video. https://ffmpeg.org/. [36] K ING , J. C. Symbolic execution and program testing.
Communications of the ACM 19, 7 (1976), 385–394.
[25] F U , C., RYDER , B. G., M ILANOVA , A., AND W ONNA -
COTT, D. Testing of Java web services for robustness. [37] L AWALL , J., L AURIE , B., H ANSEN , R. R., PALIX , N.,
In Proceedings of the 2004 International Symposium on AND M ULLER , G. Finding error handling bugs in
Software Testing and Analysis (ISSTA) (2004), pp. 23– openssl using Coccinelle. In Proceedings of the 2010
34. European Dependable Computing Conference (EDCC)
(2010), pp. 191–196.
[26] G AN , S., Z HANG , C., Q IN , X., T U , X., L I , K., P EI ,
Z., AND C HEN , Z. CollAFL: path sensitive fuzzing. In [38] L EMIEUX , C., AND S EN , K. FairFuzz: a targeted muta-
Proceedings of the 39th IEEE Symposium on Security tion strategy for increasing greybox fuzz testing cover-
and Privacy (2018), pp. 679–696. age. In Proceedings of the 33rd International Confer-
ence on Automated Software Engineering (ASE) (2018),
[27] G ODEFROID , P., K IEZUN , A., AND L EVIN , M. Y. pp. 475–485.
Grammar-based whitebox fuzzing. In Proceedings
of the 29th International Conference on Programming [39] M ARINESCU , P. D., AND C ANDEA , G. LFI: a practical
Language Design and Implementation (PLDI) (2008), and general library-level fault injector. In Proceedings
pp. 206–215. of the 39th International Conference on Dependable
Systems and Networks (DSN) (2009), pp. 379–388.
[28] G UNAWI , H. S., RUBIO -G ONZÁLEZ , C., A RPACI -
D USSEAU , A. C., A RPACI -D USSEAU , R. H., AND L I - [40] M ENDONCA , M., AND N EVES , N. Robustness test-
BLIT, B. EIO: error handling is occasionally correct. In ing of the Windows DDK. In Proceedings of the 37th
Proceedings of the 6th International Conference on File International Conference on Dependable Systems and
and Storage Technologies (FAST) (2008), pp. 207–222. Networks (DSN) (2007), pp. 554–564.

[29] H OLLER , C., H ERZIG , K., AND Z ELLER , A. Fuzzing [41] MSan: memory sanitizer. https://github.com/google/san-
with code fragments. In Proceedings of the 21st USENIX itizers/wiki/MemorySanitizer.
Security Symposium (2012), pp. 445–458.
[42] M YTKOWICZ , T., C OUGHLIN , D., AND D IWAN , A. In-
[30] Honggfuzz: security oriented fuzzer with powerful anal- ferred call path profiling. In Proceedings of the 24th
ysis options. https://github.com/google/honggfuzz. International Conference on Object Oriented Program-
ming Systems Languages and Applications (OOPSLA)
[31] Intel Processor Tracing (PT). https://software.intel.com/ (2009), pp. 175–190.
en-us/blogs/2013/09/18/processor-tracing.
[43] NATELLA , R., C OTRONEO , D., D URAES , J., AND
[32] JANA , S., K ANG , Y. J., ROTH , S., AND R AY, B. Au- M ADEIRA , H. Representativeness analysis of injected
tomatically detecting error handling bugs using error software faults in complex software. In Proceedings
specifications. In Proceedings of the 25th USENIX Se- of the 40th International Conference on Dependable
curity Symposium (2016), pp. 345–362. Systems and Networks (DSN) (2010), pp. 437–446.

[33] K ANG , Y., R AY, B., AND JANA , S. APEx: automated [44] NATELLA , R., C OTRONEO , D., D URAES , J. A., AND
inference of error specifications for C APIs. In Proceed- M ADEIRA , H. S. On fault representativeness of soft-
ings of the 31st International Conference on Automated ware fault injection. IEEE Transactions on Software
Software Engineering (ASE) (2016), pp. 472–482. Engineering (TSE) 39, 1 (2013), 80–96.

[34] K ERY, M. B., L E G OUES , C., AND M YERS , B. A. Ex- [45] PADHYE , R., L EMIEUX , C., S EN , K., PAPADAKIS , M.,
amining programmer practices for locally handling ex- AND L E T RAON , Y. Semantic fuzzing with Zest. In
ceptions. In Proceedings of the 13th International Work- Proceedings of the 2019 International Symposium on
ing Conference on Mining Software Repositories (MSR) Software Testing and Analysis (ISSTA) (2019), pp. 329–
(2016), pp. 484–487. 340.
[46] Aacsbr: check that sample_rate is not 0 before division. [59] WANG , J., C HEN , B., W EI , L., AND L IU , Y. Skyfire:
http://github.com/ffmpeg/ffmpeg/commit/a50a5ff29e. data-driven seed generation for fuzzing. In Proceedings
of the 38th IEEE Symposium on Security and Privacy
[47] Found bug: libav: divide-by-zero in sbr_make_f_master. (2017), pp. 579–594.
https://blogs.gentoo.org/ago/2016/09/21/libav-divide-
by-zero-in-sbr_make_f_master-aacsbr-c/. [60] WANG , J., C HEN , B., W EI , L., AND L IU , Y. Superion:
grammar-aware greybox fuzzing. In Proceedings of the
[48] CVE-2016-7499. https://nvd.nist.gov/vuln/detail/CVE-
41st International Conference on Software Engineering
2016-7499.
(ICSE) (2019), pp. 724–735.
[49] Frame: fix the error path in av_frame_copy_props.
[61] W EIMER , W., AND N ECULA , G. C. Finding and pre-
http://github.com/ffmpeg/ffmpeg/commit/a53551cba8.
venting run-time error handling mistakes. In Proceed-
[50] P HAM , V.-T., B ÖHME , M., S ANTOSA , A. E., C ACI - ings of the 19th International Conference on Object-
ULESCU , A. R., AND ROYCHOUDHURY, A. Smart Oriented Programming Systems, Languages and Appli-
greybox fuzzing. IEEE Transactions on Software Engi- cations (OOPSLA) (2004), pp. 419–431.
neering (TSE) (2019).
[62] W U , W., C HEN , Y., X U , J., X ING , X., G ONG , X., AND
[51] R AWAT, S., JAIN , V., K UMAR , A., C OJOCAR , L., Z OU , W. FUZE: towards facilitating exploit generation
G IUFFRIDA , C., AND B OS , H. VUzzer: application- for kernel use-after-free vulnerabilities. In Proceed-
aware evolutionary fuzzing. In Proceedings of the 24th ings of the 27th USENIX Security Symposium (2018),
Network and Distributed Systems Security Symposium pp. 781–797.
(NDSS) (2017), pp. 1–14.
[63] X U , W., L I , J., S HU , J., YANG , W., X IE , T., Z HANG ,
[52] ROSENBERG , H. A., AND S HIN , K. G. Software fault Y., AND G U , D. From collision to exploitation: unleash-
injection and its application in distributed systems. In ing use-after-free vulnerabilities in Linux kernel. In
Proceedings of the 23rd International Symposium on Proceedings of the 22nd International Conference on
Fault-Tolerant Computing (FTCS) (1993), pp. 208–217. Computer and Communications Security (CCS) (2015),
pp. 414–425.
[53] S AHA , S., L OZI , J., T HOMAS , G., L AWALL , J. L., AND
M ULLER , G. Hector: detecting resource-release omis- [64] YANG , X., C HEN , Y., E IDE , E., AND R EGEHR , J. Find-
sion faults in error-handling code for systems software. ing and understanding bugs in C compilers. In Proceed-
In Proceedings of the 43rd International Conference ings of the 32nd International Conference on Program-
on Dependable Systems and Networks (DSN) (2013), ming Language Design and Implementation (PLDI)
pp. 1–12. (2011), pp. 283–294.
[54] S HAH , H., G ÖRG , C., AND H ARROLD , M. J. Why [65] Y UN , I., L EE , S., X U , M., JANG , Y., AND K IM , T.
do developers neglect exception handling? In Proceed- QSYM: a practical concolic execution engine tailored
ings of the 4th International Workshop on Exception for hybrid fuzzing. In Proceedings of the 27th USENIX
Handling (WEH) (2008), pp. 62–68. Security Symposium (2018), pp. 745–761.
[55] S USSKRAUT, M., AND F ETZER , C. Automatically find- [66] Z HANG , H., S HE , D., AND Q IAN , Z. Android ion haz-
ing and patching bad error handling. In Proceedings of ard: The curse of customizable memory management
the 2006 European Dependable Computing Conference system. In Proceedings of the 23rd International Confer-
(EDCC) (2006), pp. 13–22. ence on Computer and Communications Security (CCS)
(2016), pp. 1663–1674.
[56] TSan: thread sanitizer. https://github.com/google/san-
itizers/wiki/ThreadSanitizerCppManual. [67] Z HANG , P., AND E LBAUM , S. Amplifying tests to
validate exception handling code. In Proceedings of the
[57] UBSan: undefined behavior sanitizer. https://clang.llvm.
34th International Conference on Software Engineering
org/docs/UndefinedBehaviorSanitizer.html.
(ICSE) (2012), pp. 595–605.
[58] VAN D ER V EEN , V., F RATANTONIO , Y., L INDORFER ,
M., G RUSS , D., M AURICE , C., V IGNA , G., B OS , H., [68] Z UO , C., W U , J., AND G UO , S. Automatically detect-
R AZAVI , K., AND G IUFFRIDA , C. Drammer: deter- ing SSL error-handling vulnerabilities in hybrid mobile
ministic rowhammer attacks on mobile platforms. In web apps. In Proceedings of the 10th ACM Symposium
Proceedings of the 23rd International Conference on on Information, Computer and Communications Secu-
Computer and Communications Security (CCS) (2016), rity (2015), pp. 591–596.
pp. 1675–1689.
Appendix
We randomly select 50 of the 317 alerts reported by FIFUZZ in the 9 tested applications, and show their information in the table.
These 50 alerts are caused by 36 bugs in terms of their root causes.
The column “Error points” shows the call stacks of error points (ErrPtx) that trigger the alert. A call stack presents the information of
each function call in the stack, including the name of the called function and code line number of this function call.
The columns “Source file” and “Line” respectively show the source file name and code line number where the alert occurs.
The column “State” shows the current state of our bug report. “F” means that the bug has been confirmed and fixed; “C” means that the
bug has been confirmed but not fixed yet; “R” means that the bug report has not been replied.

Program Error points Source file Line Alert type State

vim ErrPt1: main -> common_init(173) -> alloc(934) -> lalloc(827) -> malloc(924) message.c 4334 null-pointer dereference F
vim ErrPt1: main -> mch_early_init(115) -> alloc(3212) -> lalloc(827) -> malloc(924) message.c 4334 null-pointer dereference F
ErrPt1: main -> termcapinit(384) -> set_termname(2571) -> set_shellsize(2069) -> screenclear(3466) -> screenalloc(8744)
vim screen.c 8664 null-pointer dereference F
-> lalloc(8495) -> malloc(924)
ErrPt1: main -> termcapinit(384) -> set_termname(2571) -> set_shellsize(2069) -> screenclear(3466) -> screenalloc(8744)
vim screen.c 8664 null-pointer dereference F
-> win_alloc_lines(8507) -> alloc_clear(5085) -> lalloc(851) -> malloc(924)
ErrPt1: main -> vim_main2(444) -> create_windows(728) -> open_buffer(2750) -> ml_open(167) -> ml_new_data(392) ->
vim misc2.c 4446 null-pointer dereference F
mf_new(4015) -> mf_alloc_bhdr(379) -> alloc(898) -> lalloc(827) -> malloc(924)
ErrPt1: main -> common_init(173) -> set_init_1(1010) -> set_options_default(3522) -> set_option_default(3847) ->
vim charset.c 1456 null-pointer dereference F
set_string_option_direct(3769) -> vim_strsave(5976) -> alloc(1279) -> lalloc(827) -> malloc(924)
ErrPt1: main -> common_init(173) -> set_init_1(1010) -> set_options_default(3522) -> set_option_default(3847) ->
vim set_string_option_direct(3769) -> set_string_option_global(5987) -> vim_strsave(6083) -> alloc(1279) -> lalloc(827) charset.c 1456 null-pointer dereference F
-> malloc(924)
ErrPt1: main -> command_line_scan(200) -> alist_add(2495) -> buflist_add(6688) -> buflist_new(3309) ->
vim option.c 8422 null-pointer dereference F
buf_copy_options(2036) -> vim_strsave(11649) -> alloc(1279) -> lalloc(827) -> malloc(924)
ErrPt1: main -> command_line_scan(200) -> save_typebuf(2365) -> alloc_typebuf(1332) -> alloc(1286) -> lalloc(827) ->
vim getchar.c 1313 double free F
malloc(924)
ErrPt1: main -> command_line_scan(200) -> save_typebuf(2365) -> alloc_typebuf(1332) -> alloc(1287) -> lalloc(827) ->
vim getchar.c 1317 double free F
malloc(924)
ErrPt1: main -> init_highlight(413) -> do_highlight(415) -> syn_check_group(859) -> vim_strsave_up(3066) -> lalloc(827) ->
vim highlight.c 871 null-pointer dereference F
malloc(924)
ErrPt1: main -> vim_main2(444) -> create_windows(728) -> open_buffer(2750) -> readfile(233) -> next_fenc(893) ->
vim fileio.c 2320 freeing invalid pointer F
enc_canonize(2789) -> alloc(4323) -> lalloc(827) -> malloc(924)
ErrPt1: main -> vim_main2(444) -> main_loop(903) -> msg_attr(1286) -> msg_attr_keep(122) -> set_vim_var_string(142) ->
vim message.c 1437 use after free F
vim_strsave(7119) -> alloc(1279) -> lalloc(827) -> malloc(924)
ErrPt1: main -> vim_main2(444) -> main_loop(903) -> normal_cmd(1370) -> do_pending_operator(1133) ->
vim ops.c 4559 null-pointer dereference F
op_delete(1816) -> do_join(2079) -> alloc(4557) -> lalloc(827) -> malloc(924)
ErrPt1: main -> vim_main2(444) -> load_start_packages(492) -> do_in_path(2317) -> alloc(1864) -> lalloc(827) ->
malloc(924)
vim message.c 2589 null-pointer dereference F
ErrPt2: main -> vim_main2(444) -> wait_return(680) -> hit_return_msg(1078) -> msg_puts_attr_len(1961) -> alloc(2588) ->
lalloc(827) -> malloc(924)
ErrPt1: main -> source_startup_scripts(432) -> do_source(3051) -> fix_fname(2759) -> FullName_save(4817) ->
vim_FullName(3082) -> mch_FullName(4479) -> fchdir(2589)
vim ErrPt2: main -> vim_main2(444) -> wait_return(680) -> hit_return_msg(1078) -> msg_putchar(1267) -> message.c 2589 null-pointer dereference F
msg_putchar_attr(1369) -> msg_puts_attr(1386) -> msg_puts_attr_len(1961) -> msg_puts_printf(2008) ->
alloc(2588) -> lalloc(827) -> malloc(924)
bison ErrPt1: main -> uniqstrs_new(86) -> hash_initialize(160) -> malloc(605) hash.c 251 null-pointer dereference F
bison ErrPt1: main -> reader(104) -> symbols_new(714) -> hash_initialize(776) -> malloc(605) hash.c 251 null-pointer dereference F
bison ErrPt1: main -> muscle_init(87) -> hash_initialize(129) -> calloc(626) hash.c 251 null-pointer dereference F
bison ErrPt1: main -> generate_states(124) -> allocate_storage(358) -> state_hash_new(168) -> hash_initialize(362) -> calloc(626) hash.c 251 null-pointer dereference F
ErrPt1: main -> tables_generate(152) -> pack_table(802) -> bitset_create(727) -> bitset_alloc(163) -> bitset_init(138) ->
bison vector.c 81 null-pointer dereference F
vbitset_init(88) -> vbitset_resize(989) -> realloc(77)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
ffmpeg dict.c 205 null-pointer dereference R
avformat_alloc_context(1041) -> av_mallocz(151) -> av_malloc(238) -> posix_memalign(87)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3331) -> open_output_file(3277) ->
ffmpeg avformat_alloc_output_context2(2152) -> avformat_alloc_context(151) -> av_mallocz(151) -> av_malloc(238) -> dict.c 205 null-pointer dereference R
posix_memalign(87)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3331) -> open_output_file(3277) -> new_audio_stream(2236)
-> new_output_stream(1859) -> avcodec_alloc_context3(1387) -> init_context_defaults(163) -> av_opt_set(141) ->
ffmpeg options.c 141 assertion failure R
set_string_number(484) -> av_expr_parse_and_eval(292) -> av_expr_parse(751) -> av_malloc(687) ->
posix_memalign(87)
ErrPt1: main -> transcode(4894) -> transcode_step(4692) -> process_input(4638) -> process_input_packet(4518) ->
ffmpeg decode_audio(2619) -> send_frame_to_filters(2337) -> ifilter_send_frame(2270) -> configure_filtergraph(2189) -> avstrings.c 87 null-pointer dereference R
avfilter_graph_parse2(1056) -> parse_filter(427) -> av_get_token(184) -> av_malloc(151) -> posix_memalign(87)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
ffmpeg avformat_find_stream_info(1126) -> avcodec_open2(3674) -> av_mallocz(624) -> av_malloc(238) -> utils.c 491 null-pointer dereference R
posix_memalign(87)
ErrPt1: main -> transcode(4894) -> transcode_step(4692) -> reap_filters(4648) -> init_output_stream(1442) ->
ffmpeg avcodec_open2(3517) -> ff_ac3_float_encode_init(935) -> ff_ac3_encode_init(138) -> allocate_buffers(2481) -> mem.c 223 null-pointer dereference R
ff_ac3_float_allocate_sample_buffers(2331) -> av_mallocz(49) -> av_malloc(238) -> posix_memalign(87)
Program Error points File name Line Bug type State
ErrPt1: main -> transcode(4894) -> transcode_step(4692) -> process_input(4638) -> process_input_packet(4518) ->
decode_audio(2619) -> send_frame_to_filters(2337) -> ifilter_send_frame(2270) -> configure_filtergraph(2189) ->
ffmpeg avfiltergraph.c 583 use after free R
avfilter_graph_config(1109) -> graph_config_formats(1275) -> query_formats(1164) ->
ff_merge_channel_layouts(499) -> av_realloc_array(242) -> av_realloc(202) -> realloc(144)
ErrPt1: main -> transcode(4894) -> transcode_step(4692) -> process_input(4638) -> process_input_packet(4518) ->
decode_audio(2619) -> send_frame_to_filters(2337) -> ifilter_send_frame(2270) -> configure_filtergraph(2189) ->
ffmpeg configure_output_filter(1106) -> configure_output_audio_filter(685) -> choose_channel_layouts(606) -> ffmpeg_filter.c 179 null-pointer dereference R
avio_close_dyn_buf(194) -> avio_flush(1431) -> flush_buffer(241) -> writeout(184) -> dyn_buf_write(163) ->
av_reallocp(1319) -> av_realloc(173) -> realloc(144)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3331) -> open_output_file(3277) -> avio_open2(2558) ->
ffmpeg mem.c 233 double free R
ffio_open_whitelist(1180) -> ffio_fdopen(1169) -> av_strdup(1007) -> av_realloc(256) -> realloc(144)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) -> avformat_open_input(1104)
ffmpeg -> init_input(573) -> io_open_default(438) -> ffio_open_whitelist(124) -> ffio_fdopen(1169) -> av_strdup(1007) -> mem.c 233 double free R
av_realloc(256) -> realloc(144)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> avcodec_open2(3674) -> av_opt_set_dict(634) -> av_opt_set_dict2(1605) ->
av_dict_set(1590) -> av_strdup(87) -> av_realloc(256) -> realloc(144)
ffmpeg decode.c 2059 freeing dangling pointer R
ErrPt2: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> try_decode_frame(3903) -> avcodec_open2(3050) ->
ff_decode_bsfs_init(736) -> av_bsf_alloc(232) -> av_mallocz(86) -> av_malloc(238) -> posix_memalign(87)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> avcodec_open2(3674) -> av_opt_set_dict(634) -> av_opt_set_dict2(1605) ->
av_dict_set(1590) -> av_strdup(87) -> av_realloc(256) -> realloc(144)
ffmpeg aacdec_template.c 2659 null-pointer dereference R
ErrPt2: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> try_decode_frame(3903) -> avcodec_open2(3050) -> aac_decode_init(935) ->
ff_mdct_init(1226) -> ff_fft_init(61) -> av_malloc(224) -> posix_memalign(87)
ErrPt1: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> avcodec_open2(3674) -> av_opt_set_dict(634) -> av_opt_set_dict2(1605) ->
av_dict_set(1590) -> av_strdup(87) -> av_realloc(256) -> realloc(144)
ffmpeg aacdec_template.c 2659 null-pointer dereference R
ErrPt2: main -> ffmpeg_parse_options(4872) -> open_files(3317) -> open_input_file(3277) ->
avformat_find_stream_info(1126) -> try_decode_frame(3903) -> avcodec_open2(3050) -> aac_decode_init(935) ->
ff_mdct_init(1226) -> av_malloc_array(64) -> av_malloc(188) -> posix_memalign(87)
nasm ErrPt1: main -> saa_init(479) -> nasm_malloc(56) -> malloc(75) nasm.c 1909 null-pointer dereference F
nasm ErrPt1: main -> init_labels(476) -> nasm_malloc(563) -> malloc(75) nasm.c 1909 null-pointer dereference F
nasm ErrPt1: main -> saa_init(479) -> nasm_zalloc(47) -> calloc(85) nasm.c 1909 null-pointer dereference F
nasm ErrPt1: main -> init_labels(476) -> hash_init(561) -> alloc_table(66) -> nasm_zalloc(60) -> calloc(85) nasm.c 1909 null-pointer dereference F
catdoc ErrPt1: main -> read_charset(112) -> calloc(93) charsets.c 95 null-pointer dereference R
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_loadftm(5184) -> cli_parse_add(2156) ->
clamav cli_ac_addsig(497) -> cli_ac_addpatt(2835) -> cli_ac_addpatt_recursive(340) -> add_new_node(299) -> matcher-ac.c 578 use after free C
cli_calloc(236) -> calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_loadftm(5184) -> cli_parse_add(2156) ->
clamav cli_ac_addsig(608) -> cli_ac_addpatt(2835) -> cli_ac_addpatt_recursive(340) -> cli_ac_addpatt_recursive(305) -> matcher-ac.c 578 use after free C
add_new_node(299) -> cli_calloc(229) -> calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_load(833) -> cli_loaddbdir(4726) -> cli_load(4581) -> cli_cvdload(4341) ->
cli_tgzload(706) -> cli_load(345) -> cli_loadcbc(4392) -> load_oneldb(2020) -> cli_parse_add(1876) ->
clamav matcher-ac.c 578 use after free C
cli_ac_addsig(497) -> cli_ac_addpatt(2835) -> cli_ac_addpatt_recursive(340) -> add_new_node(299) ->
cli_calloc(236) -> calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_load(833) -> cli_loaddbdir(4726) -> cli_load(4581) -> cli_cvdload(4341) ->
clamav cli_tgzload(706) -> cli_load(345) -> cli_loadcbc(4392) -> cli_initroots(1961) -> cli_bm_init(678) -> cli_calloc(147) -> matcher-bm.c 224 double free C
calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_bytecode_prepare2(5250) -> selfcheck(2683) ->
clamav bytecode.c 1931 null-pointer dereference C
add_selfcheck(2479) -> cli_calloc(2397) -> calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_load(833) -> cli_loaddbdir(4726) -> cli_load(4581) -> cli_cvdload(4341) ->
clamav dsig.c 136 null-pointer dereference C
cli_cvdverify(625) -> cli_versig(566) -> cli_str2hex(131) -> cli_calloc(242) -> calloc(216)
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_loadftm(5184) -> cli_parse_add(2156) ->
clamav cli_ac_addsig(608) -> cli_ac_addpatt(2835) -> cli_ac_addpatt_recursive(340) -> add_new_node(299) -> matcher-ac.c 578 use after free C
cli_realloc(245) -> realloc(235)
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_loadftm(5184) -> cli_parse_add(2156) ->
clamav cli_ac_addsig(497) -> cli_ac_addpatt(2835) -> cli_ac_addpatt_recursive(340) -> cli_ac_addpatt_recursive(305) -> matcher-ac.c 578 use after free C
cli_ac_addpatt_recursive(305) -> insert_list(268) -> cli_realloc(106) -> realloc(235)
ErrPt1: main -> scanmanager(161) -> cl_engine_compile(861) -> cli_bytecode_prepare2(5250) -> selfcheck(2683) ->
clamav bytecode.c 1919 null-pointer dereference C
add_selfcheck(2479) -> cli_realloc2(2348) -> realloc(254)
ErrPt1: main -> scanmanager(161) -> cl_load(833) -> cli_loaddbdir(4726) -> cli_load(4581) -> cli_cvdload(4341) ->
clamav cli_tgzload(706) -> cli_load(345) -> cli_loadcbc(4392) -> cli_initroots(1961) -> cli_ac_init(670) -> cli_malloc(521) -> matcher-ac.c 614 use after free C
malloc(197)
clamav ErrPt1: main -> scanmanager(161) -> scanfile(205) -> cl_scandesc_callback(391) -> scan_common(4324) -> malloc(4128) scanners.c 4129 null-pointer dereference R