FuzzBuilder: Automated building greybox

fuzzing environment for C/C++ library

Joonun Jang Huy Kang Kim
Samsung Research Korea University
Seoul, Republic of Korea Seoul, Republic of Korea
[email protected] [email protected]

ABSTRACT vulnerabilities. Until today, greybox fuzzing has been researched

Fuzzing is an effective method to find bugs in software. Many extensively, and many fuzzers have been implemented, such as
security communities are interested in fuzzing as an automated AFL [35], AFLFast [13], AFLGo [12], VUzzer [24], T-Fuzz [23], An-
approach to verify software security because most of the bugs dis- gora [14], and CollAFL [17]. The fuzzers showed their performance
covered by fuzzing are related to security vulnerabilities. However, by discovering bugs in programs. Table 1 shows the number of
not all software can be tested by fuzzing because fuzzing requires unknown bugs found by each greybox fuzzer.
a running environment, especially an executable. Notably, in the
case of libraries, most of the libraries do not have a relevant exe-
cutable in practice. Thus, state-of-the-art fuzzers have a limitation Table 1: The number of unknown bugs found by greybox
to test an arbitrary library. To overcome this problem, we propose fuzzers
FuzzBuilder to provide an automated fuzzing environment for li-
braries. FuzzBuilder generates an executable that calls library API greybox fuzzer # of bugs
functions to enable library fuzzing. Moreover, any executable gen- AFL more than 370
erated by FuzzBuilder is compatible with existing fuzzers such as AFLFast 12
AFL. We evaluate the overall performance of FuzzBuilder by testing AFLGo 39
open source libraries. Consequently, we discovered unknown bugs VUzzer 8
in libraries while achieving high code coverage. We believe that T-Fuzz 3
FuzzBuilder helps security researchers to save both setup cost and Angora 175
learning cost for library fuzzing. CollAFL 157

CCS CONCEPTS
• Security and privacy → Software security engineering; Greybox fuzzing is well known for its advantage of being able
to test with only binaries. However, most of the bugs in Table 1 are
KEYWORDS in open source software. This implies that greybox fuzzing is also
useful when source code is available. Therefore, it is necessary to
greybox fuzzing, library fuzzing, software development, unit test apply greybox fuzzing to a development process to prevent security
ACM Reference Format: vulnerabilities at an early stage. In practice, large IT companies such
Joonun Jang and Huy Kang Kim. 2019. FuzzBuilder: Automated building as Google and Microsoft enforce fuzzing in their development pro-
greybox fuzzing environment for C/C++ library. In 2019 Annual Computer cess. Notably, Google developed OSS-Fuzz [27], which is a greybox
Security Applications Conference (ACSAC ’19), December 9–13, 2019, San fuzzing service platform for open source software.
Juan, PR, USA. ACM, New York, NY, USA, 11 pages. https://doi.org/10.1145/
Greybox fuzzing requires an execution of a program. This implies
3359789.3359846
that it is not easy to apply greybox fuzzing to libraries. Nonetheless,
libraries need to be tested with greybox fuzzing, as vulnerabilities
1 INTRODUCTION in libraries potentially affect many programs by being reused.
Greybox fuzzing is a test method used to find bugs residing in A simple approach for fuzzing a library is to generate an exe-
software. Security researchers often use greybox fuzzing as an au- cutable that calls library API functions and then fuzzing the gener-
tomated method to find security flaws in programs because most ated executable. The executable enables library fuzzing by calling
of the bugs found by greybox fuzzing can be related to security library API functions with arbitrary input values generated by a
fuzzer. To this end, source code for an executable should be written
Permission to make digital or hard copies of all or part of this work for personal or manually. Generally, writing code can be done by developers or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full citation testers.
on the first page. Copyrights for components of this work owned by others than ACM Developers can write code with in-depth knowledge of a library.
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or a
Thus, a library API can be used correctly, which helps to reduce false
fee. Request permissions from [email protected]. positive bugs. Additionally, various combinations of library API
ACSAC ’19, December 9–13, 2019, San Juan, PR, USA functions can be used, which helps to achieve high code coverage.
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-7628-0/19/12. . . $15.00
However, writing code is an additional task for developers. Fur-
https://doi.org/10.1145/3359789.3359846 thermore, it is difficult for developers to write code for fuzzing

627
because they may not know fuzzing sufficiently. Therefore, devel- Library Executable Bugs
opers need to be educated about fuzzing. These things make it
difficult for developers to support greybox fuzzing in practice. FuzzBuilder Fuzzing researches
Table 2 shows the status of both unit test and fuzzing of the top
10 stars of open source libraries obtained from GitHub by using
Figure 1: Research area of this study
search filter: C, C++, and parse. Source code for fuzzing was found
only in a few project’s repositories, while source code for unit tests
was found in all project’s repository. Even if the number of projects generate executables for library fuzzing, other types of software
is increased up to top 30, 24 projects have source code for unit tests, also can be a target of the proposed method.
while only five projects have source code for fuzzing. This shows For example, daemon, which is a program that runs as a back-
that developers do not support fuzzing sufficiently in practice. ground process, is one of the programs that cannot be tested by
most of the greybox fuzzers as greybox fuzzers require guaran-
Table 2: The status of supporting unit test and fuzzing in teed termination of a program. This is because fuzzers should con-
library projects tinue to execute a program with many input values they generated.
FuzzBuilder can help to do fuzzing a daemon by generating an
project unittest fuzzing executable that calls functions which are defined in the daemon.
libphonenumber ✔ ✗ Although a generated executable and an original program are not
rapidjson ✔ ✗ the same, this is sufficient to provide an opportunity to prevent
http-parser ✔ ✗ security problems by fuzzing several essential functions. In this
redcarpet ✔ ✗ way, FuzzBuilder lends a way to extend a type of target that cannot
grbl ✔ ✗ be tested by greybox fuzzers initially without any modification.
tinyxml2 ✔ ✗ In summary, our contributions are as follows.
cJSON ✔ ✔ • We propose a novel approach to generate executables auto-
yajl ✔ ✗ matically to fuzz libraries.
libpostal ✔ ✗ • For practicality, generated executables are compatible with
pugixml ✔ ✔ various greybox fuzzers such as AFL.
• We implemented this approach as a tool, FuzzBuilder that
is based on LLVM framework. Source code is available at
In the case of testers, testers can write code that is fuzzing-
https://github.com/hksecurity/FuzzBuilder.
friendly. However, it takes much time for testers to understand
library API functions for using them correctly. Especially it is chal- 2 BACKGROUND
lenging for testers to write code to achieve high code coverage
because in-depth knowledge of libraries is required. This way is 2.1 Library Fuzzing
not also practical considering that testers may test more than one A library is a collection of implementation of features that has a well-
program in parallel. defined interface. Various independent programs can use prebuilt
Above all, writing code is a labor-intensive job. Moreover, it is features of libraries developed by other developers. This reuse of
required to maintain this source code whenever library code change. features contributes to the field of computer science because of its
To overcome these problems, we propose an approach to generate high productivity caused by low implementation time and cost.
executables automatically for application of greybox fuzzing to However, vulnerabilities in libraries can be more critical than
libraries. The main idea is to generate executables through static bugs in other software because they can affect various programs
and dynamic analysis of unit tests in projects. that use the libraries. Therefore, it is required to test libraries with
Our research helps developers by eliminating their additional greybox fuzzing to prevent security vulnerabilities preemptively.
task to support greybox fuzzing. Further, testers can do fuzzing An interface of a library, which is referred to as a library API, is
libraries with minimized effort. We implemented this method as defined as a set of functions that a library exports. Library API func-
a tool, FuzzBuilder that is based on LLVM framework [21]. Exe- tions are functions from a library API. A program can use a library
cutables generated by FuzzBuilder can be tested by various grey- by calling its API functions along with parameters. Instructions
box fuzzers such as AFL without any customization. We evaluated in a library are executed based on a value passed as parameters.
FuzzBuilder by measuring code coverage and by discovering un- This means that library fuzzing requires a set of instructions to call
known bugs. library API functions with input values.
To clarify the scope of this study, Fig 1 is shown to display a dif- Libfuzzer [26] is a renowned greybox fuzzer for a library. To
ference between general fuzzing researches and our research. Most use Libfuzzer, developers should define LLVMFuzzerTestOneInput
of the fuzzing researches have focused on discovering bugs in given function, which is used as an interface between a fuzzer and a
executables. They did not consider how executables were prepared. target library. This function has two parameters; the first parameter
Thus, in the case of libraries, if there is no relevant executable for a denotes a buffer address, which stores a random value generated
library, the library cannot be tested by greybox fuzzing. Our study by a fuzzer, and the second parameter denotes a buffer size.
focuses on generating executables that enable any software to be Developers should write a body of this function. When writing
tested by fuzzers. Although the main target of our approach is to this function, it should be included that calling library API functions

628
 1 static bool initflag = false ; 5,000
2 static char * buffer = NULL ;
3 static int buffer_len = 0; 4,000

# of visited lines
4
5 init () { 3,000
6 ... // omitted logic about an initialization
7 initflag = true ;
2,000
8 }
9
10 insert ( char * input , int len ) {
1,000
11 if ( initflag == false ) return ;
12 0
13 buffer = calloc ( len + 1, 1) ; 0 1 2 3 4 5 6
14 memcpy ( buffer , input , len ); time(h)
15 buffer_len = len ; with seeds
16 }
17 without seed
18 parse_A () {
19 ... // omitted logic about a parsing
Figure 3: Line coverage comparison between when seeds are
20 }
21 given and when not given
22 parse_B () {
23 ... // omitted logic about a parsing
24 } be called before calling insert function. Further, insert function is
only used for storing a value passed through a parameter. Actual
usage of the value is defined in parse_A, and parse_B. Therefore,
Figure 2: The example for understanding why function se- to achieve a higher code coverage in this example, it is required to
quence is required to achieve higher code coverage write code for calling two function sequences, init - insert - parse_A,
and init - insert - parse_B.

with parameters of this function. For this, at first, developers should 2.3 Seed in greybox fuzzing
select a base function to pass a value from a fuzzer to a library. A Low test coverage is a representative weakness of greybox fuzzing.
base function is a function that has instructions for loading a value Because greybox fuzzing depends on randomness, it is not easy
to the memory where a library will use. Then, developers should to explore deep program paths. Therefore, it is recommended to
choose other library API functions to be tested jointly for testing provide seeds to a greybox fuzzer to alleviate this problem.
various features of a library. A seed is a sample input. Seeds can explore various and deep
Our suggestion is to automate this job. Thus, generated executa- program paths without any input generation mechanism of grey-
bles should include instructions to do the followings. box fuzzing. Seeds can help to increase test coverage because, for
• Obtaining a value from a fuzzer. exploring more program paths, it is better to manipulate seeds than
• Passing the value to a target library through a base function. to start from scratch. Fig 3 is the result of fuzzing of pkcs12 exe-
• Calling various library API functions related to the base cutable for 6 hours by AFL. When seeds are provided, higher code
function. coverage is achieved from the beginning, and coverage continues
to increase during fuzzing. After 6 hours of fuzzing, code coverage
2.2 Function Sequence of around 10 times higher than that of without seeds was achieved.
Writing code for library fuzzing is initiated by selecting a base Consequently, it is necessary to provide seeds to greybox fuzzing
function. Additionally, calling other functions related to a base for efficiency. Therefore, automation in this study includes generat-
function is desired to achieve sufficient code coverage. For example, ing seeds.
an implementation of a specific feature in a library can be divided
into one or more functions. Further, a specific function in a library 2.4 Unit Test
cannot be tested alone if the function requires a return value of A unit test is conducted to verify whether a program works correctly
other function. In these cases, all of the related functions should be or not. For a successful unit test, it is recommended for a unit test
tested jointly. Thus, it is considered to call a sequence of functions to cover every possible program path at least once. This means that
when writing code for library fuzzing. Han et al. [18] introduced a unit test also needs high code coverage.
a similar concept as an API model which includes two types of A unit test should consider function sequences to explore various
dependency; ordering dependencies and value dependencies in program paths. Further, a unit test needs a variety of test inputs,
their kernel fuzzer. that are used as a parameter to call a specific function. Test inputs
In this paper, we define a function sequence as a set of functions can affect code coverage as a control flow of a function depends on
to be tested jointly. Fig 2 shows that why a function sequence is its parameters.
required to write code for library fuzzing. In other words, if a unit test is written well, it means that various
In this example, a base function is insert function. insert function function sequences and test inputs are already available in the unit
works well only if initflag set to true. Therefore, init function should test. These attributes satisfy the requirements of our suggestion.

629
Therefore, we propose a method to generate executables and seeds Test functions. Users can choose test functions. A test function
by using prepared function sequences and test inputs in unit tests. is one that implements a unit test case, which is a target that is to be
Although this approach requires a project to have a unit test, it is analyzed by FuzzBuilder. There are many functions in source code
practical because most of the projects already have it, as the Table of unit tests, and only some of them are test functions. Therefore,
2 shows. FuzzBuilder needs to know which functions are test functions. If a
unit test is based on a popular unit test framework such as google
2.5 Fuzzable API test [3], test function names have a specific pattern. Hence, it is
We define FA (Fuzzable API) as a base library API function for not challenging to identify them. However, if a unit test is written
library fuzzing. FA is used to pass input values generated by fuzzers without any convention, it is needed to specify their name manually
to a target library. Further, FA is used to extract function sequences because there is no clear way to identify test functions.
and seeds from unit tests. CBS_init and yr_compiler_add_string in Thus, FuzzBuilder supports configurable option for identifying
Table 3 are examples of FA. test functions by specifying their names one by one. However, this
configuration is time-consuming because there are a lot of test
functions in unit tests. Fortunately, most projects use test func-
Table 3: The example of Fuzzable API tion names with a specific prefix or postfix such as test_A, test_B,
A_test, and B_test. Therefore, to alleviate this problem, FuzzBuilder
library function parameters supports asterisk (*).
[1] CBS* cbs
boringssl CBS_init [2] const uint8* data Functions to be skipped. Users can choose functions that need
[3] size_t len to be skipped. Unfortunately, execution of some test functions may
[1] YR_COMPILER* compiler take a lot of time. For example, excessive loops in a test function
yr_compiler can cause this problem. The efficiency of greybox fuzzing decreases
yara [2] const char* rules_string
_add_string if execution speed is slow. It is because greybox fuzzing needs a lot
[3] const char* namespace_
of execution of a program to test it with various input values that
they generated. Thus, this configuration helps in efficient fuzzing
For fuzzing boringssl, it is necessary to call CBS_init function with by skipping such functions.
a buffer address that stores random values and its size through the
second and the third parameter. In the case of yara, it is necessary 3.2 Automated generation of an executable
to call yr_compiler_add_string function with only a buffer address It is assumed that the following two conditions are satisfied to
through the second or the third parameter. generate executables through unit tests.

3 METHODOLOGY • Each test is implemented as a function.

• Each test is independent of each other.
3.1 User Configuration
These conditions are mentioned in the best practice guides of
At first, users should provide unit test files and FA to FuzzBuilder. A JUnit [6], which is a popular unit test framework for Java. More-
unit test file is an LLVM bitcode file whose format is an intermediate over, most of unit tests that are based on google test satisfy these
representation of LLVM framework. Although this step requires conditions. Hence, these conditions do not affect the practicality of
human interaction, this is a relatively simple step that does not the proposed method adversely.
require in-depth knowledge of a target library. Therefore, this does
not affect the overall automation process.
Algorithm 1 The overall process of FuzzBuilder to generate an
FA. In the case of FA, users should provide information about FA executable from unit tests
and its parameters that are used for input values. Some FAs require
1: procedure FuzzBuilder(f unctions)
both a buffer address and its size, while others require only a buffer
2: tests, entry ← preprocess(f unctions)
address as a parameter. Below is an example to specify FAs in Table
3: entry ← insert_interface(entry)
3.
4: for all test ∈tests do
(1) CBS_init 2 3 5: if is_necessary(test) then
(2) yr_compiler_add_string 2 6: test ← insert_operands(test)
The first example shows how to specify CBS_init in user con- 7: else
figuration. In this example, FA name is CBS_init, the second pa- 8: test ← remove_test(test)
rameter is specified for a buffer address of an input value, and the 9: end if
third parameter is specified for its size. The second example is for 10: end for
specifying yr_compiler_add_string. In this example, FA name is 11: modify(entry, tests)
yr_compiler_add_string and the second parameter is specified for
a buffer address of an input value. It is not necessary to provide
a buffer size for yr_compiler_add_string. In this case, a buffer is Algorithm 1 shows the overall process. A detailed description of
considered a string, which is null-terminated. each process is provided in the followings.

630
 3.2.1 preprocess. This process extracts the entry function and • Script files for executing unit testing.
test functions from every function in LLVM bitcode files. Test func- • Extra files in a project repository.
tions are extracted by using specific patterns if a unit test is based Various analyzers are required to extract test inputs from a unit
on a popular test framework. Otherwise which, they are extracted test statically. For example, if test inputs are in C source code,
based on user configuration. An entry function is main function. then static analyzer for C source code is required. However, it is
3.2.2 insert_interface. This process inserts an interface that not practical to prepare all types of the static analyzer. Thus, we
loads input values that are generated by fuzzers to memory. At use a dynamic approach to collect test inputs during a unit test is
first, two new global variables are added to store both an input working.
value and its size. Then, a set of instructions is inserted in the For this, FuzzBuilder instruments FA in a library to store values
extracted entry function to achieve the following goals. of its parameter to a specific file. Figure 5 shows how this instru-
• Obtaining an input value from a fuzzer. mentation works. In this code, input parameter denotes a specific
• Allocating sufficient space to an added global variable. buffer address and size parameter denotes its size. Instrumented
• Copying the input value to the added global variable. instructions are for storing values in input buffer to “file.txt”. By
• Storing an input size in another added global variable. this instrumentation, every test input is collected into a specific
file while a unit test is working. Finally, seed files are generated
stdin or a specific file can be used to obtain an input value from a
by dividing each seed into separated files. The overall process is
fuzzer because most of the greybox fuzzers use them as an interface.
shown in Fig 4.
This feature can ensure the compatibility of generated executables
with popular greybox fuzzers such as AFL.
4 EVALUATION
3.2.3 is_necessary. This process identifies test functions that Experiments in this paper are designed to evaluate FuzzBuilder
include instructions to call a specified FA. This can be achieved by under the following criteria.
traversing instructions in test functions. Then, insert_operands is
performed if a test function has an instruction to call a specified • The efficiency of generated seeds by FuzzBuilder.
FA, otherwise which remove_test is performed. If a test function is • The effectiveness of generated executables by FuzzBuilder
one that specified in Functions to be skipped in User Configuration, for library fuzzing.
remove_test is always performed. • The effectiveness of FuzzBuilder as a bug finding tool.
For the experiment, we chose a few library projects from OSS-
3.2.4 insert_operands. This process changes parameters of FA Fuzz. Each project had executables and seeds to fuzz their libraries.
in test functions into the added global variables in 3.2.2. User Config- Most of the executables and seeds were prepared by developers who
uration that is explained in 3.1 is used to identify which parameters have in-depth knowledge of libraries. FuzzBuilder was evaluated
should be changed. By this process, instructions in a library can be by comparing with these executables and seed files.
executed with an input value generated by a fuzzer. We chose projects under the following conditions.
3.2.5 remove_test. This process removes unnecessary test func- (1) Project should support 32-bit build.
tions. It is necessary to remove these functions because they can (2) Project should support at least one unit test that includes FA
decrease execution speed of generated executables. Execution speed used in fuzzing.
has a significant impact on fuzzing performance because fuzzing (3) Project should have a unit test that satisfies the conditions
needs to be performed with input values as many as possible. It is that are addressed in 3.2.
safe to delete test functions because we have assumed that one test
In this experiment, AFL 2.52b, which is the latest version of
function does not affect the other test functions.
a popular greybox fuzzer was used. To discover unknown bugs
3.2.6 modify. This process stores the instrumented functions efficiently, we built libraries with an address sanitizer [28]. Unfor-
in bitcode files where they were defined. tunately, it seems that AFL does not support a 64-bit executable
with an address sanitizer [10]. Therefore, we chose only projects
These instrumented files are built to an executable according to that satisfy the first condition.
a build process of a project. As a result, an executable is generated The second condition is considered to compare FuzzBuilder with
with an interface to obtain input values from fuzzers. These input OSS-Fuzz. Executables prepared for fuzzing in OSS-Fuzz were based
values are used as parameters of FA. Finally, input values from on a specific FA. To compare FuzzBuilder with them, FuzzBuilder
fuzzers can be used to explore library code to discover bugs. should generate an executable, which is based on the same FA.
Note that, the above processes work during a compilation of unit This means that unit tests in the selected projects should have test
test executables. Therefore, several executables can be generated if functions, which calls the same FA. If such test functions do not exist,
a project has more than one unit test executable. then FuzzBuilder cannot generate an executable for comparison.
Therefore, this condition was considered to choose projects. The
3.3 Automated generation of seed files third condition is to remind the conditions for unit tests that are
Seeds can be extracted from test inputs that are used as parameters addressed in 3.2.
of FA in unit tests. Most of the projects usually store test inputs in Four projects were selected from OSS-Fuzz under these condi-
the following way for their unit tests. tions. They are c-ares [2], expat [1], boringssl [8], and yara [4]. Table
• Source code. 4 lists commits of each project used in this experiment.

631
 User Instrumented Unit test
FuzzBuilder Linker Logs
Configuration Library Code Executable

Unit test
Library Code Seed files
Objects

Figure 4: Overall process to generate seed files

1 FA ( char * input , size_t size , ...) { Table 5: Prepared executables in OSS-Fuzz and their ID and
2 int fd = open (" file . txt " , ...) ; // instrumented FA
3 write ( fd , input , size ); // instrumented
4 close ( fd ) ; // instrumented
5 ... project c-ares
6 } ID executable FA
c1 ares_create_query_fuzzer ares_create_query
c2 ares_parse_reply_fuzzer ares_parse_*
Figure 5: The example of instrumentation for seed genera-
project expat
tion
ID executable FA
e1 parse_ISO_8859_1_fuzzer XML_Parse
Table 4: Tested projects and commit number e2 parse_US_ASCII_fuzzer XML_Parse
e3 parse_UTF_16BE_fuzzer XML_Parse
project commit e4 parse_UTF_16_fuzzer XML_Parse
OSS-Fuzz a55a1276d9e0c453f588160b7e3581cdf6236013 e5 parse_UTF_16LE_fuzzer XML_Parse
c-ares a9c2068e25a107bf535b1fc988eec47384b86dc6 e6 parse_UTF_8_fuzzer XML_Parse
expat 39e487da353b20bb3a724311d179ba0fddffc65b project boringssl
boringssl d2a0ffdfa781dd6fde482ccb924b4a756731f238
ID executable FA
yara a3784d3855029bd0ad24071e72746cc0c31b8cba
b1 bn_div CBS_init
b2 bn_mod_exp CBS_init
b3 client CBS_init
Line coverage and the number of discovered bugs are used as met-
b4 dtls_client CBS_init
rics for evaluating FuzzBuilder. These two metrics are commonly
b5 dtls_server CBS_init
used to evaluate fuzzing performance in recent fuzzing researches
b6 pkcs12 CBS_init
[20]. Line coverage was measured by llvm-cov gcov. Note that, only
b7 pkcs8 CBS_init
library code coverage was measured because each executable is
b8 read_pem BIO_new_mem_buf
generated from different test code.
b9 server CBS_init
This experiment was held on Intel (R) Core (TM) i7-9700K CPU,
b10 session SSL_SESSION_from_bytes
Debian GNU / Linux 9.5 OS.
b11 spki CBS_init
b12 ssl_ctx_api CBS_init
4.1 Automatically generated seeds
project yara
For this experiment, we collected all FAs, which were used in both
unit tests and fuzzing. Then, we generated seed files based on the ID executable FA
collected FAs by using FuzzBuilder. We then used AFL-cmin, which y1 dex_fuzzer yr_rules_scan_mem
is a corpus minimization tool provided along with AFL to remove y2 dotnet_fuzzer yr_rules_scan_mem
duplicate seeds. Duplicated seeds are ones that explore the same y3 elf_fuzzer yr_rules_scan_mem
path in a program. FuzzBuilder can generate such duplicated seeds y4 pe_fuzzer yr_rules_scan_mem
because seeds are originally used as test inputs for unit tests. y5 rules_fuzzer yr_compiler_add_string
We measured line coverage of libraries when seed files of FuzzBui-
lder and OSS-Fuzz were used, respectively. Table 5 provides informa-
tion about executables used in this experiment. executable column
denotes executable names provided by OSS-Fuzz. FA column de- prepared seed in OSS-Fuzz. Thus, we provided an arbitrary seed
notes FA names. Because there are many executables and their to OSS-Fuzz because AFL requires at least one seed file to work.
names are quite long, we have assigned an ID to each of them to re- Therefore, the executables of expat are considered to have 1 seed
fer to them conveniently. Note that ares_parse_* which is the FA of in Table 6.
c2, refers to an abbreviation of the 9 FAs starting with ares_parse_. The result shows that seeds generated by FuzzBuilder achieved
Fig 6 shows line coverage of seeds from FuzzBuilder and OSS- higher code coverage for 15 executables and a slightly lower one
Fuzz respectively. Note that, in the case of expat, there was no for 4 executables when compared to OSS-Fuzz. This result infers

632
 # of visited lines

600 1,000 2,000

10,000

400
500 5,000 1,000

200
0 0 0
c1 c2 e1 e2 e3 e4 e5 e6 b1 b2 b3 b4 b5 b6 b7 b8 b9 b10 b11 b12 y1 y2 y3 y4 y5
test programs
FuzzBuilder
OSS-Fuzz

Figure 6: Line coverage comparison when seed files from FuzzBuilder and OSS-Fuzz are used respectively

Table 6: Comparison of the number of seed files by OSS-Fuzz generated by FuzzBuilder and that prepared in OSS-Fuzz. Because
and FuzzBuilder duplicated seeds were removed, the number of seeds led to higher
line coverage in most cases.
# of seed files Of course, a large number of seed files do not always achieve
program
OSS-Fuzz FuzzBuilder high code coverage like b7, y2 and y4 because paths in AFL are
c1 6 12 based on basic blocks [36]. The number of basic blocks is not the
c2 25 57 same as the number of code lines.
e1, e2 1 14 It is evident that seeds prepared in OSS-Fuzz outperformed in b2,
e3, e4 1 6 b3, b4, b5, and b9. This is because of the role of CBS_init. CBS_init,
e5 1 7 is only used to store a value in a specific memory structure defined
e6 1 13 in a library. There is no other instruction for processing this data.
b1 42 12 How this data is used depends on which function is called along
b2 57 6 with this data structure. This means that a format of an input value
b3 281 7 depends on a function sequence, not FA. Unfortunately, a unit test
b4 85 6 did not have the same function sequence that is used in fuzzing of
b5 84 6 OSS-Fuzz. Therefore, FuzzBuilder cannot extract test inputs for the
b6 4 60 function sequence.
b7 14 39
b8 41 13
b9 269 8 4.2 Automatically generated executables
b10 2 13 Additionally, we assigned an ID to each executable that is generated
b11 8 65 by FuzzBuilder as shown in Table 7. In this table, executable column
b12 87 153 denotes executable names of unit tests. Several executables can be
y1 6 15 generated from several unit test executables even if they are based
y2 3 14 on the same FA such as fb2 and fb3. Further, several executables
y3 15 15 can be generated from one unit test executable, if the executables
y4 8 15 are based on different FA such as fb1 and fb3.
y5 3 216 Test sets for comparing FuzzBuilder with OSS-Fuzz were grouped
by FA. Table 8 shows executables that were included in each test set.
For evaluation, we did fuzzing each executable for 6 hours. However,
that seeds generated by FuzzBuilder can help in efficient fuzzing
if the number of executables differs between FuzzBuilder and OSS-
while satisfying practicality.
Fuzz like T3, more time is allocated to the one with less number of
This result is obtained because FuzzBuilder can generate a higher
executables. For example, in the case of T3, total fuzzing time of
number of seed files than the number of seeds that prepared in OSS-
OSS-Fuzz is six times more than FuzzBuilder. For fair evaluation,
Fuzz. Unit tests have been established better than fuzzing in the
we should do fuzzing fe1 in T3 for 36 hours. The policy of time
field of software testing as Table 2 shows. Thus, developers know
allocation in this experiment is reasonable because several function
how unit tests work and what has to be done to make unit tests
sequences exist in one executable in FuzzBuilder, while they are
successful. This implies that developers prepare many test inputs
distributed over several executables in OSS-Fuzz. In this way, each
for unit tests because it is one of the things should be satisfied for
function sequence is tested under the same condition. Exact fuzzing
successful unit tests. Table 6 shows the number of seed files that

633
Table 7: Generated executables by FuzzBuilder and their ID while an executable is working. callable column denotes the number
and FA of all library API functions in the libraries that are specified in the
library column. In the case of c-ares and expat, unit tests called
project c-ares most of library API functions. Further, at least 44% of library API
ID executable FA functions were called while unit tests are working. This implies
fc1 ares_test ares_create_query that unit tests can give a chance to explore various parts of library
fc2 ares_test ares_parse_* code.
project expat Table 10 shows how many library API functions were called
ID executable FA during each executable generated by FuzzBuilder is being tested.
fe1 runtests XML_Parse The number of library API functions that are called by executa-
bles generated by FuzzBuilder is lower than that by unit tests be-
project boringssl
cause FuzzBuilder deletes test functions that are not related to
ID executable FA FA. Nonetheless, FuzzBuilder still provides an opportunity to call
fb1 crypto_test CBS_init various library API functions for library fuzzing without in-depth
fb2 ssl_test BIO_new_mem_buf knowledge of libraries.
fb3 crypto_test BIO_new_mem_buf
fb4 ssl_test SSL_session_from_bytes
Table 9: The number of library API functions that are called
project yara
by unit test
ID executable FA
fy1 test-api yr_rules_scan_mem
unit test
fy2 test-api yr_compiler_add_string project library called callable
executable
c-ares libcares.so ares_test 58 61
time is specified in parentheses in Table 8 for executables that are expat libexpat.so runtests 79 84
assigned more than 6 hours. libcrypto.so crypto_test
boringssl 1220 2727
libssl.so ssl_test
Table 8: Test sets for line coverage comparison of libraries yara libyara.so test-api 47 79
by using executables

set FuzzBuilder OSS-Fuzz FA

T1 fc1 c1 ares_create_query Table 10: The number of library API functions that are called
T2 fc2 c2 ares_parse_* by executables generated by FuzzBuilder
e1, e2, e3
T3 fe1 (36) XML_Parse
e4, e5, e6 FuzzBuilder
project library called callable
b1, b2, b3 executable
b4, b5, b6 c-ares libcares.so fc1, fc2 23 61
T4 fb1 (60) CBS_init
b7, b9, b11 expat libexpat.so fe1 70 84
b12 libcrypto.so fb1, fb2
T5 fb2, fb3 b8 (12) BIO_new_mem_buf boringssl 707 2727
libssl.so fb3, fb4
T6 fb4 b10 SSL_SESSION_from_bytes yara libyara.so fy1, fy2 44 79
y1, y2, y3
T7 fy1 (24) yr_rules_scan_mem
y4
T8 fy2 y5 yr_compiler_add_string Naturally, coverage of FuzzBuilder may include coverage that
is not affected by fuzzing directly because a test function may call
Fig 7 shows line coverage change during 6 hours of fuzzing for library API functions that are not directly related to FA. This is
T1, T2, T6, and T8. Fig 8 shows results for T3, T4, T5, and T7, whose because FuzzBuilder copies the whole body of test functions that
executables were tested for different time ranges. In these sets, most have FA. However, coverage of FuzzBuilder still provides a valid
of the executables that are generated by FuzzBuilder can do fuzzing opportunity for library fuzzing because such API functions can
libraries with higher code coverage than OSS-Fuzz. affect exploration of library code or discovering bugs potentially
In T7, line coverage of FuzzBuilder is lower than OSS-Fuzz. How- by changing internal state.
ever, this result is acceptable, considering the advantage of FuzzBuil-
der. The result of T4 is because of CBS_init function, which can be 4.3 Discovered vulnerabilities
explained in the same way as 4.1. We added two projects from GitHub, cJSON [5] and mpc [7], to
High coverage of FuzzBuilder is based on high coverage of unit discover unknown bugs using AFL and FuzzBuilder. As projects
tests. Table 9 shows how many library API functions were called in OSS-Fuzz have been fuzzed significantly, other projects need
during each unit test executable is working. In this table, called to be fuzzed to discover unknown bugs under limited resource.
column denotes the number of library API functions that are called Moreover, this can be an example to show that testers can easily

634
 500 1,260 14,700 7,200
# of visited lines

400 1,050 12,250 6,000

840 9,800 4,800
300
630 7,350 3,600
200 4,900 2,400
420
100 210 2,450 1,200
0 0 0 0
0 1 2 3 4 5 6 0 1 2 3 4 5 6 0 1 2 3 4 5 6 0 1 2 3 4 5 6
time(h) time(h) time(h) time(h)
(a) T1 (b) T2 (c) T6 (d) T8

FuzzBuilder
OSS-Fuzz

Figure 7: Line coverage comparison between FuzzBuilder and OSS-Fuzz for 6 hours fuzzing

20,000 Input Generation Target Program Execution

FuzzBuilder
OSS-Fuzz
15,000 Update Seeds Execution Analysis
# of visited lines

Figure 9: General process of greybox fuzzing

10,000

5,000 Especially, heap buffer overflow found in expat is an impressive

result because it is a bug that OSS-Fuzz could not find even though
it has used a large resource. The bug was found because FuzzBuilder
0 could generate an executable that includes a function sequence to
T3 T4 T5 T7 reach a state where the bug could be disclosed. Other bugs are
test set significant because they have not found owing to a difficulty in
applying greybox fuzzing.
Figure 8: Line coverage comparison between FuzzBuilder
and OSS-Fuzz for different time ranges fuzzing 5 RELATED WORKS
5.1 Greybox fuzzing
Table 11: Tested projects from GitHub
We will address a simple background of greybox fuzzing in this
section. Although there are various types of greybox fuzzing, we
project commit will focus on evolutionary mutation-based greybox fuzzing such as
cJSON 08103f048e5f54c8f60aaefda16761faf37114f2 AFL.
mpc b31e02e427f55d4ce69c33ed9936a1b396628440 Fuzzing is a dynamic test that finds bugs in programs with ar-
bitrary input values. Greybox fuzzing uses lightweight program
Table 12: Found bugs via automatically generated executa- analysis to generate arbitrary input values efficiently for exploring
bles various program paths. Generally, greybox fuzzing works, as shown
in Fig 9.
project bug type Greybox fuzzing generates arbitrary input values based on pro-
expat heap buffer overflow vided seeds. When a program terminates, greybox fuzzing analyzes
cJSON NULL dereference execution information such as basic block coverage and then up-
stack buffer overflow dates seeds if a generated input value covered new coverage. These
mpc steps continue to repeat.
heap buffer overflow
It is difficult for greybox fuzzing to achieve high code coverage
if a program requires a complex structure for its input value. There-
apply fuzzing to library projects by using FuzzBuilder without in- fore, seeds that can explore various program paths are used to solve
depth knowledge of libraries. Project names and commits are shown this problem.
in Table 11. Seeds that are used for fuzzing have been researched actively:
Consequently, we found four bugs from three projects, and all which seeds should get more energy [29, 32], how to generate seeds
the bugs we found are summarized in Table 12. efficiently [31, 33], and seed optimization [25].

635
 Some researches suggested new input generation mechanisms code. However, as future work, we will automate this functionality
to solve this problem: Angora[14], VUzzer[24], lafintel [9], steelix for a fully automated process.
[22], eclipser [15], TIFF [19], memfuzz [16]. Additionally, there are
hybrid approaches that use both fuzzing and symbolic execution Optimization of generated executable. FuzzBuilder eliminates
[30, 34]. However, it is still useful to provide seeds because they help unnecessary test functions when generating executables from unit
to achieve sufficient code coverage without any runtime resources, tests. However, there are many instructions that are not related to
in contrast to the above mechanisms. fuzzing in generated executables. It is because most of the instruc-
Fuzzers should find bugs as many and as fast as possible to tions were copied from unit tests directly. Further, even necessary
increase the efficiency of greybox fuzzing. One of the ways to test functions may include calls to library API functions that are
achieve this is increasing the number executed instructions during not directly related to FA. These unnecessary parts slow down exe-
fuzzing. If more instructions are executed, then a larger part of cution speed, which affects fuzzing performance. As future work,
code is visited, thereby increasing opportunities to discover bugs. we will consider a transformation of unit tests to generate a more
Thus, the number of visited code lines is measured to evaluate efficient executable for library fuzzing.
the performance of greybox fuzzing along with the number of
discovered bugs. Errors in unit test. Unit test functions that implement test
Most of the greybox fuzzers require an executable and the ex- cases may use an instruction to terminate program forcefully, such
ecutable should be guaranteed to terminate within a short time. as assert and abort for handling unexpected results. Some test func-
Greybox fuzzers should execute a program with as many input tions do not consider an unexpected value; hence, many bugs may
values as possible to explore various program paths. Thus, execu- arise if an unexpected value is given. Unfortunately, most of the
tion time is significant as it determines how many input values values from fuzzers are an unexpected value. Thus, these cases
can be tested within a specific time. These make it challenging to frequently occur during fuzzing and can cause two problems. First,
apply greybox fuzzing to some programs, such as a library or a because of forced termination, most of the instructions in unit tests
server program which is waiting a request. Hence, an additional may not be executed sufficiently. Second, fuzzers will recognize
executable should be prepared to apply greybox fuzzing and addi- such termination as a bug. As bugs in test code may not be a bug
tional executables should include calling functions that are defined of a target library, this can be considered as a false alarm. One of
in a target program and terminate within a short time. the advantages of greybox fuzzing is low false alarms. Therefore,
this topic should be considered to get benefits of greybox fuzzing.
5.2 OSS-Fuzz
Expansion of input value types. FuzzBuilder considers an in-
OSS-Fuzz is a fuzzing service framework that was managed by
put parameter type as char pointer. This data type is sufficient to
Google. This framework was released to share open source projects
show the performance of the proposed method. This is because
with a successful fuzzing experience of chrome component. Until
many functions for parsing, a primary test target of greybox fuzzing,
today, OSS-Fuzz has tested more than 100 open source projects and
receive input values in the form of char pointer data type. Further,
has found more than 10000 bugs.
most of the fuzzing in OSS-Fuzz use this data type for passing
To register an open source project to OSS-Fuzz, developers in
random input to a library. However, input value can be a string
charge of a project should write code for fuzzing. The written source
data type in C++, or any user-defined type. Fuzzers have a higher
code is stored in OSS-Fuzz, and then it is distributed to Clusterfuzz
chance to explore various paths of libraries if more data types are
[11] environment. Clusterfuzz is a distributed environment where
supported.
fuzzing is performed. OSS-Fuzz uses Libfuzzer as a fuzzer.
Libfuzzer is a greybox fuzzer for a library, that allows developers
to write code for fuzzing by using a simple interface. The written 7 CONCLUSION
source code is built to an executable with Libfuzzer code. Fuzzing In this study, we proposed a method for applying greybox fuzzing
starts by executing this executable. to a library. We implemented this method as a tool, FuzzBuilder,
Although Libfuzzer provides a way for library fuzzing, source which is based on LLVM framework. Executables generated by
code for fuzzing should be written manually by one who has in- FuzzBuilder can be used in any other fuzzers such as AFL. This
depth knowledge of a target project. These make it challenging to method helps developers by eliminating writing and maintenance
use Libfuzzer in practice because it requires an understanding of a of source code for fuzzing. Additionally, this method helps testers
library to write code correctly, understanding of fuzzing to write to perform greybox fuzzing without in-depth knowledge of a target
code that is fuzzing-friendly, and maintenance of source code. program. Further, seeds can be generated by this method for effi-
cient greybox fuzzing. Thus, FuzzBuilder helps to enhance software
6 DISCUSSION security by applying greybox fuzzing to various types of a program
The followings are discussion topics for generating an executable easily.
more precisely.
Automated specification of FA. FA is chosen manually in this
ACKNOWLEDGMENTS
study. User intervention is required to provide information about FA This work was supported under the framework of international
to FuzzBuilder. Fortunately, it is not difficult to select them because cooperation program managed by National Research Foundation
various information is available in the form of manual or example of Korea(No.2017K1A3A1A17092614).

636
REFERENCES [32] Maverick Woo, Sang Kil Cha, Samantha Gottlieb, and David Brumley. 2013.
[1] 1997. expat. https://github.com/libexpat/libexpat Scheduling black-box mutational fuzzing. In Proceedings of the 2013 ACM SIGSAC
[2] 2003. c-ares. https://github.com/c-ares/c-ares conference on Computer & communications security. ACM, 511–522.
[3] 2008. Google Test. https://github.com/google/googletest [33] Wei You, Xuwei Liu, Shiqing Ma, David Perry, Xiangyu Zhang, and Bin Liang.
[4] 2008. yara. https://github.com/VirusTotal/yara 2019. SLF: Fuzzing without Valid Seed Inputs. In Proceedings of the 41st Interna-
[5] 2009. cJSON. https://github.com/DaveGamble/cJSON tional Conference on Software Engineering, ICSE, Vol. 2019.
[6] 2012. JUnit Best Practices Guide. https://howtodoinjava.com/best-practices/ [34] Insu Yun, Sangho Lee, Meng Xu, Yeongjin Jang, and Taesoo Kim. 2018. {QSYM }:
unit-testing-best-practices-junit-reference-guide/ A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing. In 27th
[7] 2013. mpc. https://github.com/orangeduck/mpc {USENIX } Security Symposium ( {USENIX } Security 18). 745–761.
[8] 2014. boringssl. https://github.com/google/boringssl [35] Michal Zalewski. 2014. American Fuzzy Lob. http://lcamtuf.coredump.cx/afl/
[9] 2016. lafintel. https://lafintel.wordpress.com/ [36] Michal Zalewski. 2015. american fuzzy lop technical whitepaper.
[10] 2017. notes for asan. https://github.com/mirrorer/afl/blob/master/docs/notes_ http://lcamtuf.coredump.cx/afl/technical_details.txt (2015).
for_asan.txt
[11] 2019. clusterfuzz. https://github.com/google/clusterfuzz
[12] Marcel Böhme, Van-Thuan Pham, Manh-Dung Nguyen, and Abhik Roychoudhury.
2017. Directed greybox fuzzing. In Proceedings of the 2017 ACM SIGSAC Conference
on Computer and Communications Security. ACM, 2329–2344.
[13] Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2017. Coverage-
based greybox fuzzing as markov chain. IEEE Transactions on Software Engineering
(2017).
[14] Peng Chen and Hao Chen. 2018. Angora: Efficient fuzzing by principled search.
In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 711–725.
[15] Jaeseung Choi, Joonun Jang, Choongwoo Han, and Sang Kil Cha. 2019. Grey-
box Concolic Testing on Binary Code. In International Conference on Software
Engineering (ICSE).
[16] Nicolas Coppik, Oliver Schwahn, and Neeraj Suri. 2019. MemFuzz: Using Memory
Accesses to Guide Fuzzing. In 2019 12th IEEE Conference on Software Testing,
Validation and Verification (ICST). IEEE, 48–58.
[17] Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and
Zuoning Chen. 2018. CollAFL: Path sensitive fuzzing. In 2018 IEEE Symposium
on Security and Privacy (SP). IEEE, 679–696.
[18] HyungSeok Han and Sang Kil Cha. 2017. Imf: Inferred model-based fuzzer. In
Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications
Security. ACM, 2345–2358.
[19] Vivek Jain, Sanjay Rawat, Cristiano Giuffrida, and Herbert Bos. 2018. TIFF: Using
Input Type Inference To Improve Fuzzing. In Proceedings of the 34th Annual
Computer Security Applications Conference. ACM, 505–517.
[20] George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018.
Evaluating fuzz testing. In Proceedings of the 2018 ACM SIGSAC Conference on
Computer and Communications Security. ACM, 2123–2138.
[21] Chris Lattner and Vikram Adve. 2004. LLVM: A compilation framework for
lifelong program analysis & transformation. In Proceedings of the international
symposium on Code generation and optimization: feedback-directed and runtime
optimization. IEEE Computer Society, 75.
[22] Yuekang Li, Bihuan Chen, Mahinthan Chandramohan, Shang-Wei Lin, Yang Liu,
and Alwen Tiu. 2017. Steelix: program-state based binary fuzzing. In Proceedings
of the 2017 11th Joint Meeting on Foundations of Software Engineering. ACM,
627–637.
[23] Hui Peng, Yan Shoshitaishvili, and Mathias Payer. 2018. T-Fuzz: fuzzing by
program transformation. In 2018 IEEE Symposium on Security and Privacy (SP).
IEEE, 697–710.
[24] Sanjay Rawat, Vivek Jain, Ashish Kumar, Lucian Cojocar, Cristiano Giuffrida,
and Herbert Bos. 2017. VUzzer: Application-aware Evolutionary Fuzzing.. In
NDSS, Vol. 17. 1–14.
[25] Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David
Warren, Gustavo Grieco, and David Brumley. 2014. Optimizing seed selection for
fuzzing. In 23rd {USENIX } Security Symposium ( {USENIX } Security 14). 861–875.
[26] Kosta Serebryany. 2016. Continuous fuzzing with libfuzzer and addresssanitizer.
In 2016 IEEE Cybersecurity Development (SecDev). IEEE, 157–157.
[27] Kostya Serebryany. 2017. OSS-Fuzz-Google’s continuous fuzzing service for open
source software. (2017).
[28] Konstantin Serebryany, Derek Bruening, Alexander Potapenko, and Dmitriy
Vyukov. 2012. AddressSanitizer: A fast address sanity checker. In Presented as
part of the 2012 {USENIX } Annual Technical Conference ( {USENIX } {ATC } 12).
309–318.
[29] Lingyun Situ, Linzhang Wang, Xuandong Li, Le Guan, Wenhui Zhang, and Peng
Liu. 2019. Energy distribution matters in greybox fuzzing. In Proceedings of the
41st International Conference on Software Engineering: Companion Proceedings.
IEEE Press, 270–271.
[30] Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang,
Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna.
2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution.. In
NDSS, Vol. 16. 1–16.
[31] Junjie Wang, Bihuan Chen, Lei Wei, and Yang Liu. 2017. Skyfire: Data-driven
seed generation for fuzzing. In 2017 IEEE Symposium on Security and Privacy (SP).
IEEE, 579–594.

637