Team for Research in
Ubiquitous Secure Technology

Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs
Marjan Aslani, NGA CHUNG, Jason Doherty, Nichole Stockman, William Quach

Citation
Marjan Aslani, NGA CHUNG, Jason Doherty, Nichole Stockman, William Quach. "Comparison of Blackbox and Whitebox Fuzzers in Finding Software Bugs". Talk or presentation, 12, November, 2008.

Abstract
Both blackbox and whitebox fuzzing techniques have been widely used to uncover security vulnerabilities in software applications, but there have been few studies comparing each technique. Our approach was to use Zzuf, a blackbox fuzzer, and Catchconv, a whitebox fuzzer, to generate test cases that were then run on open source and commercial software to compare both fuzzers efciency in terms of the number of unique bugs found per test case. An analysis of our results showed that Zzuf found an average of 2:69 unique errors per 100 unique test cases, while Catchconv found an average of 2:63 unique errors per 100 test cases. In terms of unique errors per total errors, 22 percent of the total errors found by Catchconv were unique, while 0:05 percent of the total errors found by Zzuf were unique. From the analysis of the data we collected, we identified metrics which we suggest for future comparison between fuzzers, but we did not collect enough information to evaluate in our study.

Electronic downloads

Citation formats  
  • HTML
    Marjan Aslani, NGA CHUNG, Jason Doherty, Nichole Stockman,
    William Quach. <a
    href="http://www.truststc.org/pubs/493.html"
    ><i>Comparison of Blackbox and Whitebox Fuzzers in
    Finding Software Bugs</i></a>, Talk or
    presentation,  12, November, 2008.
  • Plain text
    Marjan Aslani, NGA CHUNG, Jason Doherty, Nichole Stockman,
    William Quach. "Comparison of Blackbox and Whitebox
    Fuzzers in Finding Software Bugs". Talk or
    presentation,  12, November, 2008.
  • BibTeX
    @presentation{AslaniCHUNGDohertyStockmanQuach08_ComparisonOfBlackboxWhiteboxFuzzersInFindingSoftware,
        author = {Marjan Aslani and NGA CHUNG and Jason Doherty and
                  Nichole Stockman and William Quach},
        title = {Comparison of Blackbox and Whitebox Fuzzers in
                  Finding Software Bugs},
        day = {12},
        month = {November},
        year = {2008},
        abstract = {Both blackbox and whitebox fuzzing techniques have
                  been widely used to uncover security
                  vulnerabilities in software applications, but
                  there have been few studies comparing each
                  technique. Our approach was to use Zzuf, a
                  blackbox fuzzer, and Catchconv, a whitebox fuzzer,
                  to generate test cases that were then run on open
                  source and commercial software to compare both
                  fuzzers efciency in terms of the number of unique
                  bugs found per test case. An analysis of our
                  results showed that Zzuf found an average of 2:69
                  unique errors per 100 unique test cases, while
                  Catchconv found an average of 2:63 unique errors
                  per 100 test cases. In terms of unique errors per
                  total errors, 22 percent of the total errors found
                  by Catchconv were unique, while 0:05 percent of
                  the total errors found by Zzuf were unique. From
                  the analysis of the data we collected, we
                  identified metrics which we suggest for future
                  comparison between fuzzers, but we did not collect
                  enough information to evaluate in our study. },
        URL = {http://www.truststc.org/pubs/493.html}
    }
    

Posted by Jessica Gamble on 23 Jan 2009.
For additional information, see the Publications FAQ or contact webmaster at www truststc org.

Notice: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright.