FuzzBench: symsan report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
symsan 98.40
fuzzolic 97.06
aflplusplus 96.60
symqemu 96.00
honggfuzz 95.93
symcc 95.24
eclipser 93.32
entropic 92.91
aflsmart 90.83
afl 90.51
mopt 88.71
libfuzzer 87.85
aflfast 86.54
lafintel 84.91
fairfuzz 83.60
By avg. rank
average rank
fuzzer
fuzzolic 4.03
symqemu 4.71
symsan 4.75
aflplusplus 4.82
symcc 4.85
honggfuzz 6.91
entropic 7.26
eclipser 7.53
afl 7.79
aflsmart 8.29
mopt 9.09
libfuzzer 11.00
aflfast 11.88
fairfuzz 11.97
lafintel 12.38
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    symsan fuzzolic aflplusplus honggfuzz symqemu symcc eclipser entropic aflsmart afl mopt libfuzzer aflfast lafintel fairfuzz
    FuzzerMedian 96.39 96.86 95.53 96.71 96.80 95.81 93.79 92.78 94.82 94.71 94.64 86.75 93.29 89.15 84.92
    FuzzerMean 93.37 92.22 91.89 91.32 91.21 90.49 88.83 88.54 86.40 86.08 84.43 83.66 82.50 80.92 79.68
    curl_curl_fuzzer_http 98.66 96.86 98.78 98.94 96.67 96.98 97.06 92.78 96.90 96.99 96.59 88.86 96.18 96.31 90.33
    freetype2-2017 91.72 86.14 91.32 94.38 78.48 83.28 73.90 75.43 67.14 67.41 67.33 62.93 66.36 63.26 67.35
    harfbuzz-1.3.2 94.76 93.21 92.25 94.50 92.86 92.86 91.78 93.34 91.42 91.53 91.34 91.23 90.90 90.60 93.07
    jsoncpp_jsoncpp_fuzzer 99.69 nan 99.53 99.84 nan nan 99.53 100.00 99.53 99.53 99.69 100.00 99.69 99.69 99.53
    lcms-2017-03-21 88.91 95.59 87.37 70.11 92.70 93.85 66.43 86.30 66.39 66.39 61.95 80.97 60.70 63.89 61.08
    libjpeg-turbo-07-2017 98.92 99.22 98.80 97.76 99.35 99.13 97.81 99.09 97.84 98.43 98.06 97.63 97.59 89.15 91.64
    libpng-1.2.56 97.22 97.70 94.70 98.02 97.84 87.14 81.27 91.49 73.55 68.73 68.73 90.87 68.73 73.15 68.73
    libxml2-v2.9.2 86.94 82.01 84.43 75.70 83.59 75.38 72.99 52.23 74.03 73.88 58.96 44.04 48.26 41.83 50.45
    libxslt_xpath 98.02 96.54 98.88 98.59 96.94 96.86 98.28 85.28 98.39 98.37 97.30 82.46 97.37 96.82 96.44
    mbedtls_fuzz_dtlsclient 71.86 71.12 70.99 70.37 70.92 71.05 70.91 70.50 70.91 71.10 71.00 69.64 69.76 70.20 71.09
    openssl_x509 99.91 99.87 99.96 99.62 99.94 99.94 99.94 99.89 99.96 99.94 99.94 99.69 99.93 99.77 99.80
    openthread-2019-12-23 82.70 83.81 82.75 82.57 83.11 84.14 82.99 83.60 82.57 82.68 83.03 83.46 74.45 74.57 58.68
    proj4-2017-08-14 91.46 73.51 70.07 96.71 66.93 67.57 89.62 98.10 62.71 60.96 54.47 96.81 48.53 52.74 60.40
    re2-2014-12-09 98.24 99.04 98.27 97.93 98.32 98.39 98.27 99.33 98.21 98.55 98.18 98.18 98.16 97.90 97.92
    sqlite3_ossfuzz nan 97.01 99.10 82.07 96.80 95.81 93.79 84.72 94.82 94.71 94.64 64.73 93.43 71.55 64.18
    vorbis-2017-12-11 99.36 99.45 99.45 98.74 99.45 99.43 99.33 99.31 99.38 99.36 99.40 83.97 99.24 98.90 99.04
    woff2-2016-05-06 95.56 98.85 95.53 96.63 98.72 98.72 96.12 93.77 95.10 94.90 94.74 86.75 93.29 95.28 84.92
    zlib_zlib_uncompress_fuzzer nan 97.77 nan nan 97.87 97.77 nan nan nan nan nan nan nan nan nan
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 17965.10 74.632362 17824.0 17920.75 17977.0 18000.75 18108.0
    aflplusplus 82800 20.0 17959.10 106.579795 17794.0 17892.50 17948.5 18011.00 18170.0
    symsan 82800 20.0 17946.60 93.264988 17834.0 17877.50 17926.5 18022.75 18157.0
    eclipser 82800 20.0 17628.65 73.138135 17503.0 17606.00 17636.0 17650.00 17840.0
    afl 82800 20.0 17622.05 58.351453 17521.0 17595.75 17623.0 17654.00 17755.0
    symcc 82800 20.0 17626.75 63.351879 17464.0 17602.25 17622.0 17663.25 17732.0
    aflsmart 82800 20.0 17624.15 70.812038 17534.0 17581.25 17607.0 17645.50 17796.0
    fuzzolic 82800 20.0 17610.50 39.907129 17555.0 17577.00 17599.5 17640.25 17698.0
    symqemu 82800 20.0 17561.70 73.968770 17357.0 17550.75 17564.5 17586.75 17714.0
    mopt 82800 20.0 17544.35 53.320555 17392.0 17527.50 17551.0 17572.50 17614.0
    lafintel 82800 20.0 17489.25 83.833717 17244.0 17452.50 17500.0 17536.25 17642.0
    aflfast 82800 20.0 17468.35 109.788445 17159.0 17426.00 17475.5 17557.50 17627.0
    entropic 82800 20.0 16896.35 550.903304 15772.0 16638.75 16859.0 17435.25 17656.0
    fairfuzz 82800 20.0 16586.80 586.559783 15595.0 16197.00 16412.5 17036.50 17681.0
    libfuzzer 82800 20.0 16098.60 905.373805 14277.0 15875.50 16145.0 16594.00 17452.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 28798.400000 497.285092 27453.0 28423.25 28895.5 29100.00 29772.0
    symsan 82800 20.0 28529.350000 989.008503 27502.0 27960.00 28080.0 28532.50 30615.0
    aflplusplus 82800 20.0 27881.950000 1331.397781 25562.0 27377.25 27956.5 28307.50 30145.0
    fuzzolic 82800 19.0 26480.157895 557.185413 25242.0 26170.50 26371.0 26989.50 27275.0
    symcc 82800 20.0 25448.550000 538.330702 23564.0 25337.25 25496.0 25750.75 26033.0
    symqemu 82800 20.0 24090.150000 546.996899 22806.0 23800.00 24028.0 24286.50 25330.0
    entropic 82800 20.0 23005.650000 1147.494950 20512.0 22598.50 23093.0 23620.50 25274.0
    eclipser 82800 20.0 22806.400000 668.781248 21825.0 22500.25 22623.5 22866.00 24576.0
    afl 82800 20.0 20599.800000 301.860128 19471.0 20588.75 20639.0 20692.00 20931.0
    fairfuzz 82800 20.0 20635.600000 173.254422 20314.0 20545.75 20619.5 20701.75 20942.0
    mopt 82800 20.0 20524.050000 357.815224 19483.0 20469.25 20614.5 20694.75 20923.0
    aflsmart 82800 20.0 20509.600000 344.140027 19604.0 20471.50 20554.0 20680.75 20909.0
    aflfast 82800 20.0 20109.350000 470.047173 19162.0 19584.50 20317.5 20451.50 20575.0
    lafintel 82800 20.0 19573.850000 562.152000 18813.0 19123.25 19367.0 20161.25 20393.0
    libfuzzer 82800 20.0 19401.700000 1060.361413 17071.0 18812.50 19267.0 20064.00 21653.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 8676.95 193.707994 8419.0 8567.50 8656.0 8702.75 9132.0
    honggfuzz 82800 20.0 8642.00 83.504176 8492.0 8620.25 8633.0 8658.25 8914.0
    entropic 82800 20.0 8521.85 100.352681 8365.0 8432.50 8527.0 8611.25 8672.0
    fuzzolic 82800 20.0 8541.45 115.423009 8407.0 8437.50 8515.0 8645.25 8730.0
    fairfuzz 82800 20.0 8456.20 101.686101 8184.0 8435.75 8502.0 8515.00 8552.0
    symcc 82800 20.0 8577.00 204.458206 8420.0 8428.00 8482.5 8666.75 9135.0
    symqemu 82800 20.0 8551.10 107.523755 8441.0 8453.25 8482.5 8660.25 8704.0
    aflplusplus 82800 20.0 8495.60 144.377211 8367.0 8411.75 8427.0 8515.50 8820.0
    eclipser 82800 20.0 8414.15 106.771411 8269.0 8324.00 8384.5 8492.50 8598.0
    afl 82800 20.0 8402.95 86.791629 8314.0 8339.50 8361.0 8498.25 8560.0
    aflsmart 82800 20.0 8386.80 83.909475 8302.0 8343.75 8351.5 8381.75 8578.0
    mopt 82800 20.0 8363.70 62.238167 8287.0 8322.75 8343.5 8385.50 8498.0
    libfuzzer 82800 20.0 8351.30 93.435089 8112.0 8319.25 8333.5 8371.75 8547.0
    aflfast 82800 20.0 8335.10 95.835879 8201.0 8267.00 8304.0 8427.00 8579.0
    lafintel 82800 20.0 8333.95 91.739032 8238.0 8263.00 8276.0 8413.50 8493.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    libfuzzer 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    honggfuzz 82800 20.0 640.20 0.410391 640.0 640.0 640.0 640.0 641.0
    aflfast 82800 20.0 638.65 0.489360 638.0 638.0 639.0 639.0 639.0
    lafintel 82800 20.0 639.00 0.000000 639.0 639.0 639.0 639.0 639.0
    mopt 82800 20.0 638.60 0.502625 638.0 638.0 639.0 639.0 639.0
    symsan 82800 20.0 638.80 0.410391 638.0 639.0 639.0 639.0 639.0
    afl 82800 20.0 638.30 0.470162 638.0 638.0 638.0 639.0 639.0
    aflplusplus 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    aflsmart 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    eclipser 82800 20.0 638.35 0.489360 638.0 638.0 638.0 639.0 639.0
    fairfuzz 82800 20.0 638.45 0.510418 638.0 638.0 638.0 639.0 639.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fuzzolic 82800 20.0 3740.40 94.020938 3535.0 3679.00 3770.0 3797.00 3887.0
    symcc 82800 20.0 3677.15 173.210026 3233.0 3651.00 3701.5 3774.75 3895.0
    symqemu 82800 20.0 3686.05 136.799690 3438.0 3605.75 3656.0 3764.75 3944.0
    symsan 82800 20.0 3363.65 411.125832 2523.0 3093.25 3506.5 3626.75 3823.0
    aflplusplus 82800 20.0 3214.15 401.521485 2499.0 2783.75 3446.0 3514.50 3707.0
    entropic 82800 20.0 3407.90 124.575447 3208.0 3314.50 3403.5 3514.50 3587.0
    libfuzzer 82800 20.0 3190.65 138.313250 2920.0 3111.50 3193.5 3289.25 3394.0
    honggfuzz 82800 20.0 2997.25 454.180103 2559.0 2579.00 2765.0 3445.00 3776.0
    eclipser 82800 20.0 2545.05 95.750484 2430.0 2443.00 2620.0 2626.25 2655.0
    afl 82800 20.0 2495.10 264.741761 1444.0 2434.75 2618.5 2624.50 2659.0
    aflsmart 82800 20.0 2540.10 107.336608 2311.0 2444.50 2618.5 2624.75 2661.0
    lafintel 82800 20.0 2592.80 146.742739 2450.0 2489.75 2520.0 2688.25 3057.0
    mopt 82800 20.0 2509.05 100.323936 2431.0 2435.00 2443.5 2629.75 2680.0
    fairfuzz 82800 20.0 2358.70 228.442346 1419.0 2385.25 2409.0 2410.25 2617.0
    aflfast 82800 20.0 1912.05 578.709497 1150.0 1333.00 2394.0 2423.00 2431.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 82800 20.0 3814.300000 16.403145 3777.0 3801.00 3819.0 3825.00 3842.0
    fuzzolic 82800 20.0 3814.950000 19.261292 3779.0 3803.25 3814.0 3829.00 3844.0
    symcc 82800 20.0 3795.900000 72.646875 3494.0 3803.50 3810.5 3821.25 3841.0
    entropic 82800 19.0 3811.526316 11.379164 3785.0 3805.00 3809.0 3815.50 3833.0
    symsan 82800 20.0 3802.100000 17.344497 3773.0 3788.00 3802.5 3820.25 3827.0
    aflplusplus 82800 20.0 3797.150000 19.762471 3754.0 3783.75 3798.0 3816.00 3824.0
    afl 82800 20.0 3735.500000 119.489044 3453.0 3758.75 3783.5 3800.00 3820.0
    mopt 82800 20.0 3764.150000 40.766698 3637.0 3757.75 3769.5 3784.50 3806.0
    aflsmart 82800 20.0 3645.650000 168.960374 3405.0 3457.75 3761.0 3784.00 3807.0
    eclipser 82800 20.0 3658.200000 152.117199 3443.0 3472.25 3760.0 3779.25 3811.0
    honggfuzz 82800 20.0 3671.000000 159.270706 3430.0 3439.75 3758.0 3779.00 3802.0
    libfuzzer 82800 20.0 3633.800000 225.909763 3186.0 3651.75 3753.0 3755.75 3790.0
    aflfast 82800 20.0 3652.750000 154.978734 3410.0 3452.00 3751.5 3777.00 3796.0
    fairfuzz 82800 20.0 3560.750000 187.942285 3326.0 3409.25 3522.5 3758.50 3802.0
    lafintel 82800 20.0 3507.250000 167.315295 3328.0 3407.50 3427.0 3665.25 3758.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 2149.50 22.765394 2103.0 2145.75 2153.5 2158.25 2197.0
    symqemu 82800 20.0 2142.45 18.420455 2108.0 2127.00 2149.5 2156.00 2166.0
    fuzzolic 82800 20.0 2138.15 23.454940 2097.0 2112.50 2146.5 2153.25 2174.0
    symsan 82800 20.0 2137.60 18.198901 2107.0 2130.75 2136.0 2144.75 2177.0
    aflplusplus 82800 20.0 2080.45 16.433198 2056.0 2071.75 2080.5 2088.00 2135.0
    entropic 82800 20.0 2015.85 16.255444 2003.0 2008.00 2010.0 2014.50 2067.0
    libfuzzer 82800 20.0 1987.70 21.491981 1943.0 1993.50 1996.5 2000.00 2002.0
    symcc 82800 20.0 1928.75 28.728997 1900.0 1909.00 1914.5 1940.50 2008.0
    eclipser 82800 20.0 1772.30 58.041818 1526.0 1783.50 1785.5 1787.00 1793.0
    aflsmart 82800 20.0 1617.70 6.641972 1614.0 1615.75 1616.0 1617.00 1645.0
    lafintel 82800 20.0 1615.20 12.684844 1606.0 1607.00 1607.0 1618.75 1642.0
    afl 82800 20.0 1510.85 3.166851 1508.0 1510.00 1510.0 1510.00 1520.0
    aflfast 82800 20.0 1509.60 0.598243 1508.0 1509.00 1510.0 1510.00 1510.0
    fairfuzz 82800 20.0 1510.95 3.103055 1509.0 1510.00 1510.0 1510.00 1520.0
    mopt 82800 20.0 1509.85 0.366348 1509.0 1510.00 1510.0 1510.00 1510.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symcc.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 12894.100000 583.658597 12061.0 12555.00 12799.0 13160.00 14721.0
    aflplusplus 82800 20.0 11947.500000 1428.663582 7521.0 11818.25 12429.5 12716.50 13189.0
    symqemu 82800 19.0 12203.210526 470.727992 11295.0 11796.50 12305.0 12574.00 12910.0
    fuzzolic 82800 19.0 11936.631579 984.269228 8527.0 11773.50 12072.0 12558.50 12937.0
    honggfuzz 82800 20.0 11019.750000 1006.297742 9128.0 10324.00 11144.0 11576.25 12878.0
    symcc 82800 1.0 11097.000000 NaN 11097.0 11097.00 11097.0 11097.00 11097.0
    aflsmart 82800 20.0 10297.600000 1493.633274 6839.0 9998.50 10897.5 11146.75 12143.0
    afl 82800 20.0 10405.200000 1374.587195 6835.0 10319.25 10875.5 11207.50 11761.0
    eclipser 82800 20.0 10030.950000 1682.728157 6674.0 9722.00 10745.5 11320.75 11912.0
    mopt 82800 20.0 8871.600000 1136.880237 6860.0 8187.75 8679.5 9777.75 10844.0
    entropic 82800 20.0 7616.000000 629.329635 6493.0 7431.25 7688.5 8013.00 9232.0
    fairfuzz 82800 20.0 7716.850000 1978.305662 4984.0 6068.50 7426.5 9784.25 10627.0
    aflfast 82800 20.0 7107.250000 720.566070 6167.0 6483.00 7104.0 7498.00 9076.0
    libfuzzer 82800 20.0 6880.250000 550.392671 6418.0 6460.00 6482.5 7570.50 7622.0
    lafintel 82800 20.0 6153.300000 55.904147 5967.0 6135.50 6157.5 6178.75 6246.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symsan.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 18957.00 128.081800 18736.0 18888.75 18963.5 19056.50 19178.0
    honggfuzz 82800 20.0 18901.05 143.575572 18685.0 18778.25 18908.0 18996.75 19179.0
    aflsmart 82800 20.0 18880.30 60.963406 18765.0 18830.00 18870.5 18923.50 18977.0
    afl 82800 20.0 18850.15 71.677843 18686.0 18822.25 18865.5 18898.00 18976.0
    eclipser 82800 20.0 18850.15 65.984268 18748.0 18797.00 18850.0 18894.75 19022.0
    symsan 82800 1.0 18799.00 NaN 18799.0 18799.00 18799.0 18799.00 18799.0
    aflfast 82800 20.0 18673.10 73.754857 18456.0 18646.00 18674.5 18696.00 18819.0
    mopt 82800 20.0 18652.60 69.051011 18515.0 18615.25 18661.0 18694.25 18760.0
    symqemu 82800 18.0 18560.50 78.628875 18394.0 18508.50 18592.5 18618.50 18663.0
    symcc 82800 20.0 18566.75 101.998388 18409.0 18480.50 18577.0 18621.75 18789.0
    lafintel 82800 20.0 18563.90 173.383633 18063.0 18525.25 18570.0 18664.50 18854.0
    fuzzolic 82800 20.0 18516.50 96.095621 18303.0 18452.25 18515.0 18561.00 18712.0
    fairfuzz 82800 20.0 18240.55 859.615855 15799.0 18005.75 18496.5 18851.75 19004.0
    entropic 82800 20.0 16407.50 178.394448 16151.0 16287.75 16355.5 16539.50 16767.0
    libfuzzer 82800 20.0 15849.00 123.141767 15700.0 15754.25 15815.5 15917.25 16110.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 9281.050000 1131.663307 8210.0 8307.00 8353.5 10436.50 10828.0
    fuzzolic 82800 20.0 8263.500000 49.827598 8151.0 8243.50 8268.0 8277.00 8396.0
    afl 82800 20.0 8595.350000 1016.725460 8205.0 8233.50 8265.0 8313.25 11625.0
    fairfuzz 82800 20.0 8333.350000 287.818341 8187.0 8224.50 8264.5 8323.25 9530.0
    symcc 82800 17.0 8400.588235 676.300789 8143.0 8193.00 8260.0 8279.00 11016.0
    mopt 82800 20.0 8582.800000 979.570883 8215.0 8230.75 8253.5 8326.00 11503.0
    aflplusplus 82800 20.0 8239.900000 46.769198 8118.0 8211.75 8252.5 8277.75 8291.0
    symqemu 82800 20.0 8464.400000 689.521909 8159.0 8215.50 8244.5 8274.00 10525.0
    aflsmart 82800 20.0 8259.900000 46.873739 8204.0 8227.00 8243.5 8278.25 8356.0
    eclipser 82800 20.0 8395.100000 673.758031 8206.0 8220.50 8243.5 8267.75 11255.0
    entropic 82800 20.0 8198.200000 43.124788 8141.0 8166.75 8196.0 8215.00 8302.0
    honggfuzz 82800 20.0 8215.200000 228.845294 8091.0 8122.25 8181.0 8198.75 9168.0
    lafintel 82800 20.0 8266.400000 518.700644 8100.0 8122.75 8161.0 8169.00 10466.0
    aflfast 82800 20.0 8108.600000 41.314455 8048.0 8075.25 8110.0 8147.25 8179.0
    libfuzzer 82800 20.0 7932.700000 239.825616 7602.0 7702.00 8095.5 8137.25 8212.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 13774.95 8.369588 13752.0 13775.25 13779.0 13779.00 13785.0
    aflsmart 82800 20.0 13778.50 1.933091 13775.0 13777.00 13779.0 13779.00 13783.0
    afl 82800 20.0 13777.35 1.348488 13775.0 13777.00 13777.0 13779.00 13779.0
    eclipser 82800 20.0 13778.20 3.396593 13775.0 13775.00 13777.0 13779.00 13785.0
    mopt 82800 20.0 13775.50 5.907177 13751.0 13775.00 13777.0 13777.00 13779.0
    symcc 82800 20.0 13773.60 6.003508 13759.0 13773.25 13777.0 13777.00 13779.0
    symqemu 82800 20.0 13774.90 4.700504 13759.0 13775.00 13777.0 13777.00 13779.0
    aflfast 82800 20.0 13770.85 8.106365 13751.0 13771.25 13775.0 13775.00 13777.0
    symsan 82800 20.0 13767.80 11.781341 13752.0 13757.25 13772.5 13779.00 13779.0
    entropic 82800 20.0 13766.60 11.962221 13745.0 13755.00 13769.5 13778.00 13779.0
    fuzzolic 82800 20.0 13769.20 9.294028 13753.0 13761.50 13767.5 13778.00 13780.0
    fairfuzz 82800 20.0 13758.55 8.605231 13750.0 13752.00 13757.0 13761.00 13777.0
    lafintel 82800 20.0 13757.95 16.113006 13726.0 13750.75 13753.0 13773.25 13775.0
    libfuzzer 82800 20.0 13744.00 6.316228 13724.0 13742.00 13742.0 13750.00 13751.0
    honggfuzz 82800 20.0 13738.40 11.690752 13727.0 13729.00 13732.5 13752.25 13758.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symcc 82800 20.0 6071.95 378.441746 5850.0 5900.75 5935.0 5950.50 7054.0
    fuzzolic 82800 20.0 5909.10 32.884007 5856.0 5890.25 5912.0 5933.25 5949.0
    entropic 82800 20.0 5892.90 14.472842 5850.0 5890.00 5897.0 5901.00 5916.0
    libfuzzer 82800 20.0 5891.15 8.875009 5877.0 5886.00 5887.5 5895.50 5910.0
    symqemu 82800 20.0 5903.30 200.671268 5816.0 5843.25 5862.5 5881.25 6750.0
    mopt 82800 20.0 5815.90 140.108191 5244.0 5826.25 5857.0 5871.00 5897.0
    eclipser 82800 20.0 5847.95 37.276315 5736.0 5838.00 5854.0 5882.25 5889.0
    aflplusplus 82800 20.0 5841.95 23.493504 5822.0 5827.75 5837.5 5840.25 5921.0
    symsan 82800 20.0 5836.15 31.422880 5735.0 5828.00 5833.5 5839.25 5898.0
    afl 82800 20.0 5829.55 39.662358 5716.0 5814.75 5832.0 5854.00 5911.0
    aflsmart 82800 20.0 5782.10 132.662572 5250.0 5789.25 5824.5 5836.75 5856.0
    honggfuzz 82800 20.0 5828.40 49.508532 5720.0 5812.75 5824.5 5872.75 5888.0
    lafintel 82800 20.0 5412.25 247.306517 5233.0 5252.75 5260.5 5673.25 5890.0
    aflfast 82800 20.0 5499.60 293.157834 5221.0 5241.75 5252.0 5816.50 5863.0
    fairfuzz 82800 20.0 4523.90 560.570001 4139.0 4139.00 4139.0 5085.25 5860.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 7793.10 78.750706 7697.0 7727.25 7789.0 7834.50 7940.0
    libfuzzer 82800 20.0 7688.10 111.489863 7406.0 7624.00 7687.0 7773.75 7875.0
    honggfuzz 82800 20.0 7695.05 84.786093 7542.0 7638.75 7678.5 7715.00 7864.0
    symsan 82800 20.0 7241.10 334.799846 6621.0 7035.50 7262.0 7452.25 7928.0
    eclipser 82800 20.0 7073.20 165.101755 6603.0 7038.25 7115.5 7160.50 7289.0
    fuzzolic 82800 20.0 5843.40 126.113316 5625.0 5786.00 5836.5 5902.50 6154.0
    aflplusplus 82800 20.0 5602.85 120.008015 5452.0 5509.00 5563.5 5686.50 5861.0
    symcc 82800 20.0 5374.25 157.883660 5149.0 5252.00 5365.0 5496.00 5620.0
    symqemu 82800 20.0 5355.75 154.432024 5178.0 5229.25 5314.0 5435.00 5657.0
    aflsmart 82800 20.0 4581.90 721.583194 3154.0 4303.25 4979.0 5105.25 5267.0
    afl 82800 20.0 4541.60 682.244315 3133.0 4198.25 4840.5 5043.75 5158.0
    fairfuzz 82800 20.0 4799.05 212.722571 4292.0 4678.75 4796.0 4985.00 5096.0
    mopt 82800 20.0 4221.75 655.483339 3094.0 3729.25 4325.0 4707.25 5066.0
    lafintel 82800 20.0 4287.55 756.823102 2826.0 3821.00 4187.5 5032.50 5216.0
    aflfast 82800 20.0 3804.80 846.694974 2583.0 3025.50 3853.0 4430.50 5084.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symsan.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 3542.100000 18.951184 3516.0 3523.00 3555.0 3558.00 3564.0
    fuzzolic 82800 20.0 3535.300000 18.719818 3510.0 3517.75 3544.5 3552.50 3559.0
    afl 82800 20.0 3527.450000 19.030377 3503.0 3509.75 3527.0 3545.25 3555.0
    symcc 82800 20.0 3531.250000 17.905821 3507.0 3517.25 3521.5 3548.75 3554.0
    symqemu 82800 20.0 3525.800000 15.746679 3507.0 3517.00 3519.0 3528.50 3555.0
    aflplusplus 82800 20.0 3524.150000 15.267009 3508.0 3514.00 3517.0 3540.75 3553.0
    eclipser 82800 20.0 3527.550000 21.421767 3507.0 3511.00 3517.0 3549.00 3579.0
    symsan 82800 6.0 3520.333333 15.933194 3508.0 3512.75 3516.0 3517.75 3552.0
    aflsmart 82800 20.0 3522.300000 16.157449 3506.0 3511.00 3515.0 3541.25 3550.0
    libfuzzer 82800 20.0 3512.100000 15.231201 3474.0 3511.50 3514.0 3517.00 3552.0
    mopt 82800 20.0 3522.650000 21.492410 3472.0 3508.50 3514.0 3544.00 3547.0
    aflfast 82800 20.0 3516.700000 18.719818 3471.0 3505.00 3513.0 3537.25 3549.0
    honggfuzz 82800 20.0 3510.150000 12.807379 3500.0 3504.00 3505.0 3506.75 3545.0
    fairfuzz 82800 20.0 3489.050000 53.283230 3343.0 3502.00 3504.5 3508.25 3541.0
    lafintel 82800 20.0 3506.150000 26.332439 3462.0 3501.00 3504.0 3518.50 3545.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symcc.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 36684.80 193.007663 36163.0 36604.00 36699.0 36793.25 37031.0
    fuzzolic 82800 20.0 35783.20 476.576011 34425.0 35524.25 35922.5 36062.75 36451.0
    symqemu 82800 20.0 35836.40 362.761253 35046.0 35621.75 35845.5 36134.25 36344.0
    symcc 82800 4.0 35471.50 367.079011 35088.0 35196.75 35478.5 35753.25 35841.0
    aflsmart 82800 20.0 35095.70 336.804504 34459.0 34949.25 35114.5 35303.25 35724.0
    afl 82800 20.0 34877.70 909.362825 32344.0 34776.25 35071.0 35434.25 35723.0
    mopt 82800 20.0 35012.80 469.740869 34109.0 34805.50 35045.0 35328.75 35670.0
    eclipser 82800 20.0 34675.45 408.497951 33871.0 34381.00 34733.0 34996.50 35434.0
    aflfast 82800 20.0 34658.40 339.990155 34098.0 34486.25 34598.0 34774.25 35439.0
    entropic 82800 20.0 31296.45 654.333367 30130.0 30875.75 31372.5 31766.00 32352.0
    honggfuzz 82800 20.0 30340.25 604.276677 28827.0 30082.50 30390.5 30602.25 31677.0
    lafintel 82800 20.0 26622.25 1226.556535 24828.0 25599.50 26496.5 27756.50 29156.0
    libfuzzer 82800 20.0 23851.30 1056.883856 22099.0 23139.50 23970.0 24521.50 25417.0
    fairfuzz 82800 20.0 24543.10 2704.278869 21771.0 22724.75 23765.0 25786.75 30784.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 2168.400000 2.909151 2165.0 2167.00 2168.0 2169.00 2177.0
    fuzzolic 82800 20.0 2168.800000 4.323741 2162.0 2166.00 2168.0 2170.25 2178.0
    symqemu 82800 20.0 2168.350000 3.602265 2164.0 2167.00 2168.0 2169.00 2180.0
    symcc 82800 20.0 2168.250000 4.024595 2163.0 2165.75 2167.5 2170.00 2180.0
    mopt 82800 20.0 2167.950000 5.889107 2158.0 2164.50 2167.0 2173.25 2178.0
    aflsmart 82800 20.0 2166.850000 4.934039 2159.0 2163.00 2166.5 2168.25 2177.0
    afl 82800 20.0 2165.700000 5.694873 2152.0 2163.75 2166.0 2167.00 2179.0
    symsan 82800 20.0 2166.750000 3.024027 2161.0 2165.00 2166.0 2169.00 2172.0
    eclipser 82800 20.0 2165.200000 4.237676 2157.0 2162.00 2165.5 2167.00 2176.0
    entropic 82800 20.0 2131.700000 103.035559 1830.0 2163.00 2165.0 2165.25 2172.0
    aflfast 82800 20.0 2162.750000 4.722566 2153.0 2160.75 2163.5 2164.25 2177.0
    fairfuzz 82800 19.0 2134.473684 38.013694 2047.0 2104.00 2159.0 2163.00 2167.0
    lafintel 82800 20.0 2149.100000 18.378763 2103.0 2148.50 2156.0 2160.00 2169.0
    honggfuzz 82800 20.0 2150.750000 7.765477 2135.0 2150.00 2152.5 2156.00 2160.0
    libfuzzer 82800 20.0 1976.150000 165.904943 1829.0 1830.00 1830.5 2155.50 2159.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    fuzzolic 82800 20.0 1932.60 18.636128 1871.0 1926.00 1936.5 1942.50 1959.0
    symcc 82800 20.0 1933.10 17.873532 1898.0 1920.00 1934.0 1950.25 1956.0
    symqemu 82800 20.0 1927.70 21.006515 1890.0 1906.75 1934.0 1947.75 1953.0
    honggfuzz 82800 20.0 1888.30 21.940350 1836.0 1873.50 1893.0 1907.25 1915.0
    eclipser 82800 20.0 1880.70 23.468007 1849.0 1858.50 1883.0 1894.00 1929.0
    symsan 82800 20.0 1868.80 13.563108 1840.0 1859.50 1872.0 1877.50 1889.0
    aflplusplus 82800 20.0 1869.10 16.631929 1836.0 1864.25 1871.5 1881.00 1890.0
    lafintel 82800 20.0 1864.10 16.029250 1837.0 1854.00 1866.5 1870.50 1906.0
    aflsmart 82800 20.0 1861.45 24.191071 1789.0 1846.75 1863.0 1883.00 1890.0
    afl 82800 20.0 1861.30 18.067418 1839.0 1843.75 1859.0 1876.50 1887.0
    mopt 82800 20.0 1860.75 16.764232 1829.0 1851.25 1856.0 1876.00 1891.0
    entropic 82800 20.0 1841.65 87.216171 1698.0 1757.00 1837.0 1922.75 1959.0
    aflfast 82800 20.0 1825.15 10.668572 1805.0 1818.75 1827.5 1832.00 1846.0
    libfuzzer 82800 20.0 1697.10 55.169500 1599.0 1654.00 1699.5 1745.00 1799.0
    fairfuzz 82800 20.0 1661.40 32.809498 1612.0 1645.00 1663.5 1672.75 1765.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

zlib_zlib_uncompress_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symqemu 82800 20.0 964.30 7.671547 946.0 962.25 964.0 965.0 983.0
    fuzzolic 82800 20.0 965.60 8.100682 960.0 961.00 963.0 964.0 985.0
    symcc 82800 20.0 962.75 1.970172 960.0 961.00 963.0 964.0 966.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

from cached data