FuzzBench: symsan report

warning
Please consider this as a preliminary report to demonstrate the capabilities of FuzzBench. While we have tried our best, we have not confirmed that we configured everything correctly. We are hoping to work together with the community to validate results and improve the set of fuzzers, benchmarks, and their configurations in the future. See FAQ for more details.

experiment summary

We show two different aggregate (cross-benchmark) rankings of fuzzers. The first is based on the average of per-benchmarks scores, where the score represents the percentage of the highest reached median code-coverage on a given benchmark (higher value is better). The second ranking shows the average rank of fuzzers, after we rank them on each benchmark according to their median reached code-covereges (lower value is better).
By avg. score
average normalized score
fuzzer
symsan 99.03
aflplusplus 97.19
honggfuzz 96.44
eclipser 93.81
entropic 93.49
aflsmart 91.32
afl 91.00
mopt 89.18
libfuzzer 88.39
aflfast 87.00
lafintel 85.38
fairfuzz 84.04
By avg. rank
average rank
fuzzer
symsan 3.38
aflplusplus 3.41
entropic 5.15
honggfuzz 5.21
eclipser 5.35
afl 5.68
aflsmart 5.94
mopt 6.62
libfuzzer 8.47
aflfast 9.29
fairfuzz 9.38
lafintel 9.62
  • Critical difference diagram
    The diagram visualizes the average rank of fuzzers (second ranking above) while showing the significance of the differences as well. What is considered a "critical difference" (CD) is based on the Friedman/Nemenyi post-hoc test. See more in the documentation.
    Note: If a fuzzer does not support all benchmarks, its ranking as shown in this diagram can be lower than it should be. So please check the list of supported benchmarks for the fuzzer(s) of your interest. The list could be specified in the fuzzer's README.md like this.
  • Median relative code-coverages on each benchmark

    Note: The relative coverage summary table shows the median relative performance of each fuzzer to the experiment maximum. Thus the highest relative performance may not be 100%.
    trial_relative_coverage = trial_coverage / experiment_max_coverage

    symsan aflplusplus honggfuzz eclipser entropic aflsmart afl mopt libfuzzer aflfast lafintel fairfuzz
    FuzzerMedian 97.62 98.27 97.93 96.12 93.37 95.10 94.90 94.74 88.86 93.29 89.41 84.92
    FuzzerMean 94.56 93.01 92.40 89.90 89.66 87.48 87.16 85.50 84.77 83.48 81.90 80.48
    curl_curl_fuzzer_http 98.66 98.78 98.94 97.06 92.78 96.90 96.99 96.59 88.86 96.18 96.31 90.33
    freetype2-2017 91.72 91.32 94.38 73.90 75.43 67.14 67.41 67.33 62.93 66.36 63.26 67.35
    harfbuzz-1.3.2 94.79 92.28 94.54 91.81 93.37 91.45 91.56 91.37 91.26 90.93 90.63 93.10
    jsoncpp_jsoncpp_fuzzer 99.69 99.53 99.84 99.53 100.00 99.53 99.53 99.69 100.00 99.69 99.69 99.53
    lcms-2017-03-21 91.72 90.14 72.33 68.53 89.03 68.49 68.49 63.92 83.53 62.62 65.92 63.01
    libjpeg-turbo-07-2017 99.20 99.09 98.04 98.10 99.37 98.12 98.71 98.34 97.91 97.87 89.41 91.90
    libpng-1.2.56 97.22 94.70 98.02 81.27 91.49 73.55 68.73 68.73 90.87 68.73 73.15 68.73
    libxml2-v2.9.2 86.94 84.43 75.70 72.99 52.23 74.03 73.88 58.96 44.04 48.26 41.83 50.45
    libxslt_xpath 98.02 98.88 98.59 98.28 85.28 98.39 98.37 97.30 82.46 97.37 96.82 96.44
    mbedtls_fuzz_dtlsclient 71.86 70.99 70.37 70.91 70.50 70.91 71.10 71.00 69.64 69.76 70.20 71.09
    openssl_x509 99.91 99.96 99.62 99.94 99.89 99.96 99.94 99.94 99.69 99.93 99.77 99.80
    openthread-2019-12-23 98.52 98.59 98.37 98.87 99.59 98.37 98.50 98.92 99.43 88.70 88.84 69.90
    proj4-2017-08-14 91.46 70.07 96.71 89.62 98.10 62.71 60.96 54.47 96.81 48.53 52.74 60.40
    re2-2014-12-09 98.24 98.27 97.93 98.27 99.33 98.21 98.55 98.18 98.18 98.16 97.90 97.92
    sqlite3_ossfuzz nan 99.10 82.07 93.79 84.72 94.82 94.71 94.64 64.73 93.43 71.55 64.18
    vorbis-2017-12-11 99.40 99.50 98.78 99.38 99.36 99.43 99.40 99.45 84.01 99.29 98.94 99.08
    woff2-2016-05-06 95.56 95.53 96.63 96.12 93.77 95.10 94.90 94.74 86.75 93.29 95.28 84.92
    • Fuzzers are sorted by "FuzzerMean" (average median relative coverage), highest on the left.
    • Green background = highest relative median coverage.
    • Blue gradient background = greater than 95% relative median coverage.

curl_curl_fuzzer_http summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 17965.10 74.632362 17824.0 17920.75 17977.0 18000.75 18108.0
    aflplusplus 82800 20.0 17959.10 106.579795 17794.0 17892.50 17948.5 18011.00 18170.0
    symsan 82800 20.0 17946.60 93.264988 17834.0 17877.50 17926.5 18022.75 18157.0
    eclipser 82800 20.0 17628.65 73.138135 17503.0 17606.00 17636.0 17650.00 17840.0
    afl 82800 20.0 17622.05 58.351453 17521.0 17595.75 17623.0 17654.00 17755.0
    aflsmart 82800 20.0 17624.15 70.812038 17534.0 17581.25 17607.0 17645.50 17796.0
    mopt 82800 20.0 17544.35 53.320555 17392.0 17527.50 17551.0 17572.50 17614.0
    lafintel 82800 20.0 17489.25 83.833717 17244.0 17452.50 17500.0 17536.25 17642.0
    aflfast 82800 20.0 17468.35 109.788445 17159.0 17426.00 17475.5 17557.50 17627.0
    entropic 82800 20.0 16896.35 550.903304 15772.0 16638.75 16859.0 17435.25 17656.0
    fairfuzz 82800 20.0 16586.80 586.559783 15595.0 16197.00 16412.5 17036.50 17681.0
    libfuzzer 82800 20.0 16098.60 905.373805 14277.0 15875.50 16145.0 16594.00 17452.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

freetype2-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 28798.40 497.285092 27453.0 28423.25 28895.5 29100.00 29772.0
    symsan 82800 20.0 28529.35 989.008503 27502.0 27960.00 28080.0 28532.50 30615.0
    aflplusplus 82800 20.0 27881.95 1331.397781 25562.0 27377.25 27956.5 28307.50 30145.0
    entropic 82800 20.0 23005.65 1147.494950 20512.0 22598.50 23093.0 23620.50 25274.0
    eclipser 82800 20.0 22806.40 668.781248 21825.0 22500.25 22623.5 22866.00 24576.0
    afl 82800 20.0 20599.80 301.860128 19471.0 20588.75 20639.0 20692.00 20931.0
    fairfuzz 82800 20.0 20635.60 173.254422 20314.0 20545.75 20619.5 20701.75 20942.0
    mopt 82800 20.0 20524.05 357.815224 19483.0 20469.25 20614.5 20694.75 20923.0
    aflsmart 82800 20.0 20509.60 344.140027 19604.0 20471.50 20554.0 20680.75 20909.0
    aflfast 82800 20.0 20109.35 470.047173 19162.0 19584.50 20317.5 20451.50 20575.0
    lafintel 82800 20.0 19573.85 562.152000 18813.0 19123.25 19367.0 20161.25 20393.0
    libfuzzer 82800 20.0 19401.70 1060.361413 17071.0 18812.50 19267.0 20064.00 21653.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

harfbuzz-1.3.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 8676.95 193.707994 8419.0 8567.50 8656.0 8702.75 9132.0
    honggfuzz 82800 20.0 8642.00 83.504176 8492.0 8620.25 8633.0 8658.25 8914.0
    entropic 82800 20.0 8521.85 100.352681 8365.0 8432.50 8527.0 8611.25 8672.0
    fairfuzz 82800 20.0 8456.20 101.686101 8184.0 8435.75 8502.0 8515.00 8552.0
    aflplusplus 82800 20.0 8495.60 144.377211 8367.0 8411.75 8427.0 8515.50 8820.0
    eclipser 82800 20.0 8414.15 106.771411 8269.0 8324.00 8384.5 8492.50 8598.0
    afl 82800 20.0 8402.95 86.791629 8314.0 8339.50 8361.0 8498.25 8560.0
    aflsmart 82800 20.0 8386.80 83.909475 8302.0 8343.75 8351.5 8381.75 8578.0
    mopt 82800 20.0 8363.70 62.238167 8287.0 8322.75 8343.5 8385.50 8498.0
    libfuzzer 82800 20.0 8351.30 93.435089 8112.0 8319.25 8333.5 8371.75 8547.0
    aflfast 82800 20.0 8335.10 95.835879 8201.0 8267.00 8304.0 8427.00 8579.0
    lafintel 82800 20.0 8333.95 91.739032 8238.0 8263.00 8276.0 8413.50 8493.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

jsoncpp_jsoncpp_fuzzer summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    libfuzzer 82800 20.0 641.00 0.000000 641.0 641.0 641.0 641.0 641.0
    honggfuzz 82800 20.0 640.20 0.410391 640.0 640.0 640.0 640.0 641.0
    aflfast 82800 20.0 638.65 0.489360 638.0 638.0 639.0 639.0 639.0
    lafintel 82800 20.0 639.00 0.000000 639.0 639.0 639.0 639.0 639.0
    mopt 82800 20.0 638.60 0.502625 638.0 638.0 639.0 639.0 639.0
    symsan 82800 20.0 638.80 0.410391 638.0 639.0 639.0 639.0 639.0
    afl 82800 20.0 638.30 0.470162 638.0 638.0 638.0 639.0 639.0
    aflplusplus 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    aflsmart 82800 20.0 638.40 0.502625 638.0 638.0 638.0 639.0 639.0
    eclipser 82800 20.0 638.35 0.489360 638.0 638.0 638.0 639.0 639.0
    fairfuzz 82800 20.0 638.45 0.510418 638.0 638.0 638.0 639.0 639.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

lcms-2017-03-21 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 3363.65 411.125832 2523.0 3093.25 3506.5 3626.75 3823.0
    aflplusplus 82800 20.0 3214.15 401.521485 2499.0 2783.75 3446.0 3514.50 3707.0
    entropic 82800 20.0 3407.90 124.575447 3208.0 3314.50 3403.5 3514.50 3587.0
    libfuzzer 82800 20.0 3190.65 138.313250 2920.0 3111.50 3193.5 3289.25 3394.0
    honggfuzz 82800 20.0 2997.25 454.180103 2559.0 2579.00 2765.0 3445.00 3776.0
    eclipser 82800 20.0 2545.05 95.750484 2430.0 2443.00 2620.0 2626.25 2655.0
    afl 82800 20.0 2495.10 264.741761 1444.0 2434.75 2618.5 2624.50 2659.0
    aflsmart 82800 20.0 2540.10 107.336608 2311.0 2444.50 2618.5 2624.75 2661.0
    lafintel 82800 20.0 2592.80 146.742739 2450.0 2489.75 2520.0 2688.25 3057.0
    mopt 82800 20.0 2509.05 100.323936 2431.0 2435.00 2443.5 2629.75 2680.0
    fairfuzz 82800 20.0 2358.70 228.442346 1419.0 2385.25 2409.0 2410.25 2617.0
    aflfast 82800 20.0 1912.05 578.709497 1150.0 1333.00 2394.0 2423.00 2431.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libjpeg-turbo-07-2017 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 19.0 3811.526316 11.379164 3785.0 3805.00 3809.0 3815.50 3833.0
    symsan 82800 20.0 3802.100000 17.344497 3773.0 3788.00 3802.5 3820.25 3827.0
    aflplusplus 82800 20.0 3797.150000 19.762471 3754.0 3783.75 3798.0 3816.00 3824.0
    afl 82800 20.0 3735.500000 119.489044 3453.0 3758.75 3783.5 3800.00 3820.0
    mopt 82800 20.0 3764.150000 40.766698 3637.0 3757.75 3769.5 3784.50 3806.0
    aflsmart 82800 20.0 3645.650000 168.960374 3405.0 3457.75 3761.0 3784.00 3807.0
    eclipser 82800 20.0 3658.200000 152.117199 3443.0 3472.25 3760.0 3779.25 3811.0
    honggfuzz 82800 20.0 3671.000000 159.270706 3430.0 3439.75 3758.0 3779.00 3802.0
    libfuzzer 82800 20.0 3633.800000 225.909763 3186.0 3651.75 3753.0 3755.75 3790.0
    aflfast 82800 20.0 3652.750000 154.978734 3410.0 3452.00 3751.5 3777.00 3796.0
    fairfuzz 82800 20.0 3560.750000 187.942285 3326.0 3409.25 3522.5 3758.50 3802.0
    lafintel 82800 20.0 3507.250000 167.315295 3328.0 3407.50 3427.0 3665.25 3758.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libpng-1.2.56 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 2149.50 22.765394 2103.0 2145.75 2153.5 2158.25 2197.0
    symsan 82800 20.0 2137.60 18.198901 2107.0 2130.75 2136.0 2144.75 2177.0
    aflplusplus 82800 20.0 2080.45 16.433198 2056.0 2071.75 2080.5 2088.00 2135.0
    entropic 82800 20.0 2015.85 16.255444 2003.0 2008.00 2010.0 2014.50 2067.0
    libfuzzer 82800 20.0 1987.70 21.491981 1943.0 1993.50 1996.5 2000.00 2002.0
    eclipser 82800 20.0 1772.30 58.041818 1526.0 1783.50 1785.5 1787.00 1793.0
    aflsmart 82800 20.0 1617.70 6.641972 1614.0 1615.75 1616.0 1617.00 1645.0
    lafintel 82800 20.0 1615.20 12.684844 1606.0 1607.00 1607.0 1618.75 1642.0
    afl 82800 20.0 1510.85 3.166851 1508.0 1510.00 1510.0 1510.00 1520.0
    aflfast 82800 20.0 1509.60 0.598243 1508.0 1509.00 1510.0 1510.00 1510.0
    fairfuzz 82800 20.0 1510.95 3.103055 1509.0 1510.00 1510.0 1510.00 1520.0
    mopt 82800 20.0 1509.85 0.366348 1509.0 1510.00 1510.0 1510.00 1510.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxml2-v2.9.2 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 12894.10 583.658597 12061.0 12555.00 12799.0 13160.00 14721.0
    aflplusplus 82800 20.0 11947.50 1428.663582 7521.0 11818.25 12429.5 12716.50 13189.0
    honggfuzz 82800 20.0 11019.75 1006.297742 9128.0 10324.00 11144.0 11576.25 12878.0
    aflsmart 82800 20.0 10297.60 1493.633274 6839.0 9998.50 10897.5 11146.75 12143.0
    afl 82800 20.0 10405.20 1374.587195 6835.0 10319.25 10875.5 11207.50 11761.0
    eclipser 82800 20.0 10030.95 1682.728157 6674.0 9722.00 10745.5 11320.75 11912.0
    mopt 82800 20.0 8871.60 1136.880237 6860.0 8187.75 8679.5 9777.75 10844.0
    entropic 82800 20.0 7616.00 629.329635 6493.0 7431.25 7688.5 8013.00 9232.0
    fairfuzz 82800 20.0 7716.85 1978.305662 4984.0 6068.50 7426.5 9784.25 10627.0
    aflfast 82800 20.0 7107.25 720.566070 6167.0 6483.00 7104.0 7498.00 9076.0
    libfuzzer 82800 20.0 6880.25 550.392671 6418.0 6460.00 6482.5 7570.50 7622.0
    lafintel 82800 20.0 6153.30 55.904147 5967.0 6135.50 6157.5 6178.75 6246.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

libxslt_xpath summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symsan.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 18957.00 128.081800 18736.0 18888.75 18963.5 19056.50 19178.0
    honggfuzz 82800 20.0 18901.05 143.575572 18685.0 18778.25 18908.0 18996.75 19179.0
    aflsmart 82800 20.0 18880.30 60.963406 18765.0 18830.00 18870.5 18923.50 18977.0
    afl 82800 20.0 18850.15 71.677843 18686.0 18822.25 18865.5 18898.00 18976.0
    eclipser 82800 20.0 18850.15 65.984268 18748.0 18797.00 18850.0 18894.75 19022.0
    symsan 82800 1.0 18799.00 NaN 18799.0 18799.00 18799.0 18799.00 18799.0
    aflfast 82800 20.0 18673.10 73.754857 18456.0 18646.00 18674.5 18696.00 18819.0
    mopt 82800 20.0 18652.60 69.051011 18515.0 18615.25 18661.0 18694.25 18760.0
    lafintel 82800 20.0 18563.90 173.383633 18063.0 18525.25 18570.0 18664.50 18854.0
    fairfuzz 82800 20.0 18240.55 859.615855 15799.0 18005.75 18496.5 18851.75 19004.0
    entropic 82800 20.0 16407.50 178.394448 16151.0 16287.75 16355.5 16539.50 16767.0
    libfuzzer 82800 20.0 15849.00 123.141767 15700.0 15754.25 15815.5 15917.25 16110.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

mbedtls_fuzz_dtlsclient summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    symsan 82800 20.0 9281.05 1131.663307 8210.0 8307.00 8353.5 10436.50 10828.0
    afl 82800 20.0 8595.35 1016.725460 8205.0 8233.50 8265.0 8313.25 11625.0
    fairfuzz 82800 20.0 8333.35 287.818341 8187.0 8224.50 8264.5 8323.25 9530.0
    mopt 82800 20.0 8582.80 979.570883 8215.0 8230.75 8253.5 8326.00 11503.0
    aflplusplus 82800 20.0 8239.90 46.769198 8118.0 8211.75 8252.5 8277.75 8291.0
    aflsmart 82800 20.0 8259.90 46.873739 8204.0 8227.00 8243.5 8278.25 8356.0
    eclipser 82800 20.0 8395.10 673.758031 8206.0 8220.50 8243.5 8267.75 11255.0
    entropic 82800 20.0 8198.20 43.124788 8141.0 8166.75 8196.0 8215.00 8302.0
    honggfuzz 82800 20.0 8215.20 228.845294 8091.0 8122.25 8181.0 8198.75 9168.0
    lafintel 82800 20.0 8266.40 518.700644 8100.0 8122.75 8161.0 8169.00 10466.0
    aflfast 82800 20.0 8108.60 41.314455 8048.0 8075.25 8110.0 8147.25 8179.0
    libfuzzer 82800 20.0 7932.70 239.825616 7602.0 7702.00 8095.5 8137.25 8212.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openssl_x509 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 13774.95 8.369588 13752.0 13775.25 13779.0 13779.00 13785.0
    aflsmart 82800 20.0 13778.50 1.933091 13775.0 13777.00 13779.0 13779.00 13783.0
    afl 82800 20.0 13777.35 1.348488 13775.0 13777.00 13777.0 13779.00 13779.0
    eclipser 82800 20.0 13778.20 3.396593 13775.0 13775.00 13777.0 13779.00 13785.0
    mopt 82800 20.0 13775.50 5.907177 13751.0 13775.00 13777.0 13777.00 13779.0
    aflfast 82800 20.0 13770.85 8.106365 13751.0 13771.25 13775.0 13775.00 13777.0
    symsan 82800 20.0 13767.80 11.781341 13752.0 13757.25 13772.5 13779.00 13779.0
    entropic 82800 20.0 13766.60 11.962221 13745.0 13755.00 13769.5 13778.00 13779.0
    fairfuzz 82800 20.0 13758.55 8.605231 13750.0 13752.00 13757.0 13761.00 13777.0
    lafintel 82800 20.0 13757.95 16.113006 13726.0 13750.75 13753.0 13773.25 13775.0
    libfuzzer 82800 20.0 13744.00 6.316228 13724.0 13742.00 13742.0 13750.00 13751.0
    honggfuzz 82800 20.0 13738.40 11.690752 13727.0 13729.00 13732.5 13752.25 13758.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

openthread-2019-12-23 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 5892.90 14.472842 5850.0 5890.00 5897.0 5901.00 5916.0
    libfuzzer 82800 20.0 5891.15 8.875009 5877.0 5886.00 5887.5 5895.50 5910.0
    mopt 82800 20.0 5815.90 140.108191 5244.0 5826.25 5857.0 5871.00 5897.0
    eclipser 82800 20.0 5847.95 37.276315 5736.0 5838.00 5854.0 5882.25 5889.0
    aflplusplus 82800 20.0 5841.95 23.493504 5822.0 5827.75 5837.5 5840.25 5921.0
    symsan 82800 20.0 5836.15 31.422880 5735.0 5828.00 5833.5 5839.25 5898.0
    afl 82800 20.0 5829.55 39.662358 5716.0 5814.75 5832.0 5854.00 5911.0
    aflsmart 82800 20.0 5782.10 132.662572 5250.0 5789.25 5824.5 5836.75 5856.0
    honggfuzz 82800 20.0 5828.40 49.508532 5720.0 5812.75 5824.5 5872.75 5888.0
    lafintel 82800 20.0 5412.25 247.306517 5233.0 5252.75 5260.5 5673.25 5890.0
    aflfast 82800 20.0 5499.60 293.157834 5221.0 5241.75 5252.0 5816.50 5863.0
    fairfuzz 82800 20.0 4523.90 560.570001 4139.0 4139.00 4139.0 5085.25 5860.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

proj4-2017-08-14 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 7793.10 78.750706 7697.0 7727.25 7789.0 7834.50 7940.0
    libfuzzer 82800 20.0 7688.10 111.489863 7406.0 7624.00 7687.0 7773.75 7875.0
    honggfuzz 82800 20.0 7695.05 84.786093 7542.0 7638.75 7678.5 7715.00 7864.0
    symsan 82800 20.0 7241.10 334.799846 6621.0 7035.50 7262.0 7452.25 7928.0
    eclipser 82800 20.0 7073.20 165.101755 6603.0 7038.25 7115.5 7160.50 7289.0
    aflplusplus 82800 20.0 5602.85 120.008015 5452.0 5509.00 5563.5 5686.50 5861.0
    aflsmart 82800 20.0 4581.90 721.583194 3154.0 4303.25 4979.0 5105.25 5267.0
    afl 82800 20.0 4541.60 682.244315 3133.0 4198.25 4840.5 5043.75 5158.0
    fairfuzz 82800 20.0 4799.05 212.722571 4292.0 4678.75 4796.0 4985.00 5096.0
    mopt 82800 20.0 4221.75 655.483339 3094.0 3729.25 4325.0 4707.25 5066.0
    lafintel 82800 20.0 4287.55 756.823102 2826.0 3821.00 4187.5 5032.50 5216.0
    aflfast 82800 20.0 3804.80 846.694974 2583.0 3025.50 3853.0 4430.50 5084.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

re2-2014-12-09 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
error
The following fuzzers do not have enough samples: symsan.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    entropic 82800 20.0 3542.100000 18.951184 3516.0 3523.00 3555.0 3558.00 3564.0
    afl 82800 20.0 3527.450000 19.030377 3503.0 3509.75 3527.0 3545.25 3555.0
    aflplusplus 82800 20.0 3524.150000 15.267009 3508.0 3514.00 3517.0 3540.75 3553.0
    eclipser 82800 20.0 3527.550000 21.421767 3507.0 3511.00 3517.0 3549.00 3579.0
    symsan 82800 6.0 3520.333333 15.933194 3508.0 3512.75 3516.0 3517.75 3552.0
    aflsmart 82800 20.0 3522.300000 16.157449 3506.0 3511.00 3515.0 3541.25 3550.0
    libfuzzer 82800 20.0 3512.100000 15.231201 3474.0 3511.50 3514.0 3517.00 3552.0
    mopt 82800 20.0 3522.650000 21.492410 3472.0 3508.50 3514.0 3544.00 3547.0
    aflfast 82800 20.0 3516.700000 18.719818 3471.0 3505.00 3513.0 3537.25 3549.0
    honggfuzz 82800 20.0 3510.150000 12.807379 3500.0 3504.00 3505.0 3506.75 3545.0
    fairfuzz 82800 20.0 3489.050000 53.283230 3343.0 3502.00 3504.5 3508.25 3541.0
    lafintel 82800 20.0 3506.150000 26.332439 3462.0 3501.00 3504.0 3518.50 3545.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

sqlite3_ossfuzz summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 36684.80 193.007663 36163.0 36604.00 36699.0 36793.25 37031.0
    aflsmart 82800 20.0 35095.70 336.804504 34459.0 34949.25 35114.5 35303.25 35724.0
    afl 82800 20.0 34877.70 909.362825 32344.0 34776.25 35071.0 35434.25 35723.0
    mopt 82800 20.0 35012.80 469.740869 34109.0 34805.50 35045.0 35328.75 35670.0
    eclipser 82800 20.0 34675.45 408.497951 33871.0 34381.00 34733.0 34996.50 35434.0
    aflfast 82800 20.0 34658.40 339.990155 34098.0 34486.25 34598.0 34774.25 35439.0
    entropic 82800 20.0 31296.45 654.333367 30130.0 30875.75 31372.5 31766.00 32352.0
    honggfuzz 82800 20.0 30340.25 604.276677 28827.0 30082.50 30390.5 30602.25 31677.0
    lafintel 82800 20.0 26622.25 1226.556535 24828.0 25599.50 26496.5 27756.50 29156.0
    libfuzzer 82800 20.0 23851.30 1056.883856 22099.0 23139.50 23970.0 24521.50 25417.0
    fairfuzz 82800 20.0 24543.10 2704.278869 21771.0 22724.75 23765.0 25786.75 30784.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

vorbis-2017-12-11 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    aflplusplus 82800 20.0 2168.400000 2.909151 2165.0 2167.00 2168.0 2169.00 2177.0
    mopt 82800 20.0 2167.950000 5.889107 2158.0 2164.50 2167.0 2173.25 2178.0
    aflsmart 82800 20.0 2166.850000 4.934039 2159.0 2163.00 2166.5 2168.25 2177.0
    afl 82800 20.0 2165.700000 5.694873 2152.0 2163.75 2166.0 2167.00 2179.0
    symsan 82800 20.0 2166.750000 3.024027 2161.0 2165.00 2166.0 2169.00 2172.0
    eclipser 82800 20.0 2165.200000 4.237676 2157.0 2162.00 2165.5 2167.00 2176.0
    entropic 82800 20.0 2131.700000 103.035559 1830.0 2163.00 2165.0 2165.25 2172.0
    aflfast 82800 20.0 2162.750000 4.722566 2153.0 2160.75 2163.5 2164.25 2177.0
    fairfuzz 82800 19.0 2134.473684 38.013694 2047.0 2104.00 2159.0 2163.00 2167.0
    lafintel 82800 20.0 2149.100000 18.378763 2103.0 2148.50 2156.0 2160.00 2169.0
    honggfuzz 82800 20.0 2150.750000 7.765477 2135.0 2150.00 2152.5 2156.00 2160.0
    libfuzzer 82800 20.0 1976.150000 165.904943 1829.0 1830.00 1830.5 2155.50 2159.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

woff2-2016-05-06 summary

Ranking by median reached code coverage
Reached code coverage distribution
Mean code coverage growth over time
Mean code coverage growth over time
* The error bands show the 95% confidence interval around the mean code coverage.
  • Sample statistics and statistical significance (code coverage)
    Code coverage sample statistics
    count mean std min 25% median 75% max
    fuzzer time
    honggfuzz 82800 20.0 1888.30 21.940350 1836.0 1873.50 1893.0 1907.25 1915.0
    eclipser 82800 20.0 1880.70 23.468007 1849.0 1858.50 1883.0 1894.00 1929.0
    symsan 82800 20.0 1868.80 13.563108 1840.0 1859.50 1872.0 1877.50 1889.0
    aflplusplus 82800 20.0 1869.10 16.631929 1836.0 1864.25 1871.5 1881.00 1890.0
    lafintel 82800 20.0 1864.10 16.029250 1837.0 1854.00 1866.5 1870.50 1906.0
    aflsmart 82800 20.0 1861.45 24.191071 1789.0 1846.75 1863.0 1883.00 1890.0
    afl 82800 20.0 1861.30 18.067418 1839.0 1843.75 1859.0 1876.50 1887.0
    mopt 82800 20.0 1860.75 16.764232 1829.0 1851.25 1856.0 1876.00 1891.0
    entropic 82800 20.0 1841.65 87.216171 1698.0 1757.00 1837.0 1922.75 1959.0
    aflfast 82800 20.0 1825.15 10.668572 1805.0 1818.75 1827.5 1832.00 1846.0
    libfuzzer 82800 20.0 1697.10 55.169500 1599.0 1654.00 1699.5 1745.00 1799.0
    fairfuzz 82800 20.0 1661.40 32.809498 1612.0 1645.00 1663.5 1672.75 1765.0

    Vargha-Delaney A12 measure
    The table summarizes the A12 values from the pairwise Vargha-Delaney A measure of effect size. Green cells indicate the probability the fuzzer in the row will outperform the fuzzer in the column.
    Mann-Whitney U test
    The table summarizes the p values of pairwise Mann-Whitney U tests. Green cells indicate that the reached coverage distribution of a given fuzzer pair is significantly different.

experiment data

You can download the raw data for this report here.

Check out the documentation on how to create customized reports using this data. Also see some example Colab notebooks for doing custom analysis on the data here.

Experiment Description:

from cached data