Merge `cpython3` and `python3-libraries` fuzzers?

[StanFromIreland]

Our fuzzers are currently split between Modules/_xxtestfuzz/ and this repository. I think we should merge them into one cpython project,

• There is the idea of cpython3 fuzzing the core interpreter, and python3-libraries fuzzing the stdlib, but that isn't the case. In fact, more than half of cpython3s fuzzers are for stdlib modules. Even worse so, we have overlap, for example, both projects fuzz csv.reader().

• I'm writing this issue because after opening Test building in CI #25, I considered doing the same in CPython for _xxtestfuzz, but that would just duplicate infrastructure. We would have two almost identical workflows in two very different places. This split/duplication also increases our maintenance burden, we have to maintain quite similar (even more so with Refactor OSS-Fuzz integration #23) scripts for two projects in google/oss-fuzz.

• We actually already share resources between the fuzzers by using the py_compile.dict from cpython3 for python3-libraries's fuzzer-ast, but this is quite hacky and easy to forget about, as such I worry it may break some day.

CC @python/fuzzers

[sethmlarson]

I am in favor of merging the fuzzers and am not married to having fuzzers live in a separate repository to the code being fuzzed. Would be great to have them all in the CPython repository. 👍

[StanFromIreland]

Would be great to have them all in the CPython repository.

I'd argue the other way, we'd be burdening everyone with some quite large corpus collections. And, we also have more freedom in another repository.

[ammaraskar]

Just for some historical context, originally the fuzzers inside CPython were the first set added to oss-fuzz. The ones that currently exist here were originally made by a third-party person Guido Kraken, after they were deleted Hugo took them over and eventually Seth transferred over Hugo's repo here.

I'm in favor of consolidation, as far as some of my notes/ideas on consolidation:

• I was able to get memory sanitizer working but only with building C fuzzers, I think it has found a fair few heap memory issues so I would love for that to stay working.
• Python fuzzers are very easy to write but might be slower than their C equivalents, ideally if we have a high speed C fuzzer covering a module with the same coverage, we don't need the python version.
• Having the C fuzzers live in cpython was semi-important so they could be kept updated and maintained when people made changes to the C API over time, as seen e.g python/cpython@1887a95 and python/cpython@d5bd32f, this is much less of a concern for python-based fuzzers.

[StanFromIreland]

I was able to get memory sanitizer working but only with building C fuzzers, I think it has found a fair few heap memory issues so I would love for that to stay working.

I recently got python3-libraries set up with MSAN, it works, but not ideally since we don't instrument dependencies and as a result we get false positives (I've yet to get around to opening a PR for this), e.g. this testcase. It is also quite slow, so I excluded it from my CI build proposal.

I also presume we could have a hybrid approach, with some fuzzers in Python and others in C/C++.

[ammaraskar]

Yeah that's the main issue with MSAN, stuff has to be transitively built with it otherwise you get the false positives you're seeing. I think the C based fuzzers avoid this because they only go into the C library and some core interpreter machinery.

Hybrid approach sounds good, only caution would be making sure the fuzzers stay building and healthy which we got for free with them being under the main repo.

[ammaraskar]

Maybe dictionaries/corpus living here and fuzzers living in CPython repo could be good?

[StanFromIreland]

Hybrid approach sounds good, only caution would be making sure the fuzzers stay building and healthy which we got for free with them being under the main repo.

Not exactly... some time ago I broke a target, and there is nothing in the CI that checks for that¹, as normal debug builds just build _xxtestfuzz as an extension module, not as stand-alone libFuzzer binaries (which we do by passing -D_Py_FUZZ_<target> IIRC).

I think the C based fuzzers avoid this because they only go into the C library and some core interpreter machinery

They don't target any modules with dependencies, python3-libraries fuzzes modules like {zip,tar}file.

Maybe dictionaries/corpus living here and fuzzers living in CPython repo could be good?

This is better than having everything in CPython, though it would still be a little painful to deal with two repos. OTOH, we rarely change them.

Footnotes

1. Technically there is CIFuzz, which caught the failure, but only after the PR was merged. I suppose this could be solved by adding a job to check, like https://github.com/python/library-fuzzers/pull/25. ↩