Feat: Add FuzzTesting scaffold for ResponseStreamEvent decoding#421
Open
Krivoblotsky wants to merge 2 commits into
Open
Feat: Add FuzzTesting scaffold for ResponseStreamEvent decoding#421Krivoblotsky wants to merge 2 commits into
Krivoblotsky wants to merge 2 commits into
Conversation
…ness First step toward OSS-Fuzz integration. Adds a separate SwiftPM package under FuzzTesting/ (kept out of the main Package.swift so library consumers are unaffected) with one libFuzzer harness targeting ResponseStreamEvent JSON decoding -- a high-value surface that runs several fallible decode paths on untrusted bytes from the network. The harness ships with two modes: - Replay (default `swift run` build): decodes a single input file and exits. Useful for reproducing crash artifacts locally without a libFuzzer toolchain. - Fuzzer (`-sanitize=fuzzer,address -parse-as-library -DFUZZING_ENABLED`): libFuzzer's runtime drives `LLVMFuzzerTestOneInput`. Includes a small seed corpus drawn from real event shapes (`response.output_text.delta`, `response.function_call_arguments.done`, `response.mcp_call_arguments.done`). Refs #241. Upstream `google/oss-fuzz` project submission is left for a follow-up. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Extend the FuzzTesting scaffold with libFuzzer harnesses for the remaining bytes-off-the-wire Codable types: - FuzzChatResultDecoder — /v1/chat/completions non-streaming response - FuzzChatStreamResultDecoder — streaming chunk type - FuzzResponseObjectDecoder — /v1/responses non-streaming response - FuzzAudioTranscriptionStreamResultDecoder — audio transcription streaming events (hand-written init(from:) with type-branching) Each ships with 1-3 realistic seed JSONs. All five harnesses build with plain `swift build` and all 11 seeds replay without crashing. The package only depends on the public OpenAI module surface — no @testable / internal access — so it stays cross-package-clean. Refs #241. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a
FuzzTesting/directory at the repo root containing a separate SwiftPM package with one libFuzzer harness:FuzzTesting/Package.swift— declares the package and one.executableTarget.FuzzTesting/Sources/FuzzResponseStreamEventDecoder/FuzzResponseStreamEventDecoder.swift—LLVMFuzzerTestOneInputharness fortry JSONDecoder().decode(ResponseStreamEvent.self, …), plus a@mainreplay entry point gated behind!FUZZING_ENABLEDfor use without a libFuzzer toolchain.FuzzTesting/FuzzCorpus/FuzzResponseStreamEventDecoder/— three seed inputs (output_text.delta, function_call_arguments.done, mcp_call_arguments.done).FuzzTesting/README.md— replay-mode and fuzzer-mode invocations, plus how to add more harnesses.The main
Package.swiftis unchanged; library consumers see no impact.Why
First in-repo step toward #241 — Integrate OSS-Fuzz.
ResponseStreamEvent.init(from:)is a good first target — it runs several fallible decode paths (early-decode forResponseEvent,OutputItem,MCPCallArguments,ResponseFunctionCallArgumentsDoneEvent, then a generated raw event with ~50 oneOf cases) over bytes coming straight off the network. Historical bug fixes in stream decoding (#383, #387) suggest this surface benefits from coverage-guided mutation.The harness deliberately ignores decode errors — we're looking for crashes, hangs, and sanitizer findings.
Affected Areas
FuzzTesting/. Separate package, separate.gitignore, no changes to the main package, build, or tests.google/oss-fuzzupstream project submission is separate maintainer work).More Info
Replay mode (no libFuzzer toolchain):
cd FuzzTesting swift build swift run FuzzResponseStreamEventDecoder FuzzCorpus/FuzzResponseStreamEventDecoder/seed-function-call-args-done.jsonI ran this against all three seeds locally — each decoded without crashing.
Fuzzer mode (requires Swift toolchain with libFuzzer runtime):
cd FuzzTesting swift build -c debug \ -Xswiftc -sanitize=fuzzer,address \ -Xswiftc -parse-as-library \ -Xswiftc -DFUZZING_ENABLED ./.build/debug/FuzzResponseStreamEventDecoder FuzzCorpus/FuzzResponseStreamEventDecoder -max_total_time=60Open questions for review:
ServerSentEventsStreamParser),ChatStreamResultdecode, and theModelResponseEventsStreamInterpreterdata path. Happy to extend in this PR or follow-ups — preference?google/oss-fuzzupstream submission: that lives in google/oss-fuzz and needs maintainer primary/auto-CC email contacts I shouldn't put in. I left it for you. Note: OSS-Fuzz Swift support is lighter than C/C++/Rust/Go — worth a check before committing to it.🤖 Generated with Claude Code