chore(tests): antithesis harness, durability harness scenario#25562
Open
blt wants to merge 4 commits into
Open
chore(tests): antithesis harness, durability harness scenario#25562blt wants to merge 4 commits into
blt wants to merge 4 commits into
Conversation
Contributor
Author
This stack of pull requests is managed by Graphite. Learn more about stacking. |
github-actions
Bot
added
the
domain: ci
9 tasks
blt
changed the title
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
ec466b3 to
8fe00bc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds an Antithesis-based deterministic simulation testing harness to Vector, centered on a new multi-node “vector → vector” end-to-end scenario for exercising disk_v2 durability/conservation and liveness under injected faults. It also checks in a large “scratchbook” (property/evidence catalog) and introduces Antithesis SDK instrumentation behind a new antithesis Cargo feature.
Changes:
- Add Antithesis scratchbook documentation (semantic claims ledger + property/evidence catalog) under
tests/antithesis/scratchbook/. - Add a runnable Antithesis scenario
vector_to_vector_e2e_disk(compose config, workload/oracle binaries, launch script, reload fault helper). - Add
antithesisfeature wiring + SDK-based assertions/instrumentation inlib/vector-buffers(disk_v2reader/writer/ledger), plus workspace and dependency updates.
Reviewed changes
Copilot reviewed 76 out of 77 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/antithesis/scratchbook/semantic-claims-ledger.md | Semantics-first claims vs code reality ledger for the disk buffer e2e scenario. |
| tests/antithesis/scratchbook/properties/throughput-progresses-under-contention.md | Property/evidence note describing throughput under contention. |
| tests/antithesis/scratchbook/properties/sink-failure-not-silently-acked.md | Property/evidence note for sink-error ack semantics. |
| tests/antithesis/scratchbook/properties/recovery-completes-after-crash.md | Property/evidence note for crash recovery completion. |
| tests/antithesis/scratchbook/properties/record-id-monotonicity-holds.md | Property/evidence note for record-id monotonicity invariants. |
| tests/antithesis/scratchbook/properties/overflow-chain-no-unaccounted-gap.md | Property/evidence note for overflow buffer chain behavior. |
| tests/antithesis/scratchbook/properties/no-corrupted-record-delivered.md | Property/evidence note for corruption validation and delivery gating. |
| tests/antithesis/scratchbook/properties/multi-hop-conservation-no-loss.md | Property/evidence note for N-hop conservation expectations. |
| tests/antithesis/scratchbook/properties/fsync-window-bounded-under-clock-jitter.md | Property/evidence note for fsync window and clock jitter faults. |
| tests/antithesis/scratchbook/properties/foreign-data-file-no-writer-stall.md | Property/evidence note for foreign .dat file impact. |
| tests/antithesis/scratchbook/properties/file-id-rollover-stays-coordinated.md | Property/evidence note for file-id rollover coordination. |
| tests/antithesis/scratchbook/properties/every-written-event-eventually-delivered.md | Property/evidence note for at-least-once delivery across crashes. |
| tests/antithesis/scratchbook/properties/durable-unacked-events-survive-crash.md | Property/evidence note for durability of fsync’d-but-unacked events. |
| tests/antithesis/scratchbook/properties/dropped-events-are-counted.md | Property/evidence note for drop accounting metrics expectations. |
| tests/antithesis/scratchbook/properties/delivery-is-at-least-once-not-exactly-once.md | Clarification note: duplicates are expected (anti-vacuity). |
| tests/antithesis/scratchbook/properties/corruption-skip-record-id-accounting-consistent.md | Evidence note linking corruption skip to accounting/monotonicity. |
| tests/antithesis/scratchbook/properties/corruption-skip-loss-is-counted.md | Evidence note for corruption skip loss accounting/metrics. |
| tests/antithesis/scratchbook/properties/corruption-skip-loss-bounded.md | Evidence note for bounding loss on corruption skip behavior. |
| tests/antithesis/scratchbook/properties/corruption-is-detected-and-recovered.md | Reachability note for corruption detection/recovery execution. |
| tests/antithesis/scratchbook/properties/ack-is-per-hop-not-transitive.md | Clarification note about per-hop vs transitive acks. |
| tests/antithesis/scratchbook/properties/ack-does-not-imply-durability.md | Evidence note about 200/ack not implying fsync durability. |
| tests/antithesis/scratchbook/grind-plan.md | Execution plan for running/triaging Antithesis “shots”. |
| tests/antithesis/scratchbook/existing-assertions.md | Inventory of committed Antithesis assertions and harness checks. |
| tests/antithesis/scratchbook/.markdownlint.jsonc | Scratchbook-scoped markdownlint relaxations. |
| tests/antithesis/scratchbook/_external-references-digest.md | Digest of in-repo/external references and known bug targets. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/tail.yaml | Tail node Vector config for scenario. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/parallel_driver_produce.rs | Workload producer command: claim ID, POST to head, report ack. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/oracle.rs | Oracle service: issues IDs, validates delivery + payload integrity. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/src/bin/eventually_conservation.rs | Eventually-phase judge: drain + conservation + liveness probe. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/README.md | Scenario documentation. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/launch.sh | Pinned snouty launch wrapper with fault profile. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/head.yaml | Head node Vector config for scenario. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/head.b.yaml | Alternate head config swapped by reload fault. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/Dockerfile | Builds SUT Vector + workload binaries with Antithesis coverage flags. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/docker-compose.yaml | Compose topology for head/tail/oracle + volumes/healthchecks. |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/Cargo.toml | Scenario crate manifest (workload/oracle binaries). |
| tests/antithesis/scenarios/vector_to_vector_e2e_disk/anytime_reload.sh | In-container config swap + SIGHUP reload helper. |
| tests/antithesis/README.md | Top-level Antithesis test directory README. |
| tests/antithesis/harness/src/lib.rs | Shared harness helpers (payload generation/validation). |
| tests/antithesis/harness/Cargo.toml | Harness crate manifest. |
| tests/antithesis/harness/.gitignore | Harness build artifacts ignore. |
| tests/antithesis/AGENTS.md | Agent guidance for running Antithesis scenarios and launching shots. |
| LICENSE-3rdparty.csv | Adds Antithesis crates to license inventory. |
| lib/vector-lib/Cargo.toml | Adds antithesis feature forwarding to vector-buffers/antithesis. |
| lib/vector-buffers/src/variants/disk_v2/writer.rs | Adds Antithesis assertions around file size, blocking, rollovers, large records. |
| lib/vector-buffers/src/variants/disk_v2/reader.rs | Adds Antithesis assertions for monotonicity, bad read roll, delivery invariants. |
| lib/vector-buffers/src/variants/disk_v2/mod.rs | Exposes common module + antithesis env override for max data file size. |
| lib/vector-buffers/src/variants/disk_v2/ledger.rs | Adds Antithesis assertions for underflow detection and reopen reachability. |
| lib/vector-buffers/src/lib.rs | Links Antithesis instrumentation + re-exports write buffer size under feature. |
| lib/vector-buffers/Cargo.toml | Adds optional Antithesis deps + antithesis feature definition. |
| Cargo.toml | Adds workspace members + Antithesis deps + top-level antithesis feature wiring. |
| Cargo.lock | Locks new crates and dependencies. |
| .github/actions/spelling/excludes.txt | Excludes tests/antithesis/ from spelling checks. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fe00bcb6c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
2 times, most recently
from
615dda3 to
18fa9da
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 18fa9da292
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
2 times, most recently
from
14630a6 to
3614c5a
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3614c5a966
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9 tasks
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
3614c5a to
5209ae6
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5209ae6df4
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
9 tasks
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
5209ae6 to
24d8e0f
Compare
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24d8e0f1ec
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
24d8e0f to
2445cb8
Compare
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
2445cb8 to
5e79a45
Compare
Antithesis research artifacts for disk buffer v2: SUT analysis, 29-property catalog (incl. the silent data-loss cluster), property relationships, evaluation synthesis, per-property evidence files, and the run log. Internal references genericized.
Self-driving exercisers (disk_v2_antithesis: #21683 accounting cluster; disk_v2_lossfinder: 7-scenario data-loss fault menu with a no-silent-loss oracle) plus the full Antithesis harness: Dockerfiles, docker-compose configs (block / drop variants / direct / lossfinder), the Vector-SUT workload, and the test templates under tests/antithesis/test/v1. Registers the examples + serde_json dev-dep.
Adds antithesis_sdk + antithesis-instrumentation deps and three #[inline] assert_always! guards that fold away in production builds (no-ops outside Antithesis) at the unchecked-arithmetic sites: ledger.rs decrement_total_buffer_size and get_total_records (#21683 / 0-1 underflow), and reader.rs:524 data-file size-delta. These guide Antithesis to the accounting-underflow cluster (organically reproduced #21683 in run D0).
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5e79a45c3a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
2 times, most recently
from
487ed7e to
1dbdb4e
Compare
blt
force-pushed
the
blt/antithesis-harness-first-scenario
branch
from
1dbdb4e to
b2c70cd
Compare
blt
added
the
no-changelog
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR introduces a new antithesis harness for Vector, similar to the work I have done recently for saluki. The goal here is to provide a stable foundation for this project to add deterministic simulation testing. Vector project has a good practice of internal property testing and the like but it's tricky to demonstrate good behavior across multiple, networked Vector instances.
To that end and to provide a demonstration about what we can do with Antithesis I've put together a test scenario
vector_to_vector_e2e_diskthat is meant to demonstrate a Vector -> Vector maintains conservation of input and liveness. There's more detail in that scenario about how it functions, seetests/antithesis/scenarios/. The scenario is not rigged up into CI in this PR. Submission requires Antithesis credentials and is done now manually.Finally, I have added antithesis_sdk assertions through the code. I don't yet have a House Style for these that I like yet, open to feedback here.
Vector configuration
N/A
How did you test this PR?
The change is itself test code.
Change Type
Is this a breaking change?
Does this PR include user facing changes?
no-changeloglabel to this PR.References
Notes
@vectordotdev/vectorto reach out to us regarding this PR.pre-pushhook, please see this template.make fmtmake check-clippy(if there are failures it's possible some of them can be fixed withmake clippy-fix)make testgit merge origin masterandgit push.Cargo.lock), pleaserun
make build-licensesto regenerate the license inventory and commit the changes (if any). More details on the dd-rust-license-tool.