Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
532 changes: 287 additions & 245 deletions e2e/FAULT_TESTING.md

Large diffs are not rendered by default.

13 changes: 7 additions & 6 deletions e2e/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -18,23 +18,24 @@ FAULT_SCRIPT := $(CURDIR)/scripts/fault-test.sh
MANIFEST := $(CURDIR)/Cargo.toml
FAULT_BUILD_JOBS ?= 1

.PHONY: help fault-check fault-preflight fault-run fault-run-regular fault-run-dm fault-cleanup
.PHONY: help fault-check fault-list fault-preflight fault-run fault-run-dm fault-cleanup

help:
@echo "RustFS e2e fault-test package"
@echo ""
@echo "Usage:"
@echo " make -C e2e fault-check"
@echo " make -C e2e fault-preflight [SCENARIO=io-eio]"
@echo " make -C e2e fault-list"
@echo " make -C e2e fault-run SCENARIO=io-eio"
@echo " make -C e2e fault-run-regular"
@echo " make -C e2e fault-run-dm"
@echo " make -C e2e fault-cleanup"
@echo ""
@echo "Required runtime environment:"
@echo " RUSTFS_FAULT_TEST_EXPECTED_CONTEXT"
@echo " RUSTFS_FAULT_TEST_STORAGE_CLASS"
@echo " RUSTFS_FAULT_TEST_SERVER_IMAGE"
@echo "Optional safety guard:"
@echo " RUSTFS_FAULT_TEST_EXPECTED_CONTEXT"
@echo ""
@echo "See e2e/FAULT_TESTING.md for cluster preparation and safety requirements."

Expand All @@ -44,16 +45,16 @@ fault-check:
CARGO_BUILD_JOBS=$(FAULT_BUILD_JOBS) cargo test --manifest-path $(MANIFEST)
CARGO_BUILD_JOBS=$(FAULT_BUILD_JOBS) cargo clippy --manifest-path $(MANIFEST) --all-targets -- -D warnings

fault-list:
@bash $(FAULT_SCRIPT) list

fault-preflight:
@bash $(FAULT_SCRIPT) preflight "$(or $(SCENARIO),io-eio)"

fault-run:
@test -n "$(SCENARIO)" || (echo "SCENARIO is required" >&2; exit 2)
@bash $(FAULT_SCRIPT) run "$(SCENARIO)"

fault-run-regular:
@bash $(FAULT_SCRIPT) run-regular

fault-run-dm:
@bash $(FAULT_SCRIPT) run dm-flakey

Expand Down
14 changes: 12 additions & 2 deletions e2e/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ The harness is intentionally separated from the main operator crate so e2e-only

## Architecture

The harness is split into four top-level domains:
The harness is split into five top-level domains:

- `manifests/`: e2e-owned static manifests such as the dedicated Kind config.
- `framework/`: reusable infrastructure primitives.
- `fault/`: real-cluster fault-test orchestration and fault-specific helpers.
- `cases/`: release test-case inventory grouped by product boundary.
- `tests/`: executable suite entrypoints; live tests are ignored by default and run only through explicit Make targets.

Expand All @@ -24,9 +25,18 @@ e2e/
src/
lib.rs
bin/rustfs-e2e.rs Makefile-internal helper for live workflow steps
fault/
config.rs real-cluster fault-test configuration and safety checks
scenarios.rs fault scenario catalog
plan.rs fault plan expansion for one or more fault injections
runner.rs destructive fault-test orchestration
fixture.rs fault namespace ownership and real-cluster Tenant fixture
backends/ Chaos Mesh and host-side fault backends
workload.rs S3 workload generation and execution
history.rs workload operation recorder
checker.rs committed-object correctness checker
framework/
config.rs dedicated Kind e2e configuration
fault_config.rs real-cluster fault-test configuration and safety checks
command.rs safe subprocess wrapper for kind/docker/kubectl
kind.rs Kind cluster lifecycle and host mount preparation
kubectl.rs kubectl command construction boundary
Expand Down
Loading
Loading