Product-limits stress, dual-outage matrix, resume, and lblk integrity… by RaunakJalan · Pull Request #1357 · simplyblock/sbcli

RaunakJalan · 2026-09-14T21:06:40Z

… primitives

Rebuilt directly on main now that #1356 has landed, so this carries only the work that PR did not: 17 files, all in e2e/ and .github/, no product code.

Contents:

product_limits_stress.py, a docker/k8s stress test for the advertised object limits, plus the 70TiB variants.
dual_outage_matrix.py, a 90-case table covering every outage-type pair at every distance along the secondary chain. Existing multi-outage tests pick victims through _pick_outage_nodes, which deliberately refuses to take a node and its own secondary together, so the topology that matters most has never been reachable. Separation 0 is exactly that pair.
run_state.py and the resume path, so a stress run that dies at hour 20 can re-enter where it stopped instead of wiping the cluster and starting over. The wipe is gated at all three sites, because eleven classes override setup() without calling super() and a base-class guard alone would be bypassed by most of the tests that need it. Mass-create declares itself unsupported rather than pretending: it is phase-shaped, and re-entering at phase 5 with an empty registry would delete nothing and report a pass.
raw_device_verify.py and md_journal.py, the integrity primitives for non-NVMe (lblk) clusters. crc32c on the raw device with a disjoint churn region, and the journal's own statistics over raw JSON-RPC, since neither journal method is registered in scripts/rpc.py.
RESUME and case-selection inputs on the two stress pipelines.

Rebuilt rather than merged for the same reason #1356 was. feat/lblk-e2e still carries the R26.3-original commits that main holds as adapted copies, so a merge or rebase fights them rather than dropping them as already-applied. This applies only the delta between backup-rework's tip and lblk's, three-way, onto current main.

Verified: main's own TestObjectLimits registration and device-claim guard survive the patch, the two NameErrors #1356 fixed are not reintroduced, ruff reports nothing in e2e/ or .github/ (all 45 findings are in main's product test files), e2e compiles on 3.11, and discovery finds all six new classes.

… primitives Rebuilt directly on main now that #1356 has landed, so this carries only the work that PR did not: 17 files, all in e2e/ and .github/, no product code. Contents: - product_limits_stress.py, a docker/k8s stress test for the advertised object limits, plus the 70TiB variants. - dual_outage_matrix.py, a 90-case table covering every outage-type pair at every distance along the secondary chain. Existing multi-outage tests pick victims through _pick_outage_nodes, which deliberately refuses to take a node and its own secondary together, so the topology that matters most has never been reachable. Separation 0 is exactly that pair. - run_state.py and the resume path, so a stress run that dies at hour 20 can re-enter where it stopped instead of wiping the cluster and starting over. The wipe is gated at all three sites, because eleven classes override setup() without calling super() and a base-class guard alone would be bypassed by most of the tests that need it. Mass-create declares itself unsupported rather than pretending: it is phase-shaped, and re-entering at phase 5 with an empty registry would delete nothing and report a pass. - raw_device_verify.py and md_journal.py, the integrity primitives for non-NVMe (lblk) clusters. crc32c on the raw device with a disjoint churn region, and the journal's own statistics over raw JSON-RPC, since neither journal method is registered in scripts/rpc.py. - RESUME and case-selection inputs on the two stress pipelines. Rebuilt rather than merged for the same reason #1356 was. feat/lblk-e2e still carries the R26.3-original commits that main holds as adapted copies, so a merge or rebase fights them rather than dropping them as already-applied. This applies only the delta between backup-rework's tip and lblk's, three-way, onto current main. Verified: main's own TestObjectLimits registration and device-claim guard survive the patch, the two NameErrors #1356 fixed are not reintroduced, ruff reports nothing in e2e/ or .github/ (all 45 findings are in main's product test files), e2e compiles on 3.11, and discovery finds all six new classes.

Three changes asked for after the first cut. The matrix gains multipath as a fourth axis and stops needing a case selector. 15 pairs x 3 separations = 45 topologies, x drain/inflight x mp/nomp = 180 per platform, 360 across docker and k8s. Multipath is now a property driven by the case rather than a class attribute, so it varies within a single run; it is exposed under the base class's MULTIPATH_MODE name so the inherited _multipath_selected keeps working untouched. Each platform class sweeps its whole table in one invocation. A case is a few minutes of outage and recovery, so scheduling 180 pipeline runs per platform would spend more time bootstrapping clusters than injecting faults. The base run() is an open-ended stress loop, so completion is signalled by DualOutageMatrixComplete from the seam and caught in run(). A sweep that dies at case 120 resumes there through the existing checkpoint. --case still runs exactly one, which is the debugging path once the sweep has found something. Topology skips no longer consume an iteration: a case the cluster cannot express is stepped over and reported at the end, so a 4-node cluster reports 180 skips rather than aborting on the first one. lblk gets actual tests, not just the primitives from the previous commit. Ten classes across five lanes: a functional smoke, raw-crc32c integrity across steady state and restart and three outage types, block-device hot-remove and hung IO (which have no NVMe-PCIe equivalent, since lblk devices are never PCI-bound to SPDK), journal replay forced by pausing the drain, and a reproducer for the missing leadership fence the journal header documents -- the drain stops on demotion but a demoted node can still append, and their own test F3 saw a thawed leader write 15 further md ops. Every class asserts device_mode == "lblk" first. A cluster created with an older control plane silently reverts to nvme, comes up healthy, and would let all ten pass while testing the wrong storage path. One bug worth naming: the platform methods were first declared as NotImplementedError stubs on the mixins. The mixins precede the platform bases in the MRO, so the stubs shadowed the real implementations and every call would have raised on a live cluster. Caught by checking __qualname__ resolution; the stubs are gone and the contract is documented in the class docstring instead. Verified without a cluster: the table is 180 unique ids with the axes evenly split; a simulated 6-node ring sweeps all 180 with no skips; a 4-node ring and an npcs=1 run skip all 180 and still terminate; multipath resolves per case; --case runs exactly one; all twelve classes are discoverable (104 stress tests) and dispatch resolves to the platform bases. ruff reports nothing in e2e/, and e2e compiles on 3.11.

You were right: all ten went into stress_test/ when only the soak belongs there. The convention across the tree is clear -- e2e_tests/ is 72 classes on TestClusterBase, stress_test/ is the failover and loop bases -- and none of the ten were stress-shaped. They are finite scenarios, which is the e2e_tests/ shape, the same as TestMultiNodeOutage*. e2e_tests/lblk/test_lblk.py now holds the functional and integration classes on TestClusterBase: the smoke, integrity across steady state and three outage types, device hot-remove, journal replay, and the unfenced-journal reproducer. Outages go through sbcli_utils.shutdown_node/restart_node rather than the stress helpers, which is what let them come off the stress bases at all; the same route TestMultiNodeOutage* already uses. stress_test/lblk_stress.py is what actually belongs in stress_test/: an open-ended soak that hooks the parent loop's seam and re-verifies the raw device after every outage set, until something breaks. It adopts a volume the parent already built rather than creating its own, so it measures the objects the rest of the test is exercising. Registration follows the existing split: `--testname lblk` runs the ten via a new get_lblk_tests(), matching how security and backup are grouped, and the two soaks sit in get_stress_tests(). Stress drops from 104 to 96 as a result. The platform hooks are mixins applied before the scenario base, so _spdk_exec_prefix resolves to _LblkDockerMixin or _LblkK8sMixin rather than being shadowed -- the MRO trap from the previous commit, avoided by ordering this time rather than by deleting stubs. Verified: 10 classes under `--testname lblk` all inheriting TestClusterBase, 2 under stress, platform hooks and the stress seam resolving correctly, ruff clean in e2e/, and e2e compiling on 3.11.

"sep0", "nomp" and "drain" were shorthand from the design notes that leaked into permanent test ids, and they told a reader nothing. A case id is what shows up in a pipeline input, a Slack failure and an RCA six months later, so it should say what the case actually does. dual_graceful_shutdown_container_stop_sep0_drain_nomp dual_graceful_shutdown_with_container_stop_secondary_await_migration_allpaths The pair-distance axis is now named for what the second victim IS rather than for a hop count: secondary, tertiary, quaternary. "secondary" says the pair serves the same lvstore, which is the entire reason that case exists -- it is the one _pick_outage_nodes deliberately refuses to produce. "sep0" said only that a counter was zero. Migration becomes await_migration / during_migration, and multipath becomes allpaths / onepathdown, both of which read without a glossary. The numeric separation is still carried on the case as "separation" because the chain walk needs the hop count; "separation_name" carries the label. Also updated the two workflow help strings that used the old id as their example, so nobody copies a case id that no longer exists. Still 180 unique cases per platform, 360 across both. ruff clean in e2e/, and e2e compiles on 3.11.

Run on actions-runner-3 died before a single test ran: File "e2e/utils/ssh_utils.py", line 252, in SshUtils def _try_connect(self, ..., pkey: paramiko.PKey | None, ...): TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' `X | None` is PEP 604 and is evaluated when the def executes, so it needs Python 3.10+ at runtime. That annotation is on main, not from this branch, and the file carries no `from __future__ import annotations`. It has not bitten before because e2e.py is invoked as a bare `python3` on the self-hosted runner and the workflow pins no interpreter, so which Python runs is whatever that host happens to have. actions-runner2 ran the suite for 7h51m last week; actions-runner-3 is older than 3.10 and fails at import. Adding the future import to every e2e file carrying such an annotation makes the suite independent of that, which it should be: a test suite that only runs on some of the runners is a coin toss, and the failure mode is an import error hours into a bootstrap rather than anything diagnosable. Safe here: PEP 563 only defers annotations to strings, and nothing in e2e reads them back -- no get_type_hints, no __annotations__, no pydantic, no dataclass. Verified by importing utils/ssh_utils.py under a real 3.9: it raises the same TypeError without the future import and imports cleanly with it. 3.11 still compiles, ruff is clean, and discovery still finds 10 lblk and 96 stress tests. This is a workaround, not the fix. pyproject declares requires-python >= 3.11 and the CI lint/type jobs use 3.11, so actions-runner-3 is out of spec and should be brought up to 3.11 regardless.

+    for attempt in range(retries):
+        try:
+            ssh = paramiko.SSHClient()
+            ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())


+            key = _load_private_key(KEY_PATH)
+            if bastion_ip:
+                bastion = paramiko.SSHClient()
+                bastion.set_missing_host_key_policy(paramiko.AutoAddPolicy())


+def exec_command(ssh: paramiko.SSHClient, command: str, retries=3) -> tuple[str, str]:
+    for attempt in range(retries):
+        try:
+            print(f"[INFO] Executing: {command}")


The workflows invoke the suite as a bare `python3`, so which interpreter runs is whatever the self-hosted runner happens to have. actions-runner2 ran the suite for 7h51m last week; actions-runner-3 is on an older Python and died at import: def _try_connect(self, ..., pkey: paramiko.PKey | None, ...) TypeError: unsupported operand type(s) for |: 'type' and 'NoneType' PEP 604 is evaluated at def time and needs 3.10+. pyproject declares requires-python >= 3.11 and CI's lint/type jobs use 3.11, so 3.11 is the contract -- the runners were just never held to it, and the failure surfaced hours into a bootstrap on one runner and not another. e2e/scripts/ensure_python.sh finds an interpreter >= 3.11, fetches a standalone one if the host has none, builds a venv with the e2e requirements, and puts it first on PATH. Every later `python3` in the job then resolves to it, so the nine workflows that run the suite needed only their dependency-install step swapped, not every invocation. It never touches a Python the host already has. uv comes from the standalone installer into our own cache with INSTALLER_NO_MODIFY_PATH=1, not `pip install --user`, which would write into the host user's site-packages and can shadow what the system depends on; `uv python install` unpacks a standalone build into uv's own directory and creates no shim, because `uv python update-shell` is never called. Nothing is written outside $HOME/.cache/sbcli-e2e, and the PATH change is scoped to the GitHub job. Compulsory on purpose: if no suitable interpreter can be obtained it exits non-zero rather than silently falling back to the host's, which is how this went unnoticed. The uv technique is taken from simplyBlockDeploy PR #221, which installs sbctl against a uv-managed 3.11 because the storage nodes do not ship one either -- same problem, other side of the ssh connection. That PR installs to /usr/local/bin; this does not, since a runner job should own nothing shared. Verified against a host with only Python 3.9 on PATH: it reports none suitable, fetches uv, fetches CPython 3.11.16, builds the venv, installs paramiko / requests / boto3 into it, and leaves the 3.9 untouched. On a host that already has 3.12 it reuses it and fetches nothing. A second run reuses the venv. Two bugs found while testing and fixed: log() wrote to stdout while install_with_uv is consumed via $(...), so log lines were being captured as the interpreter path; and the venv bin directory is Scripts/ on Windows, which blocked local testing entirely.

The first cut preferred whatever Python the host had and only reached for uv when there was none. That still leaves runner2 on 3.12, runner3 on 3.9 and a third on 3.13 -- the same inconsistency that let a PEP 604 annotation sit unnoticed for weeks and then fail hours into a bootstrap on one runner only. uv now comes first, as simplyBlockDeploy PR #221 already does for the nodes, so every runner uses the same pinned 3.11. A host interpreter is used only if uv cannot be obtained at all, which keeps an offline runner with a good 3.11 working. Both uv and its CPython are cached under our own directory, so the network is touched once per runner, not once per run. Also corrects the failure message. This is not a style preference that a __future__ import can paper over: e2e_tests/cluster_test_base.py, which every test inherits, does `from datetime import UTC` -- added in 3.11, a hard ImportError below it, and not polyfillable. 14 e2e files use it. Verified: on a host with 3.12 it still builds a 3.11 venv; a rerun reuses uv, the interpreter and the venv; and with the network blocked it reports the uv failure and falls back to the host's 3.12 rather than dying.

test_object_limits.py asserts against simplyblock_core.constants.MAX_LVOL_SIZE rather than restating the number, which is right -- a copied constant silently stops matching the day the product changes it. But e2e runs with e2e/ as the working directory, so the repo root is not on sys.path and simplyblock_core is not importable. That went unnoticed because the runners had an sbcli pip-installed globally at some point and the import was satisfied by accident. Moving e2e into a clean venv removed the accident, and __init__.py died at line 230 with ModuleNotFoundError, taking all 230 test classes with it before any test ran. simplyblock_core.constants imports nothing but stdlib and _version, so putting the repo root on the path is enough -- far cheaper than installing the product and its dependency tree into the e2e venv for one constant. Appended rather than inserted, deliberately. The repo root has a scripts/ directory and so does e2e, and on Python 3 any directory is importable as a namespace package, so prepending would let the repo root shadow e2e's own modules. Appending also means a properly installed sbcli still wins, since site-packages comes first, so this changes nothing for anyone who has one. Verified in a clean 3.11 venv with no product installed: __init__ imports, all 230 classes load, LblkFunctionalDocker resolves through ALL_TESTS (which is what --testname searches, not get_lblk_tests), and the repo root lands at sys.path index 7 of 8.

argparse treats any token starting with a dash as an option, so `--extra_sn_args --lblk` dies with "expected one argument" before a single test runs. These two options are unusual in that their value is itself a string of flags, which walks straight into that rule. --extra_cluster_args survived only by accident: argparse exempts a token containing a space from being read as an option, and its value happened to be '--device-mode lblk --enable-node-affinity'. A single-flag value there would have failed the same way, so fix both rather than only the one that broke. The = form has no such edge, whatever the value looks like. Also says so in --help for both, since anyone running e2e.py by hand hits the same rule and argparse's message does not hint at the fix. Verified: the separated form still fails exactly as reported; the = form parses to '--device-mode lblk --enable-node-affinity' and '--lblk'; the workflow is valid YAML and the edited block is valid bash emitting both args.

Every call site builds a request as `cluster_api_url + "/some/path"`, and API_BASE_URL defaults to "http://192.168.10.210/" in the workflows. The two concatenate to "//mgmtnode/". requests sends that verbatim, haproxy's wep_api_services backend forwards the path untouched (unlike nginx it does not merge slashes), and it reaches the API as a path matching no route: 404 {"detail":"Not Found"}, ten retries, and the run dies in check_for_dumps before a single test starts. Verified against the real route shapes rather than by reading them. With simplyblock_web's mount of v1 under /api/v1 and the legacy redirect shim: /mgmtnode/ -> 308 /api/v1//mgmtnode/ -> 200 /mgmtnode -> 307 -> 308 -> 200 //mgmtnode/ -> 404 {"detail":"Not Found"} <- what we hit So the product is fine. The shim does emit its own doubled slash, but v1 sets `api.url_map.strict_slashes = False` and Werkzeug tolerates it; only a doubled slash at the *front* misses the FastAPI route entirely. Normalised in one place rather than at the five call sites, and rather than in the workflow defaults, so e2e is correct whichever form of the URL it is handed. Empty base preserved as empty -- stress-run-bootstrap-k8s sets API_BASE_URL="".

E722, pre-existing and unrelated to the URL fix, but it sits four lines from it and keeps ruff red on the branch. Bare except also swallows KeyboardInterrupt, so a Ctrl-C landing inside get_lvol_id was silently turned into "Lvol not found, continuing without delete" rather than stopping the run.

Answering "are we sure every call has a / in front?" -- I audited all of them and yes, every endpoint in the suite is written with a leading slash: the ~60 literal and f-string paths, the two built into a variable first, the ones passed positionally, and the direct concat in _wait_for_api_recovery. But that is a convention, not a guarantee, and rstrip()ing the base makes the convention load-bearing: a future call site written as "pool" instead of "/pool" would produce "http://192.168.10.210pool", which is a worse failure than the doubled slash it replaced because it only appears at runtime and does not look like a URL problem. So normalise both halves in one _url() helper and route all five join sites through it. Now neither end has to be spelled a particular way. cluster_test_base.py's v2 alerts URL and SbcliUtilsV2 already rstrip the base themselves, so they are unaffected and stay correct.

…ailed Two defects, one visible and one that hid it. The RPC never worked. The payload runs `python3 -c` inside the SPDK container and opens /mnt/ramdisk/spdk_<port>/spdk.sock directly, but the container's default user cannot open that socket, so every call died on s.connect() with "PermissionError: [Errno 13] Permission denied" before sending a byte. The rest of the suite already knew this -- TestClusterBase._rpc_via_docker_exec runs "sudo python spdk/scripts/rpc.py" inside the container on both platforms -- and this helper simply did not. The container's own shell now picks sudo up if it is there, so a container running as root with no sudo installed still works. The second defect is why the run was hard to read. call_rpc embedded the raw remote stderr in the exception, and that stderr is a full Python traceback whose middle is a 2000-character hex blob and a run of carets. Everything downstream greps one line out of output.log -- the workflow summary and the Slack message both do -- so the reported failure was MultipleExceptions: LblkFunctionalDocker: ). On an lblk cluster the journal is what replaces the 4K atomic-write guarantee, so its absence invalidates the run. which opens with a stray ")" and names no cause at all. Condensing the stderr to its last meaningful line makes that same failure read ... : PermissionError: [Errno 13] Permission denied. On an lblk cluster ... Verified the command survives ssh -> sh -c -> python quoting with sudo present and absent, and that the condenser handles a real traceback, empty stderr, and a single-line error.

TestClusterBase defaults test_name to "" and builds the run's log directory as f"{test_name}-{timestamp}". None of the ten lblk classes set it, so every lblk run wrote to "<nfs>/-20260915-201040/" -- a directory starting with a dash, with nothing to say which test produced it, and the "Logs Path:" line that the workflow summary and Slack message grep out of output.log came back equally nameless. Derived from the class name in _LblkBase rather than hardcoded in each leaf, so a new scenario cannot be added nameless by forgetting a line. Same source as the test_name already handed to setup(), so the two agree. LblkFunctionalDocker now logs to <nfs>/lblk_functional_docker-<ts>/.

Asked for multi-iteration outage soaks on lblk matching the docker multi-client multi-node multi-outage family and the k8s resilient family. Docker was already on RandomMultiClientMultiFailoverAllNodesTest, which is that family. K8s was on the plain K8sNativeFailoverTest, which is not: without the resilient base's permanent PVCs, snapshots and clones, PVC provisioning blocks whenever ndcs + npcs > online_nodes, so a degraded cluster drops to zero IO and every iteration after the first outage proves nothing about the journal. Keeping IO on permanent volumes is what makes a multi-iteration lblk soak mean anything. K8sNativeResilientFailoverTest has its own run() loop, but it calls both perform_n_plus_k_outages() and restart_nodes_after_failover(), which are exactly the two hooks _LblkStressMixin brackets -- so the raw crc32c verify and the metadata-corruption log scan fire once per outage as they do on docker. Confirmed the mixin still wins the MRO for both hooks while each leaf keeps its own _spdk_exec_prefix. Both classes also set test_name. Inheriting it meant an lblk soak's logs landed in "n_plus_k_failover_multi_client_ha_all_nodes-<ts>/", indistinguishable from an NVMe run of the same family after the fact.

Correcting the previous commit, which swapped LblkStressK8s onto the resilient base rather than adding a case. That silently dropped the single-outage k8s soak. Both now exist: LblkStressDocker all-nodes loop, widest outage mix LblkStressK8s single-outage loop LblkMultiOutageStressDocker K parallel outages/iteration, primaries only LblkResilientStressK8s resilient loop, IO survives the degraded window The multi-outage pair is where a metadata journal is actually put under pressure: concurrent failovers mean concurrent lvstore metadata mutation, which is what the journal serialises. A single-outage loop rarely produces more than one writer. Also assert every device is an AIO bdev before soaking, not just that the cluster says device_mode=lblk. Those are two different layers and the words differ -- the cluster says "lblk", the device says "aio". device_mode records what was asked for; bdev_type records what each node ended up with. Checking only the former can commit hours to a soak that is exercising the NVMe path. The lvols these soaks create are ordinary lvols reached over NVMe-oF, so the client always sees /dev/nvmeXnY whatever the backend is. That is inherent, not a defect, and it is exactly why the backend has to be asserted on the storage nodes rather than inferred from anything visible on the client. Platform accessors moved to _LblkDockerPlatform/_LblkK8sPlatform mixins so the four leaves share one copy, ordered before the scenario base so they win the MRO. Verified: mixin owns both outage hooks, platform mixin owns the exec prefix, all four registered once in ALL_TESTS and get_stress_tests.

LblkIntegrity created two lvols, stamped them and verified across three outage types -- all data path. It never took a snapshot or a clone, so it never produced a metadata mutation, so it never wrote to the journal. That showed up in the first passing run: every lvstore reported used_slots 0, mem_head 5, mem_tail 5. The journal was enabled and completely inert. We were proving the journal exists, not that it works, which on a device with no atomic-write guarantee is most of the point. Snapshot and clone are the metadata-heavy operations, so _metadata_churn() now runs one of each before the first outage -- so recovery has a non-empty ring to replay rather than an empty one -- and again before each subsequent outage, alternating lvols so both the primary and the secondary lvstore see some. Journal movement is sampled on mem_head, not used_slots. The drain runs continuously and returns used_slots to ~0 between samples, which is exactly why the ring looked untouched; mem_head advances monotonically and does not lie. Not fatal when the head does not advance. That metadata operations must move the ring is a reasonable expectation, not a contract this suite has verified, and failing on it would be inventing one. It logs a warning naming the before and after heads instead -- if that shows up on every run, the journal is enabled and inert, and that is worth escalating on evidence. Also adds the assert_devices_are_aio() call this class was missing. The functional class checked all three layers; this one checked only two, so it would have soaked happily on a cluster whose devices were not aio.

A fresh operator install serves v1alpha2 as the storage version, and the conversion webhook that would accept a v1alpha1 CR is only deployed by an upgrade. So every k8s-native workflow authoring at v1alpha1 fails at apply on current main, lblk or not. Verified against simplyblock-operator origin/main (b10db7a1) rather than inferred from the sample: StorageCluster -> v1alpha2. Keeps stripe, fabricType, backup, kms, thresholds, maxSubsystemCount, vcpuCount -- the migration is far smaller than the sample suggests -- and gains deviceClass, enum NVMe|LogicalBlock. StoragePool -> v1alpha2, and clusterName is renamed clusterRef. StorageNodeSet stays v1alpha1: it has no v1alpha2, and it already carries enableLblk/blkNames/blkSerials from operator#437, merged. kms hashicorpVaultSettings.baseURL -> kms.vault.baseURL. lblk rides in cluster_params as device_class=LogicalBlock, because these workflows are at the 25-input GitHub limit. DEVICE_CLASS is defaulted where it is CONSUMED, not where it is parsed -- there are three different "Set parameters" shapes across the six files, and a missed one would render an empty deviceClass that the enum rejects. Defaulting at the point of use holds however the value arrived. The StorageNodeSet CEL rule makes lblk selectors mutually exclusive with pcieAllowList/pcieDenyList/pcieModel/driveSizeRange/deviceNames, but it tests size(...) > 0, so blanking is enough and the keys may stay. The lblk branch therefore blanks driveSizeRange and pcieModel and sets enableLblk, and is placed AFTER the existing pcieModel conditional, which would otherwise overwrite the blanking. Backup is refused rather than migrated. v1alpha2's BackupStoreSpec is not a rename: it requires an explicit bucket, and ours is simplyblock-backup-<CLUSTER_ID>, which does not exist until after the CR is applied. Emitting the v1alpha1 keys would not fail -- unknown fields are PRUNED by the apiserver, not rejected -- so the CR would apply cleanly and leave a cluster with no backup store while every backup test ran against it and reported nonsense. The workflows now exit with a clear message instead. That needs the backup owner's decision. k8s-native-upgrade.yaml deliberately untouched. Also adds filesystem FIO to the lblk integrity test, on a formatted and mounted clone. md5 is demoted to a warning there and only there: run_fio_test hardcodes --verify=md5 and auto-enables verify_backlog, which its own comment says bypasses the rand_seed check, so on a device with no 4K atomic guarantee a mismatch is as likely to be a harness artefact as a defect. The raw crc32c verify on the parent stays the gate. Demoting is not ignoring -- it is still logged and the hdr_fail dumps are still collected. All seven k8s workflows still parse as YAML.

The workflow CRs moved to v1alpha2, but k8s_utils creates pools at runtime and was still authoring v1alpha1. Authoring at v1alpha2 is mandatory -- the CRDs' conversion stanza names simplyblock-operator-conversion-webhook-service in namespace simplyblock-operator-system, and nothing in the chart creates it (6 CRDs reference it, zero non-CRD references), so a v1alpha1 apply has no webhook to call. v1alpha2 StoragePool is a reshape, not a rename: clusterName -> clusterRef dhchap: true -> volumeDefaults.enableDHCHAP storageClassParameters -> typed volumeDefaults action/status/qos/capacityLimit/logicalVolumeMaxSize -> gone so the values are translated rather than passed through. An unknown key is PRUNED by the apiserver, not rejected, which would otherwise leave a pool that applies cleanly and quietly lacks its encryption or its filesystem. _pool_volume_defaults raises on a key it does not recognise rather than dropping it, for the same reason: a pool that looks right and behaves differently is the failure mode the translation exists to prevent. It also renders one volumeDefaults block for both sources, since two separate keys would be a duplicate mapping and the second would silently win. Not migrated here, and still on v1alpha1: StorageNode, BackupRestore, BackupImport, BackupPolicy have no v1alpha2 at all, so they are correct as they are. StorageNodeOps and StorageBackup do have one and are reshaped rather than renamed (storageNodeRef -> nodeRef, drain/newSsdPcie/targetWorkerNode replaced by migrate/remove; StorageBackup becomes backupID + clusterRef). Those need their own pass -- StorageBackup behind the same backup decision already outstanding.

Both migration passes added the same line. Same value either way, so the behaviour was never wrong, but two identical writes to GITHUB_ENV read as a mistake to anyone diffing the file.

LblkJournalRecovery ended with self._scan_spdk_logs("journal recovery") self._verify_all("after journal recovery") self.logger.info("[lblk] journal recovery replayed and data intact") and nothing in those two calls looks at whether a replay happened. The run that just passed proves the setup works -- drain paused, 6 entries in the ring, node killed, restarted, data clean -- but the headline claim in that log line was never checked. A cluster with no journal at all would have passed identically. assert_journal_recovered reads it out of SPDK's own log instead. The three outcomes are distinguished because they are not equally good: "md journal recovery: N entries to drain (tail=T head=H)" -> pass, N >= 1 "md journal recovery: ring empty" -> FAIL "md journal recovery failed: ..." -> FAIL "ring empty" is the one worth failing loudly on: it is not an error, the node comes up healthy, and it means either the entries never reached the disk or recovery did not read them -- the exact failure the test exists to catch, and invisible from a clean data verify. The exec prefix is rebuilt after the restart. The container the node was killed in is gone, which is also why the drain-resume in the finally block logged "No such container: spdk_4420" on this run. That is harmless -- a restarted node comes up with the drain unpaused anyway -- but it means the finally block is not what guarantees the invariant its comment claims. Regex verified against the format string at blob_md_journal.c:1038, including the multi-lvstore case and the tail/head-less variant.

The new recovery assertion failed with "no 'md journal recovery:' line ... Either the log rotated past it or recovery did not run". Neither. It was reading /var/log/spdk.log inside the container, and that file does not exist. SPDK logs to the container's stdout. The product's own collector is the proof: collect_logs.py pulls spdk_<rpc_port> out of Graylog by container_name rather than reading any path, and nothing else in this suite references /var/log/spdk.log -- only the code added here did. The assertion was therefore right to fail, and it failed for the right reason: it refuses to pass on absence of evidence. But the same bad path is in _scan_spdk_logs, which every lblk test calls after every phase, and which swallowed the error with "2>/dev/null || true". So it has been scanning an empty string since it was written. Every "no blobstore metadata-CRC errors" result in every lblk run so far was vacuous -- it never read a log. _spdk_log_cmd now supplies the real source per platform, "docker logs" or "kubectl logs", alongside the exec prefix used for the RPC socket. Both are needed: the socket lives inside the container, the log does not. _journal_targets grew rpc_port, since the log command needs it and the exec prefix had been hiding it. Verified the rendered command for all four platform combinations.

First run with the scanner actually reading logs, and it failed on 192.168.10.202 with 'crc mismatch for blob.' and 'metadata page is all zero'. Checked against the SPDK source before reporting it, and it is not a defect. bs_update_cur_md_page_valid (blobstore.c:14467) is a predicate, not an error path. It is how the lvstore-update scan walks the md region: an unallocated page is zeroed, a zeroed page's CRC does not match, so it logs both lines at ERRLOG and returns false. The caller at :14897 then falls straight through to bs_update_replay_md_chain_cpl and carries on. Nothing propagates, and the scan runs on every failover and restart -- so as written this failed every recovery test on a perfectly healthy cluster. The distinction is one trailing word: "crc mismatch for blobid 0x..." blobstore.c:2187, blob_load_final(-EINVAL) -> the blob fails to load. FATAL, kept. "crc mismatch for blob." blobstore.c:14479, scan predicate -> page skipped. BENIGN, now informational. "Metadata page is all zero." only ever accompanies the benign one, so it moves with it. Still reported in the info list rather than dropped, so a run shows they occurred without failing on them. The fatal pair is unchanged and still gates the run, and is the reason this was worth getting right rather than silencing the check: a real blob-load CRC failure on a device with no atomic-write guarantee is exactly what the journal exists to prevent.

…ot read Root cause of the repeated failures, finally established rather than guessed: the spdk_<port> container is created with the GELF log driver (simplyblock_web/api/internal/storage_node/docker.py:135). `docker logs` cannot read GELF -- it returns "configured logging driver does not support reading" instead of output. The product's own collector goes to Graylog for exactly this reason. So there was never a way to read SPDK's log from the node. /var/log/spdk.log did not exist, and `docker logs` does not work either. Both attempts were wrong, and chasing the log a third time would be wrong again. Assert the outcome instead. The ten lvols created while the drain is paused have their metadata in the ring and nowhere else, so if the journal does not replay they do not come back. assert_staged_lvols_survived checks all ten are present after the kill and restart. That is a stronger claim than the log line: "md journal recovery: N entries" proves SPDK said it replayed, a surviving lvol proves the metadata actually came back -- which is the guarantee the journal exists to provide on a device with no atomic-write guarantee. The ring's post-recovery counters are logged alongside as corroboration. _scan_spdk_logs now says out loud when it cannot read the log, instead of scanning an empty string and reporting a clean result. That silent pass is what hid the problem for three runs: a check that cannot run has to look different from one that ran and found nothing. On docker it will now warn every time, and the raw crc32c verify is the gate. assert_journal_recovered is deleted rather than left behind, so nobody wires a dead log-scraper back in.

The run before this one reported "all 10 staged lvols survived the kill" and that result was worthless. The log timeline says why: 14:14:14,372 POST /storagenode/restart/ 14:14:14,434 all 10 staged lvols survived the kill 14:14:14,434 bdev_lvol_get_md_journal_stats -> No such container 62 milliseconds for a kill, a restart and a verify. wait_for_storage_node_status returns immediately when the node already has the status asked for, and right after a kill the control plane still reports online because its monitor has not noticed yet. So nothing waited, the node had not gone down, and the assertion ran against a cluster that was mid-crash. The stats call failed only because it was the first thing to touch the node itself. Three fixes, each closing one way that test could pass without testing: _wait_for_node_down blocks until the control plane stops reporting online, and fails loudly if it never does -- a kill that did not take means there is no crash to recover from. Only the lvols that landed on the PAUSED lvstore count as staged. The drain is paused per-lvstore but placement is the cluster's choice, so most of the ten landed elsewhere and were written through normally; the previous run had 2 entries in the ring for 10 "staged" lvols. Asserting on all ten passes whatever the journal does. It now raises if none land on the paused node, since then the kill proves nothing. The survival check reads the restarted node, not FoundationDB. /lvol lists what the control plane remembers, which is unchanged by anything the lvstore did; bdev_get_bdevs on the node is what shows the blobs actually came back. Verified the sequence and that the down-wait raises rather than looping forever.

The precondition guard added last commit did its job and refused the run: none of the ten lvols landed on the node whose drain was paused. The run before it got two. Placement is the cluster's choice, so which node receives the staged metadata was luck, and the test could only work when the coin came up right. Pause every lvstore's drain first, then pick the victim from where the lvols actually landed -- the node holding the most of them. Its ring necessarily has their metadata, whatever the placement did, so the kill always has something to recover. The resume in the finally block now walks every lvstore it paused rather than only the victim's. Pausing four and resuming one would leave three rings filling, and metadata writes block behind a full ring -- that would have degraded the cluster for the rest of the run and through teardown. The victim's own resume still fails, as before, because its container is gone; the restart clears that one. Verified the executable order by AST rather than by reading: pause all, create, group, select victim, sample its ring, kill, resume all, wait down, restart, wait online, assert on-node.

lblk is releasing from feat/lblk-26.3.1-1, cut from release 26.3.1-1 rather than from main, so the v1alpha2 migration does not apply to it. Checked that branch's CRDs rather than assuming: StorageCluster and StorageNodeSet are v1alpha1 only, v1alpha1 is the storage version, and the spec still carries stripe, fabricType, hashicorpVaultSettings and the old backup block. Our original CRs are exactly right for it. Reverted to the pre-migration state: six k8s-native workflows and the StoragePool authoring in k8s_utils. That also removes the backup guard, which existed only because v1alpha2's BackupStoreSpec requires a bucket we cannot know at create time -- v1alpha1's does not, so backup works again and the question for the CP team is no longer blocking. test_lblk.py is deliberately untouched. The filesystem-FIO work rode along in the same commit as the migration and has nothing to do with the CRD version. The v1alpha2 work is preserved on feat/lblk-k8s-v1alpha2. It will be needed the day operator main's CRDs reach a release, and re-deriving it would mean re-reading the conversion code and the CEL rules from scratch.

Separate from the revert so it can be dropped on its own. The revert restored v1alpha1 correctly but left k8s with no way to ask for lblk at all, and the point of the release branch is to test lblk. Verified against simplyblock-operator origin/feat/lblk-26.3.1-1 rather than carried over from the v1alpha2 work: StorageCluster v1alpha1, spec.deviceMode, enum nvme|lblk, default nvme. NOT deviceClass: LogicalBlock -- that is main's spelling and does not exist on this branch. StorageNodeSet v1alpha1, enableLblk alongside blkNames/blkSerials. CEL lblk selectors are mutually exclusive with pcieAllowList / pcieDenyList / pcieModel / driveSizeRange / deviceNames, and the rule tests size(...) > 0, so blanking suffices and the keys may stay. Rides in cluster_params as device_mode=lblk, since these workflows are at the 25-input GitHub limit. Defaulted to nvme where it is consumed, not only where it is parsed: there are two different "Set parameters" shapes across the five files and a missed one would emit an empty deviceMode. The lblk branch is placed after the existing pcieModel conditional, which assigns PCIE_MODEL_YAML and would otherwise overwrite the blanking. cross-cluster-restore is left alone: it has no PCIe selector block to hang the switch on, and it is a backup test rather than an lblk one. Rendered both ways through the workflow's own shell: default gives deviceMode nvme with the PCIe selectors intact, device_mode=lblk gives deviceMode lblk, enableLblk true and a blank driveSizeRange.

…t run A k8s run died in cleanup with bash: .../e2e/scripts/cleanup_upgrade_test.sh: No such file or directory Error: Process completed with exit code 127 The script is committed on the branch that ran. It was missing because the checkout never finished. From that run's log: git checkout --progress --force -B feat/lblk-e2e-clean ... error: Path 'e2e/AGENTS.md' not uptodate; will not remove from working tree. error: Path 'e2e/CLAUDE.md' not uptodate; will not remove from working tree. error: Path 'tests/unit/test_core_tls_settings.py' not uptodate; ... error: Path 'tests/unit/test_lblk_persist_node_config.py' not uptodate; ... Previous HEAD position was 7fb2f5c Deduplicate TLS context creation Switched to a new branch 'feat/lblk-e2e-clean' Those four paths exist on the previous ref and not on this one, so git wanted to delete them, found local modifications, and refused. The working tree was left only partly updated -- HEAD on the new commit, files from it never written -- and cleanup_upgrade_test.sh was one that never arrived. The step reported SUCCESS. Those are ##[error] annotations from git, not a non-zero exit, so the job ran on a tree that did not match its own HEAD and failed four hundred lines later somewhere unrelated. The files were modified because tests write into the workspace as root and the existing chown is best-effort with "|| true". Rather than chase which test dirtied what, reset the tree before actions/checkout gets it, so it always has something clean to switch. Applied to the nine workflows that carry the permissions step; bare-metal-deploy.yml indents differently and is left alone. k8s-e2e-ha.yaml still does not parse under PyYAML -- confirmed by stashing that this is true at HEAD as well, so it is pre-existing and not from this change.

12 of 15 cycles passed -- every node through graceful_shutdown, container_stop and storage_node_reboot, static data unchanged and live FIO alive throughout. Cycle 13, the first network outage, took the cluster down with the node: fdb.impl.FDBError: Operation aborted because the transaction timed out (1031) The outage dropped ALL traffic between the node and its peers. On k8s those same node IPs carry OVN's geneve overlay, so every pod-to-pod link across nodes died with it, FoundationDB lost quorum and the control plane could no longer read its own database. That is a broken test environment, not a storage outage, and nothing it reports afterwards can be trusted. Now it matches only the node's own service ports: NVMe-oF from NVMF_BASE_PORT 4420, SPDK JSON-RPC from RPC_BASE_PORT 8080, and the SNodeAPI from SNODE_API_PORT 50001, as ranges because each is a base the cluster allocates upward from. The node is isolated as a STORAGE node -- its peers stop hearing from it and its data path is gone -- while the kubelet, the API server, the overlay and the database keep working, so the cluster can still report what happened. Verified the -D spec is byte-identical to the -A spec, since iptables deletes by exact match and a near-miss would leave the rules in place. The rule check also warns if a portless DROP ever appears again.

Only k8s. A k8s node is not just a storage node -- OVN's geneve overlay runs between the same node IPs -- so cutting it off took every pod-to-pod link across nodes with it, FoundationDB lost quorum, and the control plane could no longer read its own database. FDBError 1031, on cycle 13 of 15, after twelve clean cycles. _network_outage has since been narrowed to the node's own service ports so the overlay and the database survive, but that is unproven on a real run, and an outage that breaks the cluster rather than the node invalidates every cycle after it. Better to bank the twelve cycles that work than risk the run on the one that might not. Docker keeps all four types: it cuts a machine that does nothing but storage, so dropping its NICs isolates exactly what the test means to isolate, and that path has no cluster infrastructure riding on it. k8s 12 cycles, docker 16. LBLK_MATRIX_SKIP_OUTAGES="" puts it back on k8s without a code change, and the run logs each skipped type so a green result never implies coverage it did not have.

Docker setup now completes: plain, crypto, dhchap and namespaced volumes all seeded, clone hashing identically to its parent, raw device stamped with crc32c, and three of four live FIO jobs started. The fourth failed: AssertionError: No new block device after connecting mxlivensvol191 Namespace packing is working as designed and the harness cannot follow it. The static lane's namespaced volume is the FIRST namespace in its subsystem and connects normally. A second one packs into that SAME subsystem, so the client is already attached: nvme connect answers "already connected" three times, no new controller appears, and _connect_and_mount_dual's device diff finds nothing. It ran nvme ns-rescan across every live controller twice and the namespace never surfaced. That is worth raising with the dev team: a namespace hot-added to a subsystem a client is already connected to should become discoverable, or clients cannot use packed namespaces created after they attached. Until then the docker live lane drops that flavour rather than losing a whole run to it, and logs the reason so a green run does not imply coverage it lacks. The static lane still exercises a namespaced volume on both platforms, and k8s is unaffected since each PVC gets its own attachment.

I was wrong to call this a product finding. A namespaced volume joins an existing subsystem and so has no controller of its own -- it must NOT be nvme connect-ed. The client is already attached, connect answers "already connected", no new controller appears, and a before/after device diff finds nothing: AssertionError: No new block device after connecting mxlivensvol191 The namespace was present and working the whole time; the harness was looking for it the wrong way. _create_namespaced_children in continuous_failover_ha_multi_outage.py has done this correctly all along: rescan the controllers and locate the volume by (NQN, ns_id), which is what _mount_namespaced now does, reusing SshUtils.get_nvme_device_for_nqn. So the flavour goes back into the docker live lane rather than being dropped, and the previous commit's "raise it with the dev team" note is withdrawn. It refuses to resolve a volume whose ns_id is missing or below 1: without a usable NSID the only option is "any head on this NQN", which on a shared subsystem is a sibling's device, and the next step formats whatever it is handed.

Still wrong last time, and the error said so: ns_id 1 of its own NQN. A volume with namespace=True but no parent still opens a fresh subsystem, so it was neither connected (I had stopped connecting namespaced volumes) nor joinable (nothing to join). It could never resolve. RandomMultiClientMultiFailoverAllNodesTest does it in two distinct parts, and both matter: parent ordinary lvol, max_namespace_per_subsys=30, connected normally child namespace=True, host_id=<parent's node>, never connected, located by (NQN, ns_id) The slots belong on the volume being JOINED, not on the one joining, and the pin is required because a subsystem is per-node -- an unpinned child lands elsewhere and opens its own, which is exactly what happened here. So every other flavour now carries max_namespace_per_subsys=30, the first one connected is recorded as the parent, and the namespaced volume is created against that node and resolved rather than connected. plain is created first in both lanes, so it is always the host. If the backend puts the child in a different subsystem on that node -- it picks any with room -- that is logged rather than treated as an error, since the resolve then looks somewhere other than the parent.

…rent Two separate failures. K8s ran all 12 cycles clean and then died in the final validation with "str.join() takes no keyword arguments". The handle is the Job NAME on k8s, a string, and str has a .join -- so `if hasattr(handle, "join")` was true and this called "jobname".join(timeout=...). Exactly the trap already fixed in _fs_fio in test_lblk.py; I fixed it there and left this copy alone. Guarded on threading.Thread now, and there are no other hasattr-join sites in the repo. Docker got the namespaced child working and then failed on the clone: No new block device after connecting mxclone761 Giving every volume max_namespace_per_subsys=30 had a second effect I did not intend. A clone of a parent that has free slots stays in the parent's subsystem rather than opening its own, so it arrived with no new controller and hit the same discovery problem as the namespaced child, reached from a different direction. The reference suite says as much -- it uses 30 so "clones stay on the parent's subsystem". Slots now go only on the volume that hosts the namespaced child, and the clone is taken from a different static volume, so every clone opens a subsystem of its own and connects normally.

Docker completed setup -- all four live volumes including the namespaced one at ns_id 3 -- passed cycle 1's md5 check, and then failed on my own gate: live FIO on mxliveplain490 stopped during graceful_shutdown on .201 FIO had not stopped. run_fio_test starts it in a DETACHED tmux session and returns as soon as it has confirmed the session exists, so the launching thread finishes seconds later while FIO runs for its full 15000s. Checking handle.is_alive() therefore reported every job dead one cycle in. A gate that fails on healthy runs is worse than no gate, and it would have masked a real interruption behind noise. The docker probe now asks the client: tmux has-session, falling back to pgrep. _finish_live_fio kills the sessions before reading the logs, since they were sized to outlast the matrix and waiting for them would add hours -- an io_u error during an outage is already written by then, which is what the gate reads. handles gained the job name, so it is a 4-tuple now. The audit checks that every unpack matches the append: an arity mismatch is a ValueError hours into a run, and I verified the check catches one by introducing it.

All 12 cycles passed again and the run still failed: FIO Job 'fio-mxliveplain' did not succeed (status=timeout) (pod phase=Running) validate_fio_job calls wait_job_complete(timeout=600), but this job is deliberately sized to outlast the entire matrix -- it was still Running and was always going to be. Nothing was wrong with the IO; the check asked the wrong question, the mirror image of the docker side where I had to stop the tmux session before reading. _k8s_finish_fio reads every pod log for the job, fails on anything that records a real IO error, then deletes the job. There is nothing to wait for: an io_u error raised during an outage is already in the log by the time the last cycle finishes. The markers are deliberately narrow. "error" as a substring matches a clean summary, and err= appears in every FIO log -- a healthy one says err= 0. So it matches io_u error, verify failed, bad magic header, hdr_fail, data mismatch, checksum error, and err= followed by a non-zero digit. Checked against six real clean lines and five real failure lines from previous runs: no false positives, no misses.

The rule looked for validate_fio_job in _finish_live_fio, which was deliberately removed: it waits for a job sized to outlast the matrix. It now requires _k8s_finish_fio and flags validate_fio_job coming back.

SEC_PER_CYCLE was 900, a guess. The twelve cycles on the k8s run of 2026-09-21 measured min 164s, max 401s, mean 227s, so a 16-cycle run was asking FIO for 15000s -- four times the work it had to cover. An over-estimate is not free. The job then far outlives the outages, so it can never be allowed to finish, and both platforms ended up killing it and grepping the wreckage rather than reading a completed run. That is what produced "FIO Job did not succeed (status=timeout, pod phase=Running)" on k8s and the kill-then-grep path on docker. 450s sits above the worst cycle observed. k8s now asks for 6000s against a loop of 2724s at the measured mean and 4812s at the worst; docker 7800s against 3632s and 6416s. Both cover the worst case with margin. _finish_live_fio waits for the jobs rather than killing them, bounded by the runtime plus slack, so what gets validated is a completed run with a real summary. A job still going past that bound is judged where it stands, since hanging there would hide a stuck job. _assert_fio_alive now separates "died" from "finished its runtime". If the outages outlast the job that is a sizing problem on our side and it says so; only an early exit counts as a loss of availability.

Docker passed 12 of 16 cycles -- every node through graceful_shutdown, container_stop and storage_node_reboot, static data unchanged and live FIO alive after each -- then failed on the first network outage: Timed out waiting for node status, Expected: ['offline'], Actual: online The outage worked; nobody was watching it. The branch slept out the whole 120s window AND restored the network before falling through to the shared offline wait, so the first poll landed 164s after the cut, by which time the node was back. It then polled 600 times over ten minutes, saw "online" every time, and failed the run. The offline wait now happens while the links are down. If the node never goes offline within the window that is logged plainly -- the control plane either tolerates a gap that short or is not watching -- and the cycle continues to its integrity checks rather than failing, because the outage proved less than intended but the data checks after it still mean something. When the node never left, there is nothing to restart, so the shared restart-to-online is skipped. Separately, live FIO is capped at 6000s on both platforms. 16 cycles would otherwise ask for 7800s, and every second FIO runs past the last outage is a second spent waiting for it to finish. At the measured mean a 16-cycle loop is 3632s, so 6000 covers it; a run of near-worst-case cycles would outlast it, and _assert_fio_alive reports that as a sizing note rather than a failure.

…lves Cut off from its peers, SPDK fences itself and aborts; the monitor sees the node offline and queues a restart, and it comes back once the links return. That whole path is what this outage exercises, and forcing our own restart straight afterwards papered over it -- including the case where auto-restart never fires, which is a defect worth failing on rather than hiding behind a manual restart call. So after the links are restored the test now waits up to 900s for the node to come back on its own and says plainly when it does. If it does not, it says that is worth reporting, restarts it explicitly so the remaining cycles can run, and carries on. A core dump is expected in this cycle: spdk_abort_node is how the fence is implemented. Nothing in the stress path fails the case for it -- stress.py checks after the case completes, and skips the check entirely on k8s.

No new block device after connecting mxclone348 Where a clone ends up is not decided by what it was cloned FROM. The backend puts it in any subsystem on that node with free namespace slots, and one volume on the node must have slots in order to host the namespaced child -- so a clone can be packed in beside it, with no controller of its own to find. Sourcing the clone from a differently-created volume did not avoid that, because the source was never what governed it. So try the ordinary connect, and if the clone turns out to have joined an existing subsystem, resolve it by (NQN, ns_id) exactly as a namespaced volume is resolved. _mount_namespaced gained a format_fs switch for this: a clone carries its parent's bytes and this test compares its md5 against them, so mkfs would destroy the thing being measured. Neither path formats. k8s is untouched -- each PVC is attached on its own there.

Two docker-only faults, both ours, found by correlating the run of 2026-09-21 against its cycle timeline. _docker_fio_running could not return GONE. `pgrep -f` matches whole command lines, and the shell running the check carried `fio_<job>` in its own, which `fio.*<job>` matches inside that one token: it found itself. mxliveplain died 21s into cycle 8 at 22:39:23; this reported it alive at 22:39:45 and after every one of the eight remaining cycles, so the availability gate watched nothing for two hours and the failure only appeared at final validation. _await_fio_done then waited from 22:58 to 00:27 for four processes already gone -- most of the 2h14m runtime. Both patterns are bracketed, not just the pgrep one; they share a command line, so a bare fio_<job> in the tmux half is still a match for the pgrep half. The tmux side also moves to an exact compare against list-sessions, since `has-session -t` falls back to prefix matching and fio_mxlive would have matched fio_mxlivecrypto. Second: run_fio_test defaults to --max_latency=20s and create_fio_job has no equivalent, so a single IO slower than 20s killed the job with err=110 on docker while k8s could not trip it whatever the cluster did. Stopping a storage node is exactly when IO stalls, so the live lane was failing on an outage it injected on purpose. This lane asks whether IO continues and whether the data is right; latency under outage deserves measuring, but not as a fatal trip mid-kill. Turning it off also makes the EIO recorded at that same instant interpretable on its own.

_finish_live_fio validated in a plain loop and validate_fio_test raises, so the first failure ended it. On 2026-09-21 the plain volume failed and the crypto, dhchap and namespaced jobs were never looked at -- and whether a failure hit one flavour or all four is the most useful thing a four-flavour lane has to say. Now each is judged, the verdicts are collected, and the run fails once with all of them named.

I claimed the k8s path sets no latency ceiling. That is wrong about k8s generally: k8s_native_add_node, k8s_native_node_migration, k8s_major_upgrade and large_scale_lvol_stress all set max_latency in fio configs they build by hand, and continuous_k8s_native_failover -- the k8s outage lane, the nearest analogue to this matrix -- sets 40s, with the scale-break lane documenting its own 20s as "(parent: 40s)". What was true is narrower: create_fio_job has no built-in ceiling, so THIS test had none on k8s while docker took run_fio_test's 20s default. So the previous commit's use_latency=False was the wrong correction. It removed a real signal -- a ten minute stall would have passed unnoticed -- when the actual problem was that the two platforms disagreed and nobody had said what the number should be. 40s, both platforms, stated. max_latency is now an explicit _run_fio_dual parameter rather than a kwarg, because kwargs reaches run_fio_test on docker and is read by nothing on the k8s branch: setting it through kwargs gated one platform and silently no-opped on the other, which is exactly the class of bug the no-op audit exists to catch. No other caller passes it, so nothing else changes behaviour. The audit's final-validation rules now read _judge_one_fio as well as _finish_live_fio, since the per-volume verdict moved there.

The ceiling had drifted per-test: 20s hardcoded in four k8s fio configs, 40s in continuous_k8s_native_failover, a dead 30s in the quick-outage lane (its use_latency=False had disabled it), run_fio_test defaulting to 20s from kwargs, and nothing at all on any k8s job built through create_fio_job. So "did IO stay under the ceiling" meant something different in every lane, and the two halves of a single test disagreed with each other. utils/fio_defaults.FIO_MAX_LATENCY is now the only definition. Every site imports it; the per-call max_latency kwarg is gone from run_fio_test and the parameter is gone from _run_fio_dual, so no test can set a different value to get itself passing. use_latency stays a choice, since whether to have a ceiling at all is a separate question from what it is. Two behaviour changes worth watching: continuous_k8s_native_failover tightens from 40s to 20s, and every k8s job going through _run_fio_dual now HAS a ceiling where it previously had none.

The separate log-collection step has been failing every run with /root/.local/bin/python3: No module named simplyblock_core.scripts.collect_logs while the in-test Graylog export worked fine, because the two use different mechanisms: the test queries Graylog's HTTP API from the runner, the pipeline step ssh'es to the mgmt node and runs the module. On the mgmt node `python3` resolves to /root/.local/bin/python3, which does not have sbcli. What it does have on sys.path is a stray /usr/local/lib/python3.12/site-packages/simplyblock_core/ containing only a scripts/ dir and NO __init__.py -- so `import simplyblock_core` succeeds as an empty namespace package (__file__ is None) and the submodule import then fails. That is why the error says "No module named" rather than "No module named simplyblock_core": the top-level package resolved, its contents did not. sbcli is installed in the uv tool environment behind sbctl. Confirmed on 192.168.10.210: /root/.local/share/uv/tools/sbctl/bin/python -m \ simplyblock_core.scripts.collect_logs --help -> works Rather than hardcode that path, derive the interpreter from sbctl's own shebang -- it follows wherever sbctl is installed, which is by definition an interpreter that has the package -- then the uv path, then plain python3, taking the first candidate that can actually import the module. The resolver reuses each workflow's existing ssh invocation verbatim, so no quoting is re-derived. The chosen interpreter is echoed so a future failure says which one was picked. Eight workflows, one call site each, all the same shape. Not fixed here: the stray site-packages/simplyblock_core/ on the mgmt node is leftover from an older install and should be removed, and the retry ladder masks this by degrading 15m -> 5m -> 1m windows when the real error is not a timeout.

The previous commit called /usr/local/lib/python3.12/site-packages/simplyblock_core/ a leftover from an older install and said it should be removed. That is wrong, and acting on it would have broken monitoring. The product writes it. render_and_deploy_alerting_configs (simplyblock_core/utils/__init__.py:3143) renders prometheus.yml.j2 and alert_resources.yaml and sudo-mv's them into <parent-of-package>/simplyblock_core/scripts/ -- _top_dir() being three dirnames up from the utils module. sbcli was installed in /usr/local/lib/python3.12/site-packages at deploy time, so that is where they landed, and docker-compose-swarm-monitoring.yml:81 mounts ./prometheus.yml from that same directory into the Prometheus container. It is live config, not residue. It survived the package moving to the uv tool install behind sbctl for a mundane reason: a sudo mv is not pip-managed, so uninstalling the package never removed the rendered files. The directory is left holding config and no __init__.py, which is exactly what makes it an empty namespace package that shadows the real import. No behaviour change -- the resolver from the previous commit is already the right fix precisely because it does not touch that path. Only the comments change, so nobody deletes the directory on my say-so.

The md journal was reverted from spdk on 2026-09-22 to investigate a CRC mismatch, so spdk:main-latest no longer registers bdev_lvol_get_md_journal_stats. Both matrix runs that morning died at the third precondition before provisioning anything -- docker in 15m, k8s in 11m, ClientLogs empty on both -- which reads like flakiness next to the 2h14m run the night before but is a different failure entirely. "The build has no journal" and "the journal did not come up" are now different facts. JSON-RPC -32601 (method not registered) raises the new MdJournalAbsent and the run continues with the gap recorded; anything else, including enabled=false, still fails. The second case is the one worth stopping for: a build that HAS the feature and did not start it is a defect, while a build shipped without it is a decision. Nothing in the product requires it. grep md_journal across simplyblock_core, simplyblock_cli and simplyblock_web returns nothing -- the only hard dependency was this test. The separate --enable-journal-device requirement for lblk (storage_node_ops.py:3624) is the JM journal-on-device layout, a different thing. What the gap costs, so a green run is not over-read: the journal's only safety contribution is torn-write detection on 4K metadata pages (design doc S9 -- "no transactions or multi-page atomicity are needed"). A page can only tear where the device's atomic unit is under 4K. The lab's storage is 4Kn (Samsung MZQLB1T9HAJR, logical=physical=4096), so on this hardware the journal was covering something the devices cannot do. On 512b hardware -- which is what lblk exists for -- it is the whole story. Covered: the precondition, the per-cycle head sampler, the post-recovery sampler (absence is terminal, retrying cannot help) and the soaks' precondition. LblkJournalRecovery still fails, with a message saying to drop it from the run list, because it is nothing but ring pause/fill/ replay -- and the suite has no skip that does not also fail the run, so passing it would imply replay was verified when it was not.

The k8s run of 2026-09-22 said three volumes "recorded 47 IO error(s) ... this is a loss of availability". All three had ZERO IO errors. Every line matched was err=110 -- our own --max_latency tripping. Two wrongs: The wording. A latency breach and an EIO are different findings: one says IO was slow, the other says IO did not happen. Calling the first a loss of availability overstates it, and worse, it makes a real EIO indistinguishable from a slow one in the summary -- exactly the distinction that separates this k8s run from the docker run of 2026-09-21, which did have genuine EIO. The count. It counted matching LINES inside get_pod_logs(tail=4000), and FIO's end-of-run summary repeats err=110 across its per-job blocks. That is how three volumes whose full logs hold 461, 122 and 263 err=110 lines all reported exactly "47": the number described the tail window, not the workload. Now the two are counted separately, the worst latency comes from FIO's own "latency of N nsec" line and is reported in seconds, and the message names which happened. A latency breach says plainly that every IO was eventually served and no data was lost, while keeping the finding: at --iodepth=1 a 76s worst case is one operation blocked that long, and the ceiling detected the stall rather than causing it. Replayed against the real log lines from both runs: k8s crypto -> LATENCY BREACH (0 io_err, worst 30.6s), docker plain -> IO ERRORS (2 io_err, worst 147.7s), clean run -> clean.

Three gaps, all of them k8s pretending to test something it did not. A "storage node reboot" on k8s restarted the SPDK pod. That tested SPDK coming back and nothing else -- not the kubelet, not volume re-attach, not CSI re-registration, which is most of what a reboot exercises. There IS a kubectl route to the host: the one the core-dump collector already uses. Generalised as K8sUtils.run_on_node -- `oc debug node/<n> -- chroot /host` on OpenShift, `kubectl debug node/<n>` elsewhere -- and reboot_node rides on it. The command is detached behind a short sleep because `reboot` kills the connection carrying it, and its result is not trusted either way: the node going NotReady is the only honest evidence. Talos has no host shell, so it tries talosctl and returns False if that is not on the runner; the caller then falls back to a pod restart AND SAYS SO, rather than reporting a reboot that did not happen. Nothing tested pod eviction. Every outage left the node Ready on purpose -- the existing network cut drops only the storage ports so the kubelet keeps its heartbeat. So node_network_isolation: total isolation held 420s, past the 300s unreachable toleration, so the scheduler actually moves the pods. It asserts the node went NotReady (or says the cycle proved nothing), that it rejoins, and that no Multi-Attach / FailedAttachVolume events appeared -- an RWO volume that does not detach in time leaves the rescheduled pod wedged in ContainerCreating, and nothing else in the suite would notice. That outage was previously impossible to run safely, and this is why it is now: a blanket DROP from inside the SPDK pod could not be undone, because the pod is not hostPID so there was nowhere to leave a timer, and a cut outliving the container once left a worker NotReady until it was rebooted out of band. With host access the timer goes on the HOST, so the cut restores itself while nothing can reach the node. And short_network_interrupt, 30s: the same cut too brief to evict anything, testing the data path riding out a blip. Most real blips are seconds; a suite that only ran the long one never exercised recover-without-failover at all. Planned per platform, on whichever platform each means something: docker skips node_network_isolation (no scheduler to react), k8s keeps skipping interface_full_network_interrupt (it takes OVN and FDB with it). Both now run 5 types, 18 cycles on 4 nodes. The integrity e2e lane gets the same split.

_DualOutageK8s "rebooted" a k8s storage node with ssh_obj.reboot_node(node_ip) started in a daemon thread that nothing joins. k8s storage nodes are generally not ssh-reachable -- the lblk lane says so in as many words -- so the call raised inside the thread and was swallowed. Every storage_node_reboot case in that matrix has been a no-op reporting success. It now goes through the kubectl route to the host, and where no real reboot is possible it stops the SPDK pod AND warns, rather than letting a pod restart pass for a reboot. K8sNativeFailoverTest ran exactly one outage type, graceful_shutdown. interface_full_network_interrupt sat commented out beside it and there was no node reboot at all -- so Basic, Resilient, RapidNoGap and Quick, which all inherit it, tested one outage each. Now three, and five in outage_types2. Re-enabling the network outage costs nothing new: it is already a blanket DROP restored by a host-level nsenter process, and it already picks 30, 300 or 600 seconds. Which means the 300s and 600s cases have been evicting pods all along and nothing has ever looked. They are now marked as expecting eviction, and recovery asserts no Multi-Attach or FailedAttachVolume events -- an RWO volume that does not detach from the old node leaves the rescheduled pod in ContainerCreating indefinitely, silently. Below 300s nothing is expected to move and the check no-ops, so a 30s blip cannot fail for the wrong reason. The eviction check is wired into restart_nodes_after_failover rather than only defined; an assertion nothing calls is the failure mode this suite keeps producing. Verified all six k8s failover leaves inherit both helpers, and that no ssh reboot remains on any k8s path -- the four that are left are docker.

`since` was a parameter multi_attach_errors accepted and never used: the query pulled every Warning event in the namespace with no time bound. Events live about an hour and these suites loop every few minutes, so iteration 5 would have failed on iteration 2's events -- and once a real failure landed, every later iteration would echo it and a new failure would be indistinguishable from the old one. A check that gets less trustworthy the longer the run goes is worse than no check, and these are the runs that go for hours. Now it reads lastTimestamp (falling back to eventTime and the series observation) and counts only what is inside the window, default 900s. Events with no usable timestamp are kept, not dropped: a check hunting a rare failure should not discard evidence because a field was missing. Each hit now carries its age and the object it was raised against. Checked against a synthetic event set: a 2-minute-old Multi-Attach and a 5-minute-old FailedMount flagged, a 45-minute-old one excluded, an unrelated BackOff excluded, and a timestamp-less Multi-Attach kept.

The Multi-Attach check was armed only by the network outage, because there the duration is chosen up front. A reboot is the same hazard reached a different way -- the kubelet goes away, past the 300s unreachable toleration the scheduler moves the pods, and their RWO volumes have to detach from a node that is not there to detach them -- and it was not checked at all. A reboot's downtime is not ours to pick, though: a fast one is back inside the toleration and moves nothing, a slow one crosses it. So it is timed rather than assumed. The clock starts when the node is confirmed NotReady, not when the reboot was asked for -- the gap between those is the command propagating, and nothing is evicted during it -- and the check is armed only if the node was gone long enough for eviction to have been possible. A short reboot logs its downtime and arms nothing, so it cannot fail on an echo of someone else's eviction. Both lanes: continuous_k8s_native_failover (and so Basic, Resilient, RapidNoGap and Quick) records the downtime in the reboot path and arms the check during recovery; the lblk matrix does the same inline and fails with the measured downtime in the message.

…o 40s The attach check was gated on "was this outage long enough to evict", which assumed Multi-Attach only follows a full eviction. It does not: any detach/attach cycle can strand a VolumeAttachment, and whatever wants that volume then waits in ContainerCreating indefinitely with nothing failing. A pod restarted in place re-attaches its volume too. So the gate was a blind spot over graceful_shutdown, container_stop, operator_shutdown and every short network cut -- most of the suite. It is now asserted after every outage, in both lanes: the k8s-native dispatcher arms it for all five types, and the lblk matrix checks after every cycle alongside the md5 and crc32c checks. What made the gate look necessary was staleness, and that is fixed properly instead: the event window is scoped to the outage plus its recovery rather than a flat lookback, so a real failure in one cycle cannot re-fail every cycle after it. With the window right the gate only created blind spots. Separately, FIO_MAX_LATENCY 20s -> 40s, which is the value continuous_k8s_native_failover used before the constant existed; 20s is what the scale-break lane picks deliberately for a load test. Worth saying that this changes no conclusion: the k8s run of 2026-09-22 measured a single 4K read at 76s and a write at 64s, so the stalls that matter clear either ceiling by a wide margin. One definition still, in utils/fio_defaults.py, and no literal ceiling remains anywhere.

Two simultaneous outages is the case the single-outage soaks cannot reach, and it is the one that matters most where a metadata journal is involved: concurrent failovers mean two lvstores mutating metadata at once. On nvme a 4K metadata page cannot tear, so that concurrency is a control-plane question. On a device with no 4K atomic write it is a durability question as well -- which is exactly the gap lblk exists to test, and exactly what the parents' filesystem-level md5 can mask. Composition rather than a copy. _LblkStressMixin goes FIRST in the MRO so its hooks wrap the dual matrix's instead of replacing them; both layers override perform_n_plus_k_outages and both call super(). Verified by resolution rather than by reading the MRO: perform_n_plus_k_outages _LblkStressMixin -> _DualOutageMixin -> base restart_nodes_after_failover _LblkStressMixin -> base _apply_platform_outage _DualOutageDocker -> _DualOutageMixin _spdk_exec_prefix _LblkDockerPlatform run _DualOutageMixin -> base So the crc32c bracket wraps the pair selection, the platform bindings stay with the platform, and the case-driven run() is untouched. Both leaves take their own test_name, or the logs land under the nvme matrix's directory and an lblk run is indistinguishable from an nvme one after the fact. 180 cases per platform, selected with stress.py --case.

The check I added failed the k8s run of 2026-09-23 on cycle 1, and it was wrong, not the product. What it matched: [370s ago] md5-mxnsvol242: Multi-Attach error for volume "pvc-42297..." Volume is already exclusively attached to one node md5-mxnsvol242 is OUR pod, and the event was raised during volume seeding at 09:59:13, before any outage: 09:58:42 Deleting pod 'seed-mxnsvol242' (waiting) 09:59:13 pod "seed-mxnsvol242" deleted <- 31s to release 09:59:13 Creating utility pod 'md5-mxnsvol242' <- same second Delete a pod holding an RWO volume and create another immediately, and the second is told the volume is still exclusively attached until the first attachment is released. The controller retries and it clears -- which it did: the run seeded the volume, read its md5, started live FIO and completed cycle 1. Then the check looked back 900s, found the setup-time event, and failed a cycle that had passed. So the question was wrong. A pod that is Running got its volume, whatever was logged on the way there. pods_stuck_on_volumes takes the pods that are NOT Running or Succeeded and reports only those with a recent volume complaint against them, which is what "it did not error out" actually means. Every failing check now uses it; the raw event list stays as an INFO line, because a rise in transient attach churn is worth seeing even when everything eventually attached. Replayed against the exact shape that failed: 2 Multi-Attach events, the resolved md5 pod not flagged, the still-Pending pod flagged.

offline that already came back Two bugs, both mine, that ended the k8s run of 2026-09-23 at cycle 9. The reboot never happened. The command was issued cleanly on OpenShift and took 48s: oc debug node/worker-1 -- chroot /host bash -c \ 'nohup sh -c "sleep 3; systemctl reboot -f || reboot -f" &' and worker-1 never went NotReady. `oc debug` DELETES its debug pod as soon as the command returns, and the backgrounded sleep-then-reboot is a child of that pod, so it died with the pod before the sleep elapsed. Backgrounding cannot outlive the thing that owns the process. Now the host's own init owns it: systemd-run schedules a transient unit five seconds out, which the debug pod going away cannot touch, falling back to a synchronous reboot where systemd-run is unavailable. Then the run hung for 52 minutes. The k8s reboot branch waits the node down and back up itself, and then fell through to the shared wait_for_storage_node_status(uuid, "offline") -- by which point the node is Ready and online again, so the wait could never succeed. It now returns after confirming online and healthy, like the network branch does. The pod-restart fallback still falls through, correctly: a deleted pod does take the node offline. Worth recording separately, NOT changed here: wait_for_storage_node_status decrements its `timeout` once per iteration, and each iteration is an API call plus a 1s sleep. So timeout=600 is 600 ATTEMPTS, not 600 seconds -- about 50 minutes on k8s, which is where the 52 came from. Both the docker and k8s implementations do this, so every caller in the suite is calibrated to it; making it wall-clock would shorten every wait roughly fivefold at once. That is a change to make deliberately, not as a side effect of this fix.

RaunakJalan added 5 commits September 15, 2026 02:22

github-advanced-security Bot found potential problems Sep 15, 2026

View reviewed changes

RaunakJalan added 24 commits September 15, 2026 19:41

Drop a duplicated DEVICE_CLASS export

3adb132

Both migration passes added the same line. Same value either way, so the behaviour was never wrong, but two identical writes to GITHUB_ENV read as a mistake to anyone diffing the file.

RaunakJalan added 30 commits September 21, 2026 14:54

Point the audit at the new k8s FIO check

f04423a

The rule looked for validate_fio_job in _finish_live_fio, which was deliberately removed: it waits for a job sized to outlast the matrix. It now requires _k8s_finish_fio and flags validate_fio_job coming back.

Hoist the audit script's mid-file imports so ruff is clean

d413127

Navigation Menu

Product-limits stress, dual-outage matrix, resume, and lblk integrity… - #1357

Product-limits stress, dual-outage matrix, resume, and lblk integrity…#1357
RaunakJalan wants to merge 117 commits into
mainfrom
feat/lblk-e2e-clean

RaunakJalan commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

RaunakJalan commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants