Add git / stress-ng / pjdfstest targets and a backends × targets CI matrix - #2
Merged
Merged
Conversation
…atrix
The CI ran exactly one suite (`git annex test`) per backend. That answers
"is git-annex broken on this filesystem?" but not "at which layer?" --
a rename-semantics quirk shows up as a pile of git-annex failures with
no indication whether git, or POSIX itself, is already unhappy.
Add three complementary targets underneath it:
stress-ng 20 curated filesystem stressors with --verify, one at a
time so an unsupported operation (xattr on vfat) reports
as a skip rather than poisoning the run. ~2 min, apt only.
pjdfstest POSIX conformance: 238 scripts, ~8800 assertions naming
the syscall and the errno. ~3.5 min on ext4.
git 174 scripts of git's own testsuite (t0*.sh t1*.sh, ~10k
assertions) -- the plumbing git-annex stands on. Only the
trash directories go on the mount, via the suite's --root;
the build stays on the runner's disk. ~2 min on ext4.
Structure:
- bin/ci/matrix.sh is the single source of truth: the backend rows, the
target columns, pinned upstream refs, per-target timeout / loop size /
cron minute.
- bin/ci/install-target.sh does runner-side prep (apt package, or a
shallow fetch + build at a pinned ref into $EVAL_UNDER_SRC_DIR).
- bin/ci/target-<target>.sh is the suite itself, run inside the mount.
- bin/ci/run-under.sh grows a third argument selecting the target and
applies its timeout and loop-image size.
- bin/ci/gen-dispatchers.sh generates the 20 per-cell dispatcher
workflows and the README matrix from matrix.sh; `--check` reports
stale or orphaned files. GitHub publishes one badge per workflow
file, not per matrix cell, so the grid needs one file per cell.
The five existing dispatchers are renamed to carry an explicit
-git-annex suffix so all 20 cells are named uniformly.
Two notes on pinning:
- git is pinned to the v2.55.0 tag.
- pjdfstest is pinned to a commit, not its one upstream tag. That tag
("0.1", 2016) no longer compiles: major()/minor()/makedev() moved to
<sys/sysmacros.h> in glibc 2.28 and the tree builds with -Werror, so
the implicit declarations are hard errors on any current toolchain.
Master builds clean, so pin a commit on it rather than carry a patch.
README badge URLs pointed at the pre-rename yarikoptic/beegfs-test repo;
the generated matrix points them at con/eval-under.
Validated locally: all three new targets run green on ext4 (stress-ng
20/20 through a real loop mount via run-under.sh; pjdfstest 8827/8827;
git 10366 tests, 0 failed), shellcheck clean across bin/ci/*.sh,
bin/eval-under*, provision/*.sh, all 21 workflows parse, and reuse lint
still reports 100% compliance.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
The NFS backend deliberately drops back to the invoking user and exports with the default root_squash: that is what a normal user's NFS home looks like, and reproducing it is half the point of the backend. git-annex and git want exactly that. pjdfstest and stress-ng do not. pjdfstest is half privileged-vs- unprivileged assertions and refuses to run non-root at all (the first CI run of this PR died on precisely that); stress-ng's chown and mknod stressors need CAP_CHOWN / CAP_MKNOD. Squashing every privileged syscall to nobody leaves both with nothing to measure. So give the backend an explicit opt-out: --no-root-squash / EVAL_UNDER_NFS_NO_ROOT_SQUASH exports with no_root_squash *and* keeps the wrapped command's privileges. matrix.sh grows target_needs_root(), and run-under.sh passes the flag for the two targets that need it. The loop and beegfs backends already run the wrapped command as root, so they are unaffected. Export options and mount options are separate namespaces -- sync/async are valid in both but no_root_squash is export-only and mount(8) rejects it -- so the two option strings are now built separately. Also trim the on-failure dmesg dump: `dmesg | tail -100` on a hosted runner is ~100 lines of hyperv/pci/apparmor boot spam that buries the actual failure. Dump warnings and errors, plus lines naming the filesystem under test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
`make test` hangs every per-script recipe off a single aggregate-results-and-cleanup target. Under `make -j` without `-k` the first failing script stops make from scheduling any more, so the run aborts partway through and aggregate-results -- the thing that prints the totals -- never runs at all. The job log then just ends on whichever scripts happened to still be in flight, which is why the Loop vfat cell ended on a screenful of passes with no summary and no failure count anywhere. Drive the suite through `prove` instead. It runs every selected script regardless of failures and ends with a Test Summary Report naming each failing script, its failing assertion numbers, and the totals -- the same shape the pjdfstest target already produces, since that one is prove-driven too. Three details that come with it: - UNIT_TESTS= empties git's C unit-test list, which the prove target otherwise appends to $(T). Those are built binaries exercising in-process data structures; they never touch the mount and would need a separate build step. - Parallelism moves from `make -j` to prove's own --jobs. - --verbose-log tees each script's full TAP stream to test-results/<script>.out. Under a TAP harness test-lib.sh writes no .counts files at all (they are guarded on HARNESS_ACTIVE), so the previous dump-failure-logs.sh git branch was reading files that never existed once a harness was in play -- and its `tail` of the matching .out was dead code even before that, because nothing passed --verbose-log. It now reports from the .out files: the failing assertion titles as an index, then the tail holding the actual diff/stderr. install-target.sh asserts `prove` is present rather than letting a missing TAP harness surface as an opaque make failure minutes into the run, on the mount, after the whole build. Validated against a real v2.55.0 build: a synthetic failing script plus two real ones confirms the later scripts still run after the failure, the summary names the failure, the exit status propagates, and dump-failure-logs.sh renders the assertion and its output. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
A cell's result only means something relative to how the filesystem was made and mounted, and none of those knobs were written down anywhere outside the backend scripts themselves. Collect them, plus the root cause of every red cell run down so far, and link it from the top of the README. The settings half records, per backend, what we actually pass and what it implies: the loop backend's `uid=`-only mount for vfat (and the deliberate decision to leave fmask/dmask at kernel defaults), NFS's split between export and mount option namespaces and why root_squash is the default with a per-target opt-out, BeeGFS's port choices and why sysMountSanityCheckMS is zeroed on v8 only. The known-red half explains why each failing cell fails. The one worth reading is Loop vfat / git testsuite, because "surely git's own suite handles this?" is the obvious reaction and the answer is specific: git probes the filesystem for SYMLINKS, CASE_INSENSITIVE_FS and FILEMODE, which all come out correct on vfat, but POSIXPERM is set from `uname -s` and is unconditionally true on Linux regardless of what the work tree sits on. So git runs every permission-dependent assertion against a filesystem that reports 0755 for everything. t0001-init.sh's check_config() helper fails whenever `.git/config` is executable -- which on vfat it always is -- and it has 13 call sites, matching the 13 failures that cell reports in that script exactly. That is a finding about the test suite's portability assumptions rather than about git the program, and it is not a regression: nobody upstream runs the suite on a vfat work tree. Recording it so the next person does not re-derive it. Also notes what is deliberately not covered yet, chiefly that a single localhost export with kernel-negotiated defaults is a thin proxy for "NFS" -- version, attribute caching, locking and squashing are each a plausible row of their own. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
BeeGFS 8.1.0 / stress-ng went from green to red between two runs of the
same code with this, and nothing else, in the job log:
I: mounting BeeGFS at /mnt/beegfs
+ mount --internal -t beegfs --source beegfs_nodev ...
chown: cannot access '/mnt/beegfs': Communication error on send
I: teardown
`mount -t beegfs` returns as soon as the client module has registered
with mgmtd and downloaded the node groups. That is not the same as its
connections to the meta and storage nodes being usable: dmesg shows
"BeeGFS mount ready" at 164.865s and the umount from teardown at
164.868s, three milliseconds later, because the chown on the very next
line failed with ECOMM.
start_cluster() waiting for mgmtd/meta/storage to bind does not cover
this -- a listening port only proves the server side is up, and the
`sleep 2` grace period after it is a guess that happens to be enough most
of the time. The client side needs its own readiness probe.
wait_for_mount_usable() probes with a real create + write + read-back:
the create exercises the meta node, the write a storage target, so a
success means the mount can actually do work rather than merely exist.
It polls for up to --mount-ready-wait seconds (default 60, env
EVAL_UNDER_BEEGFS_MOUNT_READY_WAIT).
This class of red is the expensive one: it looks like a finding about the
filesystem, it is intermittent, and it produces no suite output to
contradict that reading. So on timeout the error now carries the last
probe failure and a dmesg beegfs tail, and GOTCHAS.md grows a section
separating harness races from real findings.
Probe logic tested both ways in isolation: returns immediately on a
healthy directory, and on an unusable path times out with the underlying
errno reported rather than a bare "not ready".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
The prove switch did its job -- the vfat cell now runs all 174 scripts instead of aborting after two, and uploads 348 result files instead of 8. But that turned the on-failure dump into thousands of lines, which scrolls prove's Test Summary Report out of reach. The count was once again present and unfindable, which was the original complaint. So bound the detail and put the number last: - detail for the first 8 failing scripts (env EVAL_UNDER_GIT_DUMP_MAX_SCRIPTS), with a 30-line tail rather than 60 (EVAL_UNDER_GIT_DUMP_TAIL_LINES) - an explicit "N further failing script(s) not detailed here" line pointing at the artifact, so the cap is never silent - a per-script roll-up plus grand total printed last, so the end of the job log answers "how many, and where" with no scrolling Verified against a synthetic test-results directory of ten failing scripts: caps at eight, reports the two it skipped, and totals correctly. Incidental confirmation from the run that prompted this: t1700-split-index fails on vfat at "split index respects core.sharedrepository 0666" and "0642" -- core.sharedrepository is a file-mode feature, so that is the same POSIXPERM-on-a-mode-0755-filesystem story GOTCHAS.md documents for t0001-init, showing up in a second script. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
The NFS backend's version is the literal "n/a", so the artifact name
built as "logs-<backend>-<version>-<target>" came out as
"logs-nfs-n/a-git". actions/upload-artifact rejects '/' in artifact
names outright.
This was latent until now: the upload step validates the name only
after it has files to upload, and `if-no-files-found: ignore`
short-circuited first. Switching the git target to prove --verbose-log
made the cell produce 348 .out files, so the upload finally ran and the
name was rejected -- turning a cell whose suite had *passed*
("Files=174, Tests=10368 ... Result: PASS") into a red one.
bin/ci/matrix.sh already knew how to render a slash-free backend slug;
extend it with cell_slug() for a whole cell and expose it to the
workflow through bin/ci/cell-slug.sh, rather than reimplementing the
"n/a" special case in GitHub expression syntax.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
Replaces 21 workflow files (20 generated per-cell dispatchers + one reusable body) with a single .github/workflows/test.yaml whose job matrix is read from .github/matrix.yaml. Following datalad/datalad's tools/ci/test-jobs.yml pattern: a `matrix` job renders the cell list to $GITHUB_OUTPUT, and the `test` job consumes it via fromJson with fail-fast disabled. Each entry carries its own `name`, so the checks list still reads "BeeGFS 7.4.6 / git testsuite" rather than GitHub's default "test (beegfs, 7.4.6, git)". The 20 files were never the design, they were a workaround: GitHub publishes one badge per workflow *file*, so a 20-cell README grid needed 20 files. That constraint is now paid for directly instead of structurally -- each run renders an SVG per cell and force-pushes them to an orphan `badges` branch, which the README points at. No third-party badge service, no 20 requests to render the README, and the SVGs are diffable. .github/matrix.yaml is now the single source of truth for both axes, the pinned upstream refs, and the per-target knobs (timeout, loop-size-mb, needs-root, needs-git-annex). bin/ci/matrix.sh stops holding that data and becomes accessors over it, keeping its function API so every existing caller is unchanged. The workflow and the shell scripts can no longer drift, because they parse the same file. Also drops target_cron_minute: the per-target cron stagger existed so 20 separate workflows would not stampede the runner pool. One workflow fanning out needs one cron. Removed: gen-dispatchers.sh (replaced by gen-readme-matrix.sh, which now only has a table to write) and cell-slug.sh (the slug ships in the matrix entry, so the workflow no longer shells out for it). Verified locally, not just linted: - `run-under.sh loop ext4 stress-ng` through a real loop mount: 20/20 stressors pass, clean teardown, 768MB image (i.e. loop-size-mb read from the new YAML and applied) - matrix-json.sh -> fromJson simulation: 20 unique cells, 20 unique slash-free slugs, every `matrix.*` key the workflow references is provided, and the badges job's `result-*` download pattern matches the artifact names the cells upload - render-badge.sh output parses as well-formed XML in all states - publish-badges.sh --dry-run renders all 20 from a partial result set, marking the absent ones "unknown" rather than dropping them - gen-readme-matrix.sh --check idempotent; shellcheck clean Note: the `badges` branch is created by the first non-PR run, so the README badge images 404 until this lands on master. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
Follows con/git-annex's badging, adapted. Two things change. 1. State is merged, not recomputed. bin/ci/publish-badges.sh derived all 20 badges from one run's artifacts, defaulting anything absent to "unknown". That is wrong for any run that does not cover every cell -- GitHub's "Re-run failed jobs" executes only the red ones, so it would have rewritten the ~10 green badges to "unknown" and destroyed good state. A cancelled run would do the same. status.json now lives on the site branch and is merged into, the way con/git-annex's update.py merges its clients' status.json. The monotonic guard is per cell on (run_number, run_attempt) rather than global on highest_build: a re-run keeps run_number and bumps run_attempt, so the pair orders correctly, and guarding per cell is what lets a partial re-run update its own cells without touching the others. Deliberate divergence from upstream: they set a test absent from a run to UNKNOWN; here absent means "this run did not re-run that cell", which is not the same claim, so the prior result is carried over. Cells dropped from .github/matrix.yaml are pruned, so nothing lingers. 2. Badges now link somewhere useful. They previously all pointed at the workflow's run list, because GitHub publishes no stable URL for "the latest job of this matrix cell". So the indirection is ours: each run regenerates a report page on gh-pages where every cell anchors as #<slug> and links to the deep job URL for the run that produced its current state. Job URLs are resolved from `gh api repos/$REPO/actions/runs/$RUN_ID/jobs` matched on job name -- con/git-annex's set-pr-status trick -- which is exact here because the job names come from the matrix entries' `name`. (set-pr-status itself does not apply to us: it exists to synthesise statuses onto an external repo's PR. Our 20 cells are native checks that already carry their own log links.) The page also carries what a badge cannot: which run, how long ago, and a one-line reason for each of the ten cells that are red on purpose. Badge images are served from raw.githubusercontent so the README grid renders even before Pages is enabled; only the links need Pages. Site history is kept rather than force-pushed, so status.json doubles as a log of when each cell changed state. Verified locally: - full run (20 cells) then a simulated "Re-run failed jobs" (10 cells): 0 cells wiped to unknown, the 10 that did not re-run kept their state, and the re-run's cells advanced to #7.2 while the rest stayed at #7.1 - an out-of-order older run (#6 after #7) is ignored for every cell - job-name -> URL matching resolves 20/20 across a paginated `gh api` response, and 20 deep links land in the page - badge SVGs parse as well-formed XML; page has 20 anchors and 20 images - empty-string GITHUB_RUN_* (what a workflow exports for a missing value) no longer crashes int() -- found by running the publisher - shellcheck clean; python compiles; gen-readme-matrix.sh --check idempotent Note: gh-pages is created by the first non-PR run, and the report links need Pages enabled on the repo (Settings -> Pages -> branch: gh-pages). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
The .pyc files came from running python3 -m py_compile as a syntax check while developing the new bin/ci/*.py scripts; they are build output and have no business in the tree. Ignore the pattern so the next syntax check cannot repeat it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
REUSE.toml enumerates paths rather than using a `path = "**"` catchall (deliberately -- see the comment there), so a new top-level file is uncovered until it is listed. `reuse lint` would have failed on the .gitignore added in the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
PR #3 adds a codespell workflow, and its check fails on this branch: "os.environ.get's" trips the get's ==> gets rule. The possessive is correct English, but the rule has no way to know that, and adding an ignore-words-list entry for one docstring is worse than rephrasing. Verified with PR #3's own .codespellrc: the whole tree is now clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
Conflict was in README.md's file table, and it was a real one rather than a formatting clash: #3 corrected the `drafts/git-annex-test-beegfs.yaml` row from datalad/git-annex to con/git-annex, while this branch deleted the two rows directly above it (`_test-under.yaml` and `test-*.yaml`, both of which no longer exist) and added seven new ones for the matrix and status-site scripts. Resolved by keeping this branch's table and carrying #3's correction onto the surviving row, then re-padding the whole table -- con/ is four characters shorter than datalad/, so taking the row verbatim would have left the column ragged. Also covers .codespellrc in REUSE.toml. It arrived with #3 uncovered: the annotation list is an explicit enumeration rather than a `**` catchall (deliberately -- see the comment there), so a new top-level file is unlicensed until it is listed. No CI check enforces this today, but the repo is otherwise 100% compliant and worth keeping that way. Verified on the merged tree: codespell clean under the config #3 added (which is what its workflow runs); shellcheck clean; python compiles; all three workflow files parse; gen-readme-matrix.sh --check idempotent, so the generated CI matrix block survived the table re-padding; every tracked file REUSE-covered. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CI ran exactly one suite (
git annex test) per backend. That answers "is git-annex broken on this filesystem?" but not "at which layer?" — a rename-semantics quirk shows up as a pile of git-annex failures with no indication whether git, or POSIX itself, is already unhappy.This adds three complementary targets underneath it and turns the single Status column into a grid.
Targets
stress-ng--verify, one at a timeapt install stress-ngpjdfstestgitt0*.sh t1*.sh, ~10k assertions)git-annexgit annex testEach stressor runs on its own so an unsupported operation (xattr on vfat) reports as a skip, not a failure that poisons the run. For the
gittarget only the trash directories go on the mount (via the suite's own--root); the build stays on the runner's disk, so the mount sees test I/O and nothing else.Structure
bin/ci/matrix.sh— single source of truth: backend rows, target columns, pinned upstream refs, per-target timeout / loop-image size / cron minute.bin/ci/install-target.sh— runner-side prep (apt package, or shallow fetch + build at a pinned ref into$EVAL_UNDER_SRC_DIR).bin/ci/target-<target>.sh— the suite itself, run inside the mount.bin/ci/run-under.sh— grows a third argument selecting the target, and applies its timeout and loop size.bin/ci/gen-dispatchers.sh— generates the 20 per-cell dispatcher workflows and the README matrix frommatrix.sh;--checkreports stale or orphaned files. GitHub publishes one badge per workflow file, not per matrix cell, so a grid needs one file per cell.The five existing dispatchers are renamed with an explicit
-git-annexsuffix so all 20 cells are named uniformly. Crons are staggered per target so the cells don't all stampede the runner pool at once.Pinning
gitis pinned to thev2.55.0tag.pjdfstestis pinned to a commit, not to its one upstream tag. That tag (0.1, 2016) no longer compiles:major()/minor()/makedev()moved to<sys/sysmacros.h>in glibc 2.28 and the tree builds with-Werror, so the implicit declarations are hard errors on any current toolchain. Master builds clean, so pinning a commit on it beats carrying a patch. Same determinism, no fork.Also
README badge URLs still pointed at the pre-rename
yarikoptic/beegfs-testrepo; the generated matrix points them atcon/eval-under.Validation
Run locally on ext4 (loop + plain), not just written:
run-under.sh loop ext4 stress-ng→ 20/20 pass through a real loop mount, clean teardownpjdfstest→Files=238, Tests=8827,Result: PASSgit→ 174 scripts, 10366 tests,failed 0(179 upstream# TODO known breakage)run-under.sh loop ext4 git(narrow glob) → trash dirs confirmed on the mount, none in-treeshellcheck bin/ci/*.sh bin/eval-under* provision/*.shclean; all 21 workflows parse;reuse lintstill 100% compliantvfat and the network backends are untested locally — this container's kernel has no vfat module and no NFS server / BeeGFS kmod. Those cells get their first real signal from this PR's CI run, and some of them are expected to be red: vfat has no symlinks, ownership, or xattrs. That's the point of the grid.
🤖 Generated with Claude Code
https://claude.ai/code/session_01SHhrxxVz3TJFwYMXuxei2H
Generated by Claude Code