diff --git a/.claude/skills/mx-api-add-feature/SKILL.md b/.claude/skills/mx-api-add-feature/SKILL.md index d641b0b26..d47a3c996 100644 --- a/.claude/skills/mx-api-add-feature/SKILL.md +++ b/.claude/skills/mx-api-add-feature/SKILL.md @@ -63,9 +63,9 @@ write paths symmetric. See ./steps/step-3-mx-impl.md ## Step 4: Run tests -Run `make test` for the unit suites and `make test-api-roundtrip` for the corpus roundtrip gate -(Step 1 explains both, along with `make discover-api-roundtrip`). Verify everything passes. Then run -`make fmt` and `make check` so the CI quality gates will pass. +Run `make api-test` for the unit suites and `make api-roundtrip` for the corpus roundtrip gate +(Step 1 explains both, along with `make api-roundtrip-discover`). Verify everything passes. Then run +`make fmt` and `make fmt-check` so the CI quality gates will pass. ## (optional) Step 5: Open a PR diff --git a/.claude/skills/mx-api-add-feature/steps/step-1-test-strategy.md b/.claude/skills/mx-api-add-feature/steps/step-1-test-strategy.md index 930210ba9..c14ba3361 100644 --- a/.claude/skills/mx-api-add-feature/steps/step-1-test-strategy.md +++ b/.claude/skills/mx-api-add-feature/steps/step-1-test-strategy.md @@ -16,7 +16,7 @@ There are two kinds of api tests. - Location: `src/private/mxtest/api/` (e.g. `NoteDataTest.cpp`, `DirectionDataTest.cpp`, `MeasureDataTest.cpp`, `PitchDataTest.cpp`, `MetronomeApiTest.cpp`). -- They compile into the `mxtest` binary and run via `make test`. +- They compile into the `mxtest` binary and run via `make api-test`. - Every file is gated by `#include "mxtest/control/CompileControl.h"` then `#ifdef MX_COMPILE_API_TESTS ... #endif`. That flag is defined in `src/private/mxtest/control/CompileControl.h` (currently always on). Wrap any new test file the @@ -65,13 +65,13 @@ side, and run a strict DOM compare (`mxtest/corert/Compare`). Zero tolerance on - `regression `: runs only the files listed in `src/private/mxtest/api/roundtrip-baseline.txt`. Non-zero exit on any failure. This is the CI - gate. Run with `make test-api-roundtrip`. CI runs it in `.github/workflows/ci.yaml` (the - `linux-api` job step "Run corpus api roundtrip (regression mode)" and again in the docker job). + gate. Run with `make api-roundtrip`. CI runs it in `.github/workflows/ci.yaml` (the `test-linux` + job via `make test-all`, and again natively on macOS/Windows). - `discovery `: walks the whole corpus (excludes `expected/`, `testOutput/`, `generalxml/`, `smufl/`, `*.fixup.xml`, `*.features.xml`, `corpus.xml`, `api.features.xml`, and any file with a sibling `*.invalid` marker), prints one tab-separated line per file `PASS|FAIL|SKIP|LOADFAIL|GETDATAFAIL|CREATEFAIL relpath detail`, and always exits 0. - Run with `make discover-api-roundtrip`. Never a gate. + Run with `make api-roundtrip-discover`. Never a gate. `roundtrip-baseline.txt` is a growing pinned list (~140 files across ksuite/lysuite/musuite/mjbsuite/custom/synthetic; see the file's header). It grows only by @@ -90,8 +90,8 @@ genuinely survives the strict compare. ## Build note -`make test-api-roundtrip` and `make discover-api-roundtrip` depend on the `dev` target (builds -`mxtest-api-roundtrip`); `make test` builds and runs `mxtest` (the unit tests) plus examples. +`make api-roundtrip` and `make api-roundtrip-discover` depend on the `api-build` target (builds +`mxtest-api-roundtrip`); `make api-test` builds and runs `mxtest` (the unit tests) plus examples. ## Procedure @@ -103,7 +103,7 @@ Do these in order. The discovery snapshot must be captured before you implement. MusicXML shape and attribute set. 2. Capture the pre-implementation discovery baseline: - `make discover-api-roundtrip > /tmp/api-discovery-before.txt`. This is your reference for which + `make api-roundtrip-discover > /tmp/api-discovery-before.txt`. This is your reference for which corpus files fail and WHY (the `detail` column). Keep it. 3. Grep that snapshot for files whose failure looks attributable to the missing feature -- e.g. @@ -135,7 +135,7 @@ Do these in order. The discovery snapshot must be captured before you implement. 8. Make the unit test pass (green); iterate until your new `TEST(...)` cases pass (SKILL.md Step 4 owns the test/fmt/check commands). -9. Re-run discovery and diff: `make discover-api-roundtrip > /tmp/api-discovery-after.txt`, then +9. Re-run discovery and diff: `make api-roundtrip-discover > /tmp/api-discovery-after.txt`, then `diff /tmp/api-discovery-before.txt /tmp/api-discovery-after.txt`. Look for files that moved to `PASS` (or whose `FAIL` detail changed). For each newly-PASSing file, confirm the pass is actually attributable to your feature (re-read the file; check the old `detail` named your diff --git a/.claude/skills/mx-api-classify-roundtrip-failures/SKILL.md b/.claude/skills/mx-api-classify-roundtrip-failures/SKILL.md index 3c589b322..468fa5d99 100644 --- a/.claude/skills/mx-api-classify-roundtrip-failures/SKILL.md +++ b/.claude/skills/mx-api-classify-roundtrip-failures/SKILL.md @@ -52,13 +52,13 @@ to be absorbed in test normalization later (#214). If `build/api/roundtrip-dump/` is empty or stale (api/impl changed since it was written), run: ``` -make dump-api-roundtrip +make api-roundtrip-dump ``` Then always run: ``` -make classify-api-roundtrip +make api-roundtrip-classify ``` Read the stdout summary it prints -- status counts, the distance histogram, the ranked worklist, diff --git a/.github/instructions/corpus-and-tests.instructions.md b/.github/instructions/corpus-and-tests.instructions.md index e95be953a..4b407f9de 100644 --- a/.github/instructions/corpus-and-tests.instructions.md +++ b/.github/instructions/corpus-and-tests.instructions.md @@ -8,12 +8,12 @@ applyTo: "data/**,src/private/mxtest/**" - Adding or removing a corpus file requires: the pinned count bump in `src/private/mxtest/corert/CoreRoundtripTest.cpp`, regenerated audit artifacts (`make audit`: the `*.features.xml` sidecar and `data/corpus.xml`), and a green - `make test-core-dev`. Flag a corpus change missing any of these. + `make core-roundtrip-test`. Flag a corpus change missing any of these. - A deliberately unparseable file needs a sibling `.invalid` marker explaining why; a schema-invalid-but-lenient file needs a `.fixup.xml` sidecar. New fixups must follow the uniform leniency policy in `data/README.md`, not invent per-file rules. - `roundtrip-baseline.txt` (the api round-trip pass list) only grows by deliberate commits; flag removals or unexplained churn. - `ApiLoadSmokeTest` only proves a file imports without crashing; a feature claim needs a - fixture pinned in the read->write->read gate (`make test-api-roundtrip`). + fixture pinned in the read->write->read gate (`make api-roundtrip`). - Test code follows the same C++ rules as the library (no new anonymous namespaces). diff --git a/.github/instructions/cpp.instructions.md b/.github/instructions/cpp.instructions.md index 785c2868f..c47192e75 100644 --- a/.github/instructions/cpp.instructions.md +++ b/.github/instructions/cpp.instructions.md @@ -8,7 +8,7 @@ Repo-wide C++ rules (see AGENTS.md): get named functions (`tokenIsNameChar`); file-local constants get a per-type prefix (`kYesNoWire`). Existing anonymous namespaces are being retired as touched; do not demand drive-by fixes. -- Formatting is enforced by clang-format (`make fmt` / `make check`); do not comment on layout +- Formatting is enforced by clang-format (`make fmt` / `make fmt-check`); do not comment on layout the formatter controls. - Keep source ASCII. - Comments should state constraints and effects hard to ascertain from the code. diff --git a/.github/instructions/gen.instructions.md b/.github/instructions/gen.instructions.md index f84567fb4..643772ce9 100644 --- a/.github/instructions/gen.instructions.md +++ b/.github/instructions/gen.instructions.md @@ -17,6 +17,6 @@ C++ (`gen/cpp`, the product), Go, C, JSON Schema. Review against its cardinal ru - Fail loud: new error paths must exit non-zero with a `template:line` style message, never emit best-effort output. - Generated output is committed: a generator change without its regenerated output (or vice - versa) fails `make test-gen` drift checking. + versa) fails the CI drift check (`make gen` then `git diff --exit-code`). - Template identifiers must be unity-build safe: use `{{ident}}`-suffixed names for file-local symbols. diff --git a/.github/instructions/generated-core.instructions.md b/.github/instructions/generated-core.instructions.md index 63746bcbb..3db9fad94 100644 --- a/.github/instructions/generated-core.instructions.md +++ b/.github/instructions/generated-core.instructions.md @@ -7,6 +7,6 @@ Everything in this directory is GENERATED by `gen/` (`make gen-cpp`) from the Mu - Do not review style, naming, or repetition here; the generator owns those. - A diff in these files must be accompanied by a change under `gen/` (templates, config, or generator code) that produces it. Flag any hand edit -- it will be destroyed on the next - regeneration, and CI (`make test-gen`) fails on regeneration drift. + regeneration, and the CI drift check (`make gen` then `git diff --exit-code`) fails on it. - Review the paired `gen/` change instead, and check the generated diff only for evidence of an unintended blast radius (e.g. an edit meant for one type changing hundreds of files). diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml index 3519e7436..5f5d48188 100644 --- a/.github/workflows/ci.yaml +++ b/.github/workflows/ci.yaml @@ -20,52 +20,57 @@ concurrency: env: DOCKER_VOLUME: mx-build +# Coverage is intentionally absent here -- the instrumented core/api builds are +# expensive and run only on demand from .github/workflows/coverage.yaml (the +# "Run workflow" button or a `/coverage` PR comment). +# # Every Linux job runs its make targets inside the pinned mx-sdk Docker -# toolchain; only the two macOS jobs build natively. +# toolchain (ccache baked into the image); the macOS and Windows jobs build +# natively to catch AppleClang and MSVC fallout, kept incremental across runs by +# a content-addressed compiler cache -- ccache on macOS, sccache on Windows. jobs: - # Cheap up-front diff check so the gen-quality gate (score.json/report.md, - # ratcheted floor) only runs -- and only spends a PR comment -- when gen/ - # actually changed. A gen/ change that's md-only (README/AGENTS/DESIGN docs) - # doesn't move the score, so it's excluded too. Other linux-gen steps - # (fmt-check, test-gen, gen-check, regen drift) are unaffected: they're - # cheap and apply to the whole repo, not just gen/. + # Cheap up-front diff check so the gen job (generator/audit tests, Go/C + # corert, the ratcheted quality/lint gates, and the gen-quality PR comment) + # only runs -- and only spends a PR comment -- when gen/ actually changed. A + # gen/ change that's md-only (README/AGENTS/DESIGN docs) doesn't move any of + # that, so it's excluded too. changes: name: "Detect gen/ changes" runs-on: ubuntu-latest outputs: gen: ${{ steps.filter.outputs.gen }} steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v5 - uses: dorny/paths-filter@v3 id: filter with: + # predicate-quantifier 'every' requires a changed file to match ALL + # patterns to count -- i.e. be under gen/ AND not markdown. Without + # it the patterns are OR'd, and the `!gen/**/*.md` rule then matches + # every non-gen-markdown path in the repo, so the filter fired on + # essentially every PR (the gen job ran even for pure Makefile/C++ + # changes). See picomatch negation semantics. + predicate-quantifier: 'every' filters: | gen: - 'gen/**' - '!gen/**/*.md' - # Python-side gates for the gen/ generator and audit tool, plus the repo's - # only clang-format gate. Also runs plates --check for every target. No C++ - # is compiled or tested here. - linux-gen: - name: "gen: unit/audit tests + regen drift + quality/lint + fmt-check (Linux)" + # The deep gate: every C++ suite -- the mx::core substrate (corert + unit) + # and the mx::api product surface (mxtest + examples + corpus round-trip) -- + # built once through the pinned mx-sdk toolchain. This is the one place the + # core suites run, so it is the authoritative C++ correctness check. + test-linux: + name: "linux: test-all" runs-on: ubuntu-latest - needs: changes - - # Read-only token: this job builds untrusted fork PR code, so it never - # requests write. The gen-quality comment is posted by the pr-comment.yaml - # companion workflow (workflow_run) from the base-repo context instead. - permissions: - contents: read steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v5 # Container-driver builder (needed to export the gha layer cache) plus # the runtime-token export so the Makefile auto-detects # ACTIONS_RUNTIME_TOKEN and pushes/pulls the Docker layer cache. Keeps - # the toolchain-install layer warm across runs. The same `make check` - # runs locally with no cache (token absent -> plain build). + # the toolchain-install layer warm across runs. - uses: docker/setup-buildx-action@v3 - uses: crazy-max/ghaction-github-runtime@v3 @@ -84,128 +89,156 @@ jobs: uses: actions/cache@v4 with: path: build/docker/.ccache - key: ccache-${{ runner.os }}-gen-${{ github.sha }} - restore-keys: ccache-${{ runner.os }}-gen- - - - name: fmt-check (clang-format, all C++ under src/) - run: make check + key: ccache-${{ runner.os }}-test-${{ github.sha }} + restore-keys: ccache-${{ runner.os }}-test- - - name: gen unit tests (parser/IR/plates/press) - run: make test-gen + # corert file counts are pinned by an in-suite counts case (see CoreRoundtripTest.cpp). + - name: All C++ suites (core roundtrip + unit, api mxtest + examples + roundtrip) + run: make test-all + + # The api product surface, built natively with AppleClang. macOS is a + # portability check on the product tier only (the core suites run on Linux in + # test-linux); this is the only C++ job off the pinned toolchain, by design, + # to keep finding AppleClang fallout. + # + # Incremental across runs via ccache (content-addressed, like the mx-sdk image + # bakes in for Linux). This replaces an earlier build/-dir cache keyed on + # hashFiles('src/**') -- that key was busted by every source change, so it gave + # nothing on exactly the PRs that recompile. + test-macos: + runs-on: macos-latest - - name: audit tool unit tests (incl. failure classifier) - run: make test-audit + # No Docker on the macOS runners: MX_RUNNING_IN_DOCKER=1 flips the Makefile + # to its direct-invocation branch so the build runs natively. CMake reads the + # *_COMPILER_LAUNCHER env vars as launcher defaults (the Makefile passes no -D + # launcher flag), the same mechanism the mx-sdk image uses for Linux. + env: + MX_RUNNING_IN_DOCKER: 1 + CMAKE_C_COMPILER_LAUNCHER: ccache + CMAKE_CXX_COMPILER_LAUNCHER: ccache - - name: plates --check (all targets) - run: make gen-check + steps: + - uses: actions/checkout@v5 - # The committed generated output must match the generator: regenerate - # every target, then any tracked-file drift fails the build. - - name: Regenerate all targets - run: make gen + - name: ccache + uses: hendrikmuhs/ccache-action@v1.2.18 + with: + key: ${{ runner.os }}-api + max-size: 500M - - name: Generated output drift check - run: git diff --exit-code + - name: api suite (mxtest + examples) + run: make api-test - # Static-analysis gates on the gen/ generator. gen-quality only runs when - # gen/ actually changed (see the `changes` job); gen-lint stays unguarded - # since it's cheap and has no report/comment plumbing riding on it. Each - # fails the build below its floor (GEN_QUALITY_FLOOR / GEN_LINT_FLOOR in - # the Makefile). - - name: gen-quality gate - if: needs.changes.outputs.gen == 'true' - run: make gen-quality + - name: api corpus round-trip (regression mode) + run: make api-roundtrip + + # The api product surface, built natively with the real MSVC toolchain to keep + # finding Windows portability fallout. Windows has no make/Unix tools, so this + # job drives CMake directly. + # + # Generator is Ninja, not the Visual Studio generator: the VS generator ignores + # CMAKE__COMPILER_LAUNCHER, so sccache (the MSVC-capable compiler cache) + # can't hook in under it. Ninja honors the launcher and is faster besides. + # ilammy/msvc-dev-cmd puts cl.exe + the VS-bundled ninja on PATH via vcvars. + # Debug info is forced to Embedded (/Z7, not the default /Zi): sccache can't + # cache the shared-PDB writes /Zi makes, so /Z7 is what lets the object cache + # actually hit. Ninja is single-config, so binaries land in build/api/ (no + # Debug/ subdir). + test-windows: + runs-on: windows-latest + + # Git Bash so multi-command run steps stop on first failure (set -e) and + # forward-slash paths + .exe invocation work uniformly. + defaults: + run: + shell: bash - - name: gen-lint gate - run: make gen-lint + steps: + - uses: actions/checkout@v5 - # Surface the score in the run summary when the gate actually ran. - # always() so a gate failure still reports why; skipped runs (gen/ - # unchanged) leave this section out entirely rather than printing a - # misleading "(no report produced)". - - name: gen-quality report - if: always() && needs.changes.outputs.gen == 'true' - run: | - { - echo '### gen-quality `gen/`' - echo '' - cat data/testOutput/gen-quality/report.md 2>/dev/null || echo '(no report produced)' - } >> "$GITHUB_STEP_SUMMARY" + # Puts cl.exe, link.exe, and the VS-bundled ninja on PATH -- exported to + # $GITHUB_ENV, so the later bash steps inherit the MSVC environment. + - uses: ilammy/msvc-dev-cmd@v1 - # Fork PRs run with a read-only token, so this job can't post the comment - # itself. Stage the body + PR number as an artifact; pr-comment.yaml posts - # it after CI completes. always() so a gate failure still reports why. - # Guarded on the report file existing (not just needs.changes.outputs.gen) - # so the comment-sending job gets no artifact -- and gracefully posts - # nothing -- whenever gen-quality didn't run or crashed before writing it. - - name: Stage gen-quality PR comment - if: always() && github.event_name == 'pull_request' + - name: sccache + uses: hendrikmuhs/ccache-action@v1.2.18 + with: + variant: sccache + key: ${{ runner.os }}-msvc + max-size: 1G + + - name: Configure (MSVC, Ninja, MX_API) + run: >- + cmake -S . -B build/api -G Ninja + -DCMAKE_BUILD_TYPE=Debug + -DMX_API=on + -DCMAKE_C_COMPILER=cl + -DCMAKE_CXX_COMPILER=cl + -DCMAKE_C_COMPILER_LAUNCHER=sccache + -DCMAKE_CXX_COMPILER_LAUNCHER=sccache + -DCMAKE_POLICY_DEFAULT_CMP0141=NEW + -DCMAKE_MSVC_DEBUG_INFORMATION_FORMAT=Embedded + + - name: Build + run: >- + cmake --build build/api + --target mx mxtest mxtest-api-roundtrip mxread mxwrite mxhide + --parallel + + - name: api examples run: | - if [ ! -f data/testOutput/gen-quality/report.md ]; then - echo 'No gen-quality report to stage (check skipped or produced nothing).' - exit 0 - fi - mkdir -p pr-comment - { - echo '### gen-quality `gen/`' - echo '' - cat data/testOutput/gen-quality/report.md - echo '' - echo 'Commit `${{ github.sha }}`.' - } > pr-comment/body.md - echo '${{ github.event.pull_request.number }}' > pr-comment/number + build/api/mxread.exe + build/api/mxwrite.exe build/api/example.musicxml + build/api/mxhide.exe - - name: Upload gen-quality PR comment - if: always() && github.event_name == 'pull_request' - uses: actions/upload-artifact@v4 - with: - name: pr-comment-gen - path: pr-comment/ - if-no-files-found: ignore + - name: mxtest suite (api/impl/file/control) + run: build/api/mxtest.exe - # The mx::api/mx::impl product surface. Core-layer suites (corert, unit, - # xmllint, probes) run in linux-core, not here. - linux-api: - name: "api: mxtest suite + examples + corpus round-trip (Linux)" - runs-on: ubuntu-latest + - name: api corpus round-trip (regression mode) + run: >- + build/api/mxtest-api-roundtrip.exe regression + data src/private/mxtest/api/roundtrip-baseline.txt + + # Proves mx compiles as a Swift package under AppleClang/SPM on every PR -- + # the path komp takes when it depends on a sibling mx via `.package(path:)`. + # Runs no test suites; the binary xcframework release arm is follow-up work. + # + # SwiftPM has no compiler-launcher hook, so incremental builds come from caching + # its .build tree: llbuild tracks inputs by content signature, so a restored + # tree rebuilds only what changed. The per-commit key suffix makes each run save + # an updated tree while restore-keys pulls the newest prior one. + swift: + runs-on: macos-latest steps: - - uses: actions/checkout@v4 - - - uses: docker/setup-buildx-action@v3 - - uses: crazy-max/ghaction-github-runtime@v3 + - uses: actions/checkout@v5 - - name: Create bind-backed build volume - run: | - mkdir -p build/docker - docker volume create --driver local \ - --opt type=none --opt o=bind \ - --opt device="$GITHUB_WORKSPACE/build/docker" mx-build - - - name: Cache ccache + - name: Cache SwiftPM build uses: actions/cache@v4 with: - path: build/docker/.ccache - key: ccache-${{ runner.os }}-api-${{ github.sha }} - restore-keys: ccache-${{ runner.os }}-api- - - - name: Build mx library - run: make lib + path: .build + key: ${{ runner.os }}-spm-${{ hashFiles('Package.swift', 'Package.resolved') }}-${{ github.sha }} + restore-keys: | + ${{ runner.os }}-spm-${{ hashFiles('Package.swift', 'Package.resolved') }}- + ${{ runner.os }}-spm- - - name: mxtest suite (api/impl/file/control) + examples - run: make test + - name: Swift toolchain + run: swift --version - - name: api corpus round-trip (regression mode) - run: make test-api-roundtrip + - name: Build the Mx Swift package (local-checkout source mode) + run: swift build --product Mx - # The strongly-typed mx::core layer. Note: check-core-dev (warning-free - # build gate) is a local-only gate; CI does not run it. - linux-core: - name: "core: corert round-trip + unit + xmllint + compile probes (Linux)" + # Repo-wide, compiles no C++: the clang-format gate plus a generated-output + # drift check (regenerate every target, then fail on any tracked-file diff). + # Always runs -- both apply to the whole repo, not just gen/. + quality: runs-on: ubuntu-latest + permissions: + contents: read + steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v5 - uses: docker/setup-buildx-action@v3 - uses: crazy-max/ghaction-github-runtime@v3 @@ -221,73 +254,41 @@ jobs: uses: actions/cache@v4 with: path: build/docker/.ccache - key: ccache-${{ runner.os }}-core-${{ github.sha }} - restore-keys: ccache-${{ runner.os }}-core- - - # corert file counts are pinned by an in-suite counts case (see CoreRoundtripTest.cpp). - - name: corert round-trip suite - run: make test-core-dev - - - name: mx::core unit tests - run: make test-cpp-unit - - - name: validate-cpp (serialize corpus + xmllint vs XSD) - run: make validate-cpp - - - name: probe-cpp (must-not-compile probes) - run: make probe-cpp - - # corert round-trip suites for the generator's emitted Go and C targets. - linux-targets: - name: "corert: Go + C generated targets (Linux)" - runs-on: ubuntu-latest - - steps: - - uses: actions/checkout@v4 + key: ccache-${{ runner.os }}-quality-${{ github.sha }} + restore-keys: ccache-${{ runner.os }}-quality- - - uses: docker/setup-buildx-action@v3 - - uses: crazy-max/ghaction-github-runtime@v3 - - - name: Create bind-backed build volume - run: | - mkdir -p build/docker - docker volume create --driver local \ - --opt type=none --opt o=bind \ - --opt device="$GITHUB_WORKSPACE/build/docker" mx-build + - name: fmt-check (clang-format, all C++ under src/) + run: make fmt-check - - name: Go corert suite - run: make test-go + # The committed generated output must match the generator: regenerate + # every target, then any tracked-file drift fails the build. + - name: Regenerate all targets + run: make gen - - name: C corert suite - run: make test-c + - name: Generated output drift check + run: git diff --exit-code - # Re-runs the core (corert + unit) and api (mxtest + round-trip) suites in - # instrumented builds to produce gcovr reports. Coverage % is advisory (no - # floor), but a test failure still fails the job. Split into two jobs so the - # two instrumented builds run in parallel instead of as serial steps in one - # job; coverage-report merges their outputs into a single job summary + - # PR comment once both finish. - coverage-core: - name: "coverage: core gcovr report, advisory (Linux)" + # Generator + audit gates, gated on gen/ changes (see the `changes` job): the + # generator/audit Python tests, the Go and C corert suites (they exercise the + # generator's emitted targets), and the ratcheted quality/lint gates. Also + # stages the gen-quality report as a PR comment, posted by pr-comment.yaml. + gen: runs-on: ubuntu-latest + needs: changes + if: needs.changes.outputs.gen == 'true' + # Read-only token: this job builds untrusted fork PR code, so it never + # requests write. The gen-quality comment is posted by the pr-comment.yaml + # companion workflow (workflow_run) from the base-repo context instead. permissions: contents: read - outputs: - table: ${{ steps.table.outputs.table }} - artifact-url: ${{ steps.upload.outputs.artifact-url }} - steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@v5 - # Same container-driver + runtime-token setup as linux-gen so the - # coverage stage reuses the warm toolchain layer cache. - uses: docker/setup-buildx-action@v3 - uses: crazy-max/ghaction-github-runtime@v3 - # Bind-back mx-build so its ccache dir is cacheable, as in linux-gen. A - # distinct key prefix keeps this job's save from racing the others. - name: Create bind-backed build volume run: | mkdir -p build/docker @@ -299,222 +300,67 @@ jobs: uses: actions/cache@v4 with: path: build/docker/.ccache - key: ccache-${{ runner.os }}-coverage-core-${{ github.sha }} - restore-keys: ccache-${{ runner.os }}-coverage-core- - - - name: coverage-core-dev (corert + unit, instrumented) - run: make coverage-core-dev - - # Turn gcovr's 3-line summary (data/testOutput/coverage/summary.txt, e.g. - # "lines: 52.1% (17624 out of 33846)") into a Markdown table for the job - # summary, and pass it to coverage-report as a job output for the - # merged PR comment. - - name: Build coverage table - id: table - if: always() - run: | - make_table() { - local file="$1" - echo '| Metric | Coverage | Covered / Total |' - echo '|---|---:|---:|' - awk '/^(lines|functions|branches):/ { - m=$1; sub(":","",m); - c=$3; sub("[(]","",c); - t=$6; sub("[)]","",t); - M=toupper(substr(m,1,1)) substr(m,2); - printf "| %s | %s | %s / %s |\n", M, $2, c, t - }' "$file" - } - { - echo '### Core-dev coverage `src/private/mx/core/`' - echo '' - make_table data/testOutput/coverage/summary.txt - } > table.md - cat table.md >> "$GITHUB_STEP_SUMMARY" - { - echo 'table<> "$GITHUB_OUTPUT" - - # Single self-contained page, stored (compression-level 0). GitHub still - # delivers artifacts as a .zip -- there is no raw-file download -- so the - # zip just holds one index.html to unzip and open. - - name: Upload core HTML report - id: upload - if: always() - uses: actions/upload-artifact@v4 - with: - name: coverage-core-dev - path: data/testOutput/coverage/index.html - compression-level: 0 - - coverage-api: - name: "coverage: api gcovr report, advisory (Linux)" - runs-on: ubuntu-latest - - permissions: - contents: read + key: ccache-${{ runner.os }}-gen-${{ github.sha }} + restore-keys: ccache-${{ runner.os }}-gen- - outputs: - table: ${{ steps.table.outputs.table }} - artifact-url: ${{ steps.upload.outputs.artifact-url }} + - name: gen unit tests (parser/IR/plates/press) + run: make gen-test - steps: - - uses: actions/checkout@v4 + - name: audit tool unit tests (incl. failure classifier) + run: make audit-test - - uses: docker/setup-buildx-action@v3 - - uses: crazy-max/ghaction-github-runtime@v3 + - name: Go corert suite + run: make test-go - - name: Create bind-backed build volume - run: | - mkdir -p build/docker - docker volume create --driver local \ - --opt type=none --opt o=bind \ - --opt device="$GITHUB_WORKSPACE/build/docker" mx-build + - name: C corert suite + run: make test-c - - name: Cache ccache - uses: actions/cache@v4 - with: - path: build/docker/.ccache - key: ccache-${{ runner.os }}-coverage-api-${{ github.sha }} - restore-keys: ccache-${{ runner.os }}-coverage-api- + # Static-analysis gates on the gen/ generator. Each fails the build below + # its floor (GEN_QUALITY_FLOOR / GEN_LINT_FLOOR in the Makefile). + - name: gen-quality gate + run: make gen-quality - - name: coverage-api (mxtest + round-trip, instrumented) - run: make coverage-api + - name: gen-lint gate + run: make gen-lint - - name: Build coverage table - id: table + # Surface the score in the run summary. always() so a gate failure still + # reports why. + - name: gen-quality report if: always() run: | - make_table() { - local file="$1" - echo '| Metric | Coverage | Covered / Total |' - echo '|---|---:|---:|' - awk '/^(lines|functions|branches):/ { - m=$1; sub(":","",m); - c=$3; sub("[(]","",c); - t=$6; sub("[)]","",t); - M=toupper(substr(m,1,1)) substr(m,2); - printf "| %s | %s | %s / %s |\n", M, $2, c, t - }' "$file" - } { - echo '### API coverage `src/private/mx/{api,impl,utility}/`' + echo '### gen-quality `gen/`' echo '' - make_table data/testOutput/coverage/api/summary.txt 2>/dev/null || echo '(api coverage not produced)' - } > table.md - cat table.md >> "$GITHUB_STEP_SUMMARY" - { - echo 'table<> "$GITHUB_OUTPUT" - - - name: Upload api HTML report - id: upload - if: always() - uses: actions/upload-artifact@v4 - with: - name: coverage-api - path: data/testOutput/coverage/api/index.html - compression-level: 0 - - # Merges the two coverage jobs' tables into one PR comment. always() so it - # still posts (with whatever tables it has) if one side failed; gated to - # pull_request since push events have no PR to comment on. - coverage-report: - name: "coverage: PR comment (core + api)" - runs-on: ubuntu-latest - needs: [coverage-core, coverage-api] - if: always() && github.event_name == 'pull_request' - - # Read-only token: posting the coverage comment is delegated to the - # pr-comment.yaml companion workflow so fork PRs get a comment too (their - # GITHUB_TOKEN is capped to read-only here). - permissions: - contents: read + cat data/testOutput/gen-quality/report.md 2>/dev/null || echo '(no report produced)' + } >> "$GITHUB_STEP_SUMMARY" - steps: - # Stage the body + PR number as an artifact; pr-comment.yaml posts it - # after CI completes (this job's fork-PR token is read-only). It posts a - # fresh comment each run, so the PR keeps a per-push coverage history. - - name: Stage PR comment + # Fork PRs run with a read-only token, so this job can't post the comment + # itself. Stage the body + PR number as an artifact; pr-comment.yaml posts + # it after CI completes. always() so a gate failure still reports why. + # Guarded on the report file existing so the comment-sending job gets no + # artifact -- and gracefully posts nothing -- if gen-quality crashed + # before writing it. + - name: Stage gen-quality PR comment + if: always() && github.event_name == 'pull_request' run: | + if [ ! -f data/testOutput/gen-quality/report.md ]; then + echo 'No gen-quality report to stage (gate produced nothing).' + exit 0 + fi mkdir -p pr-comment { - echo '### Coverage report' - echo '' - echo '${{ needs.coverage-core.outputs.table }}' - echo '' - echo '${{ needs.coverage-api.outputs.table }}' + echo '### gen-quality `gen/`' echo '' - echo '[Core HTML report](${{ needs.coverage-core.outputs.artifact-url }}) | [API HTML report](${{ needs.coverage-api.outputs.artifact-url }})' + cat data/testOutput/gen-quality/report.md echo '' echo 'Commit `${{ github.sha }}`.' } > pr-comment/body.md echo '${{ github.event.pull_request.number }}' > pr-comment/number - - name: Upload PR comment + - name: Upload gen-quality PR comment + if: always() && github.event_name == 'pull_request' uses: actions/upload-artifact@v4 with: - name: pr-comment-coverage + name: pr-comment-gen path: pr-comment/ - - # Skips validate-cpp (xmllint) and probe-cpp; those run Linux-only in - # linux-core. No gen/audit gates here either. - macos: - name: "core + api: corert + unit + mxtest + round-trip (macOS/AppleClang)" - runs-on: macos-latest - - # No Docker on the macOS runners: MX_RUNNING_IN_DOCKER=1 flips the - # Makefile to its direct-invocation branch so the build runs natively - # with AppleClang. This is the only C++ job off the pinned toolchain, - # by design: it exists to find (and keep finding) AppleClang fallout. - env: - MX_RUNNING_IN_DOCKER: 1 - - steps: - - uses: actions/checkout@v4 - - - name: Cache build - uses: actions/cache@v4 - with: - path: build/ - key: ${{ runner.os }}-core-dev-${{ hashFiles('CMakeLists.txt', 'src/**') }} - restore-keys: | - ${{ runner.os }}-core-dev- - - - name: corert round-trip suite - run: make test-core-dev - - - name: mx::core unit tests - run: make test-cpp-unit - - - name: Build mx library (api/impl) - run: make lib - - - name: mxtest suite (api/impl/file/control) + examples - run: make test - - - name: api corpus round-trip (regression mode) - run: make test-api-roundtrip - - swift-local: - name: "swift: Mx package build only (macOS/SPM)" - runs-on: macos-latest - - # By default mx builds `Mx` as a source library straight from this checkout -- - # the path komp takes when it depends on a sibling mx via `.package(path:)`. - # Building it here proves mx compiles as a Swift package under AppleClang/SPM on - # every PR; it runs no test suites. The binary xcframework release arm - # (MX_BINARY_RELEASE) is follow-up work. - - steps: - - uses: actions/checkout@v4 - - - name: Swift toolchain - run: swift --version - - - name: Build the Mx Swift package (local-checkout source mode) - run: swift build --product Mx + if-no-files-found: ignore diff --git a/.github/workflows/coverage.yaml b/.github/workflows/coverage.yaml new file mode 100644 index 000000000..066bb1cc0 --- /dev/null +++ b/.github/workflows/coverage.yaml @@ -0,0 +1,181 @@ +name: Coverage + +# Manual-only coverage. The instrumented core + api builds are expensive, so +# they never run on push or PR open -- only when explicitly asked for: +# * the Actions "Run workflow" button (workflow_dispatch), or +# * a `/coverage` comment on a PR by a repo owner/member/collaborator. +# Both paths build core AND api coverage. The comment path posts the report +# back to the PR: unlike ci.yaml, an issue_comment run executes in the +# base-repo context with a write token, so it can comment directly -- no +# pr-comment.yaml delegation needed. +# +# NOTE: issue_comment triggers only fire for the workflow file on the default +# branch, so `/coverage` starts working once this file is merged to main. +on: + workflow_dispatch: + issue_comment: + types: [created] + +permissions: + contents: read + pull-requests: write + +concurrency: + group: coverage-${{ github.event.issue.number || github.ref }} + cancel-in-progress: true + +env: + DOCKER_VOLUME: mx-build + +jobs: + coverage: + runs-on: ubuntu-latest + + # Run on the manual button, or on a `/coverage` comment made on a PR (not a + # plain issue) by someone with a write-ish association. Anything else -- a + # comment on an issue, a different command, a drive-by commenter -- is + # ignored. + if: >- + github.event_name == 'workflow_dispatch' || + (github.event.issue.pull_request && + startsWith(github.event.comment.body, '/coverage') && + contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association)) + + steps: + # Acknowledge the command so the requester knows it was accepted. + - name: Acknowledge command + if: github.event_name == 'issue_comment' + uses: actions/github-script@v8 + with: + script: | + // Cosmetic ack only -- never fail the run if the reaction can't post. + try { + await github.rest.reactions.createForIssueComment({ + owner: context.repo.owner, + repo: context.repo.repo, + comment_id: context.payload.comment.id, + content: 'eyes', + }); + } catch (e) { + core.info(`Could not add reaction: ${e.message}`); + } + + # workflow_dispatch checks out the triggering ref directly. The comment + # path carries no PR ref in the event, so resolve the PR head SHA first. + - name: Resolve PR head + id: pr + if: github.event_name == 'issue_comment' + uses: actions/github-script@v8 + with: + script: | + const pr = await github.rest.pulls.get({ + owner: context.repo.owner, + repo: context.repo.repo, + pull_number: context.issue.number, + }); + core.setOutput('sha', pr.data.head.sha); + + - uses: actions/checkout@v5 + with: + ref: ${{ steps.pr.outputs.sha || github.sha }} + + # Same container-driver + runtime-token + bind-backed volume + ccache + # setup as ci.yaml's Linux jobs so the instrumented builds reuse the warm + # toolchain layer cache. + - uses: docker/setup-buildx-action@v3 + - uses: crazy-max/ghaction-github-runtime@v3 + + - name: Create bind-backed build volume + run: | + mkdir -p build/docker + docker volume create --driver local \ + --opt type=none --opt o=bind \ + --opt device="$GITHUB_WORKSPACE/build/docker" mx-build + + - name: Cache ccache + uses: actions/cache@v4 + with: + path: build/docker/.ccache + key: ccache-${{ runner.os }}-coverage-${{ github.sha }} + restore-keys: ccache-${{ runner.os }}-coverage- + + - name: core coverage (corert + unit, instrumented) + run: make core-coverage + + - name: api coverage (mxtest + round-trip, instrumented) + run: make api-coverage + + # Turn gcovr's 3-line summaries (data/testOutput/coverage[/api]/summary.txt, + # e.g. "lines: 52.1% (17624 out of 33846)") into Markdown tables for the + # job summary and the PR comment. + - name: Build coverage tables + id: tables + if: always() + run: | + make_table() { + local file="$1" + echo '| Metric | Coverage | Covered / Total |' + echo '|---|---:|---:|' + awk '/^(lines|functions|branches):/ { + m=$1; sub(":","",m); + c=$3; sub("[(]","",c); + t=$6; sub("[)]","",t); + M=toupper(substr(m,1,1)) substr(m,2); + printf "| %s | %s | %s / %s |\n", M, $2, c, t + }' "$file" + } + { + echo '### Core-dev coverage `src/private/mx/core/`' + echo '' + make_table data/testOutput/coverage/summary.txt 2>/dev/null || echo '(core coverage not produced)' + echo '' + echo '### API coverage `src/private/mx/{api,impl,utility}/`' + echo '' + make_table data/testOutput/coverage/api/summary.txt 2>/dev/null || echo '(api coverage not produced)' + } > tables.md + cat tables.md >> "$GITHUB_STEP_SUMMARY" + + # Single self-contained page each, stored (compression-level 0). GitHub + # delivers artifacts as a .zip -- there is no raw-file download -- so each + # zip just holds one index.html to unzip and open. + - name: Upload core HTML report + id: upload-core + if: always() + uses: actions/upload-artifact@v4 + with: + name: coverage-core-dev + path: data/testOutput/coverage/index.html + compression-level: 0 + + - name: Upload api HTML report + id: upload-api + if: always() + uses: actions/upload-artifact@v4 + with: + name: coverage-api + path: data/testOutput/coverage/api/index.html + compression-level: 0 + + # Post the report to the PR. issue_comment runs in the base-repo context + # with a write token, so this comments directly. always() so a test + # failure mid-build still reports whatever coverage was produced. + - name: Post coverage comment + if: always() && github.event_name == 'issue_comment' + env: + GH_TOKEN: ${{ github.token }} + REPO: ${{ github.repository }} + PR: ${{ github.event.issue.number }} + SHA: ${{ steps.pr.outputs.sha }} + CORE_URL: ${{ steps.upload-core.outputs.artifact-url }} + API_URL: ${{ steps.upload-api.outputs.artifact-url }} + run: | + { + echo '### Coverage report' + echo '' + cat tables.md + echo '' + echo "[Core HTML report]($CORE_URL) | [API HTML report]($API_URL)" + echo '' + echo "Commit \`$SHA\`." + } > comment.md + gh pr comment "$PR" --repo "$REPO" --body-file comment.md diff --git a/AGENTS.md b/AGENTS.md index 598084c8e..fa29a1c27 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -5,7 +5,7 @@ `mx` is a MusicXML C++ library. The product surface is `mx::api`, a simplified and narrowed interface of MusicXML backed by the strongly-typed `mx::core` model. -Quickstart: `make test-core-dev` (fast corert gate) - `make fmt` (format) - `make help` (all +Quickstart: `make core-roundtrip-test` (fast corert gate) - `make fmt` (format) - `make help` (all targets). Everything runs in the `mx-sdk` Docker image unless `MX_RUNNING_IN_DOCKER=1`. ## Repository layout @@ -17,7 +17,7 @@ mx/ Makefile <- top-level build driver Dockerfile <- mx-sdk image with toolchains and dev tools CMakeLists.txt <- C++ project - Package.swift <- Swift Package Manager wrapper (built by the swift-local CI job) + Package.swift <- Swift Package Manager wrapper (built by the swift CI job) data/ <- MusicXML test corpus (large, see data/README.md) docs/ai/design/ <- design docs docs/musicxml-4.0-ed15c23.xsd <- the current musicxml.xsd schema @@ -34,8 +34,6 @@ mx/ mxtest/core/ <- mx::core unit tests (small) mxtest/corert/ <- C++ core roundtrip test (deserializes and serializes the test corpus) mxtest/import/ <- normalization support (Normalize.cpp/h + DecimalFields.h) shared by corert and api tests - mxtest/probe/ <- compile-time negative probes: constructs that MUST NOT compile (`make probe-cpp`) - mxtest/validate/ <- xmllint validation driver: parses+serializes corpus, outputs for `make validate-cpp` mxtest/api/ <- tests for the mx::api layer mxtest/impl/ <- tests for the mx::impl layer mxtest/file/ <- file-walking test infrastructure for the mx::api tests (MxFile, MxFileTest, etc.) @@ -64,7 +62,7 @@ target list. Extend the `mx-sdk` image when new tools are needed. ## The corert (core roundtrip) test -Runs with `make test-core-dev`. Deserializes and reserializes the test corpus with `mx::core`. It +Runs with `make core-roundtrip-test`. Deserializes and reserializes the test corpus with `mx::core`. It compiles without `mx::impl`/`mx::api`, so you can iterate on `gen/cpp` and the generated core and defer fixing the `mx::impl` integration until you are ready. @@ -136,15 +134,17 @@ Before designing or changing anything in `mx::api`, read the `mx-api-doctrine` s ## Quality gates -Run `make fmt` to format. `make check` is the clang-format gate **only** — it builds and tests nothing. -Run `make test-core-dev` for corert (especially `mx/core` work) and `make check-core-dev` (fmt-check -+ warning-free build) before pushing core work. `make test` runs the api/impl suite + examples; -corert, `make test-api-roundtrip`, and the gen/audit gates run separately (see `make help` / CI). -Adding/removing a `data/` file: bump the pinned count in `CoreRoundtripTest.cpp`, run `make audit` (regenerates `corpus.xml` + `*.features.xml`), confirm round-trip via `make test-core-dev`. -`ApiLoadSmokeTest` proves a file imports without crashing, not that the data is correct; the read→write→read gate (`make test-api-roundtrip` / `roundtrip-baseline.txt`) is the correctness check — pin a fixture there to defend a feature. +Run `make fmt` to format. `make fmt-check` is the clang-format gate **only** — it builds and tests nothing. +Run `make core-roundtrip-test` for corert (especially `mx/core` work) before pushing core work. +`make api-test` runs the api/impl suite + examples; `make api-roundtrip` and the gen/audit gates run +separately. `make test-all` runs every C++ suite at once (core roundtrip + unit + api-test + +api-roundtrip) — the deep gate CI runs on Linux (see `make help` / CI). +Adding/removing a `data/` file: bump the pinned count in `CoreRoundtripTest.cpp`, run `make audit` (regenerates `corpus.xml` + `*.features.xml`), confirm round-trip via `make core-roundtrip-test`. +`ApiLoadSmokeTest` proves a file imports without crashing, not that the data is correct; the read→write→read gate (`make api-roundtrip` / `roundtrip-baseline.txt`) is the correctness check — pin a fixture there to defend a feature. Look at what will run in CI `.github/workflows/ci.yaml` and anticipate issues there when coding -locally. +locally. Code coverage is not part of the normal CI run; trigger it on demand with a `/coverage` +comment on a PR or the Coverage workflow's "Run workflow" button (`.github/workflows/coverage.yaml`). ## Key files to understand diff --git a/CMakeLists.txt b/CMakeLists.txt index cbd5f3e99..08a3cf2e9 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -3,7 +3,7 @@ project(mx CXX) # The mx::core C++ typed model: pugixml + the generated src/private/mx/core # (emitted by `python3 -m gen gen/cpp/config.toml`, committed) + its -# hand-written runtime, plus the corert/unit/validate test binaries. +# hand-written runtime, plus the corert/unit test binaries. # See AGENTS.md and docs/ai/design/mx-core-plan.md. set(CMAKE_CXX_STANDARD 20) @@ -135,22 +135,6 @@ if(MX_CORE_DEV) target_include_directories(mxtest-core-dev PRIVATE ${PRIVATE_DIR}) target_link_libraries(mxtest-core-dev mx_core ${CMAKE_THREAD_LIBS_INIT}) - # mxtest-validate: parses corpus files and serializes them for xmllint validation. - # Parses every eligible corpus file, serializes the OUTPUT to - # build/validate-out/, plus the default-constructed Document; `make - # validate-cpp` then xmllint-validates every output against the 4.0 XSD. - add_executable(mxtest-validate - ${PRIVATE_DIR}/mxtest/validate/main.cpp - ${PRIVATE_DIR}/mxtest/corert/Compare.cpp - ${PRIVATE_DIR}/mxtest/corert/Compare.h - ${PRIVATE_DIR}/mxtest/corert/CoreRoundtripImpl.cpp - ${PRIVATE_DIR}/mxtest/corert/CoreRoundtripImpl.h - ${PRIVATE_DIR}/mxtest/corert/Fixer.cpp - ${PRIVATE_DIR}/mxtest/corert/Fixer.h - ${SRC_CORERT_HELPERS}) - target_include_directories(mxtest-validate PRIVATE ${PRIVATE_DIR}) - target_link_libraries(mxtest-validate mx_core) - endif() diff --git a/Makefile b/Makefile index 02b7c618f..1b0fb222c 100644 --- a/Makefile +++ b/Makefile @@ -65,44 +65,42 @@ FIND_CPP := find src \ -type f \( -name '*.cpp' -o -name '*.h' -o -name '*.hpp' \) -print .DEFAULT_GOAL := help -.PHONY: help sdk fmt check core-dev check-core-dev test-core-dev test-cpp-unit \ - validate-cpp probe-cpp coverage-core-dev test-gen test-audit gen-check \ - gen-quality gen-lint \ +.PHONY: help sdk fmt fmt-check test-all \ + core-build core-roundtrip-test core-unit core-coverage \ + api-lib api-build api-test api-examples api-roundtrip \ + api-roundtrip-discover api-roundtrip-dump api-roundtrip-classify api-coverage \ + gen-test audit-test gen-quality gen-lint \ gen gen-cpp gen-go gen-c gen-schema \ audit audit-force \ build-go build-c test-go test-c \ - lib dev test run-examples test-api-roundtrip discover-api-roundtrip \ - dump-api-roundtrip classify-api-roundtrip coverage-api \ clean clean-docker check-docker docker-volume help: @echo 'mx targets (see AGENTS.md). All run via the mx-sdk Docker toolchain;' @echo 'set MX_RUNNING_IN_DOCKER=1 to run natively on the host instead.' @echo '' - @echo ' mx::api/mx::impl:' - @echo ' make lib Build the mx static library (MX_API=ON).' - @echo ' make dev Build mx + mxtest + examples + api-roundtrip binary.' - @echo ' make test Run examples + mxtest suite (api/impl/file/control).' - @echo ' make run-examples Build and run all three api example programs.' - @echo ' make test-api-roundtrip Run the corpus api roundtrip in regression mode (CI gate).' - @echo ' make discover-api-roundtrip Run discovery mode over the full corpus (manual only).' - @echo ' make dump-api-roundtrip Dump normalized expected/actual XML for failures.' - @echo ' make classify-api-roundtrip Classify dumped failures by root cause (Python).' - @echo ' make coverage-api Instrumented api/impl/utility build + gcovr report.' + @echo ' Everything C++ (the default deep gate, run in CI on Linux):' + @echo ' make test-all core-roundtrip-test + core-unit + api-test + api-roundtrip.' @echo '' - @echo ' C++ core:' - @echo ' make core-dev Build mx_core and the corert/unit/validate binaries.' - @echo " make test-core-dev Run the core roundtrip suite. Filter: ARGS='[core-roundtrip] lysuite/*'" - @echo ' make test-cpp-unit Run the mx::core unit tests (values, shapes, rejection suite).' - @echo ' make validate-cpp Serialize every corpus file and xmllint-validate the output.' - @echo ' make probe-cpp Compile-time negative probes (invalid construction must not compile).' - @echo ' make check-core-dev fmt-check + warning-free core-dev build.' - @echo ' make coverage-core-dev Instrumented build, corert + unit suites, gcovr -> $(COV_DIR)/.' + @echo ' Product surface (mx::api / mx::impl):' + @echo ' make api-lib Build the mx static library (MX_API=ON).' + @echo ' make api-build Build mx + mxtest + examples + api-roundtrip binary.' + @echo ' make api-test Run examples + mxtest suite (api/impl/file/control).' + @echo ' make api-examples Build and run all three api example programs.' + @echo ' make api-roundtrip Run the corpus api roundtrip in regression mode (CI gate).' + @echo ' make api-roundtrip-discover Run discovery mode over the full corpus (manual only).' + @echo ' make api-roundtrip-dump Dump normalized expected/actual XML for failures.' + @echo ' make api-roundtrip-classify Classify dumped failures by root cause (Python).' + @echo ' make api-coverage Instrumented api/impl/utility build + gcovr report.' @echo '' - @echo ' Generator:' - @echo ' make test-gen Run the generator (parser + IR + plates + press) Python tests.' - @echo ' make test-audit Run the audit tool Python tests (incl. failure classifier).' - @echo ' make gen-check plates --check for every target (renames, collisions).' + @echo ' Core substrate (mx::core):' + @echo ' make core-build Build mx_core and the corert + unit test binaries.' + @echo " make core-roundtrip-test Run the core roundtrip suite. Filter: ARGS='[core-roundtrip] lysuite/*'" + @echo ' make core-unit Run the mx::core unit tests (values, shapes, rejection suite).' + @echo ' make core-coverage Instrumented build, corert + unit suites, gcovr -> $(COV_DIR)/.' + @echo '' + @echo ' Generator (gen/):' + @echo ' make gen-test Run the generator (parser + IR + plates + press) Python tests.' @echo ' make gen Run the generator for every target (cpp/go/c/schema).' @echo ' make gen-cpp Run the generator for the C++ target (src/private/mx/core/generated).' @echo ' make gen-go Run the generator for the Go target.' @@ -111,21 +109,22 @@ help: @echo ' make gen-quality Score gen/ design quality; fail below the floor.' @echo ' make gen-lint Lint gen/ with pylint; fail below the floor.' @echo '' - @echo ' Feature audit:' + @echo ' Feature audit (audit/):' @echo ' make audit Write missing *.features.xml sidecars + rebuild data/corpus.xml.' @echo ' make audit-force Rewrite every *.features.xml sidecar (use after a format change).' + @echo ' make audit-test Run the audit tool Python tests (incl. failure classifier).' @echo '' - @echo ' Go test target:' + @echo ' Generated Go target:' @echo ' make build-go Build Go corert tests.' @echo ' make test-go Run Go corert tests.' @echo '' - @echo ' C test target:' + @echo ' Generated C target:' @echo ' make build-c Build C corert test binary.' @echo ' make test-c Run C corert tests.' @echo '' @echo ' Housekeeping:' @echo ' make fmt Format C++ under src/ via mx-sdk.' - @echo ' make check fmt-check via mx-sdk.' + @echo ' make fmt-check clang-format gate (checks formatting; builds nothing).' @echo ' make sdk Build the mx-sdk Docker toolchain image.' @echo ' make clean Remove the build/ tree and gen build artifacts.' @echo ' make clean-docker Remove the sdk image and build volume.' @@ -154,56 +153,61 @@ ifdef MX_RUNNING_IN_DOCKER # ----- Inside the container (or native via MX_RUNNING_IN_DOCKER=1) ---------- -lib: +# All C++ suites: the core substrate (corert + unit) and the api product +# surface (mxtest + examples + corpus round-trip). The deep gate; CI runs it +# on Linux. gen/go/c and the audit tool are separate (see their targets). +test-all: core-roundtrip-test core-unit api-test api-roundtrip + +api-lib: $(CMAKE) -S . -B $(BUILD_ROOT)/api -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DMX_API=on $(CMAKE) --build $(BUILD_ROOT)/api --target mx --parallel $(JOBS) -dev: lib +api-build: api-lib $(CMAKE) --build $(BUILD_ROOT)/api --parallel $(JOBS) -test: run-examples +api-test: api-examples $(BUILD_ROOT)/api/mxtest $(ARGS) -run-examples: dev +api-examples: api-build $(BUILD_ROOT)/api/mxread $(BUILD_ROOT)/api/mxwrite $(BUILD_ROOT)/api/example.musicxml $(BUILD_ROOT)/api/mxhide - @echo 'run-examples: all three examples ran successfully.' + @echo 'api-examples: all three examples ran successfully.' # Corpus api roundtrip -- regression mode (CI gate): every file in the # pinned baseline must pass; exit non-zero if any fails. -test-api-roundtrip: dev +api-roundtrip: api-build $(BUILD_ROOT)/api/mxtest-api-roundtrip regression \ $(CURDIR)/data \ $(CURDIR)/src/private/mxtest/api/roundtrip-baseline.txt # Discovery mode: walk the full corpus and report pass/fail. # Never a CI gate; use to grow the pinned list (manual commits only). -discover-api-roundtrip: dev +api-roundtrip-discover: api-build $(BUILD_ROOT)/api/mxtest-api-roundtrip discovery $(CURDIR)/data # Dump normalized expected/actual XML for every failing api round-trip. # Output goes to build/api/roundtrip-dump/ (build dir, already gitignored). -# Feeds the classifier: make dump-api-roundtrip && make classify-api-roundtrip -dump-api-roundtrip: dev +# Feeds the classifier: make api-roundtrip-dump && make api-roundtrip-classify +api-roundtrip-dump: api-build rm -rf $(BUILD_ROOT)/api/roundtrip-dump mkdir -p $(BUILD_ROOT)/api/roundtrip-dump $(BUILD_ROOT)/api/mxtest-api-roundtrip discovery $(CURDIR)/data \ --dump $(CURDIR)/$(BUILD_ROOT)/api/roundtrip-dump # Classify api round-trip failures by root cause. -# Reads the dump produced by dump-api-roundtrip; writes build/api/classified.json. +# Reads the dump produced by api-roundtrip-dump; writes build/api/classified.json. # Fast (pure Python); kept separate from the slow dump step so classification # logic can be re-run without re-dumping. Pass DUMP_DIR=path to override. DUMP_DIR ?= $(BUILD_ROOT)/api/roundtrip-dump -classify-api-roundtrip: +api-roundtrip-classify: python3 -m audit classify $(DUMP_DIR) \ --data $(CURDIR)/data \ --out $(BUILD_ROOT)/api/classified.json # Instrumented api coverage: build mx+mxtest with --coverage, run all # suites, produce gcovr report for src/private/mx/{api,impl,utility}/. -coverage-api: +api-coverage: @echo "=== build (api, instrumented) ===" $(CMAKE) -S . -B $(BUILD_ROOT)/cov-api \ -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) \ @@ -229,75 +233,24 @@ coverage-api: --xml $(COV_DIR)/api/coverage.xml \ --html-self-contained --html $(COV_DIR)/api/index.html \ $(BUILD_ROOT)/cov-api | tee $(COV_DIR)/api/summary.txt - @echo "=== coverage-api written to $(COV_DIR)/api/ ===" + @echo "=== api-coverage written to $(COV_DIR)/api/ ===" -core-dev: +core-build: $(CMAKE) -S . -B $(BUILD_ROOT)/core-dev -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DMX_CORE_DEV=on $(CMAKE) --build $(BUILD_ROOT)/core-dev --parallel $(JOBS) -test-core-dev: core-dev +core-roundtrip-test: core-build $(BUILD_ROOT)/core-dev/mxtest-core-dev --allow-running-no-tests $(ARGS) -test-cpp-unit: core-dev +core-unit: core-build $(BUILD_ROOT)/core-dev/mxtest-core $(ARGS) -# Serialize every parsed corpus document and xmllint-validate the output -# against the MusicXML 4.0 XSD -- mechanical proof that import leniency -# (value clamping) still emits only schema-valid XML. -validate-cpp: core-dev - rm -rf $(BUILD_ROOT)/validate-out - $(BUILD_ROOT)/core-dev/mxtest-validate $(BUILD_ROOT)/validate-out - cd $(BUILD_ROOT)/validate-out && xmllint --noout --nonet \ - --schema $(CURDIR)/docs/musicxml-4.0-ed15c23.xsd *.xml 2>&1 \ - | grep -v 'validates$$' | grep -v 'Skipping import' || true - @cd $(BUILD_ROOT)/validate-out && xmllint --noout --nonet \ - --schema $(CURDIR)/docs/musicxml-4.0-ed15c23.xsd *.xml 2>/dev/null \ - && echo 'validate-cpp: all outputs are schema-valid.' - -# Compile-time negative probes: PROBE=0 must compile (the control); -# every numbered probe is an invalid-construction attempt that must NOT -# compile. -PROBE_COUNT := 7 -probe-cpp: - @$(CXX) -std=c++20 -fsyntax-only -I src/private -DPROBE=0 \ - src/private/mxtest/probe/Probes.cpp \ - || { echo 'probe-cpp: control failed to compile'; exit 1; } - @for i in $$(seq 1 $(PROBE_COUNT)); do \ - if $(CXX) -std=c++20 -fsyntax-only -I src/private -DPROBE=$$i \ - src/private/mxtest/probe/Probes.cpp 2>/dev/null; then \ - echo "probe-cpp: PROBE=$$i compiled but must not"; exit 1; \ - fi; \ - done - @echo 'probe-cpp: control compiles; all $(PROBE_COUNT) must-not-compile probes fail as required.' - -# fmt-check + warning-free core-dev build (a fresh tree so stale objects -# cannot mask a warning; same shape as the old master gate, new subject). -check-core-dev: - @echo "=== fmt-check ===" - @$(FIND_CPP) | xargs -r clang-format --dry-run --Werror - @echo "=== build (warning-free, core-dev) ===" - @$(CMAKE) -S . -B $(BUILD_ROOT)/check-core-dev \ - -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) -DMX_CORE_DEV=on \ - > $(BUILD_ROOT)/check-build.log 2>&1 \ - || { cat $(BUILD_ROOT)/check-build.log; \ - echo "ERROR: cmake configure failed (see above)"; exit 1; } - @$(CMAKE) --build $(BUILD_ROOT)/check-core-dev --parallel $(JOBS) \ - >> $(BUILD_ROOT)/check-build.log 2>&1; status=$$?; \ - cat $(BUILD_ROOT)/check-build.log; \ - if [ $$status -ne 0 ]; then \ - echo "ERROR: build failed (see above)"; exit $$status; \ - fi; \ - if grep -q 'warning:' $(BUILD_ROOT)/check-build.log; then \ - echo "ERROR: build emitted warnings (see above)"; exit 1; \ - fi - @echo "=== check-core-dev passed ===" - -# Instrumented core-dev coverage: build with --coverage (off ccache so the +# Instrumented core coverage: build with --coverage (off ccache so the # notes/data stay exact), run the corert AND unit suites, then gcovr. The # report lands in $(COV_DIR) on the bind-mounted workspace. Filtered to # src/private/mx/core/ -- the generated model and its runtime. -coverage-core-dev: - @echo "=== build (core-dev, instrumented) ===" +core-coverage: + @echo "=== build (core, instrumented) ===" $(CMAKE) -S . -B $(BUILD_ROOT)/cov-core-dev \ -DCMAKE_BUILD_TYPE=$(BUILD_TYPE) \ -DCMAKE_C_COMPILER_LAUNCHER= \ @@ -320,21 +273,13 @@ coverage-core-dev: $(BUILD_ROOT)/cov-core-dev | tee $(COV_DIR)/summary.txt @echo "=== coverage written to $(COV_DIR)/ ===" -test-gen: +gen-test: python3 -m unittest discover -s gen/tests -t . $(ARGS) # Audit tool Python tests (feature-audit + the round-trip failure classifier). -test-audit: +audit-test: python3 -m unittest discover -s audit/tests -t . $(ARGS) -# plates --check for every target: validates renames and detects identifier -# collisions (a CI gate, like test-gen). -gen-check: - python3 -m gen plates --config gen/cpp/config.toml --check - python3 -m gen plates --config gen/test/go/config.toml --check - python3 -m gen plates --config gen/test/c/config.toml --check - python3 -m gen plates --config gen/schema/config.toml --check - # Static analysis of the gen/ generator. quality.py measures and writes the # report tree to the workspace mount; the floor check below is the gate. gen-quality: @@ -359,7 +304,7 @@ fmt: @$(FIND_CPP) | xargs -r clang-format -i @echo "Formatted C++ under src/." -check: +fmt-check: @$(FIND_CPP) | xargs -r clang-format --dry-run --Werror @echo "fmt-check passed." @@ -416,73 +361,64 @@ docker-volume: | check-docker @$(DOCKER) volume inspect $(DOCKER_VOLUME) >/dev/null 2>&1 \ || $(DOCKER) volume create $(DOCKER_VOLUME) >/dev/null -lib: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make lib BUILD_TYPE=$(BUILD_TYPE) +test-all: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make test-all BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' + +api-lib: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-lib BUILD_TYPE=$(BUILD_TYPE) -dev: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make dev BUILD_TYPE=$(BUILD_TYPE) +api-build: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-build BUILD_TYPE=$(BUILD_TYPE) -test: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make test BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' +api-test: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-test BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' -run-examples: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make run-examples BUILD_TYPE=$(BUILD_TYPE) +api-examples: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-examples BUILD_TYPE=$(BUILD_TYPE) -test-api-roundtrip: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make test-api-roundtrip BUILD_TYPE=$(BUILD_TYPE) +api-roundtrip: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-roundtrip BUILD_TYPE=$(BUILD_TYPE) -discover-api-roundtrip: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make discover-api-roundtrip BUILD_TYPE=$(BUILD_TYPE) +api-roundtrip-discover: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-roundtrip-discover BUILD_TYPE=$(BUILD_TYPE) -dump-api-roundtrip: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make dump-api-roundtrip BUILD_TYPE=$(BUILD_TYPE) +api-roundtrip-dump: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-roundtrip-dump BUILD_TYPE=$(BUILD_TYPE) @rm -rf $(BUILD_ROOT)/api/roundtrip-dump @mkdir -p $(BUILD_ROOT)/api/roundtrip-dump $(DOCKER) run --rm -v $(DOCKER_VOLUME):/vol:ro -v $(CURDIR)/$(BUILD_ROOT)/api/roundtrip-dump:/out \ alpine sh -c 'cp -a /vol/api/roundtrip-dump/. /out/' -classify-api-roundtrip: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make classify-api-roundtrip BUILD_TYPE=$(BUILD_TYPE) +api-roundtrip-classify: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make api-roundtrip-classify BUILD_TYPE=$(BUILD_TYPE) @mkdir -p $(BUILD_ROOT)/api $(DOCKER) run --rm -v $(DOCKER_VOLUME):/vol:ro -v $(CURDIR)/$(BUILD_ROOT)/api:/out \ alpine cp /vol/api/classified.json /out/classified.json -coverage-api: $(DOCKER_STAMP) docker-volume +api-coverage: $(DOCKER_STAMP) docker-volume @rm -rf $(COV_DIR)/api - $(DOCKER_RUN) make coverage-api BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' + $(DOCKER_RUN) make api-coverage BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' @echo "API coverage written to $(COV_DIR)/api/ (open $(COV_DIR)/api/index.html)" -core-dev: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make core-dev BUILD_TYPE=$(BUILD_TYPE) +core-build: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make core-build BUILD_TYPE=$(BUILD_TYPE) -test-core-dev: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make test-core-dev BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' +core-roundtrip-test: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make core-roundtrip-test BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' -test-cpp-unit: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make test-cpp-unit BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' +core-unit: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make core-unit BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' -validate-cpp: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make validate-cpp BUILD_TYPE=$(BUILD_TYPE) - -probe-cpp: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make probe-cpp - -check-core-dev: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make check-core-dev BUILD_TYPE=$(BUILD_TYPE) - -coverage-core-dev: $(DOCKER_STAMP) docker-volume +core-coverage: $(DOCKER_STAMP) docker-volume @rm -rf $(COV_DIR) - $(DOCKER_RUN) make coverage-core-dev BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' + $(DOCKER_RUN) make core-coverage BUILD_TYPE=$(BUILD_TYPE) ARGS='$(ARGS)' @echo "Coverage written to $(COV_DIR)/ (open $(COV_DIR)/index.html)" -test-gen: $(DOCKER_STAMP) - $(DOCKER_RUN) make test-gen ARGS='$(ARGS)' - -test-audit: $(DOCKER_STAMP) - $(DOCKER_RUN) make test-audit ARGS='$(ARGS)' +gen-test: $(DOCKER_STAMP) + $(DOCKER_RUN) make gen-test ARGS='$(ARGS)' -gen-check: $(DOCKER_STAMP) - $(DOCKER_RUN) make gen-check +audit-test: $(DOCKER_STAMP) + $(DOCKER_RUN) make audit-test ARGS='$(ARGS)' gen-quality: $(DOCKER_STAMP) @rm -rf $(QUALITY_DIR) @@ -494,8 +430,8 @@ gen-lint: $(DOCKER_STAMP) fmt: $(DOCKER_STAMP) docker-volume $(DOCKER_RUN) make fmt -check: $(DOCKER_STAMP) docker-volume - $(DOCKER_RUN) make check +fmt-check: $(DOCKER_STAMP) docker-volume + $(DOCKER_RUN) make fmt-check gen: $(DOCKER_STAMP) docker-volume $(DOCKER_RUN) make gen diff --git a/README.md b/README.md index 526df7702..b03d948b9 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Building and running the core roundtrip suite should be as simple as: ``` git clone https://github.com/webern/mx.git mx cd mx -make test-core-dev +make core-roundtrip-test ``` Run `make` (or `make help`) to list every target. @@ -45,20 +45,21 @@ and `MX_COVERAGE` (gcov instrumentation). The useful workflows are exposed as Ma | Target | What it does | |-----------------------------|--------------------------------------------------------------------| -| `make lib` | build the `mx` static library (`mx::api` + `mx::impl`) | -| `make dev` | build `mx` + mxtest + example + api-roundtrip binaries | -| `make test` | run api examples, then the `mxtest` suite (api/impl/file/control) | -| `make test-api-roundtrip` | corpus api roundtrip in regression mode (CI gate) | -| `make run-examples` | build and run all three api example programs | -| `make core-dev` | build `mx_core` and the corert/unit/validate test binaries | -| `make test-core-dev` | run the core roundtrip suite over the `data/` corpus | -| `make test-cpp-unit` | run the `mx::core` unit tests (values, shapes, rejection suite) | -| `make validate-cpp` | serialize every corpus file and `xmllint`-validate the output | -| `make probe-cpp` | compile-time probes: invalid constructions must NOT compile | +| `make test-all` | every C++ suite: core roundtrip + unit + api-test + api-roundtrip | +| `make api-lib` | build the `mx` static library (`mx::api` + `mx::impl`) | +| `make api-build` | build `mx` + mxtest + example + api-roundtrip binaries | +| `make api-test` | run api examples, then the `mxtest` suite (api/impl/file/control) | +| `make api-roundtrip` | corpus api roundtrip in regression mode (CI gate) | +| `make api-examples` | build and run all three api example programs | +| `make core-build` | build `mx_core` and the corert + unit test binaries | +| `make core-roundtrip-test` | run the core roundtrip suite over the `data/` corpus | +| `make core-unit` | run the `mx::core` unit tests (values, shapes, rejection suite) | +| `make core-coverage` | instrumented core build + gcovr report (manual; also `/coverage`) | +| `make api-coverage` | instrumented api build + gcovr report (manual; also `/coverage`) | `make clean` removes the build tree. Knobs: `JOBS` (parallelism, auto-detected), `BUILD_TYPE` (default `Debug`), and `ARGS` (forwarded to test binaries, e.g. -`make test-core-dev ARGS='[core-roundtrip] lysuite/*'`). +`make core-roundtrip-test ARGS='[core-roundtrip] lysuite/*'`). ### Build Tenets @@ -82,9 +83,9 @@ value to a valid one. For example, in test files I have discovered many cases wh So in this case, `mx` will load the file and replace -1.11 with 0. Unfortunately this is silent for now, but we may surface a message system to let the caller know that this has happened. -The `make validate-cpp` gate is the permanent mechanical proof of this tenet: every parsed corpus -document is serialized and the *output* is validated against the MusicXML 4.0 XSD, so clamp -leniency on import can only ever emit schema-valid XML. +This clamp-on-import behavior is exercised by the core roundtrip suite (`make core-roundtrip-test`), +which parses and reserializes every eligible corpus file and compares the result against a +normalized form of the input. ### Using `mx` in a Cmake Project diff --git a/audit/README.md b/audit/README.md index df0e1e34e..f5365a6de 100644 --- a/audit/README.md +++ b/audit/README.md @@ -40,13 +40,13 @@ common case (a new corpus file was added) only writes the new sidecar. Use ## Classifying api round-trip failures ``` -make dump-api-roundtrip # C++: write normalized expected/actual XML pairs -make classify-api-roundtrip # Python: diff each pair, rank a worklist +make api-roundtrip-dump # C++: write normalized expected/actual XML pairs +make api-roundtrip-classify # Python: diff each pair, rank a worklist python3 -m audit classify [--data DIR] [--out FILE] ``` -`classify` reads the dump directory produced by `make dump-api-roundtrip` +`classify` reads the dump directory produced by `make api-roundtrip-dump` (`build/api/roundtrip-dump/`) and diffs each expected/actual pair structurally. Drops/adds come from an order-free element **multiset** (`Counter(expected) - Counter(actual)`); value/attribute/reorder differences come @@ -67,7 +67,7 @@ re-dumping. See `docs/ai/design/api-roundtrip-classifier.md`. ## Tests ``` -make test-audit # python3 -m unittest discover -s audit/tests -t . +make audit-test # python3 -m unittest discover -s audit/tests -t . ``` ## Audited set diff --git a/audit/classify.py b/audit/classify.py index f802b7be2..f667fb723 100644 --- a/audit/classify.py +++ b/audit/classify.py @@ -1,6 +1,6 @@ """Classify api round-trip failures by *measured* divergence, and rank a worklist. -Reads the dump directory produced by ``make dump-api-roundtrip`` (Phase 1, +Reads the dump directory produced by ``make api-roundtrip-dump`` (Phase 1, issue #210): pairs of ``.expected.xml`` / ``.actual.xml`` plus a ``.status`` sidecar for crashes. Diffs each pair structurally, records the set of divergences that keep the file from round-tripping, and writes a diff --git a/data/testOutput/.gitignore b/data/testOutput/.gitignore index f515f8e20..78e93d30f 100644 --- a/data/testOutput/.gitignore +++ b/data/testOutput/.gitignore @@ -3,7 +3,7 @@ *.txt *.xml -# gcovr report tree from `make coverage-core-dev` (.html not matched above) +# gcovr report tree from `make core-coverage` / `make api-coverage` (.html not matched above) coverage/ # report tree from `make gen-quality` (.json/.md not matched above) diff --git a/gen/AGENTS.md b/gen/AGENTS.md index 838b3e0bd..117559d95 100644 --- a/gen/AGENTS.md +++ b/gen/AGENTS.md @@ -17,7 +17,7 @@ python3 -m gen # full emit: render t - `--resolve` prints the collapsed IR view (attribute groups flattened, group refs spliced). - `--config C` applies that target's companion patches before lowering. -- `--check` on `plates` validates renames and detects identifier collisions; exits non-zero on failure -- a CI gate. +- `--check` on `plates` validates renames and detects identifier collisions; exits non-zero on failure. The same checks run during full generation (`build_plates`), so `make gen` + the CI drift check enforce them -- `--check` is a fast standalone way to run just them. - Full emit shortcuts: `make gen-cpp` (C++ only), `make gen` (all targets). ## Quality gates @@ -25,17 +25,16 @@ python3 -m gen # full emit: render t Run all of these after any generator change: ``` -make test-gen # unit tests + test_agnosticism.py + regen + git diff --exit-code -make gen-check # plates --check for all targets (rename validation, collision detection) +make gen-test # unit tests + test_agnosticism.py + regen + git diff --exit-code make gen-quality # design-quality scorer (floor enforced; see quality.py) make gen-lint # pylint (config: .pylintrc) ``` After changing a target's `config.toml` or `templates/`, also run that target's corert suite -(`make test-core-dev`, `make test-go`, `make test-c`). +(`make core-roundtrip-test`, `make test-go`, `make test-c`). If generated output changes, commit the regenerated files in the same commit as the change -that produced them (`make test-gen` fails CI on drift). +that produced them (the CI drift check -- `make gen` then `git diff --exit-code` -- fails on drift). ## Cardinal rules @@ -60,14 +59,14 @@ collisions -- all exit non-zero with a clear `template:line` message. ## Workflows -**Modifying IR or Plates** -- change `gen/ir/` or `gen/plates/`. Run `make test-gen` and -`make gen-check`. +**Modifying IR or Plates** -- change `gen/ir/` or `gen/plates/`. Run `make gen-test` (its regen + +drift check runs the rename/collision validation as part of generation). **Modifying a target** -- edit only that target's `config.toml` and `templates/`. Regenerate, run the target's corert, commit the updated output. **Adding a target** -- create a new directory with `config.toml` and `templates/`. Run -`make test-gen` to confirm `test_agnosticism.py` passes. +`make gen-test` to confirm `test_agnosticism.py` passes. **Renaming a schema type or element** -- use `[rename.type]`, `[rename.element]`, or `[rename.attribute]` in the target's `config.toml`. Keys are validated against the IR; a typo or diff --git a/gen/README.md b/gen/README.md index 8b7110fd3..d5afa2d2b 100644 --- a/gen/README.md +++ b/gen/README.md @@ -23,8 +23,8 @@ there. Adding a new language touches no generator Python. ## Running it Commands, gates, and workflows: see `gen/AGENTS.md`. Full runs: `make gen-cpp` (C++ target), -`make gen` (all targets). Generated output is committed; `make test-gen` regenerates and -asserts `git diff --exit-code`. +`make gen` (all targets). Generated output is committed; the CI drift check regenerates +(`make gen`) and asserts `git diff --exit-code`. ## For agents diff --git a/src/private/mxtest/core/ValueTest.cpp b/src/private/mxtest/core/ValueTest.cpp index 9a7c39c30..fcc4626bf 100644 --- a/src/private/mxtest/core/ValueTest.cpp +++ b/src/private/mxtest/core/ValueTest.cpp @@ -4,8 +4,7 @@ // Construction-safety tests for representative generated value types: // clamping on construct and set, named-factory enums, structural bespoke -// shapes, and union dispatch. The compile-time must-NOT-compile probes live -// in mxtest/probe (script-driven). +// shapes, and union dispatch. #include "cpul/cpulTestHarness.h" diff --git a/src/private/mxtest/probe/Probes.cpp b/src/private/mxtest/probe/Probes.cpp deleted file mode 100644 index fc155605c..000000000 --- a/src/private/mxtest/probe/Probes.cpp +++ /dev/null @@ -1,75 +0,0 @@ -// MusicXML Class Library -// Copyright (c) by Matthew James Briggs -// Distributed under the MIT License - -// Compile-time negative probes: each PROBE number selects one construct -// that MUST NOT compile -- the unrepresentability -// half of valid-by-construction, verified by `make probe-cpp`, which -// asserts that compilation fails for every PROBE > 0 and succeeds for the -// PROBE=0 control (proving the probe failures are not include/setup rot). - -#include "mx/core/Token.h" -#include "mx/core/generated/AboveBelow.h" -#include "mx/core/generated/Color.h" -#include "mx/core/generated/Note.h" -#include "mx/core/generated/NoteChoice.h" -#include "mx/core/generated/Pitch.h" -#include "mx/core/generated/ScorePart.h" - -#include -#include - -#ifndef PROBE -#define PROBE 0 -#endif - -using namespace mx::core; - -int main() -{ -#if PROBE == 0 - // Control: the public API spellings these probes subvert. - AboveBelow enumValue = AboveBelow::above(); - (void)enumValue; - Color color{0xFF, 0x00, 0x00}; - (void)color; - Note note; - (void)note.addBeam(Beam{}); -#elif PROBE == 1 - // An enum wrapper cannot be constructed from a raw int. - AboveBelow v{0}; - (void)v; -#elif PROBE == 2 - // ...nor from its own Tag (the named factories are the only door, so a - // static_cast-forged Tag cannot enter). - AboveBelow v{AboveBelow::Tag::above}; - (void)v; -#elif PROBE == 3 - // A bounded repeated member exposes no mutable container: the span is - // read-only and the only append is the Result-returning add. - Note note; - note.beam()[0] = Beam{}; -#elif PROBE == 4 - // A choice's variant storage is private; alternatives cannot be reached - // (or corrupted) positionally from outside. - NoteChoice c; - (void)std::get<0>(c); -#elif PROBE == 5 - // Color has no lexical constructor: a malformed hex string has no door. - Color c{std::string{"#NOTHEX"}}; - (void)c; -#elif PROBE == 6 - // Sequence fields have no public members: a required child cannot be - // overwritten except through the typed setter. - Pitch p; - p.m_step = Step::c(); -#elif PROBE == 7 - // An xs:ID attribute is a Token, not a raw string: a bare string - // literal cannot enter (Token's converting constructor is explicit, so - // the lexically-unchecked door the audit found is closed at compile - // time). The caller must say Token{...}, which repairs. - ScorePart sp; - sp.setID("has spaces and #bad!"); -#endif - return 0; -} diff --git a/src/private/mxtest/validate/main.cpp b/src/private/mxtest/validate/main.cpp deleted file mode 100644 index c1b1b7221..000000000 --- a/src/private/mxtest/validate/main.cpp +++ /dev/null @@ -1,82 +0,0 @@ -// MusicXML Class Library -// Copyright (c) by Matthew James Briggs -// Distributed under the MIT License - -// Every parsed corpus document is serialized and the output is xmllint- -// validated against the 4.0 XSD -- the mechanical proof that the clamp -// policy emits only valid XML, end to end. This tool does the -// parse+serialize half: it writes each round-tripped document (plus the -// default-constructed Document) to the given output directory; `make -// validate-cpp` runs xmllint over the results. - -#include "mx/core/generated/DefaultsProbe.h" -#include "mx/core/generated/Document.h" -#include "mxtest/corert/CoreRoundtripImpl.h" - -#include "pugixml.hpp" - -#include -#include -#include - -int main(int argc, char **argv) -{ - if (argc != 2) - { - std::fprintf(stderr, "usage: mxtest-validate \n"); - return 2; - } - const std::filesystem::path outDir{argv[1]}; - std::filesystem::create_directories(outDir); - - // Mechanical default-construction probes first: every type's - // natural zero must survive its own strict parser... - mx::core::roundTripDefaults(); - // ...and the default Document must be schema-valid (xmllint checks it - // with the corpus outputs below). - { - pugi::xml_document doc; - mx::core::serialize(mx::core::Document{}, doc); - doc.save_file((outDir / "default-document.xml").string().c_str(), " "); - } - - int failures = 0; - int written = 0; - for (const auto &path : mxtest::corert::discoverInputFiles()) - { - pugi::xml_document input; - if (!input.load_file(path.c_str(), pugi::parse_default | pugi::parse_doctype)) - { - std::fprintf(stderr, "LOAD FAIL %s\n", path.c_str()); - ++failures; - continue; - } - auto parsed = mx::core::parse(input); - if (!parsed) - { - if (parsed.error().code == mx::core::ErrorCode::unsupportedVersion) - { - continue; // version-gated: not this model's document - } - std::fprintf(stderr, "PARSE FAIL %s [%s] %s\n", path.c_str(), parsed.error().path.c_str(), - parsed.error().message.c_str()); - ++failures; - continue; - } - pugi::xml_document output; - mx::core::serialize(parsed.value(), output); - std::string flat = mxtest::corert::toTestName(path); - for (auto &c : flat) - { - if (c == '/' || c == '\\') - { - c = '_'; - } - } - output.save_file((outDir / flat).string().c_str(), " "); - ++written; - } - - std::printf("mxtest-validate: %d outputs written, %d failures\n", written, failures); - return failures == 0 ? 0 : 1; -}