ci(benchmarks): install the modules the JMH job measures - #578
Merged
Conversation
The JMH workflow installed the engine from source and let Maven resolve the rest from Central. One of them is not there to resolve: the benchmarks read their document fixtures out of the tests-classifier jar of graph-compose-templates, which the templates release profile unbinds so it is never published. The job died at dependency resolution before a single benchmark ran — five of its last six weekly runs, across four versions, each looking for the jar under the version the release before it had just published. It now installs render-pdf and templates from source, the same sequence the two benchmark jobs in ci.yml already run. BenchmarkDependencyInstallGuardTest reads every workflow and fails when a job builds benchmarks/pom.xml without first installing each first-party dependency the benchmarks pom declares. It takes that list from the pom rather than a copy of it, so a new sibling dependency is guarded the day it is added, and it runs in the guard job on every pull request — which is what the workflow itself cannot do, firing only from the default branch on a schedule. ./mvnw clean -Dtest=<guard list> test -pl :graph-compose-core — BUILD SUCCESS, 52 tests. The install sequence verified end to end against the failing step: the JMH runner jar builds and lists all 11 benchmarks. The guard was confirmed red three ways first — templates install deleted, downgraded to a comment, and the benchmark build renamed so no job matches it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The weekly JMH run never reaches a benchmark. It dies in
Build the JMH runner jar,resolving a dependency that Maven Central does not have and never will:
benchmarks/pom.xmldepends on thetests-classifier jar ofgraph-compose-templates—
CanonicalBenchmarkSupportbuilds its fixtures fromCvDataFixtures,InvoiceDataFixturesandProposalDataFixtures, which live in the templates testscope because a fixture producing template data cannot sit in the core test scope
without making core depend on templates. That jar is a local build aid: the templates
releaseprofile unbindsattach-test-jar(phasenone) so it is never deployed.The job installed
:graph-compose-coreand left Maven to resolve the rest, so the onlyversion of that jar it could ever find is one a source build put in the local
repository. Five of its last six weekly runs failed this way, across four versions —
2.0.0, 2.1.0, 2.1.1 and 2.2.0 — each looking for the jar under the version the release
before it had just published
(run 32004336307).
render-pdfis resolved the same way and survives only because it is published.Nothing on a pull request could catch it: the workflow runs on a schedule from the
default branch and on manual dispatch, so the job goes red on a Monday, far from the
change that broke it.
What changed
benchmarks-jmh.ymlinstallsgraph-compose-render-pdfandgraph-compose-templatesfrom source between the core install and the runner-jar build — the same sequence
perf-smokeandbenchmark-diffinci.ymlalready run. The-DskipTestsinstallstill compiles test sources, which is what produces the
testsjar. The step nameInstall root artifact (graph-compose + test-jar)also loses its stale half: theartifact is
graph-compose-coresince the module split, and no core test-jar isinvolved.
BenchmarkDependencyInstallGuardTestfails when a job buildsbenchmarks/pom.xmlwithout an earlier step in the same job installing each first-party dependency the
benchmarks pom declares. The required set is read out of the pom, not a copy of it,
so a new sibling dependency is guarded the day it is added. Job scoping is the point:
a file-level check would let one benchmark job borrow another's install steps and
pass.
wrong, and it fails when it finds no benchmark-building job at all — so a workflow
shape that moves fails the guard instead of silently emptying it. YAML comments are
stripped before matching, so an install commented out does not still count as one.
A job that would rely on
-pl … -amto pull a module in transitively reads asmissing, deliberately: every install in this repository is explicit, and a guard that
reasons about reachability is one that can be argued with.
-Dtest=list in the architecture-and-documentation-guards job,which has no path filter and so runs on a workflow-only pull request — the shape that
introduces this failure.
Verification
./mvnw -B -ntp clean "-Dtest=…" test -pl :graph-compose-core→ BUILD SUCCESS,52 tests, 0 failures (exit code read unpiped).
CiGuardListGuardTestresolves theadded name.
The install sequence was run end to end against the failing step: fonts → emoji → core
→ render-pdf → templates, then
./mvnw -B -ntp -f benchmarks/pom.xml clean package -DskipTests→ BUILD SUCCESS,with
graph-compose-templates-…-tests.jaron the shade classpath. The runner jar itproduces starts:
java -jar benchmarks/target/benchmarks.jar -llists all 11benchmarks, so the step after the one that was failing works too.
The guard was confirmed red before it was trusted, three ways: the templates install
deleted from the JMH job (reports
job 'jmh' … without first installing graph-compose-templates), the same install downgraded to a YAML comment (samefailure), and
-f benchmarks/pom.xmlrenamed everywhere (reports that no job runs it,rather than passing over nothing). Dropping both
ci.ymltemplates installs reportsperf-smokeandbenchmark-diffby name, so its coverage is all three benchmark jobs,not one.
Notes
The cron fires from the default branch, so this does not turn the weekly run green on
its own — the job stays red until the commit reaches
mainwith the next release.No production code, no public API, no
@since. The CHANGELOG entry opens## v2.2.1 — Plannedunder### Build, matching the### Buildentries #481, #493 and#512 shipped for comparable CI fixes.
Lane: build/CI — one workflow job corrected against the pattern its two siblings
already follow, plus the guard that holds all three to it.