feat(#1284): benchmark unlint-non-existing-defect with +unlint metas - #1285
feat(#1284): benchmark unlint-non-existing-defect with +unlint metas#1285VasilevNStas wants to merge 1 commit into
Conversation
|
@volodya-lombrozo plz review this |
volodya-lombrozo
left a comment
There was a problem hiding this comment.
@VasilevNStas Currently we create S ... XXL benchmarks using jeo-maven-plugin which works pretty well, but it generates XMIR that is a bit of artifical. Maybe we can reconsider our strategy to generating these files? Maybe it's better to generate them by parsing real EO programs? At least, we can substitute S, M examples. What do u think?
There was a problem hiding this comment.
Good suggestion about using real EO programs instead of bytecode-derived XMIR for the S/M benchmarks. That would make the benchmark data closer to what the lints actually process in production. I see two ways to do it and would like your call on the scope:
- A focused follow-up that only adds this
scansXmirWithUnlintsbenchmark for the+unlintpath (what this PR does), and - A bigger refactor that replaces
jeo-maven-pluginfixtures for S/M (or all sizes) with parsed EO programs.
I kept this PR to option 1 because switching the existing S..XXL fixtures touches SourceSize, BenchmarkState and the README table, and it is a separate concern from #1284 (which is specifically about the +unlint path having no coverage at all).
If you prefer, I can prepare option 2 as another PR after this one lands. Meanwhile, note this PR also surfaced a pre-existing infra issue: the proguard profile (active on JDK < 22) fails on the rultor merge (JDK 21) and locally on JDK 21 with Obfuscation failed, unrelated to the diff — I described it in #1238 with the MAVEN_OPTS/-P!proguard suggestion.
Happy to proceed either way.
79e3121 to
de21d00
Compare
|
@volodya-lombrozo |
fix #1284
What
Adds a JMH benchmark that measures
LtUnlintNonExistingDefecton an XMIR that actually contains+unlintmetas.Why
The existing
scansXmirbenchmark is driven by real-world bytecode-derived XMIR (com.sun.jna.*,org.apache.hadoop...FSNamesystem), none of which contains a single+unlintmeta. So it only measures the "no+unlintat all" fast path ofLtUnlintNonExistingDefect.#1233 changed the other path too: when
+unlintmetas are present, the lint now filtersthis.lintsdown to the referenced names before computingexisting(). Until now there was no fixture exercising that path, so a regression there would go unnoticed by the benchmark.How
New benchmark method
scansXmirWithUnlintsinSourceBench, which parses the existing fixtureunlint-ascii-only.eo(it has+unlint ascii-onlyand a non-ASCII comment that produces a real defect) and runs the full lint set on it.Measured result (JMH, average time):
This gives a timing signal for the
+unlintcode path, so future changes toLtUnlintNonExistingDefect(faster, same or slower) become visible.Both
mvn test(606 tests) andmvn clean install -Pqulicepass.