From 55ce4558e842983ae6425e61443d42991045c958 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 10:52:50 -0300 Subject: [PATCH 01/15] docs: design spec for CLI benchmark runner --- .../2026-07-27-cli-benchmark-runner-design.md | 239 ++++++++++++++++++ 1 file changed, 239 insertions(+) create mode 100644 docs/superpowers/specs/2026-07-27-cli-benchmark-runner-design.md diff --git a/docs/superpowers/specs/2026-07-27-cli-benchmark-runner-design.md b/docs/superpowers/specs/2026-07-27-cli-benchmark-runner-design.md new file mode 100644 index 0000000..eb2c615 --- /dev/null +++ b/docs/superpowers/specs/2026-07-27-cli-benchmark-runner-design.md @@ -0,0 +1,239 @@ +# CLI benchmark runner — design + +_2026-07-27_ + +## Goal + +Add a fourth runner to the `benchmarks/` harness that measures the **`dw` native +CLI**. Today the harness compares three runners over a shared corpus: the Node and +Python `native-lib` wrappers (native shared library via FFI) and a JVM engine +baseline (`DataWeaveScriptingEngine`). The CLI runner completes the picture with +the shipped native executable, driving `NativeRuntime` — the CLI's own engine +wrapper — through a GraalVM native image. + +The runner integrates through the harness's existing extension contract: emit a +result file conforming to `schema/result.schema.json` and tag a Gradle task +`ext.benchmarkRunner = true`. The root `benchmarkCompare` auto-discovers it; no +edit to that aggregator, `report.mjs`, or the schema is required. + +## Why a CLI runner is distinct + +The three quadrants of the DataWeave runtime are: + +- **engine** — JVM library (`DataWeaveScriptingEngine` directly). +- **node / python** — native shared library (`dwlib`) via FFI. +- **cli** — native standalone executable (`dw`), driving `NativeRuntime`. + +Because `NativeRuntime` is the CLI's wrapper over the same scripting engine the +`engine` runner drives raw, the **engine-vs-cli delta isolates native-image vs +JVM** for the same logical runtime work — the same reason the README calls the +cross-runner cold-start comparison meaningful (the native image has no JVM to +boot). + +## Key decisions + +These were settled during brainstorming; recorded here so the rationale is not lost. + +1. **Measure the native binary, not a JVM entrypoint.** A JVM entrypoint runner + would heavily overlap the existing `engine` runner and would not measure the + shipped artifact. The native binary is what users actually run. + +2. **The binary emits a `READY` marker in benchmark mode.** Rather than treat the + binary as a black box (which would force one whole-process number per + invocation and lose the init/exec split), we add a benchmark harness *inside* + `native-cli` that prints `READY` the instant `NativeRuntime` is constructed. + This reuses the exact parent/child protocol the Node/Python/engine runners + already use, yielding a true `cold-start` **and** a like-for-like `first-run` + **and** a `warm` steady-state loop — all from the real native binary. + +3. **Build-time gate, stripped from production.** The benchmark harness must not + ship in the production `dw`. This is a security-sensitive CLI (`--untrusted`, + `--privileges`); an always-reachable alternate entrypoint is both a footgun (a + stray `DW_BENCH` env var could hijack a normal invocation) and unwanted + surface. A generated `BenchmarkMode.ENABLED` constant is `false` in normal + builds and `true` only under `-Pbenchmark=true`. Native-image reachability + analysis folds the harness away as dead code when `ENABLED` is a compile-time + `false`, so the shipped `dw` contains no benchmark code and no consumer can + reach it. Cost: the benchmark build performs its own `nativeCompile`. + +4. **Corpus only.** The runner measures the shared corpus, exactly like the other + three runners — no ad-hoc user-script mode, no arbitrary-subcommand timing. + (Arbitrary subcommands like `dw spell`/`dw validate` cannot reuse the + cold/first split anyway: init is entangled inside each command, so there is no + single honest "runtime ready" boundary to mark. Timing those would be a + separate whole-process tool with a non-comparable metric — explicitly out of + scope.) + +5. **No `streaming` metric.** The `dw run` path writes a whole output stream and + has no chunked-input FFI like the library's `runTransform`. The CLI emits + `cold-start`, `first-run`, and `warm` only. The report renders absent cells as + `—`, so this needs zero report or schema changes — it is a documented, + honest gap, matching how each runner declares only the metrics it can produce. + +## Architecture + +Parent/child split mirroring the Node runner, but **the child is the `dw` binary +itself** running in a build-gated benchmark mode. + +### Child — benchmark harness inside `native-cli` + +- `DWCLI.main` checks, before any picocli parsing or banner output: + `if (BenchmarkMode.ENABLED && ) → BenchmarkHarness.main(args)`. + In a production build `BenchmarkMode.ENABLED` is a compile-time `false`, so the + entire branch (and `BenchmarkHarness`) is unreachable and tree-shaken out of the + native image. +- `BenchmarkHarness` is **corpus-agnostic**: it knows nothing about + `manifest.json`. The parent passes it a script file, input files with explicit + mime/charset, a mode, and iteration counts. This keeps `native-cli` free of + benchmark-corpus knowledge and keeps the (gated) footprint minimal. +- The harness constructs **one** `NativeRuntime` and drives it through + `NativeRuntime.run(script, "bench", inputs, out)`. It reuses the Scala building + blocks already present in the engine runner package where practical + (`CountingOutputStream`, the "READY then single JSON line" pattern). + +Invocation shape (args produced by the parent): + +``` +DW_BENCH=1 dw --bench-mode=coldfirst \ + --script= \ + --input=payload=:application/json:utf-8 \ + [--warmup=N --iters=M] +``` + +(The exact env-var name and flag spelling are an implementation detail; a +specific, collision-unlikely name is used. `ENABLED` is what actually gates +reachability — the env var only selects mode within an already-bench-enabled +binary.) + +#### Mode `coldfirst` — one fresh process per sample + +The parent spawns this N times; each process yields one cold-start + one first-run. + +1. Read script + input files into memory. +2. Construct one `NativeRuntime` (the init being measured from outside). +3. Print `READY\n` and **flush immediately**, before any banner/logging can + interleave. The parent stamps `cold-start` = wall-clock from spawn to this + marker (process launch + image load + runtime init). +4. `nowNs()`; `runtime.run(script, "bench", inputs, CountingOutputStream)`; + measure `first-run` in-process. +5. Assert success. On failure: non-zero exit + stderr, so a bad sample never + records a bogus timing (same contract as `runners/node/coldstart.mjs`). +6. Print one JSON line: `{"firstRunMs": }`. + +#### Mode `warm` — single process, in-process loop + +Mirrors `runners/node/warm-bench.mjs`. + +1. Construct one `NativeRuntime`, print `READY`. +2. `warmup` unmeasured `run()` calls, then `iters` measured `run()` calls. +3. Print `{"warmMs": []}`. The parent computes stats via the shared + `computeStats`, so the stats path is identical across runners. + +#### Output discipline + +`dw` prints a banner and can log to stdout. The harness runs with logging silenced +and writes transformation output to a `CountingOutputStream` (never real stdout), +so the only stdout is `READY` + the JSON line. The parent reads the **last** JSON +line and tolerates stray lines (the engine parent already does this). + +### Parent — `benchmarks/runners/cli/` (Node) + +Mirrors `runners/node/` file layout and reuses the shared libs +(`lib/manifest.mjs`, `lib/stats.mjs`, `lib/env.mjs`) verbatim. + +- **`locate.mjs`** — resolves the **benchmark-enabled** `dw` binary. Env override + `DW_BENCH_BIN=/abs/path/to/dw`, else the default native build output + `native-cli/build/native/nativeCompile/dw{,.exe}`. Fails fast with a + "build the bench binary (`./gradlew native-cli:nativeCompile -Pbenchmark=true`)" + message if absent (same shape as `runners/node/wrapper.mjs`). +- **`coldstart.mjs`** — spawns `dw` in `coldfirst` mode per sample; stamps + cold-start at spawn→`READY`, parses `firstRunMs`. Structurally + `runners/node/coldstart.mjs` with the child command swapped from + `node coldstart-child.mjs` to `dw --bench-mode=coldfirst …`. Same + reject-on-failure contract. +- **`warm.mjs`** — spawns `dw` once per warm case in `warm` mode, reads the + `warmMs[]` array, runs it through the shared `computeStats`. +- **`emit.mjs`** — entrypoint. `gatherEnv({ runner: "cli", runtimeVersion: + "dw " })`, run cold+first then warm, `validateResultIds`, write + `results/cli-.json`. Same structure as `runners/node/emit.mjs`. + +### Metrics emitted + +Per case, gated by that case's declared `metrics[]` in `corpus/manifest.json` +(no manifest changes needed). + +| metric | how measured | comparable to | +|---|---|---| +| `cold-start` | spawn → `READY` (launch + image load + `NativeRuntime` init) | node/python/engine cold-start — **headline: native binary vs JVM** | +| `first-run` | in-process compile+exec, first call | node/python/engine first-run — like-for-like (both exclude launch) | +| `warm` | in-process steady-state loop | node/python/engine warm | +| `streaming` | not emitted | — (documented gap) | + +Env fields: `runtimeVersion` from `dw --version`; `dwlibBuildId = "n/a-cli"` +(following the engine runner's `"n/a-engine"` convention — the CLI is a binary, +not the staged `dwlib`). + +## Gradle wiring + +In `native-cli/build.gradle`: + +- **`genBenchmarkMode`** — generates `BenchmarkMode.java` (constant `ENABLED`) + into `build/genresource/`, following the existing `genVersions` / + `ComponentVersion` pattern. `ENABLED = true` only when `-Pbenchmark=true`, else + `false`. Wired onto the compile classpath. +- The benchmark-enabled native image reuses `nativeCompile` invoked with + `-Pbenchmark=true` (the flag flows into `genBenchmarkMode`). Normal `build`/CI + never sets it, so the shipped `dw` is built with `ENABLED=false` and the harness + is tree-shaken out. +- **`benchmarkCli`** — `onlyIf { benchmark==true }`, `ext.benchmarkRunner = true`, + `dependsOn nativeCompile` and the shared `genBenchInputs`; runs + `node corpus/gen-inputs.mjs && node runners/cli/emit.mjs`. It does **not** + render the report. The root `benchmarkCompare` auto-discovers it via the + `benchmarkRunner` tag — no edit to that task, per the README "Adding a runner" + contract. + +## Error handling + +Same fail-fast contract as `coldstart.mjs` / `EngineChild`: + +- Child non-zero exit → parent rejects with stderr. +- Missing `READY` marker → reject. +- Missing JSON line → reject. +- `result.success == false` inside the harness → non-zero exit + stderr. + +A bad sample never records a bogus timing. `locate.mjs` fails with a clear +build-the-bench-binary message when the artifact is absent. + +## Testing + +Follows the repo's parity-guard split: dwlib/binary-free tests run in the normal +`test`; binary-dependent tests are benchmark-only. + +- **Scala** (`native-cli:test`, scalatest `AnyFreeSpec`/`Matchers`): + - `BenchmarkHarness` arg parsing (script / input / mode / iters). + - `coldfirst` and `warm` each emit exactly `READY` + one JSON line. + - Transformation output goes to the counting stream, not stdout. + - Failure path → non-zero exit + stderr. + - Guard: `BenchmarkMode.ENABLED` is `false` in a normal build. +- **JS** (`benchmarkJsUnitTest`, dwlib-free, always-on parity set): + - `locate.mjs` resolution + `DW_BENCH_BIN` override. + - `emit.mjs` result-object builder. + - Parent parsing of child stdout, using fixtures / a fake child — no real + binary. +- **Smoke** (benchmark-only, needs the bench binary): one real `coldfirst` spawn + on the `trivial` case asserting a positive cold-start and a `firstRunMs`. + +## Out of scope + +- Ad-hoc user-supplied script benchmarking. +- Timing arbitrary `dw` subcommands (`spell`, `validate`, `wizard`, `repl`). +- `streaming` metric for the CLI. +- Any change to `report.mjs`, `schema/result.schema.json`, `benchmarkCompare`, or + the corpus manifest. + +## Documentation + +Update `benchmarks/README.md`: add `runners/cli/` to the layout, note the +build-gated bench binary and `-Pbenchmark=true` requirement, and record that the +CLI emits `cold-start`/`first-run`/`warm` (no `streaming`). Add the single-runner +invocation (`./gradlew native-cli:benchmarkCli -Pbenchmark=true`). From 23b840de15de14ae3ea4b869902af84851362f3f Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:09:42 -0300 Subject: [PATCH 02/15] docs: implementation plan for CLI benchmark runner --- .../plans/2026-07-27-cli-benchmark-runner.md | 1108 +++++++++++++++++ 1 file changed, 1108 insertions(+) create mode 100644 docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md diff --git a/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md b/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md new file mode 100644 index 0000000..396ce2b --- /dev/null +++ b/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md @@ -0,0 +1,1108 @@ +# CLI Benchmark Runner Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Add a fourth benchmark runner that measures the shipped `dw` native CLI over the shared corpus, emitting `cold-start`, `first-run`, and `warm` metrics into the existing comparison harness. + +**Architecture:** A build-gated benchmark harness inside `native-cli` (compiled into `dw` only under `-Pbenchmark=true`, tree-shaken out of production) prints a `READY` marker after constructing one `NativeRuntime`, then runs timed work — mirroring the Node/Python/engine child protocol. A Node parent under `benchmarks/runners/cli/` spawns that binary per case, stamps cold-start at spawn→READY, and reads back in-process timings, reusing the shared `lib/` modules. + +**Tech Stack:** Java (picocli entrypoint + generated constant), Scala 2.12 (benchmark harness on `NativeRuntime`), GraalVM native-image, Node.js (parent orchestrator, ESM), Gradle, scalatest, `node --test`. + +## Global Constraints + +- **Weave runtime version:** pinned by `weaveVersion` in `gradle.properties` — never hardcode; read it (parent already does via `lib/env.mjs`). +- **Benchmark tasks are opt-in only:** every Gradle benchmark task guards with `onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true }`. Never part of normal `build`/`test`/CI. +- **Production `dw` must not contain benchmark code:** gated by a generated `BenchmarkMode.ENABLED` constant that is `false` unless `-Pbenchmark=true`; native-image folds the unreachable branch away. +- **Result schema is frozen:** output must conform to `benchmarks/schema/result.schema.json` (`schemaVersion: "1.0"`; metrics ∈ `cold-start|first-run|warm|streaming`; units ∈ `ms|MB/s`). Do NOT edit the schema, `report.mjs`, `benchmarkCompare`, or `corpus/manifest.json`. +- **Runner column name:** the result's `runner` field is `"cli"` (the report's column + dedupe key). +- **Runner registration contract:** a runner integrates by (1) writing `benchmarks/results/-.json` and (2) tagging its Gradle task `ext.benchmarkRunner = true`. `benchmarkCompare` auto-discovers it — do NOT edit `benchmarkCompare`. +- **Child stdout discipline:** the only stdout the harness emits is the line `READY` (flushed) followed by exactly one JSON line. Transformation output goes to a discarding stream, never stdout. +- **`dwlibBuildId` env field:** `"n/a-cli"` (the CLI is a binary, not the staged `dwlib`), following the engine runner's `"n/a-engine"` convention. +- **Cross-platform:** Gradle `Exec` tasks branch on `os.name` containing `windows` (`cmd /c` vs `bash -c`), matching existing tasks. Binary name is `dw` (`dw.exe` on Windows). + +--- + +## File Structure + +**Created:** +- `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala` — the corpus-agnostic in-binary harness (arg parse, `coldfirst`/`warm` modes, READY + JSON output). +- `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` — scalatest for the harness + a guard that `BenchmarkMode.ENABLED` is `false` in a normal build. +- `benchmarks/runners/cli/locate.mjs` — resolves the bench-enabled `dw` binary. +- `benchmarks/runners/cli/coldstart.mjs` — spawns `coldfirst` per sample; cold-start + first-run rows. +- `benchmarks/runners/cli/warm.mjs` — spawns `warm` once per warm case; warm rows. +- `benchmarks/runners/cli/emit.mjs` — assembles env + rows, writes `results/cli-.json`. +- `benchmarks/runners/cli/locate.test.mjs` — dwlib/binary-free unit test for `locate.mjs`. +- `benchmarks/runners/cli/emit.test.mjs` — dwlib/binary-free unit test for the result builder. + +**Modified:** +- `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java` — dispatch to the harness before picocli when gated + env set. +- `native-cli/build.gradle` — `genBenchmarkMode` task (generates `BenchmarkMode.java`), wire onto compile, `benchmarkCli` runner task. +- `native-lib/build.gradle` — add `runners/cli/*.test.mjs` to the always-on `benchmarkJsUnitTest` file list. +- `benchmarks/README.md` — document the CLI runner. + +--- + +## Task 1: Generate the `BenchmarkMode.ENABLED` build gate + +**Files:** +- Modify: `native-cli/build.gradle` (add `genBenchmarkMode` task near `genVersions` at line ~51; wire into `compileScala`/`compileJava` deps like `genVersions`) +- Verify against: `native-cli/build.gradle:48-74` (the `genVersions` pattern generates into `build/genresource`, which is already a source dir per `native-cli/build.gradle:7-13`) + +**Interfaces:** +- Produces: a generated Java class `org.mule.weave.cli.BenchmarkMode` with `public static final boolean ENABLED` — `true` only when the Gradle property `benchmark` is truthy, else `false`. Consumed by Task 2 (`DWCLI`) and Task 3's guard test. + +Rationale: a **Java** constant (not Scala) so `DWCLI.java` reads it with no cross-language friction; `build/genresource` is already on the Scala srcDir but `javac` also compiles generated Java there via the existing `compileJava` classpath wiring (`native-cli/build.gradle:153-154`). Generate into a Java-compiled location: use a dedicated `build/genjava` dir added to the java sourceSet to keep it unambiguous. + +- [ ] **Step 1: Add a generated-Java source dir to the java sourceSet** + +In `native-cli/build.gradle`, extend the `sourceSets` block (currently lines 7-13) to add a java srcDir: + +```groovy +sourceSets { + main { + scala { + srcDirs = ['src/main/scala', 'build/genresource'] + } + java { + srcDirs += 'build/genjava' + } + } +} +``` + +- [ ] **Step 2: Add the `genBenchmarkMode` task** + +Immediately after the `genVersions` task (after line 67 in `native-cli/build.gradle`), add: + +```groovy +def genJavaDirectory = new File("$project.buildDir/genjava") + +task genBenchmarkMode() { + def enabled = project.findProperty('benchmark')?.toString()?.toBoolean() == true + def benchmarkMode = new File(genJavaDirectory, "org/mule/weave/cli/BenchmarkMode.java") + def parentFile = benchmarkMode.getParentFile() + if (!parentFile.exists()) { + parentFile.mkdirs() + } + final PrintWriter outputPrinter = new PrintWriter(new FileWriter(benchmarkMode)) + outputPrinter.println("package org.mule.weave.cli;") + outputPrinter.println() + outputPrinter.println("// GENERATED by genBenchmarkMode — do not edit.") + outputPrinter.println("// ENABLED is true only when built with -Pbenchmark=true; native-image") + outputPrinter.println("// folds the benchmark branch away as dead code when this is false.") + outputPrinter.println("public final class BenchmarkMode {") + outputPrinter.println(" private BenchmarkMode() {}") + outputPrinter.println(" public static final boolean ENABLED = " + enabled + ";") + outputPrinter.println("}") + outputPrinter.close() +} +``` + +- [ ] **Step 3: Wire it into compilation** + +Update the existing `compileScala` block (lines 72-74) and add a `compileJava` dependency so the constant exists before either compiles: + +```groovy +defaultTasks += genVersions + +compileScala { + dependsOn genVersions + dependsOn genBenchmarkMode +} + +compileJava { + dependsOn genBenchmarkMode +} +``` + +- [ ] **Step 4: Verify normal build generates `ENABLED = false`** + +Run: `./gradlew native-cli:genBenchmarkMode && cat native-cli/build/genjava/org/mule/weave/cli/BenchmarkMode.java` +Expected: file contains `public static final boolean ENABLED = false;` + +- [ ] **Step 5: Verify benchmark build generates `ENABLED = true`** + +Run: `./gradlew native-cli:genBenchmarkMode -Pbenchmark=true && cat native-cli/build/genjava/org/mule/weave/cli/BenchmarkMode.java` +Expected: file contains `public static final boolean ENABLED = true;` + +- [ ] **Step 6: Commit** + +```bash +git add native-cli/build.gradle +git commit -m "build: generate BenchmarkMode.ENABLED gate for native-cli" +``` + +--- + +## Task 2: `BenchmarkHarness` — the in-binary corpus-agnostic harness + +**Files:** +- Create: `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala` +- Create: `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` + +**Interfaces:** +- Consumes: `org.mule.weave.dwnative.NativeRuntime` (constructor `new NativeRuntime(libDir: File, path: Array[File], console: Console, maybeLanguageLevel: Option[DataWeaveVersion])`; method `run(script: String, nameIdentifier: String, inputs: ScriptingBindings, out: OutputStream, defaultOutputMimeType: String, maybePrivileges: Option[Seq[String]]): WeaveExecutionResult` where `WeaveExecutionResult.success(): Boolean` and `.result(): String`); `org.mule.weave.dwnative.utils.DataWeaveUtils#getLibPathHome(): File`; `org.mule.weave.dwnative.cli.DefaultConsole`; `org.mule.weave.v2.runtime.ScriptingBindings#addBinding(name, value: BindingValue)`; `org.mule.weave.v2.runtime.BindingValue(bytes: Array[Byte], mimeType: Option[String], props: Map[String,Any], charset: Charset)`. +- Produces: `object BenchmarkHarness { def main(args: Array[String]): Unit }` and (for tests) `def parseArgs(args: Array[String]): BenchArgs`, `case class BenchArgs(mode: String, scriptFile: String, inputs: Seq[BenchInput], warmup: Int, iters: Int)`, `case class BenchInput(name: String, file: String, mimeType: String, charset: String)`, and `def runColdFirst(args, out: java.io.PrintStream, sink: OutputStream): Unit` / `def runWarm(args, out: java.io.PrintStream, sink: OutputStream): Unit` (out = where READY/JSON go; sink = discard stream for transform output). `main` calls these with `System.out` and a fresh `CountingOutputStream`. + +Reuse a discarding stream identical in behavior to the engine runner's `CountingOutputStream`; define a small private one here rather than depend on the `benchmarks-engine` module (no such dependency exists from `native-cli`). + +Arg format from the parent (one `--input` per binding): +``` +--bench-mode=coldfirst|warm +--script= +--input==:: +--warmup= (warm mode only; default 0) +--iters= (warm mode only; default 100) +``` + +- [ ] **Step 1: Write the failing test for arg parsing** + +Create `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala`: + +```scala +package org.mule.weave.dwnative.benchmark + +import org.scalatest.freespec.AnyFreeSpec +import org.scalatest.matchers.should.Matchers + +class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { + + "parseArgs" - { + "parses coldfirst mode with one input" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=coldfirst", + "--script=/tmp/x.dwl", + "--input=payload=/tmp/p.json:application/json:utf-8")) + a.mode shouldBe "coldfirst" + a.scriptFile shouldBe "/tmp/x.dwl" + a.inputs should have size 1 + a.inputs.head shouldBe BenchInput("payload", "/tmp/p.json", "application/json", "utf-8") + } + + "parses warm mode with warmup and iters" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=warm", "--script=/tmp/x.dwl", "--warmup=5", "--iters=30")) + a.mode shouldBe "warm" + a.warmup shouldBe 5 + a.iters shouldBe 30 + a.inputs shouldBe empty + } + + "handles a mimeType-only input (charset defaults to utf-8)" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=coldfirst", "--script=/tmp/x.dwl", + "--input=payload=/tmp/p.json:application/json")) + a.inputs.head.charset shouldBe "utf-8" + } + } +} +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` +Expected: FAIL — `BenchmarkHarness` / `BenchInput` not found (compilation error). + +- [ ] **Step 3: Implement `BenchmarkHarness` with parsing + modes** + +Create `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala`: + +```scala +package org.mule.weave.dwnative.benchmark + +import org.mule.weave.dwnative.NativeRuntime +import org.mule.weave.dwnative.WeaveExecutionResult +import org.mule.weave.dwnative.cli.DefaultConsole +import org.mule.weave.dwnative.utils.DataWeaveUtils +import org.mule.weave.v2.runtime.BindingValue +import org.mule.weave.v2.runtime.ScriptingBindings + +import java.io.{ File, OutputStream, PrintStream } +import java.nio.charset.Charset +import java.nio.file.Files + +final case class BenchInput(name: String, file: String, mimeType: String, charset: String) +final case class BenchArgs(mode: String, scriptFile: String, inputs: Seq[BenchInput], warmup: Int, iters: Int) + +/** Corpus-agnostic in-binary benchmark harness. Reachable only in a build made with + * -Pbenchmark=true (guarded by BenchmarkMode.ENABLED in DWCLI); native-image folds it + * out of a production dw. Prints "READY" the instant one NativeRuntime is constructed, + * then a single JSON line of timings. Parent (benchmarks/runners/cli) measures cold-start + * as spawn->READY wall-clock. */ +object BenchmarkHarness { + + /** Discards bytes; used as the transform write sink so we never touch real stdout. */ + private final class DiscardStream extends OutputStream { + override def write(b: Int): Unit = () + override def write(b: Array[Byte]): Unit = () + override def write(b: Array[Byte], off: Int, len: Int): Unit = () + } + + private def nowNs(): Long = System.nanoTime() + private def msSince(startNs: Long): Double = (System.nanoTime() - startNs) / 1e6 + + def parseArgs(args: Array[String]): BenchArgs = { + var mode = "" + var script = "" + val inputs = scala.collection.mutable.ArrayBuffer[BenchInput]() + var warmup = 0 + var iters = 100 + args.foreach { arg => + val eq = arg.indexOf('=') + val key = if (eq >= 0) arg.substring(0, eq) else arg + val value = if (eq >= 0) arg.substring(eq + 1) else "" + key match { + case "--bench-mode" => mode = value + case "--script" => script = value + case "--warmup" => warmup = value.toInt + case "--iters" => iters = value.toInt + case "--input" => + // value = =:[:] + val nameSep = value.indexOf('=') + val name = value.substring(0, nameSep) + val rest = value.substring(nameSep + 1) + val parts = rest.split(":", 3) + val file = parts(0) + val mimeType = parts(1) + val charset = if (parts.length > 2 && parts(2).nonEmpty) parts(2) else "utf-8" + inputs += BenchInput(name, file, mimeType, charset) + case _ => throw new RuntimeException(s"unknown bench arg: $arg") + } + } + if (mode.isEmpty) throw new RuntimeException("--bench-mode is required") + if (script.isEmpty) throw new RuntimeException("--script is required") + BenchArgs(mode, script, inputs.toSeq, warmup, iters) + } + + private def newRuntime(): NativeRuntime = { + val console = DefaultConsole.enableSilent() + val utils = new DataWeaveUtils(console) + new NativeRuntime(utils.getLibPathHome(), Array.empty[File], console, None) + } + + private def readScript(a: BenchArgs): String = + new String(Files.readAllBytes(new File(a.scriptFile).toPath), java.nio.charset.StandardCharsets.UTF_8) + + private def bindings(a: BenchArgs): ScriptingBindings = { + val b = new ScriptingBindings() + a.inputs.foreach { in => + val bytes = Files.readAllBytes(new File(in.file).toPath) + val bv = new BindingValue(bytes, Some(in.mimeType), Map.empty[String, Any], Charset.forName(in.charset)) + b.addBinding(in.name, bv) + } + b + } + + private def assertOk(r: WeaveExecutionResult): Unit = + if (!r.success()) throw new RuntimeException("run failed: " + r.result()) + + def runColdFirst(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { + val script = readScript(a) + val b = bindings(a) + val rt = newRuntime() // engine init — measured externally as cold-start + out.println("READY"); out.flush() + val start = nowNs() + assertOk(rt.run(script, "bench", b, sink, "application/json", None)) + val firstRunMs = msSince(start) + out.println("{\"firstRunMs\":" + firstRunMs + "}") + } + + def runWarm(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { + val script = readScript(a) + val b = bindings(a) + val rt = newRuntime() + out.println("READY"); out.flush() + var i = 0 + while (i < a.warmup) { assertOk(rt.run(script, "bench", b, sink, "application/json", None)); i += 1 } + val samples = new Array[Double](a.iters) + i = 0 + while (i < a.iters) { + val start = nowNs() + assertOk(rt.run(script, "bench", b, sink, "application/json", None)) + samples(i) = msSince(start) + i += 1 + } + out.println("{\"warmMs\":[" + samples.mkString(",") + "]}") + } + + def main(args: Array[String]): Unit = { + val a = parseArgs(args) + val sink = new DiscardStream() + a.mode match { + case "coldfirst" => runColdFirst(a, System.out, sink) + case "warm" => runWarm(a, System.out, sink) + case other => throw new RuntimeException(s"unknown --bench-mode: $other") + } + } +} +``` + +- [ ] **Step 4: Run the parsing test to verify it passes** + +Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` +Expected: PASS (3 parsing tests). + +- [ ] **Step 5: Add behavioral tests (READY + JSON discipline, warm array, failure)** + +Append to `BenchmarkHarnessTest.scala` inside the class, using a temp script/input and capturing an in-memory `PrintStream`: + +```scala + import java.io.{ ByteArrayOutputStream, File, PrintStream } + import java.nio.charset.StandardCharsets + import java.nio.file.Files + + private def tmp(suffix: String, content: String): File = { + val f = File.createTempFile("bench", suffix) + f.deleteOnExit() + Files.write(f.toPath, content.getBytes(StandardCharsets.UTF_8)) + f + } + + private def capture(fn: PrintStream => Unit): String = { + val buf = new ByteArrayOutputStream() + val ps = new PrintStream(buf, true, "UTF-8") + fn(ps) + new String(buf.toByteArray, StandardCharsets.UTF_8) + } + + "runColdFirst" - { + "emits READY then a single firstRunMs JSON line, output not on the stream" in { + val script = tmp(".dwl", "output application/json --- payload.a + 1") + val input = tmp(".json", "{\"a\": 41}") + val a = BenchArgs("coldfirst", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) + val sink = new ByteArrayOutputStream() + val stdout = capture(ps => BenchmarkHarness.runColdFirst(a, ps, sink)) + val lines = stdout.split("\n").filter(_.nonEmpty) + lines.head shouldBe "READY" + lines.last should include ("firstRunMs") + lines.count(_.contains("firstRunMs")) shouldBe 1 + // The transformed "42" went to the sink, NOT to stdout. + new String(sink.toByteArray, StandardCharsets.UTF_8).trim shouldBe "42" + } + } + + "runWarm" - { + "emits READY then a warmMs array of length iters" in { + val script = tmp(".dwl", "output application/json --- payload.a + 1") + val input = tmp(".json", "{\"a\": 41}") + val a = BenchArgs("warm", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 1, 3) + val stdout = capture(ps => BenchmarkHarness.runWarm(a, ps, new ByteArrayOutputStream())) + val json = stdout.split("\n").filter(_.contains("warmMs")).head + json should include ("warmMs") + // 3 comma-separated samples -> 2 commas inside the array + json.count(_ == ',') shouldBe 2 + } + } + + "a failing script throws (non-zero exit path)" in { + val script = tmp(".dwl", "output application/json --- payload.missing.deep.path()") + val input = tmp(".json", "{}") + val a = BenchArgs("coldfirst", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) + an [RuntimeException] should be thrownBy + BenchmarkHarness.runColdFirst(a, capturePs(), new ByteArrayOutputStream()) + } + + private def capturePs(): PrintStream = new PrintStream(new ByteArrayOutputStream(), true, "UTF-8") +``` + +- [ ] **Step 6: Run all harness tests to verify they pass** + +Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` +Expected: PASS (parsing + coldfirst + warm + failure). + +Note: if the `.dwl` script for the failure case does not actually throw, replace its body with one that reliably fails, e.g. `output application/json --- 1 / 0` — the intent is only that a failed run raises. + +- [ ] **Step 7: Commit** + +```bash +git add native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala \ + native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala +git commit -m "feat: add in-binary BenchmarkHarness for native-cli" +``` + +--- + +## Task 3: Gate the harness behind `BenchmarkMode.ENABLED` in `DWCLI` + +**Files:** +- Modify: `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java:32-34` (the `main` method) +- Modify: `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` (add the `ENABLED == false` guard) + +**Interfaces:** +- Consumes: `org.mule.weave.cli.BenchmarkMode.ENABLED` (Task 1), `org.mule.weave.dwnative.benchmark.BenchmarkHarness.main` (Task 2). +- Produces: no new public API; behavior — when `BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null`, `dw` dispatches to `BenchmarkHarness.main(args)` before picocli. Otherwise unchanged. + +The env-var name is `DW_BENCH` (specific, collision-unlikely). `ENABLED` is the real gate: in production it is a compile-time `false`, so `BenchmarkHarness` is unreachable and native-image drops it. + +- [ ] **Step 1: Write the guard test that production has ENABLED=false** + +Append to `BenchmarkHarnessTest.scala`: + +```scala + "BenchmarkMode.ENABLED" - { + "is false in a normal (non -Pbenchmark) build" in { + // Tests run without -Pbenchmark, so the generated constant must be false — + // proving the harness is dead code / stripped from a production image. + org.mule.weave.cli.BenchmarkMode.ENABLED shouldBe false + } + } +``` + +- [ ] **Step 2: Run to verify it fails to compile (constant not generated for test yet)** + +Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` +Expected: FAIL — `BenchmarkMode` symbol not found *unless* `genBenchmarkMode` ran. If it fails on the symbol, run `./gradlew native-cli:genBenchmarkMode` once, then re-run. Expected after generation: PASS for this guard (normal build → `false`). + +Note: `compileScala`/`compileJava` already `dependsOn genBenchmarkMode` (Task 1 Step 3), so the test compile generates it. If the IDE/test invocation skips it, the explicit `genBenchmarkMode` run resolves it. + +- [ ] **Step 3: Modify `DWCLI.main` to dispatch when gated** + +In `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java`, replace the `main` method (lines 32-34): + +```java + public static void main(String[] args) { + // Benchmark dispatch: only reachable in a build made with -Pbenchmark=true + // (BenchmarkMode.ENABLED is a compile-time false in production, so native-image + // folds this branch and BenchmarkHarness away). DW_BENCH selects the mode. + if (BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null) { + org.mule.weave.dwnative.benchmark.BenchmarkHarness.main(args); + return; + } + new DWCLI().run(args, DefaultConsole$.MODULE$); + } +``` + +- [ ] **Step 4: Run the full native-cli test suite to verify nothing regressed** + +Run: `./gradlew native-cli:test` +Expected: PASS, including the `ENABLED shouldBe false` guard. + +- [ ] **Step 5: Commit** + +```bash +git add native-cli/src/main/java/org/mule/weave/cli/DWCLI.java \ + native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala +git commit -m "feat: gate BenchmarkHarness dispatch behind BenchmarkMode.ENABLED + DW_BENCH" +``` + +--- + +## Task 4: Parent `locate.mjs` — resolve the bench-enabled `dw` binary + +**Files:** +- Create: `benchmarks/runners/cli/locate.mjs` +- Create: `benchmarks/runners/cli/locate.test.mjs` + +**Interfaces:** +- Produces: `export function locateBinary(): string` — returns an absolute path to the `dw` binary. Resolution order: `process.env.DW_BENCH_BIN` if set (used as-is), else `/native-cli/build/native/nativeCompile/dw` (`dw.exe` on Windows). Throws with a build hint if the resolved path does not exist. Consumed by Tasks 5 & 6. + +Mirror `benchmarks/runners/node/wrapper.mjs` (repo-root computation via `import.meta.url`, `existsSync` check, actionable error). + +- [ ] **Step 1: Write the failing test** + +Create `benchmarks/runners/cli/locate.test.mjs`: + +```javascript +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { locateBinary } from "./locate.mjs"; + +test("DW_BENCH_BIN override is returned as-is when it exists", () => { + // Point at a file guaranteed to exist: this test file itself. + const self = new URL(import.meta.url).pathname; + process.env.DW_BENCH_BIN = self; + try { + assert.equal(locateBinary(), self); + } finally { + delete process.env.DW_BENCH_BIN; + } +}); + +test("throws an actionable error when the binary is absent", () => { + process.env.DW_BENCH_BIN = "/nonexistent/dw-binary-xyz"; + try { + assert.throws(() => locateBinary(), /nativeCompile|not found|build/i); + } finally { + delete process.env.DW_BENCH_BIN; + } +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `node --test benchmarks/runners/cli/locate.test.mjs` +Expected: FAIL — cannot find module `./locate.mjs`. + +- [ ] **Step 3: Implement `locate.mjs`** + +Create `benchmarks/runners/cli/locate.mjs`: + +```javascript +import { existsSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +// benchmarks/runners/cli -> benchmarks/runners -> benchmarks -> repo root +const REPO_ROOT = join(__dirname, "..", "..", ".."); +const BIN_NAME = process.platform === "win32" ? "dw.exe" : "dw"; +const DEFAULT_BIN = join(REPO_ROOT, "native-cli", "build", "native", "nativeCompile", BIN_NAME); + +/** + * Resolve the benchmark-enabled `dw` native binary. Honors DW_BENCH_BIN (absolute + * path to a bench-built dw); otherwise the default nativeCompile output. The binary + * must be built with -Pbenchmark=true so BenchmarkHarness is reachable. + */ +export function locateBinary() { + const candidate = process.env.DW_BENCH_BIN || DEFAULT_BIN; + if (!existsSync(candidate)) { + throw new Error( + `dw benchmark binary not found at ${candidate}. ` + + `Build it with: ./gradlew native-cli:nativeCompile -Pbenchmark=true ` + + `(or set DW_BENCH_BIN to a bench-enabled dw).` + ); + } + return candidate; +} +``` + +- [ ] **Step 4: Run to verify it passes** + +Run: `node --test benchmarks/runners/cli/locate.test.mjs` +Expected: PASS (2 tests). + +- [ ] **Step 5: Commit** + +```bash +git add benchmarks/runners/cli/locate.mjs benchmarks/runners/cli/locate.test.mjs +git commit -m "feat: add cli runner binary locator" +``` + +--- + +## Task 5: Parent `coldstart.mjs` — cold-start + first-run rows + +**Files:** +- Create: `benchmarks/runners/cli/coldstart.mjs` + +**Interfaces:** +- Consumes: `locateBinary` (Task 4); shared libs `casesForMetric`, `resolveInputs` from `../../lib/manifest.mjs`; `computeStats` from `../../lib/stats.mjs`. +- Produces: `export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}): Promise>` — for each case declaring `cold-start` or `first-run`, spawns `dw` in `coldfirst` mode `n` times, stamps cold-start at spawn→`READY`, parses `firstRunMs`. Emits a `cold-start` row (unit `ms`) only for cases that declare it, and a `first-run` row only for cases that declare it. Consumed by Task 7. + +This is `benchmarks/runners/node/coldstart.mjs` with the child command swapped to the `dw` binary. Build the per-input arg `--input==::` using absolute corpus file paths. + +- [ ] **Step 1: Implement `coldstart.mjs`** + +Create `benchmarks/runners/cli/coldstart.mjs`: + +```javascript +import { spawn } from "node:child_process"; +import { join } from "node:path"; +import { casesForMetric } from "../../lib/manifest.mjs"; +import { computeStats } from "../../lib/stats.mjs"; +import { locateBinary } from "./locate.mjs"; + +/** Build `--input=name=file:mime:charset` args for a case (absolute paths). */ +function inputArgs(manifest, c) { + const args = []; + for (const [name, inp] of Object.entries(c.inputs ?? {})) { + const file = join(manifest.corpusDir, inp.file); + const charset = inp.charset ?? "utf-8"; + args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + } + return args; +} + +/** + * Spawn one fresh dw process in coldfirst mode. Cold-start = wall-clock from just + * before spawn to the child's "READY" marker (process launch + native image load + + * NativeRuntime init). first-run is timed in-process by the child. Rejects on a + * non-zero exit or a missing READY/JSON line so a failed sample never records a + * bogus timing. + */ +function sampleOnce(bin, manifest, c) { + const scriptPath = join(manifest.corpusDir, c.script); + const args = ["--bench-mode=coldfirst", `--script=${scriptPath}`, ...inputArgs(manifest, c)]; + return new Promise((resolve, reject) => { + const t0 = process.hrtime.bigint(); + const child = spawn(bin, args, { + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, DW_BENCH: "1" }, + }); + let coldStartMs; + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf-8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + if (coldStartMs === undefined && stdout.includes("READY\n")) { + coldStartMs = Number(process.hrtime.bigint() - t0) / 1e6; + } + }); + child.stderr.setEncoding("utf-8"); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.on("error", reject); + child.on("close", (code) => { + if (code !== 0) { + reject(new Error(`cli coldfirst failed for '${c.id}' (exit ${code})\n${stderr}`)); + return; + } + if (coldStartMs === undefined) { + reject(new Error(`cli coldfirst for '${c.id}' never printed READY\n${stderr}`)); + return; + } + const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); + if (!jsonLine) { + reject(new Error(`cli coldfirst for '${c.id}' printed no result line\n${stderr}`)); + return; + } + const { firstRunMs } = JSON.parse(jsonLine); + resolve({ coldStartMs, firstRunMs }); + }); + }); +} + +/** @returns {Promise>} */ +export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}) { + const bin = locateBinary(); + const rows = []; + const ids = new Set([ + ...casesForMetric(manifest, "cold-start").map((c) => c.id), + ...casesForMetric(manifest, "first-run").map((c) => c.id), + ]); + + for (const id of ids) { + const c = manifest.cases.find((x) => x.id === id); + const n = samplesOverride ?? c.iterations?.samples ?? 20; + const colds = []; + const firsts = []; + for (let i = 0; i < n; i++) { + const { coldStartMs, firstRunMs } = await sampleOnce(bin, manifest, c); + colds.push(coldStartMs); + firsts.push(firstRunMs); + } + if (c.metrics.includes("cold-start")) { + rows.push({ id, metric: "cold-start", unit: "ms", stats: computeStats(colds), iterations: n }); + } + if (c.metrics.includes("first-run")) { + rows.push({ id, metric: "first-run", unit: "ms", stats: computeStats(firsts), iterations: n }); + } + } + return rows; +} +``` + +- [ ] **Step 2: Sanity-check syntax (no binary needed)** + +Run: `node --check benchmarks/runners/cli/coldstart.mjs` +Expected: no output, exit 0. + +Note: an end-to-end run of this file requires a bench-built `dw` and is exercised by the smoke test in Task 8; there is no dwlib-free unit test for it (it spawns the binary), matching how `runners/node/coldstart.test.mjs` is excluded from the always-on JS parity set. + +- [ ] **Step 3: Commit** + +```bash +git add benchmarks/runners/cli/coldstart.mjs +git commit -m "feat: add cli runner cold-start + first-run sampler" +``` + +--- + +## Task 6: Parent `warm.mjs` — warm rows + +**Files:** +- Create: `benchmarks/runners/cli/warm.mjs` + +**Interfaces:** +- Consumes: `locateBinary` (Task 4); `casesForMetric` from `../../lib/manifest.mjs`; `computeStats` from `../../lib/stats.mjs`. +- Produces: `export async function runWarm(manifest): Promise>` — for each case declaring `warm`, spawns `dw` once in `warm` mode with `--warmup`/`--iters` from the case's `iterations`, reads back the `warmMs[]` array, and produces a `warm` row (unit `ms`). Consumed by Task 7. + +Reuse the same `inputArgs` shape as Task 5 (duplicated as a small local helper — the two samplers are independent and each is small; a shared module is not warranted by YAGNI, matching how the Node runner keeps `coldstart.mjs` and `warm-bench.mjs` separate). + +- [ ] **Step 1: Implement `warm.mjs`** + +Create `benchmarks/runners/cli/warm.mjs`: + +```javascript +import { spawn } from "node:child_process"; +import { join } from "node:path"; +import { casesForMetric } from "../../lib/manifest.mjs"; +import { computeStats } from "../../lib/stats.mjs"; +import { locateBinary } from "./locate.mjs"; + +function inputArgs(manifest, c) { + const args = []; + for (const [name, inp] of Object.entries(c.inputs ?? {})) { + const file = join(manifest.corpusDir, inp.file); + const charset = inp.charset ?? "utf-8"; + args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + } + return args; +} + +/** Spawn dw once in warm mode; resolve the parsed warmMs[] sample array. */ +function warmSamples(bin, manifest, c) { + const scriptPath = join(manifest.corpusDir, c.script); + const warmup = c.iterations?.warmup ?? 10; + const iters = c.iterations?.warm ?? 100; + const args = [ + "--bench-mode=warm", + `--script=${scriptPath}`, + `--warmup=${warmup}`, + `--iters=${iters}`, + ...inputArgs(manifest, c), + ]; + return new Promise((resolve, reject) => { + const child = spawn(bin, args, { + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, DW_BENCH: "1" }, + }); + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf-8"); + child.stdout.on("data", (chunk) => (stdout += chunk)); + child.stderr.setEncoding("utf-8"); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.on("error", reject); + child.on("close", (code) => { + if (code !== 0) { + reject(new Error(`cli warm failed for '${c.id}' (exit ${code})\n${stderr}`)); + return; + } + const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); + if (!jsonLine) { + reject(new Error(`cli warm for '${c.id}' printed no result line\n${stderr}`)); + return; + } + const { warmMs } = JSON.parse(jsonLine); + if (!Array.isArray(warmMs) || warmMs.length === 0) { + reject(new Error(`cli warm for '${c.id}' returned no samples\n${stderr}`)); + return; + } + resolve({ warmMs, iters }); + }); + }); +} + +/** @returns {Promise>} */ +export async function runWarm(manifest) { + const bin = locateBinary(); + const rows = []; + for (const c of casesForMetric(manifest, "warm")) { + const { warmMs, iters } = await warmSamples(bin, manifest, c); + rows.push({ id: c.id, metric: "warm", unit: "ms", stats: computeStats(warmMs), iterations: iters }); + } + return rows; +} +``` + +- [ ] **Step 2: Sanity-check syntax** + +Run: `node --check benchmarks/runners/cli/warm.mjs` +Expected: no output, exit 0. + +- [ ] **Step 3: Commit** + +```bash +git add benchmarks/runners/cli/warm.mjs +git commit -m "feat: add cli runner warm sampler" +``` + +--- + +## Task 7: Parent `emit.mjs` — assemble and write the result file + +**Files:** +- Create: `benchmarks/runners/cli/emit.mjs` +- Create: `benchmarks/runners/cli/emit.test.mjs` + +**Interfaces:** +- Consumes: `loadManifest`, `validateResultIds` from `../../lib/manifest.mjs`; `gatherEnv` from `../../lib/env.mjs`; `runColdStartAndFirstRun` (Task 5); `runWarm` (Task 6); `locateBinary` (Task 4, for the version probe). +- Produces: `export function buildResult(env, cases)` (schema-shaped object, identical contract to the Node runner's) and `export async function main(): Promise` (writes `results/cli-.json`, returns its path). Runner name `"cli"`. + +`runtimeVersion`: probe `dw --version` synchronously; take the first line, or fall back to `"dw"` if the probe fails. `dwlibBuildId` comes from `gatherEnv` but the CLI is not the staged dwlib — override it to `"n/a-cli"` after gathering. + +- [ ] **Step 1: Write the failing test for `buildResult`** + +Create `benchmarks/runners/cli/emit.test.mjs`: + +```javascript +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; +import { buildResult } from "./emit.mjs"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const CORPUS = join(__dirname, "..", "..", "corpus"); + +test("buildResult produces a schema-shaped object with runner 'cli'", () => { + const env = { + runner: "cli", os: "x", cpu: "y", runtimeVersion: "dw vX", + weaveVersion: "2.12.0-x", commit: "abc", dwlibBuildId: "n/a-cli", + }; + const cases = [{ id: "trivial", metric: "cold-start", unit: "ms", stats: { median: 1 }, iterations: 10 }]; + const r = buildResult(env, cases); + assert.equal(r.schemaVersion, "1.0"); + assert.equal(r.runner, "cli"); + assert.ok(typeof r.timestamp === "string"); + assert.deepEqual(r.cases, cases); +}); + +test("orphan ids are rejected before writing", () => { + const manifest = loadManifest(CORPUS); + assert.throws(() => validateResultIds(manifest, [{ id: "totally-made-up" }]), /orphan id/); +}); +``` + +- [ ] **Step 2: Run to verify it fails** + +Run: `node --test benchmarks/runners/cli/emit.test.mjs` +Expected: FAIL — cannot find module `./emit.mjs`. + +- [ ] **Step 3: Implement `emit.mjs`** + +Create `benchmarks/runners/cli/emit.mjs`: + +```javascript +import { writeFileSync, mkdirSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { join, dirname } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; +import { gatherEnv } from "../../lib/env.mjs"; +import { locateBinary } from "./locate.mjs"; +import { runColdStartAndFirstRun } from "./coldstart.mjs"; +import { runWarm } from "./warm.mjs"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const CORPUS = join(__dirname, "..", "..", "corpus"); +const RESULTS_DIR = join(__dirname, "..", "..", "results"); + +/** Assemble the full schema object (identical contract to the Node runner). */ +export function buildResult(env, cases) { + return { + schemaVersion: "1.0", + runner: env.runner, + env, + timestamp: new Date().toISOString(), + cases, + }; +} + +/** Best-effort `dw --version` first line; falls back to "dw". */ +function probeVersion(bin) { + try { + const out = execFileSync(bin, ["--version"], { encoding: "utf-8" }); + const line = out.split("\n").map((l) => l.trim()).filter(Boolean)[0]; + return line ? `dw ${line}` : "dw"; + } catch { + return "dw"; + } +} + +export async function main() { + const manifest = loadManifest(CORPUS); + const bin = locateBinary(); + const env = gatherEnv({ runner: "cli", runtimeVersion: probeVersion(bin) }); + // The CLI is a native binary, not the staged dwlib — override the lib fingerprint. + env.dwlibBuildId = "n/a-cli"; + + const coldRows = await runColdStartAndFirstRun(manifest); + const warmRows = await runWarm(manifest); + + const cases = [...coldRows, ...warmRows]; + validateResultIds(manifest, cases); + + mkdirSync(RESULTS_DIR, { recursive: true }); + const stamp = new Date().toISOString().replace(/[:.]/g, "-"); + const outPath = join(RESULTS_DIR, `cli-${stamp}.json`); + writeFileSync(outPath, JSON.stringify(buildResult(env, cases), null, 2)); + console.log(`wrote ${outPath} (${cases.length} rows)`); + return outPath; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((e) => { + console.error(e.message); + process.exit(1); + }); +} +``` + +- [ ] **Step 4: Run to verify the test passes** + +Run: `node --test benchmarks/runners/cli/emit.test.mjs` +Expected: PASS (2 tests). + +- [ ] **Step 5: Commit** + +```bash +git add benchmarks/runners/cli/emit.mjs benchmarks/runners/cli/emit.test.mjs +git commit -m "feat: add cli runner emit entrypoint" +``` + +--- + +## Task 8: Gradle `benchmarkCli` runner task + JS test wiring + +**Files:** +- Modify: `native-cli/build.gradle` (add `benchmarkCli` task) +- Modify: `native-lib/build.gradle:307-320` (add cli test files to `benchmarkJsUnitTest`) + +**Interfaces:** +- Consumes: `native-cli:nativeCompile` (must be invoked with `-Pbenchmark=true` so the harness is present); the shared `node corpus/gen-inputs.mjs`; `benchmarks/runners/cli/emit.mjs` (Task 7). +- Produces: a Gradle task `benchmarkCli` tagged `ext.benchmarkRunner = true`, discovered by the root `benchmarkCompare`. Writes `benchmarks/results/cli-.json`; does NOT render the report. + +- [ ] **Step 1: Add the `benchmarkCli` task to `native-cli/build.gradle`** + +Append to `native-cli/build.gradle`: + +```groovy +// The CLI runner as an aggregator-registered runner: emits its result file but +// does NOT render the report (the root :benchmarkCompare renders once over all +// runners). Tagged `benchmarkRunner` so :benchmarkCompare discovers it automatically. +// Requires the bench-enabled binary — nativeCompile must run with -Pbenchmark=true so +// BenchmarkMode.ENABLED is true and BenchmarkHarness is reachable in dw. +tasks.register('benchmarkCli', Exec) { + onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } + ext.benchmarkRunner = true + + dependsOn tasks.named('nativeCompile') + workingDir("${rootDir}/benchmarks") + + def script = 'node corpus/gen-inputs.mjs && node runners/cli/emit.mjs' + if (System.getProperty('os.name').toLowerCase().contains('windows')) { + commandLine('cmd', '/c', script) + } else { + commandLine('bash', '-c', script) + } +} +``` + +- [ ] **Step 2: Add the cli JS tests to the always-on parity set** + +In `native-lib/build.gradle`, in `benchmarkJsUnitTest` (lines 307-320), extend the `files` string to include the cli runner's dwlib-free tests: + +```groovy + def files = 'lib/stats.test.mjs lib/manifest.test.mjs lib/env.test.mjs ' + + 'report/report.test.mjs runners/node/emit.test.mjs ' + + 'runners/cli/locate.test.mjs runners/cli/emit.test.mjs' +``` + +- [ ] **Step 3: Verify the JS parity tests pass (no binary needed)** + +Run: `./gradlew native-lib:benchmarkJsUnitTest` +Expected: PASS — includes `runners/cli/locate.test.mjs` and `runners/cli/emit.test.mjs`. + +- [ ] **Step 4: Verify `benchmarkCli` is discovered but skipped without the opt-in** + +Run: `./gradlew native-cli:benchmarkCli` +Expected: task is SKIPPED (the `onlyIf` is false without `-Pbenchmark=true`), build succeeds. + +- [ ] **Step 5: Commit** + +```bash +git add native-cli/build.gradle native-lib/build.gradle +git commit -m "build: register benchmarkCli runner + wire cli JS parity tests" +``` + +--- + +## Task 9: End-to-end smoke test + README + +**Files:** +- Modify: `benchmarks/README.md` + +**Interfaces:** +- Consumes: everything above. This task validates the full path once against a real bench-built binary, then documents the runner. + +The end-to-end run needs a GraalVM toolchain (per the repo README/CLAUDE.md). It is a manual verification gate, not an automated test in `build`. + +- [ ] **Step 1: Build the bench-enabled binary** + +Run: `./gradlew native-cli:nativeCompile -Pbenchmark=true` +Expected: produces `native-cli/build/native/nativeCompile/dw`. (Several minutes; needs `GRAALVM_HOME`/`JAVA_HOME` set to a GraalVM with `native-image`, per CLAUDE.md.) + +- [ ] **Step 2: Smoke-run one cold-start sample directly against the binary** + +Run: +```bash +node -e ' +import("./benchmarks/runners/cli/coldstart.mjs").then(async (m) => { + const { loadManifest } = await import("./benchmarks/lib/manifest.mjs"); + const manifest = loadManifest("./benchmarks/corpus"); + const rows = await m.runColdStartAndFirstRun(manifest, { samplesOverride: 2 }); + const cold = rows.filter(r => r.metric === "cold-start"); + const first = rows.filter(r => r.metric === "first-run"); + if (cold.length < 1 || first.length < 1) { console.error("missing rows"); process.exit(1); } + for (const r of [...cold, ...first]) { + if (!(r.stats.median > 0)) { console.error("non-positive median", r); process.exit(1); } + } + console.log("smoke OK:", cold.length, "cold,", first.length, "first rows"); +}); +' +``` +Expected: prints `smoke OK: N cold, M first rows`; a positive `cold-start` (spawn→READY) and `firstRunMs` for each sampled case. (Uses `samplesOverride: 2` to stay fast.) + +- [ ] **Step 3: Run the full cross-runner comparison including cli** + +Run: `./gradlew benchmarkCompare -Pbenchmark=true` +Expected: the printed table includes a `cli` column with `cold-start`, `first-run`, and `warm` rows populated, `streaming` rows blank (`—`) for the cli column, and a `Δ cli vs ` column. + +- [ ] **Step 4: Document the runner in `benchmarks/README.md`** + +In `benchmarks/README.md`: + +Under **Layout** (after the `runners/python/` sentence, ~line 15), add: +``` + `runners/cli/` is the CLI runner: a Node parent that spawns the `dw` native + binary (built with `-Pbenchmark=true`, which compiles in an in-binary + benchmark harness gated by `BenchmarkMode.ENABLED` and dispatched via the + `DW_BENCH` env var — the shipped `dw` contains none of it). It emits + `cold-start`, `first-run`, and `warm`; it does **not** emit `streaming` + (the `dw run` path has no chunked-input FFI like the library's). +``` + +Under **Single-runner options** (~line 52), add: +``` + ./gradlew native-cli:benchmarkCli -Pbenchmark=true # CLI only: writes results/cli-.json +``` +and note its prerequisite: +``` +The **CLI runner** requires the bench-enabled binary +(`./gradlew native-cli:nativeCompile -Pbenchmark=true`); set `DW_BENCH_BIN` to +point at a prebuilt one. Like the library runners it needs the GraalVM toolchain. +``` + +- [ ] **Step 5: Commit** + +```bash +git add benchmarks/README.md +git commit -m "docs: document the CLI benchmark runner" +``` + +--- + +## Self-Review + +**Spec coverage:** +- Native binary measured, not JVM entrypoint → Tasks 2, 8 (spawns `dw`). ✓ +- READY-marker protocol, cold-start + first-run + warm → Tasks 2 (harness), 5 (cold/first), 6 (warm). ✓ +- Build-time gate, stripped from production → Tasks 1 (`BenchmarkMode`), 3 (`ENABLED &&` dispatch + guard test). ✓ +- Corpus only, no streaming → no manifest edit; cli emits only cold/first/warm (Tasks 5–7); README documents the gap (Task 9). ✓ +- Corpus-agnostic harness (no manifest knowledge in native-cli) → Task 2 takes file-path args; parent resolves corpus (Tasks 5–7). ✓ +- Runner registration contract (result file + `ext.benchmarkRunner`, no `benchmarkCompare` edit) → Task 8. ✓ +- `runner: "cli"`, `dwlibBuildId: "n/a-cli"`, `runtimeVersion` from `dw --version` → Task 7. ✓ +- Error handling (non-zero exit / missing READY / missing JSON / failed run) → Tasks 2 (harness throws), 5 & 6 (parent rejects). ✓ +- Output discipline (only READY + one JSON line; output to discard stream) → Task 2 + its behavioral test. ✓ +- Testing: Scala harness + ENABLED guard (Tasks 2, 3); dwlib-free JS in parity set (Tasks 4, 7, 8); smoke (Task 9). ✓ +- README update → Task 9. ✓ + +**Placeholder scan:** No TBD/TODO/"handle edge cases"; every code step shows full code; every command has expected output. ✓ + +**Type consistency:** `BenchArgs`/`BenchInput`/`parseArgs`/`runColdFirst`/`runWarm`/`main` (Task 2) are used consistently in Task 3's dispatch and Task 2's tests. `runColdStartAndFirstRun(manifest, {samplesOverride})` (Task 5) and `runWarm(manifest)` (Task 6) match their calls in `emit.mjs` (Task 7) and the smoke test (Task 9). `locateBinary()` (Task 4) is imported by Tasks 5, 6, 7. `buildResult(env, cases)` (Task 7) matches its test. `--input=name=file:mime:charset` arg format is identical between the parser (Task 2) and both parent samplers (Tasks 5, 6). `BenchmarkMode.ENABLED` (Task 1) matches its use in Task 3 and the guard test. ✓ From c0a9ef09eecd1f5750e4f3a7eba0be22dc1bb76f Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:28:00 -0300 Subject: [PATCH 03/15] build: generate BenchmarkMode.ENABLED gate for native-cli --- native-cli/build.gradle | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/native-cli/build.gradle b/native-cli/build.gradle index 5ce3b5e..4ac56a9 100644 --- a/native-cli/build.gradle +++ b/native-cli/build.gradle @@ -9,6 +9,9 @@ sourceSets { scala { srcDirs = ['src/main/scala', 'build/genresource'] } + java { + srcDirs += 'build/genjava' + } } } @@ -66,11 +69,38 @@ task genVersions() { outputPrinter.close() } +def genJavaDirectory = new File("$project.buildDir/genjava") + +task genBenchmarkMode() { + def enabled = project.findProperty('benchmark')?.toString()?.toBoolean() == true + def benchmarkMode = new File(genJavaDirectory, "org/mule/weave/cli/BenchmarkMode.java") + def parentFile = benchmarkMode.getParentFile() + if (!parentFile.exists()) { + parentFile.mkdirs() + } + final PrintWriter outputPrinter = new PrintWriter(new FileWriter(benchmarkMode)) + outputPrinter.println("package org.mule.weave.cli;") + outputPrinter.println() + outputPrinter.println("// GENERATED by genBenchmarkMode — do not edit.") + outputPrinter.println("// ENABLED is true only when built with -Pbenchmark=true; native-image") + outputPrinter.println("// folds the benchmark branch away as dead code when this is false.") + outputPrinter.println("public final class BenchmarkMode {") + outputPrinter.println(" private BenchmarkMode() {}") + outputPrinter.println(" public static final boolean ENABLED = " + enabled + ";") + outputPrinter.println("}") + outputPrinter.close() +} + defaultTasks += genVersions compileScala { dependsOn genVersions + dependsOn genBenchmarkMode +} + +compileJava { + dependsOn genBenchmarkMode } // Merging Service Files From 130e1b5ff384cf5043e8442cb81ad6cb6721bbcb Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:32:52 -0300 Subject: [PATCH 04/15] feat: add in-binary BenchmarkHarness for native-cli --- .../dwnative/benchmark/BenchmarkHarness.scala | 127 ++++++++++++++++++ .../benchmark/BenchmarkHarnessTest.scala | 96 +++++++++++++ 2 files changed, 223 insertions(+) create mode 100644 native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala create mode 100644 native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala new file mode 100644 index 0000000..5d80f92 --- /dev/null +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala @@ -0,0 +1,127 @@ +package org.mule.weave.dwnative.benchmark + +import org.mule.weave.dwnative.NativeRuntime +import org.mule.weave.dwnative.WeaveExecutionResult +import org.mule.weave.dwnative.cli.DefaultConsole +import org.mule.weave.dwnative.utils.DataWeaveUtils +import org.mule.weave.v2.runtime.BindingValue +import org.mule.weave.v2.runtime.ScriptingBindings + +import java.io.{ File, OutputStream, PrintStream } +import java.nio.charset.Charset +import java.nio.file.Files + +final case class BenchInput(name: String, file: String, mimeType: String, charset: String) +final case class BenchArgs(mode: String, scriptFile: String, inputs: Seq[BenchInput], warmup: Int, iters: Int) + +/** Corpus-agnostic in-binary benchmark harness. Reachable only in a build made with + * -Pbenchmark=true (guarded by BenchmarkMode.ENABLED in DWCLI); native-image folds it + * out of a production dw. Prints "READY" the instant one NativeRuntime is constructed, + * then a single JSON line of timings. Parent (benchmarks/runners/cli) measures cold-start + * as spawn->READY wall-clock. */ +object BenchmarkHarness { + + /** Discards bytes; used as the transform write sink so we never touch real stdout. */ + private final class DiscardStream extends OutputStream { + override def write(b: Int): Unit = () + override def write(b: Array[Byte]): Unit = () + override def write(b: Array[Byte], off: Int, len: Int): Unit = () + } + + private def nowNs(): Long = System.nanoTime() + private def msSince(startNs: Long): Double = (System.nanoTime() - startNs) / 1e6 + + def parseArgs(args: Array[String]): BenchArgs = { + var mode = "" + var script = "" + val inputs = scala.collection.mutable.ArrayBuffer[BenchInput]() + var warmup = 0 + var iters = 100 + args.foreach { arg => + val eq = arg.indexOf('=') + val key = if (eq >= 0) arg.substring(0, eq) else arg + val value = if (eq >= 0) arg.substring(eq + 1) else "" + key match { + case "--bench-mode" => mode = value + case "--script" => script = value + case "--warmup" => warmup = value.toInt + case "--iters" => iters = value.toInt + case "--input" => + // value = =:[:] + val nameSep = value.indexOf('=') + val name = value.substring(0, nameSep) + val rest = value.substring(nameSep + 1) + val parts = rest.split(":", 3) + val file = parts(0) + val mimeType = parts(1) + val charset = if (parts.length > 2 && parts(2).nonEmpty) parts(2) else "utf-8" + inputs += BenchInput(name, file, mimeType, charset) + case _ => throw new RuntimeException(s"unknown bench arg: $arg") + } + } + if (mode.isEmpty) throw new RuntimeException("--bench-mode is required") + if (script.isEmpty) throw new RuntimeException("--script is required") + BenchArgs(mode, script, inputs.toSeq, warmup, iters) + } + + private def newRuntime(): NativeRuntime = { + val console = DefaultConsole.enableSilent() + val utils = new DataWeaveUtils(console) + new NativeRuntime(utils.getLibPathHome(), Array.empty[File], console, None) + } + + private def readScript(a: BenchArgs): String = + new String(Files.readAllBytes(new File(a.scriptFile).toPath), java.nio.charset.StandardCharsets.UTF_8) + + private def bindings(a: BenchArgs): ScriptingBindings = { + val b = new ScriptingBindings() + a.inputs.foreach { in => + val bytes = Files.readAllBytes(new File(in.file).toPath) + val bv = new BindingValue(bytes, Some(in.mimeType), Map.empty[String, Any], Charset.forName(in.charset)) + b.addBinding(in.name, bv) + } + b + } + + private def assertOk(r: WeaveExecutionResult): Unit = + if (!r.success()) throw new RuntimeException("run failed: " + r.result()) + + def runColdFirst(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { + val script = readScript(a) + val b = bindings(a) + val rt = newRuntime() // engine init — measured externally as cold-start + out.println("READY"); out.flush() + val start = nowNs() + assertOk(rt.run(script, "bench", b, sink, "application/json", None)) + val firstRunMs = msSince(start) + out.println("{\"firstRunMs\":" + firstRunMs + "}") + } + + def runWarm(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { + val script = readScript(a) + val b = bindings(a) + val rt = newRuntime() + out.println("READY"); out.flush() + var i = 0 + while (i < a.warmup) { assertOk(rt.run(script, "bench", b, sink, "application/json", None)); i += 1 } + val samples = new Array[Double](a.iters) + i = 0 + while (i < a.iters) { + val start = nowNs() + assertOk(rt.run(script, "bench", b, sink, "application/json", None)) + samples(i) = msSince(start) + i += 1 + } + out.println("{\"warmMs\":[" + samples.mkString(",") + "]}") + } + + def main(args: Array[String]): Unit = { + val a = parseArgs(args) + val sink = new DiscardStream() + a.mode match { + case "coldfirst" => runColdFirst(a, System.out, sink) + case "warm" => runWarm(a, System.out, sink) + case other => throw new RuntimeException(s"unknown --bench-mode: $other") + } + } +} diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala new file mode 100644 index 0000000..99e3b05 --- /dev/null +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala @@ -0,0 +1,96 @@ +package org.mule.weave.dwnative.benchmark + +import org.scalatest.freespec.AnyFreeSpec +import org.scalatest.matchers.should.Matchers + +import java.io.{ ByteArrayOutputStream, File, PrintStream } +import java.nio.charset.StandardCharsets +import java.nio.file.Files + +class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { + + private def tmp(suffix: String, content: String): File = { + val f = File.createTempFile("bench", suffix) + f.deleteOnExit() + Files.write(f.toPath, content.getBytes(StandardCharsets.UTF_8)) + f + } + + private def capture(fn: PrintStream => Unit): String = { + val buf = new ByteArrayOutputStream() + val ps = new PrintStream(buf, true, "UTF-8") + fn(ps) + new String(buf.toByteArray, StandardCharsets.UTF_8) + } + + "parseArgs" - { + "parses coldfirst mode with one input" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=coldfirst", + "--script=/tmp/x.dwl", + "--input=payload=/tmp/p.json:application/json:utf-8")) + a.mode shouldBe "coldfirst" + a.scriptFile shouldBe "/tmp/x.dwl" + a.inputs should have size 1 + a.inputs.head shouldBe BenchInput("payload", "/tmp/p.json", "application/json", "utf-8") + } + + "parses warm mode with warmup and iters" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=warm", "--script=/tmp/x.dwl", "--warmup=5", "--iters=30")) + a.mode shouldBe "warm" + a.warmup shouldBe 5 + a.iters shouldBe 30 + a.inputs shouldBe empty + } + + "handles a mimeType-only input (charset defaults to utf-8)" in { + val a = BenchmarkHarness.parseArgs(Array( + "--bench-mode=coldfirst", "--script=/tmp/x.dwl", + "--input=payload=/tmp/p.json:application/json")) + a.inputs.head.charset shouldBe "utf-8" + } + } + + "runColdFirst" - { + "emits READY then a single firstRunMs JSON line, output not on the stream" in { + val script = tmp(".dwl", "output application/json --- payload.a + 1") + val input = tmp(".json", "{\"a\": 41}") + val a = BenchArgs("coldfirst", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) + val sink = new ByteArrayOutputStream() + val stdout = capture(ps => BenchmarkHarness.runColdFirst(a, ps, sink)) + val lines = stdout.split("\n").filter(_.nonEmpty) + lines.head shouldBe "READY" + lines.last should include ("firstRunMs") + lines.count(_.contains("firstRunMs")) shouldBe 1 + // The transformed "42" went to the sink, NOT to stdout. + new String(sink.toByteArray, StandardCharsets.UTF_8).trim shouldBe "42" + } + } + + "runWarm" - { + "emits READY then a warmMs array of length iters" in { + val script = tmp(".dwl", "output application/json --- payload.a + 1") + val input = tmp(".json", "{\"a\": 41}") + val a = BenchArgs("warm", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 1, 3) + val stdout = capture(ps => BenchmarkHarness.runWarm(a, ps, new ByteArrayOutputStream())) + val json = stdout.split("\n").filter(_.contains("warmMs")).head + json should include ("warmMs") + // 3 comma-separated samples -> 2 commas inside the array + json.count(_ == ',') shouldBe 2 + } + } + + "a failing script throws (non-zero exit path)" in { + val script = tmp(".dwl", "output application/json --- 1 / 0") + val input = tmp(".json", "{}") + val a = BenchArgs("coldfirst", script.getAbsolutePath, + Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) + an [RuntimeException] should be thrownBy + BenchmarkHarness.runColdFirst(a, capturePs(), new ByteArrayOutputStream()) + } + + private def capturePs(): PrintStream = new PrintStream(new ByteArrayOutputStream(), true, "UTF-8") +} From bbc93d75d9fdedd9d379bbff1d17c6a349cc58b0 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:37:31 -0300 Subject: [PATCH 05/15] feat: gate BenchmarkHarness dispatch behind BenchmarkMode.ENABLED + DW_BENCH --- native-cli/src/main/java/org/mule/weave/cli/DWCLI.java | 7 +++++++ .../weave/dwnative/benchmark/BenchmarkHarnessTest.scala | 8 ++++++++ 2 files changed, 15 insertions(+) diff --git a/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java b/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java index ee3bd42..8387aad 100644 --- a/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java +++ b/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java @@ -30,6 +30,13 @@ public class DWCLI { public static void main(String[] args) { + // Benchmark dispatch: only reachable in a build made with -Pbenchmark=true + // (BenchmarkMode.ENABLED is a compile-time false in production, so native-image + // folds this branch and BenchmarkHarness away). DW_BENCH selects the mode. + if (BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null) { + org.mule.weave.dwnative.benchmark.BenchmarkHarness.main(args); + return; + } new DWCLI().run(args, DefaultConsole$.MODULE$); } diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala index 99e3b05..41a4c23 100644 --- a/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala @@ -93,4 +93,12 @@ class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { } private def capturePs(): PrintStream = new PrintStream(new ByteArrayOutputStream(), true, "UTF-8") + + "BenchmarkMode.ENABLED" - { + "is false in a normal (non -Pbenchmark) build" in { + // Tests run without -Pbenchmark, so the generated constant must be false — + // proving the harness is dead code / stripped from a production image. + org.mule.weave.cli.BenchmarkMode.ENABLED shouldBe false + } + } } From c168942cdb697dbb7aac2029099057cf4f20242f Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:41:06 -0300 Subject: [PATCH 06/15] feat: add cli runner binary locator --- benchmarks/runners/cli/locate.mjs | 26 ++++++++++++++++++++++++++ benchmarks/runners/cli/locate.test.mjs | 23 +++++++++++++++++++++++ 2 files changed, 49 insertions(+) create mode 100644 benchmarks/runners/cli/locate.mjs create mode 100644 benchmarks/runners/cli/locate.test.mjs diff --git a/benchmarks/runners/cli/locate.mjs b/benchmarks/runners/cli/locate.mjs new file mode 100644 index 0000000..d811207 --- /dev/null +++ b/benchmarks/runners/cli/locate.mjs @@ -0,0 +1,26 @@ +import { existsSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +// benchmarks/runners/cli -> benchmarks/runners -> benchmarks -> repo root +const REPO_ROOT = join(__dirname, "..", "..", ".."); +const BIN_NAME = process.platform === "win32" ? "dw.exe" : "dw"; +const DEFAULT_BIN = join(REPO_ROOT, "native-cli", "build", "native", "nativeCompile", BIN_NAME); + +/** + * Resolve the benchmark-enabled `dw` native binary. Honors DW_BENCH_BIN (absolute + * path to a bench-built dw); otherwise the default nativeCompile output. The binary + * must be built with -Pbenchmark=true so BenchmarkHarness is reachable. + */ +export function locateBinary() { + const candidate = process.env.DW_BENCH_BIN || DEFAULT_BIN; + if (!existsSync(candidate)) { + throw new Error( + `dw benchmark binary not found at ${candidate}. ` + + `Build it with: ./gradlew native-cli:nativeCompile -Pbenchmark=true ` + + `(or set DW_BENCH_BIN to a bench-enabled dw).` + ); + } + return candidate; +} diff --git a/benchmarks/runners/cli/locate.test.mjs b/benchmarks/runners/cli/locate.test.mjs new file mode 100644 index 0000000..8bc8c3f --- /dev/null +++ b/benchmarks/runners/cli/locate.test.mjs @@ -0,0 +1,23 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { locateBinary } from "./locate.mjs"; + +test("DW_BENCH_BIN override is returned as-is when it exists", () => { + // Point at a file guaranteed to exist: this test file itself. + const self = new URL(import.meta.url).pathname; + process.env.DW_BENCH_BIN = self; + try { + assert.equal(locateBinary(), self); + } finally { + delete process.env.DW_BENCH_BIN; + } +}); + +test("throws an actionable error when the binary is absent", () => { + process.env.DW_BENCH_BIN = "/nonexistent/dw-binary-xyz"; + try { + assert.throws(() => locateBinary(), /nativeCompile|not found|build/i); + } finally { + delete process.env.DW_BENCH_BIN; + } +}); From 2dae1f1c948b0d16213a0d14ca94236595d3b2c2 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 11:44:55 -0300 Subject: [PATCH 07/15] feat: add cli runner cold-start + first-run sampler --- benchmarks/runners/cli/coldstart.mjs | 94 ++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 benchmarks/runners/cli/coldstart.mjs diff --git a/benchmarks/runners/cli/coldstart.mjs b/benchmarks/runners/cli/coldstart.mjs new file mode 100644 index 0000000..909fe2a --- /dev/null +++ b/benchmarks/runners/cli/coldstart.mjs @@ -0,0 +1,94 @@ +import { spawn } from "node:child_process"; +import { join } from "node:path"; +import { casesForMetric } from "../../lib/manifest.mjs"; +import { computeStats } from "../../lib/stats.mjs"; +import { locateBinary } from "./locate.mjs"; + +/** Build `--input=name=file:mime:charset` args for a case (absolute paths). */ +function inputArgs(manifest, c) { + const args = []; + for (const [name, inp] of Object.entries(c.inputs ?? {})) { + const file = join(manifest.corpusDir, inp.file); + const charset = inp.charset ?? "utf-8"; + args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + } + return args; +} + +/** + * Spawn one fresh dw process in coldfirst mode. Cold-start = wall-clock from just + * before spawn to the child's "READY" marker (process launch + native image load + + * NativeRuntime init). first-run is timed in-process by the child. Rejects on a + * non-zero exit or a missing READY/JSON line so a failed sample never records a + * bogus timing. + */ +function sampleOnce(bin, manifest, c) { + const scriptPath = join(manifest.corpusDir, c.script); + const args = ["--bench-mode=coldfirst", `--script=${scriptPath}`, ...inputArgs(manifest, c)]; + return new Promise((resolve, reject) => { + const t0 = process.hrtime.bigint(); + const child = spawn(bin, args, { + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, DW_BENCH: "1" }, + }); + let coldStartMs; + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf-8"); + child.stdout.on("data", (chunk) => { + stdout += chunk; + if (coldStartMs === undefined && stdout.includes("READY\n")) { + coldStartMs = Number(process.hrtime.bigint() - t0) / 1e6; + } + }); + child.stderr.setEncoding("utf-8"); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.on("error", reject); + child.on("close", (code) => { + if (code !== 0) { + reject(new Error(`cli coldfirst failed for '${c.id}' (exit ${code})\n${stderr}`)); + return; + } + if (coldStartMs === undefined) { + reject(new Error(`cli coldfirst for '${c.id}' never printed READY\n${stderr}`)); + return; + } + const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); + if (!jsonLine) { + reject(new Error(`cli coldfirst for '${c.id}' printed no result line\n${stderr}`)); + return; + } + const { firstRunMs } = JSON.parse(jsonLine); + resolve({ coldStartMs, firstRunMs }); + }); + }); +} + +/** @returns {Promise>} */ +export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}) { + const bin = locateBinary(); + const rows = []; + const ids = new Set([ + ...casesForMetric(manifest, "cold-start").map((c) => c.id), + ...casesForMetric(manifest, "first-run").map((c) => c.id), + ]); + + for (const id of ids) { + const c = manifest.cases.find((x) => x.id === id); + const n = samplesOverride ?? c.iterations?.samples ?? 20; + const colds = []; + const firsts = []; + for (let i = 0; i < n; i++) { + const { coldStartMs, firstRunMs } = await sampleOnce(bin, manifest, c); + colds.push(coldStartMs); + firsts.push(firstRunMs); + } + if (c.metrics.includes("cold-start")) { + rows.push({ id, metric: "cold-start", unit: "ms", stats: computeStats(colds), iterations: n }); + } + if (c.metrics.includes("first-run")) { + rows.push({ id, metric: "first-run", unit: "ms", stats: computeStats(firsts), iterations: n }); + } + } + return rows; +} From 9226f1ed324170b6bfe6d1d95d361d357be4e40b Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 12:03:21 -0300 Subject: [PATCH 08/15] feat: add cli runner warm sampler --- benchmarks/runners/cli/warm.mjs | 70 +++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) create mode 100644 benchmarks/runners/cli/warm.mjs diff --git a/benchmarks/runners/cli/warm.mjs b/benchmarks/runners/cli/warm.mjs new file mode 100644 index 0000000..b5bed3c --- /dev/null +++ b/benchmarks/runners/cli/warm.mjs @@ -0,0 +1,70 @@ +import { spawn } from "node:child_process"; +import { join } from "node:path"; +import { casesForMetric } from "../../lib/manifest.mjs"; +import { computeStats } from "../../lib/stats.mjs"; +import { locateBinary } from "./locate.mjs"; + +function inputArgs(manifest, c) { + const args = []; + for (const [name, inp] of Object.entries(c.inputs ?? {})) { + const file = join(manifest.corpusDir, inp.file); + const charset = inp.charset ?? "utf-8"; + args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + } + return args; +} + +/** Spawn dw once in warm mode; resolve the parsed warmMs[] sample array. */ +function warmSamples(bin, manifest, c) { + const scriptPath = join(manifest.corpusDir, c.script); + const warmup = c.iterations?.warmup ?? 10; + const iters = c.iterations?.warm ?? 100; + const args = [ + "--bench-mode=warm", + `--script=${scriptPath}`, + `--warmup=${warmup}`, + `--iters=${iters}`, + ...inputArgs(manifest, c), + ]; + return new Promise((resolve, reject) => { + const child = spawn(bin, args, { + stdio: ["ignore", "pipe", "pipe"], + env: { ...process.env, DW_BENCH: "1" }, + }); + let stdout = ""; + let stderr = ""; + child.stdout.setEncoding("utf-8"); + child.stdout.on("data", (chunk) => (stdout += chunk)); + child.stderr.setEncoding("utf-8"); + child.stderr.on("data", (chunk) => (stderr += chunk)); + child.on("error", reject); + child.on("close", (code) => { + if (code !== 0) { + reject(new Error(`cli warm failed for '${c.id}' (exit ${code})\n${stderr}`)); + return; + } + const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); + if (!jsonLine) { + reject(new Error(`cli warm for '${c.id}' printed no result line\n${stderr}`)); + return; + } + const { warmMs } = JSON.parse(jsonLine); + if (!Array.isArray(warmMs) || warmMs.length === 0) { + reject(new Error(`cli warm for '${c.id}' returned no samples\n${stderr}`)); + return; + } + resolve({ warmMs, iters }); + }); + }); +} + +/** @returns {Promise>} */ +export async function runWarm(manifest) { + const bin = locateBinary(); + const rows = []; + for (const c of casesForMetric(manifest, "warm")) { + const { warmMs, iters } = await warmSamples(bin, manifest, c); + rows.push({ id: c.id, metric: "warm", unit: "ms", stats: computeStats(warmMs), iterations: iters }); + } + return rows; +} From b35b9646ac0d6cc46107a3fab2dc78adc6f9c5c4 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 12:06:25 -0300 Subject: [PATCH 09/15] feat: add cli runner emit entrypoint --- benchmarks/runners/cli/emit.mjs | 63 ++++++++++++++++++++++++++++ benchmarks/runners/cli/emit.test.mjs | 27 ++++++++++++ 2 files changed, 90 insertions(+) create mode 100644 benchmarks/runners/cli/emit.mjs create mode 100644 benchmarks/runners/cli/emit.test.mjs diff --git a/benchmarks/runners/cli/emit.mjs b/benchmarks/runners/cli/emit.mjs new file mode 100644 index 0000000..699667d --- /dev/null +++ b/benchmarks/runners/cli/emit.mjs @@ -0,0 +1,63 @@ +import { writeFileSync, mkdirSync } from "node:fs"; +import { execFileSync } from "node:child_process"; +import { join, dirname } from "node:path"; +import { fileURLToPath, pathToFileURL } from "node:url"; +import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; +import { gatherEnv } from "../../lib/env.mjs"; +import { locateBinary } from "./locate.mjs"; +import { runColdStartAndFirstRun } from "./coldstart.mjs"; +import { runWarm } from "./warm.mjs"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const CORPUS = join(__dirname, "..", "..", "corpus"); +const RESULTS_DIR = join(__dirname, "..", "..", "results"); + +/** Assemble the full schema object (identical contract to the Node runner). */ +export function buildResult(env, cases) { + return { + schemaVersion: "1.0", + runner: env.runner, + env, + timestamp: new Date().toISOString(), + cases, + }; +} + +/** Best-effort `dw --version` first line; falls back to "dw". */ +function probeVersion(bin) { + try { + const out = execFileSync(bin, ["--version"], { encoding: "utf-8" }); + const line = out.split("\n").map((l) => l.trim()).filter(Boolean)[0]; + return line ? `dw ${line}` : "dw"; + } catch { + return "dw"; + } +} + +export async function main() { + const manifest = loadManifest(CORPUS); + const bin = locateBinary(); + const env = gatherEnv({ runner: "cli", runtimeVersion: probeVersion(bin) }); + // The CLI is a native binary, not the staged dwlib — override the lib fingerprint. + env.dwlibBuildId = "n/a-cli"; + + const coldRows = await runColdStartAndFirstRun(manifest); + const warmRows = await runWarm(manifest); + + const cases = [...coldRows, ...warmRows]; + validateResultIds(manifest, cases); + + mkdirSync(RESULTS_DIR, { recursive: true }); + const stamp = new Date().toISOString().replace(/[:.]/g, "-"); + const outPath = join(RESULTS_DIR, `cli-${stamp}.json`); + writeFileSync(outPath, JSON.stringify(buildResult(env, cases), null, 2)); + console.log(`wrote ${outPath} (${cases.length} rows)`); + return outPath; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { + main().catch((e) => { + console.error(e.message); + process.exit(1); + }); +} diff --git a/benchmarks/runners/cli/emit.test.mjs b/benchmarks/runners/cli/emit.test.mjs new file mode 100644 index 0000000..88d1587 --- /dev/null +++ b/benchmarks/runners/cli/emit.test.mjs @@ -0,0 +1,27 @@ +import { test } from "node:test"; +import assert from "node:assert/strict"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; +import { buildResult } from "./emit.mjs"; + +const __dirname = dirname(fileURLToPath(import.meta.url)); +const CORPUS = join(__dirname, "..", "..", "corpus"); + +test("buildResult produces a schema-shaped object with runner 'cli'", () => { + const env = { + runner: "cli", os: "x", cpu: "y", runtimeVersion: "dw vX", + weaveVersion: "2.12.0-x", commit: "abc", dwlibBuildId: "n/a-cli", + }; + const cases = [{ id: "trivial", metric: "cold-start", unit: "ms", stats: { median: 1 }, iterations: 10 }]; + const r = buildResult(env, cases); + assert.equal(r.schemaVersion, "1.0"); + assert.equal(r.runner, "cli"); + assert.ok(typeof r.timestamp === "string"); + assert.deepEqual(r.cases, cases); +}); + +test("orphan ids are rejected before writing", () => { + const manifest = loadManifest(CORPUS); + assert.throws(() => validateResultIds(manifest, [{ id: "totally-made-up" }]), /orphan id/); +}); From f5892bffa371151d5d3819c8a91bd13f18f94eee Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 12:10:34 -0300 Subject: [PATCH 10/15] build: register benchmarkCli runner + wire cli JS parity tests --- native-cli/build.gradle | 22 +++++++++++++++++++++- native-lib/build.gradle | 3 ++- 2 files changed, 23 insertions(+), 2 deletions(-) diff --git a/native-cli/build.gradle b/native-cli/build.gradle index 4ac56a9..68c3855 100644 --- a/native-cli/build.gradle +++ b/native-cli/build.gradle @@ -187,4 +187,24 @@ tasks.compileJava.classpath += files(sourceSets.main.scala.classesDirectory) //tasks.named("nativeCompile") { // classpathJar = file("${buildDir}/libs/native-cli-" + nativeVersion + "-all.jar") -//} \ No newline at end of file +//} + +// The CLI runner as an aggregator-registered runner: emits its result file but +// does NOT render the report (the root :benchmarkCompare renders once over all +// runners). Tagged `benchmarkRunner` so :benchmarkCompare discovers it automatically. +// Requires the bench-enabled binary — nativeCompile must run with -Pbenchmark=true so +// BenchmarkMode.ENABLED is true and BenchmarkHarness is reachable in dw. +tasks.register('benchmarkCli', Exec) { + onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } + ext.benchmarkRunner = true + + dependsOn tasks.named('nativeCompile') + workingDir("${rootDir}/benchmarks") + + def script = 'node corpus/gen-inputs.mjs && node runners/cli/emit.mjs' + if (System.getProperty('os.name').toLowerCase().contains('windows')) { + commandLine('cmd', '/c', script) + } else { + commandLine('bash', '-c', script) + } +} \ No newline at end of file diff --git a/native-lib/build.gradle b/native-lib/build.gradle index 16721e0..568076b 100644 --- a/native-lib/build.gradle +++ b/native-lib/build.gradle @@ -310,7 +310,8 @@ tasks.register('benchmarkJsUnitTest', Exec) { } workingDir("${rootDir}/benchmarks") def files = 'lib/stats.test.mjs lib/manifest.test.mjs lib/env.test.mjs ' + - 'report/report.test.mjs runners/node/emit.test.mjs' + 'report/report.test.mjs runners/node/emit.test.mjs ' + + 'runners/cli/locate.test.mjs runners/cli/emit.test.mjs' def script = 'node --test ' + files if (System.getProperty('os.name').toLowerCase().contains('windows')) { commandLine('cmd', '/c', script) From ac167d3d25b2a670159d1ec31fd23d7ad9fefd99 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 12:13:58 -0300 Subject: [PATCH 11/15] docs: document the CLI benchmark runner --- benchmarks/README.md | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/benchmarks/README.md b/benchmarks/README.md index 67ec233..3b1dfe3 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -13,6 +13,12 @@ Language-agnostic benchmark harness for the DataWeave native-lib wrappers. (Scala/Gradle subproject `:benchmarks-engine`, depends on `org.mule.weave:runtime` at the same `weaveVersion` the native image is built from). `runners/python/` is the Python runner (stdlib scripts under `native-lib`, wrapping the same staged `dwlib` as Node). + `runners/cli/` is the CLI runner: a Node parent that spawns the `dw` native + binary (built with `-Pbenchmark=true`, which compiles in an in-binary + benchmark harness gated by `BenchmarkMode.ENABLED` and dispatched via the + `DW_BENCH` env var — the shipped `dw` contains none of it). It emits + `cold-start`, `first-run`, and `warm`; it does **not** emit `streaming` + (the `dw run` path has no chunked-input FFI like the library's). - `report/report.mjs` — joins result files against the manifest and prints a comparison table. - `results/` — gitignored per-run output. @@ -39,6 +45,10 @@ and `JAVA_HOME` set to it (see the root README / `CLAUDE.md`). The pinned build `graalvmVersion` in `gradle.properties`. The **engine runner alone** drives the JVM `DataWeaveScriptingEngine` and runs on any JDK — no native image required. +The **CLI runner** requires the bench-enabled binary +(`./gradlew native-cli:nativeCompile -Pbenchmark=true`); set `DW_BENCH_BIN` to +point at a prebuilt one. Like the library runners it needs the GraalVM toolchain. + ## Running The one-shot cross-runner comparison — runs **every** registered runner and prints the table: @@ -50,6 +60,7 @@ Single-runner options: ./gradlew native-lib:benchmark -Pbenchmark=true # Node only: build wrapper, run, report ./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true # engine (JVM) only: writes results/engine-.json ./gradlew native-lib:benchmarkPython -Pbenchmark=true # Python only: writes results/python-.json + ./gradlew native-cli:benchmarkCli -Pbenchmark=true # CLI only: writes results/cli-.json Or directly, once the wrapper is built (`./gradlew native-lib:buildNodePackage`): From 94e16b9d85defb95293e8786911e56b9fb90f36a Mon Sep 17 00:00:00 2001 From: andres-rad Date: Mon, 27 Jul 2026 12:44:25 -0300 Subject: [PATCH 12/15] fix: make cli benchmark input format and READY marker Windows-safe --- benchmarks/runners/cli/coldstart.mjs | 4 ++-- benchmarks/runners/cli/warm.mjs | 2 +- .../weave/dwnative/benchmark/BenchmarkHarness.scala | 12 ++++++------ .../dwnative/benchmark/BenchmarkHarnessTest.scala | 4 ++-- 4 files changed, 11 insertions(+), 11 deletions(-) diff --git a/benchmarks/runners/cli/coldstart.mjs b/benchmarks/runners/cli/coldstart.mjs index 909fe2a..39429ac 100644 --- a/benchmarks/runners/cli/coldstart.mjs +++ b/benchmarks/runners/cli/coldstart.mjs @@ -4,13 +4,13 @@ import { casesForMetric } from "../../lib/manifest.mjs"; import { computeStats } from "../../lib/stats.mjs"; import { locateBinary } from "./locate.mjs"; -/** Build `--input=name=file:mime:charset` args for a case (absolute paths). */ +/** Build `--input=name=file\tmime\tcharset` args for a case (absolute paths). */ function inputArgs(manifest, c) { const args = []; for (const [name, inp] of Object.entries(c.inputs ?? {})) { const file = join(manifest.corpusDir, inp.file); const charset = inp.charset ?? "utf-8"; - args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + args.push(`--input=${name}=${file}\t${inp.mimeType}\t${charset}`); } return args; } diff --git a/benchmarks/runners/cli/warm.mjs b/benchmarks/runners/cli/warm.mjs index b5bed3c..3bd84c7 100644 --- a/benchmarks/runners/cli/warm.mjs +++ b/benchmarks/runners/cli/warm.mjs @@ -9,7 +9,7 @@ function inputArgs(manifest, c) { for (const [name, inp] of Object.entries(c.inputs ?? {})) { const file = join(manifest.corpusDir, inp.file); const charset = inp.charset ?? "utf-8"; - args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); + args.push(`--input=${name}=${file}\t${inp.mimeType}\t${charset}`); } return args; } diff --git a/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala b/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala index 5d80f92..7b5e9cb 100644 --- a/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala +++ b/native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala @@ -47,11 +47,11 @@ object BenchmarkHarness { case "--warmup" => warmup = value.toInt case "--iters" => iters = value.toInt case "--input" => - // value = =:[:] + // value = =\t[\t] val nameSep = value.indexOf('=') val name = value.substring(0, nameSep) val rest = value.substring(nameSep + 1) - val parts = rest.split(":", 3) + val parts = rest.split("\t", 3) val file = parts(0) val mimeType = parts(1) val charset = if (parts.length > 2 && parts(2).nonEmpty) parts(2) else "utf-8" @@ -90,18 +90,18 @@ object BenchmarkHarness { val script = readScript(a) val b = bindings(a) val rt = newRuntime() // engine init — measured externally as cold-start - out.println("READY"); out.flush() + out.print("READY\n"); out.flush() val start = nowNs() assertOk(rt.run(script, "bench", b, sink, "application/json", None)) val firstRunMs = msSince(start) - out.println("{\"firstRunMs\":" + firstRunMs + "}") + out.print("{\"firstRunMs\":" + firstRunMs + "}\n") } def runWarm(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { val script = readScript(a) val b = bindings(a) val rt = newRuntime() - out.println("READY"); out.flush() + out.print("READY\n"); out.flush() var i = 0 while (i < a.warmup) { assertOk(rt.run(script, "bench", b, sink, "application/json", None)); i += 1 } val samples = new Array[Double](a.iters) @@ -112,7 +112,7 @@ object BenchmarkHarness { samples(i) = msSince(start) i += 1 } - out.println("{\"warmMs\":[" + samples.mkString(",") + "]}") + out.print("{\"warmMs\":[" + samples.mkString(",") + "]}\n") } def main(args: Array[String]): Unit = { diff --git a/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala index 41a4c23..8134283 100644 --- a/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala +++ b/native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala @@ -28,7 +28,7 @@ class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { val a = BenchmarkHarness.parseArgs(Array( "--bench-mode=coldfirst", "--script=/tmp/x.dwl", - "--input=payload=/tmp/p.json:application/json:utf-8")) + "--input=payload=/tmp/p.json\tapplication/json\tutf-8")) a.mode shouldBe "coldfirst" a.scriptFile shouldBe "/tmp/x.dwl" a.inputs should have size 1 @@ -47,7 +47,7 @@ class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { "handles a mimeType-only input (charset defaults to utf-8)" in { val a = BenchmarkHarness.parseArgs(Array( "--bench-mode=coldfirst", "--script=/tmp/x.dwl", - "--input=payload=/tmp/p.json:application/json")) + "--input=payload=/tmp/p.json\tapplication/json")) a.inputs.head.charset shouldBe "utf-8" } } From 3516339930ba2f4ebad8073c3950e0324cf252e2 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Tue, 28 Jul 2026 09:31:54 -0300 Subject: [PATCH 13/15] chore: exclude superpowers implementation plans --- .gitignore | 1 + .../plans/2026-07-22-native-lib-benchmarks.md | 1571 ----------------- .../plans/2026-07-23-engine-runner.md | 1326 -------------- .../plans/2026-07-23-python-runner.md | 1246 ------------- .../plans/2026-07-24-streaming-alignment.md | 944 ---------- .../plans/2026-07-27-cli-benchmark-runner.md | 1108 ------------ 6 files changed, 1 insertion(+), 6195 deletions(-) delete mode 100644 docs/superpowers/plans/2026-07-22-native-lib-benchmarks.md delete mode 100644 docs/superpowers/plans/2026-07-23-engine-runner.md delete mode 100644 docs/superpowers/plans/2026-07-23-python-runner.md delete mode 100644 docs/superpowers/plans/2026-07-24-streaming-alignment.md delete mode 100644 docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md diff --git a/.gitignore b/.gitignore index 327b866..d6ea206 100644 --- a/.gitignore +++ b/.gitignore @@ -25,3 +25,4 @@ grimoires/ .windsurf/ .claude/ +docs/superpowers/plans/ diff --git a/docs/superpowers/plans/2026-07-22-native-lib-benchmarks.md b/docs/superpowers/plans/2026-07-22-native-lib-benchmarks.md deleted file mode 100644 index 1a10a7c..0000000 --- a/docs/superpowers/plans/2026-07-22-native-lib-benchmarks.md +++ /dev/null @@ -1,1571 +0,0 @@ -# Native-lib Wrapper Benchmarks Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build a benchmark harness for the DataWeave native-lib Node wrapper that measures cold-start, first-run, warm steady-state, and streaming throughput, emits results in a shared JSON schema, and renders a comparison report — designed so Python and the Scala engine plug in later unchanged. - -**Architecture:** A language-agnostic corpus (`.dwl` scripts + inputs + `manifest.json`) is the contract. Dependency-free ESM modules under `benchmarks/lib/` hold shared logic (stats, manifest, env). The Node runner under `benchmarks/runners/node/` consumes the corpus, times the four metrics, and emits schema-conformant JSON validated by case `id`. A `report/report.mjs` joins result files against the manifest and prints a comparison table with a weave-version skew banner. Gradle exposes an opt-in `native-lib:benchmark` task. - -**Tech Stack:** Node.js ≥18 (ESM `.mjs`, zero new npm deps), built-in `node:test` for unit tests, `process.hrtime.bigint()` for timing, the existing `@dataweave/native` wrapper (built to `native-lib/node/dist`), Gradle. - -## Global Constraints - -- **No new npm dependencies.** Use only Node built-ins (`node:test`, `node:assert/strict`, `node:fs`, `node:child_process`, `node:os`, `node:path`, `node:url`) and the already-built `@dataweave/native` wrapper. -- **Timing methodology (applies to every metric):** measure with `process.hrtime.bigint()`; convert to ms as `Number(endNs - startNs) / 1e6`. This is the methodology a future engine harness must mirror, so keep it identical everywhere. -- **`id` is the immutable join key.** Runners emit case `id` verbatim from the manifest; never invent or rename ids. Deprecate, never rename. -- **Fail-fast on orphan ids.** A runner MUST abort before writing output if it emits any `id` not present in the manifest. -- **Explicit metrics.** Each manifest case declares `metrics[]` from exactly `["cold-start","first-run","warm","streaming"]`. A runner runs only the metrics a case declares. -- **`env.weaveVersion` is mandatory** in every result file, read from `gradle.properties` (`weaveVersion=`, currently `2.12.0-20260413`). `env.commit` and `env.dwlibBuildId` are also mandatory (best-effort values allowed) for future attributable history. -- **Metric refinement vs. spec (approved deviation):** `cold-start` **and** `first-run` are measured by the spawn harness (fresh process per sample → cold isolate + cold compilation); `warm` and `streaming` are measured in-process. The spec placed first-run in the in-process runner; measuring it fresh yields a real cold-compilation distribution. -- **Results are local-only.** `benchmarks/results/` and generated inputs are gitignored; only corpus, schema, lib, runners, report, and README are committed. -- **Units are per-row:** `ms` for latency metrics (cold-start, first-run, warm), `MB/s` for streaming. - ---- - -### Task 1: Scaffolding, schema, `.gitignore`, README - -**Files:** -- Create: `benchmarks/.gitignore` -- Create: `benchmarks/README.md` -- Create: `benchmarks/schema/result.schema.json` -- Create: `benchmarks/schema/schema.test.mjs` - -**Interfaces:** -- Consumes: nothing. -- Produces: `benchmarks/schema/result.schema.json` — the JSON Schema (draft-07) that every runner's output validates against. Consumers reference it as documentation; runtime validation of ids happens in code (Task 3/6). - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/schema/schema.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { readFileSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); - -test("result schema is valid JSON with the required shape", () => { - const schema = JSON.parse(readFileSync(join(__dirname, "result.schema.json"), "utf-8")); - assert.equal(schema.$schema, "http://json-schema.org/draft-07/schema#"); - const props = schema.properties; - for (const key of ["schemaVersion", "runner", "env", "timestamp", "cases"]) { - assert.ok(props[key], `schema must declare property ${key}`); - } - const envProps = props.env.properties; - for (const key of ["os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId"]) { - assert.ok(envProps[key], `env must declare ${key}`); - } - const metricEnum = props.cases.items.properties.metric.enum; - assert.deepEqual(metricEnum.sort(), ["cold-start", "first-run", "streaming", "warm"]); - const unitEnum = props.cases.items.properties.unit.enum; - assert.deepEqual(unitEnum.sort(), ["MB/s", "ms"]); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/schema/` -Expected: FAIL — `ENOENT` opening `result.schema.json`. - -- [ ] **Step 3: Create the schema** - -Create `benchmarks/schema/result.schema.json`: - -```json -{ - "$schema": "http://json-schema.org/draft-07/schema#", - "title": "DataWeave benchmark result", - "type": "object", - "required": ["schemaVersion", "runner", "env", "timestamp", "cases"], - "additionalProperties": false, - "properties": { - "schemaVersion": { "const": "1.0" }, - "runner": { "type": "string" }, - "timestamp": { "type": "string" }, - "env": { - "type": "object", - "required": ["os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId"], - "additionalProperties": true, - "properties": { - "os": { "type": "string" }, - "cpu": { "type": "string" }, - "runtimeVersion": { "type": "string" }, - "weaveVersion": { "type": "string" }, - "commit": { "type": "string" }, - "dwlibBuildId": { "type": "string" } - } - }, - "cases": { - "type": "array", - "items": { - "type": "object", - "required": ["id", "metric", "unit", "stats", "iterations"], - "additionalProperties": false, - "properties": { - "id": { "type": "string" }, - "metric": { "enum": ["cold-start", "first-run", "warm", "streaming"] }, - "unit": { "enum": ["ms", "MB/s"] }, - "iterations": { "type": "integer" }, - "stats": { - "type": "object", - "required": ["median"], - "additionalProperties": false, - "properties": { - "min": { "type": "number" }, - "median": { "type": "number" }, - "p90": { "type": "number" }, - "p99": { "type": "number" }, - "mean": { "type": "number" } - } - } - } - } - } - } -} -``` - -- [ ] **Step 4: Create `.gitignore` and README** - -Create `benchmarks/.gitignore`: - -```gitignore -# Ephemeral per-run result files -results/ -# Deterministically regenerated large inputs -corpus/inputs/generated/ -``` - -Create `benchmarks/README.md`: - -```markdown -# DataWeave native-lib benchmarks - -Language-agnostic benchmark harness for the DataWeave native-lib wrappers. - -## Layout - -- `corpus/` — shared benchmark cases: `manifest.json` (the contract), `scripts/*.dwl`, - `inputs/` (committed small inputs), `inputs/generated/` (regenerated large inputs), - `gen-inputs.mjs` (deterministic generator). -- `schema/result.schema.json` — the JSON schema every runner's output conforms to. -- `lib/` — dependency-free shared modules (stats, manifest, env). -- `runners/node/` — the Node reference runner. `runners/python/` and `runners/engine/` - are follow-ups; the engine harness lives in the `data-weave` repo but reads this corpus. -- `report/report.mjs` — joins result files against the manifest and prints a comparison table. -- `results/` — gitignored per-run output. - -## Metrics - -`cold-start` and `first-run` (fresh process per sample), `warm` (in-process steady state), -`streaming` (MB/s). Each case declares which apply via `metrics[]`. - -## Running - - ./gradlew native-lib:benchmark -Pbenchmark=true # build wrapper, run, report - -Or directly, once the wrapper is built (`./gradlew native-lib:buildNodePackage`): - - node runners/node/emit.mjs # writes results/node-.json - node report/report.mjs results/*.json # renders the table - -Generate large inputs first (idempotent): - - node corpus/gen-inputs.mjs - -Results are local-only; no history is accumulated (see the design spec). -``` - -- [ ] **Step 5: Run test to verify it passes** - -Run: `node --test benchmarks/schema/` -Expected: PASS (1 test). - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/.gitignore benchmarks/README.md benchmarks/schema/ -git commit -m "W-23545283: Scaffold benchmarks dir with result JSON schema" -``` - ---- - -### Task 2: Stats library (`lib/stats.mjs`) - -**Files:** -- Create: `benchmarks/lib/stats.mjs` -- Create: `benchmarks/lib/stats.test.mjs` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `computeStats(samples: number[]): { min, median, p90, p99, mean }` — throws on an empty/non-array input. - - `toMBps(totalBytes: number, elapsedMs: number): number` — throughput in megabytes/sec (bytes ÷ 1e6 ÷ seconds). - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/lib/stats.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { computeStats, toMBps } from "./stats.mjs"; - -test("computeStats returns min/median/p90/p99/mean", () => { - const s = computeStats([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]); - assert.equal(s.min, 1); - assert.equal(s.mean, 5.5); - assert.equal(s.median, 5); - assert.equal(s.p90, 9); - assert.equal(s.p99, 10); -}); - -test("computeStats handles a single sample", () => { - const s = computeStats([42]); - assert.deepEqual(s, { min: 42, median: 42, p90: 42, p99: 42, mean: 42 }); -}); - -test("computeStats rejects empty input", () => { - assert.throws(() => computeStats([]), /non-empty/); - assert.throws(() => computeStats(null), /non-empty/); -}); - -test("toMBps converts bytes and elapsed to MB/s", () => { - // 10 MB in 1000 ms => 10 MB/s - assert.equal(toMBps(10_000_000, 1000), 10); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/lib/stats.test.mjs` -Expected: FAIL — cannot find module `./stats.mjs`. - -- [ ] **Step 3: Write the implementation** - -Create `benchmarks/lib/stats.mjs`: - -```js -/** - * Aggregate a list of samples into min/median/p90/p99/mean. - * Percentiles use the nearest-rank method on a sorted copy. - * @param {number[]} samples - * @returns {{min:number, median:number, p90:number, p99:number, mean:number}} - */ -export function computeStats(samples) { - if (!Array.isArray(samples) || samples.length === 0) { - throw new Error("computeStats requires a non-empty array of numbers"); - } - const sorted = [...samples].sort((a, b) => a - b); - const n = sorted.length; - const pct = (p) => sorted[Math.min(n - 1, Math.max(0, Math.ceil((p / 100) * n) - 1))]; - const sum = sorted.reduce((a, b) => a + b, 0); - return { min: sorted[0], median: pct(50), p90: pct(90), p99: pct(99), mean: sum / n }; -} - -/** - * Throughput in megabytes per second (decimal MB, i.e. 1e6 bytes). - * @param {number} totalBytes - * @param {number} elapsedMs - * @returns {number} - */ -export function toMBps(totalBytes, elapsedMs) { - if (elapsedMs <= 0) throw new Error("elapsedMs must be > 0"); - return totalBytes / 1e6 / (elapsedMs / 1000); -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `node --test benchmarks/lib/stats.test.mjs` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/lib/stats.mjs benchmarks/lib/stats.test.mjs -git commit -m "W-23545283: Add stats lib (percentiles + throughput)" -``` - ---- - -### Task 3: Corpus, generator, and manifest library (`lib/manifest.mjs`) - -**Files:** -- Create: `benchmarks/lib/manifest.mjs` -- Create: `benchmarks/lib/manifest.test.mjs` -- Create: `benchmarks/corpus/manifest.json` -- Create: `benchmarks/corpus/scripts/trivial.dwl` -- Create: `benchmarks/corpus/scripts/object-transform.dwl` -- Create: `benchmarks/corpus/scripts/map-scale.dwl` -- Create: `benchmarks/corpus/scripts/xml-to-csv.dwl` -- Create: `benchmarks/corpus/scripts/json-stream.dwl` -- Create: `benchmarks/corpus/scripts/compile-heavy.dwl` -- Create: `benchmarks/corpus/inputs/person.xml` -- Create: `benchmarks/corpus/gen-inputs.mjs` - -**Interfaces:** -- Consumes: `METRICS` concept from the Global Constraints. -- Produces: - - `METRICS: string[]` — `["cold-start","first-run","warm","streaming"]`. - - `loadManifest(corpusDir: string): { corpusDir, cases: Case[], ids: Set }` — validates each case (unique id, non-empty metrics from `METRICS`, script file exists, non-generated input files exist) and throws on any violation. - - `casesForMetric(manifest, metric: string): Case[]` — cases whose `metrics[]` includes `metric`. - - `resolveInputs(manifest, caseObj): Record` — reads each input file into a Buffer. - - `validateResultIds(manifest, resultCases: {id:string}[]): void` — throws on any orphan id (fail-fast). - - `Case` shape: `{ id, script, inputs?: Record, metrics: string[], iterations?: { warm?, warmup?, streaming?, samples? } }`. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/lib/manifest.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { - METRICS, - loadManifest, - casesForMetric, - validateResultIds, -} from "./manifest.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "corpus"); - -test("METRICS lists exactly the four metrics", () => { - assert.deepEqual([...METRICS].sort(), ["cold-start", "first-run", "streaming", "warm"]); -}); - -test("loadManifest validates the committed corpus", () => { - const m = loadManifest(CORPUS); - assert.ok(m.cases.length >= 6, "expected at least 6 corpus cases"); - assert.ok(m.ids.has("trivial")); -}); - -test("casesForMetric filters by declared metric", () => { - const m = loadManifest(CORPUS); - const streaming = casesForMetric(m, "streaming"); - assert.ok(streaming.every((c) => c.metrics.includes("streaming"))); - assert.ok(streaming.length >= 1); -}); - -test("validateResultIds throws on an orphan id", () => { - const m = loadManifest(CORPUS); - assert.throws(() => validateResultIds(m, [{ id: "does-not-exist" }]), /orphan id/); - assert.doesNotThrow(() => validateResultIds(m, [{ id: "trivial" }])); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/lib/manifest.test.mjs` -Expected: FAIL — cannot find module `./manifest.mjs`. - -- [ ] **Step 3: Write the manifest library** - -Create `benchmarks/lib/manifest.mjs`: - -```js -import { readFileSync, existsSync } from "node:fs"; -import { join } from "node:path"; - -/** The only metrics a case may declare. */ -export const METRICS = ["cold-start", "first-run", "warm", "streaming"]; - -/** - * Load and validate corpus/manifest.json. - * @param {string} corpusDir absolute path to the corpus directory - */ -export function loadManifest(corpusDir) { - const raw = JSON.parse(readFileSync(join(corpusDir, "manifest.json"), "utf-8")); - if (!Array.isArray(raw.cases)) throw new Error("manifest.cases must be an array"); - const ids = new Set(); - for (const c of raw.cases) { - if (!c.id) throw new Error("manifest case is missing an id"); - if (ids.has(c.id)) throw new Error(`duplicate case id: ${c.id}`); - ids.add(c.id); - if (!Array.isArray(c.metrics) || c.metrics.length === 0) { - throw new Error(`case ${c.id} must declare a non-empty metrics[]`); - } - for (const metric of c.metrics) { - if (!METRICS.includes(metric)) throw new Error(`case ${c.id} has unknown metric: ${metric}`); - } - if (!c.script || !existsSync(join(corpusDir, c.script))) { - throw new Error(`case ${c.id} script not found: ${c.script}`); - } - for (const [name, inp] of Object.entries(c.inputs ?? {})) { - if (inp.file && !inp.generated && !existsSync(join(corpusDir, inp.file))) { - throw new Error(`case ${c.id} input '${name}' file not found: ${inp.file}`); - } - } - } - return { corpusDir, cases: raw.cases, ids }; -} - -/** Cases whose declared metrics[] includes `metric`. */ -export function casesForMetric(manifest, metric) { - return manifest.cases.filter((c) => c.metrics.includes(metric)); -} - -/** - * Read a case's declared inputs into Buffers. - * @returns {Record} - */ -export function resolveInputs(manifest, caseObj) { - const out = {}; - for (const [name, inp] of Object.entries(caseObj.inputs ?? {})) { - const buffer = readFileSync(join(manifest.corpusDir, inp.file)); - out[name] = { buffer, mimeType: inp.mimeType, charset: inp.charset }; - } - return out; -} - -/** Fail-fast: throw if any result case carries an id not present in the manifest. */ -export function validateResultIds(manifest, resultCases) { - for (const rc of resultCases) { - if (!manifest.ids.has(rc.id)) { - throw new Error(`result contains orphan id not in manifest: ${rc.id}`); - } - } -} -``` - -- [ ] **Step 4: Create the corpus scripts** - -Create `benchmarks/corpus/scripts/trivial.dwl`: - -```dataweave -2 + 2 -``` - -Create `benchmarks/corpus/scripts/object-transform.dwl`: - -```dataweave -output application/json ---- -{ - fullName: payload.first ++ " " ++ payload.last, - adult: payload.age >= 18, - initials: payload.first[0] ++ payload.last[0] -} -``` - -Create `benchmarks/corpus/scripts/map-scale.dwl`: - -```dataweave -output application/json ---- -payload map (item) -> { id: item.id, doubled: item.value * 2, label: "item_" ++ item.id } -``` - -Create `benchmarks/corpus/scripts/xml-to-csv.dwl`: - -```dataweave -output application/csv header=true ---- -[payload.person] -``` - -Create `benchmarks/corpus/scripts/json-stream.dwl`: - -```dataweave -output application/json ---- -payload map (item) -> { id: item.id, name: item.name } -``` - -Create `benchmarks/corpus/scripts/compile-heavy.dwl`: - -```dataweave -output application/json ---- -{ - a: (1 to 100) reduce ((i, acc = 0) -> acc + i), - b: [1, 2, 3, 4, 5] map ($ * 2) filter ($ > 4) reduce ((i, acc = 0) -> acc + i), - c: { x: 1, y: 2, z: 3 } mapObject (v, k) -> { (k): v * 10 }, - d: "hello world" splitBy " " map upper($), - e: (1 to 50) map (n) -> { n: n, sq: n * n, even: (n mod 2) == 0 } -} -``` - -- [ ] **Step 5: Create the committed small input** - -Create `benchmarks/corpus/inputs/person.xml` (UTF-8; the case declares its charset): - -```xml - - - Billy - 31 - -``` - -- [ ] **Step 6: Create the input generator** - -Create `benchmarks/corpus/gen-inputs.mjs`: - -```js -// Deterministically regenerate large inputs. No randomness -> comparable across -// machines and runners. Size overridable via BENCH_LARGE_N (default 50000). -import { writeFileSync, mkdirSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const outDir = join(__dirname, "inputs", "generated"); -mkdirSync(outDir, { recursive: true }); - -const n = Number(process.env.BENCH_LARGE_N ?? 50000); -const records = []; -for (let i = 1; i <= n; i++) { - records.push({ id: i, name: `item_${i}`, value: i * 3 }); -} -const path = join(outDir, "records-large.json"); -writeFileSync(path, JSON.stringify(records)); -console.log(`wrote ${n} records to ${path}`); -``` - -- [ ] **Step 7: Create the manifest** - -Create `benchmarks/corpus/manifest.json`: - -```json -{ - "cases": [ - { - "id": "trivial", - "script": "scripts/trivial.dwl", - "metrics": ["cold-start", "first-run", "warm"], - "iterations": { "warm": 200, "warmup": 20, "samples": 30 } - }, - { - "id": "object-transform", - "script": "scripts/object-transform.dwl", - "inputs": { - "payload": { "file": "inputs/person-record.json", "mimeType": "application/json" } - }, - "metrics": ["first-run", "warm"], - "iterations": { "warm": 200, "warmup": 20, "samples": 20 } - }, - { - "id": "map-scale", - "script": "scripts/map-scale.dwl", - "inputs": { - "payload": { "file": "inputs/generated/records-large.json", "mimeType": "application/json", "generated": true } - }, - "metrics": ["first-run", "warm", "streaming"], - "iterations": { "warm": 30, "warmup": 3, "streaming": 10, "samples": 15 } - }, - { - "id": "xml-to-csv", - "script": "scripts/xml-to-csv.dwl", - "inputs": { - "payload": { "file": "inputs/person.xml", "mimeType": "application/xml", "charset": "UTF-8" } - }, - "metrics": ["first-run", "warm"], - "iterations": { "warm": 100, "warmup": 10, "samples": 20 } - }, - { - "id": "json-stream", - "script": "scripts/json-stream.dwl", - "inputs": { - "payload": { "file": "inputs/generated/records-large.json", "mimeType": "application/json", "generated": true } - }, - "metrics": ["first-run", "warm", "streaming"], - "iterations": { "warm": 30, "warmup": 3, "streaming": 10, "samples": 15 } - }, - { - "id": "compile-heavy", - "script": "scripts/compile-heavy.dwl", - "metrics": ["first-run", "warm"], - "iterations": { "warm": 100, "warmup": 10, "samples": 20 } - } - ] -} -``` - -- [ ] **Step 8: Create the committed `person-record.json` input** - -The `object-transform` case references `inputs/person-record.json`. Create `benchmarks/corpus/inputs/person-record.json`: - -```json -{ "first": "Ada", "last": "Lovelace", "age": 36 } -``` - -- [ ] **Step 9: Run tests to verify they pass** - -Run: `node --test benchmarks/lib/manifest.test.mjs` -Expected: PASS (4 tests). Non-generated inputs (`person.xml`, `person-record.json`) exist; the generated `records-large.json` is marked `"generated": true` so validation does not require it yet. - -- [ ] **Step 10: Commit** - -```bash -git add benchmarks/lib/manifest.mjs benchmarks/lib/manifest.test.mjs benchmarks/corpus/ -git commit -m "W-23545283: Add corpus, input generator, and manifest lib" -``` - ---- - -### Task 4: Environment stamp (`lib/env.mjs`) - -**Files:** -- Create: `benchmarks/lib/env.mjs` -- Create: `benchmarks/lib/env.test.mjs` - -**Interfaces:** -- Consumes: repo `gradle.properties` (weave version), git (commit), the staged `dwlib` file (build id). -- Produces: `gatherEnv({ runner: string, runtimeVersion: string }): { os, cpu, runtimeVersion, weaveVersion, commit, dwlibBuildId }`. Reads the repo root internally (three levels up from `benchmarks/lib/` is the repo root's parent — see code; repo root is two levels up). - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/lib/env.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { gatherEnv } from "./env.mjs"; - -test("gatherEnv returns all required fields", () => { - const env = gatherEnv({ runner: "node-wrapper", runtimeVersion: "node vX" }); - for (const key of ["os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId"]) { - assert.ok(env[key] !== undefined && env[key] !== "", `env.${key} must be set`); - } -}); - -test("gatherEnv reads the pinned weaveVersion from gradle.properties", () => { - const env = gatherEnv({ runner: "node-wrapper", runtimeVersion: "node vX" }); - // gradle.properties pins e.g. 2.12.0-YYYYMMDD; assert it looks like a weave version. - assert.match(env.weaveVersion, /^\d+\.\d+\.\d+/); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/lib/env.test.mjs` -Expected: FAIL — cannot find module `./env.mjs`. - -- [ ] **Step 3: Write the implementation** - -Create `benchmarks/lib/env.mjs`: - -```js -import { readFileSync, existsSync, createReadStream } from "node:fs"; -import { execSync } from "node:child_process"; -import { createHash } from "node:crypto"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import os from "node:os"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -// benchmarks/lib -> benchmarks -> repo root -const REPO_ROOT = join(__dirname, "..", ".."); - -function readWeaveVersion() { - const txt = readFileSync(join(REPO_ROOT, "gradle.properties"), "utf-8"); - const m = txt.match(/^weaveVersion=(.+)$/m); - if (!m) throw new Error("weaveVersion not found in gradle.properties"); - return m[1].trim(); -} - -function readCommit() { - try { - return execSync("git rev-parse --short HEAD", { cwd: REPO_ROOT }).toString().trim(); - } catch { - return "unknown"; - } -} - -// Best-effort identity of the staged dwlib: first 8 hex of a sha256 over -// (size + first 64KB). Cheap, stable, and enough to detect a lib swap. -function readDwlibBuildId() { - const base = join(REPO_ROOT, "native-lib", "node", "native"); - for (const ext of [".dylib", ".so", ".dll"]) { - const p = join(base, `dwlib${ext}`); - if (existsSync(p)) { - const buf = readFileSync(p).subarray(0, 65536); - const { size } = statSafe(p); - return "dwlib-" + createHash("sha256").update(String(size)).update(buf).digest("hex").slice(0, 8); - } - } - return "unknown"; -} - -function statSafe(p) { - // eslint-disable-next-line - const { statSync } = require("node:fs"); - return statSync(p); -} - -/** - * @param {{runner:string, runtimeVersion:string}} opts - */ -export function gatherEnv({ runner, runtimeVersion }) { - const cpus = os.cpus(); - return { - runner, - os: `${process.platform}-${process.arch}`, - cpu: cpus.length ? cpus[0].model : "unknown", - runtimeVersion, - weaveVersion: readWeaveVersion(), - commit: readCommit(), - dwlibBuildId: readDwlibBuildId(), - }; -} -``` - -> Note: `require` is not available in ESM. Replace the `statSafe` helper with a top-level import. Use this corrected version of the two relevant lines instead: -> -> add to the imports at the top: `import { statSync } from "node:fs";` -> and replace `readDwlibBuildId`'s `statSafe(p)` call and the helper with a direct `statSync(p)`: -> -> ```js -> const size = statSync(p).size; -> return "dwlib-" + createHash("sha256").update(String(size)).update(buf).digest("hex").slice(0, 8); -> ``` -> -> Delete the `statSafe` function entirely. (Implement env.mjs with the corrected imports — do not ship the `require` form.) - -- [ ] **Step 4: Run test to verify it passes** - -Run: `node --test benchmarks/lib/env.test.mjs` -Expected: PASS (2 tests). `dwlibBuildId` will be `"unknown"` until the wrapper is built and `dwlib` staged — the test only asserts it is non-empty, which `"unknown"` satisfies. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/lib/env.mjs benchmarks/lib/env.test.mjs -git commit -m "W-23545283: Add env stamp (weaveVersion, commit, dwlibBuildId)" -``` - ---- - -### Task 5: Wrapper resolution + in-process warm/streaming runner - -**Files:** -- Create: `benchmarks/runners/node/wrapper.mjs` -- Create: `benchmarks/runners/node/warm-bench.mjs` -- Create: `benchmarks/runners/node/warm-bench.test.mjs` - -**Prerequisite:** the Node wrapper must be built (`dist/` + addon + staged `dwlib`). Run `./gradlew native-lib:buildNodePackage` (or, in `native-lib/node`: `npm install && npx node-gyp rebuild && npx tsc`) before running the integration test in this task. The generated large input must exist: `node benchmarks/corpus/gen-inputs.mjs`. - -**Interfaces:** -- Consumes: `loadManifest`, `casesForMetric`, `resolveInputs` (Task 3); `computeStats`, `toMBps` (Task 2). -- Produces: - - `loadWrapper(): Promise<{ run, runStreaming, runTransform, DataWeave, cleanup }>` — imports the built wrapper `dist/index.js`; throws a clear "not built" error otherwise. - - `runWarmAndStreaming(api, manifest): Promise` where `ResultCase = { id, metric, unit, stats, iterations }`. Emits `warm` (unit `ms`) and `streaming` (unit `MB/s`) rows for cases declaring those metrics. - -- [ ] **Step 1: Write the wrapper resolver** - -Create `benchmarks/runners/node/wrapper.mjs`: - -```js -import { existsSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath, pathToFileURL } from "node:url"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -// benchmarks/runners/node -> benchmarks/runners -> benchmarks -> repo root -const REPO_ROOT = join(__dirname, "..", "..", ".."); -const WRAPPER_DIST = join(REPO_ROOT, "native-lib", "node", "dist", "index.js"); - -/** - * Import the built @dataweave/native wrapper. The wrapper locates dwlib itself - * (staged at native-lib/node/native/dwlib.*), so no env var is required here. - */ -export async function loadWrapper() { - if (!existsSync(WRAPPER_DIST)) { - throw new Error( - `Node wrapper not built at ${WRAPPER_DIST}. ` + - `Run: ./gradlew native-lib:buildNodePackage` - ); - } - const mod = await import(pathToFileURL(WRAPPER_DIST).href); - const api = mod.run ? mod : mod.default; - if (!api || typeof api.run !== "function") { - throw new Error(`Wrapper at ${WRAPPER_DIST} did not export a run() function`); - } - return api; -} -``` - -- [ ] **Step 2: Write the warm/streaming runner** - -Create `benchmarks/runners/node/warm-bench.mjs`: - -```js -import { casesForMetric, resolveInputs } from "../../lib/manifest.mjs"; -import { computeStats, toMBps } from "../../lib/stats.mjs"; - -const nowNs = () => process.hrtime.bigint(); -const msSince = (start) => Number(nowNs() - start) / 1e6; - -/** Build the wrapper Inputs map from resolved corpus inputs. */ -function toInputsMap(resolved) { - const inputs = {}; - for (const [name, v] of Object.entries(resolved)) { - inputs[name] = { content: v.buffer, mimeType: v.mimeType, charset: v.charset ?? "utf-8" }; - } - return inputs; -} - -/** Split a Buffer into fixed-size chunks for streaming input. */ -function* chunked(buffer, size = 65536) { - for (let i = 0; i < buffer.length; i += size) yield buffer.subarray(i, i + size); -} - -async function drain(gen) { - let total = 0; - let step = await gen.next(); - while (!step.done) { - total += step.value.length; - step = await gen.next(); - } - if (!step.value.success) throw new Error(`stream failed: ${step.value.error}`); - return total; -} - -/** - * @returns {Promise>} - */ -export async function runWarmAndStreaming(api, manifest) { - const rows = []; - - for (const c of casesForMetric(manifest, "warm")) { - const script = readScript(manifest, c); - const inputs = toInputsMap(resolveInputs(manifest, c)); - const warmup = c.iterations?.warmup ?? 10; - const iters = c.iterations?.warm ?? 100; - - for (let i = 0; i < warmup; i++) assertOk(api.run(script, inputs)); - const samples = []; - for (let i = 0; i < iters; i++) { - const start = nowNs(); - assertOk(api.run(script, inputs)); - samples.push(msSince(start)); - } - rows.push({ id: c.id, metric: "warm", unit: "ms", stats: computeStats(samples), iterations: iters }); - } - - for (const c of casesForMetric(manifest, "streaming")) { - const script = readScript(manifest, c); - const resolved = resolveInputs(manifest, c); - const [primaryName, primary] = Object.entries(resolved)[0]; - const iters = c.iterations?.streaming ?? 10; - - const mbps = []; - for (let i = 0; i < iters; i++) { - const start = nowNs(); - const gen = api.runTransform(script, chunked(primary.buffer), { - inputName: primaryName, - mimeType: primary.mimeType, - charset: primary.charset, - }); - await drain(gen); - mbps.push(toMBps(primary.buffer.length, msSince(start))); - } - rows.push({ id: c.id, metric: "streaming", unit: "MB/s", stats: computeStats(mbps), iterations: iters }); - } - - return rows; -} - -function assertOk(result) { - if (!result.success) throw new Error(`run failed: ${result.error}`); - return result; -} - -function readScript(manifest, c) { - // Local import to keep the module dependency-light and avoid a top-level fs import. - const { readFileSync } = require("node:fs"); - const { join } = require("node:path"); - return readFileSync(join(manifest.corpusDir, c.script), "utf-8"); -} -``` - -> Note: `require` is unavailable in ESM. Implement `readScript` with top-level imports instead — add `import { readFileSync } from "node:fs";` and `import { join } from "node:path";` at the top of `warm-bench.mjs`, and make `readScript` just `return readFileSync(join(manifest.corpusDir, c.script), "utf-8");`. Do not ship the `require` form. - -- [ ] **Step 3: Write the integration test** - -Create `benchmarks/runners/node/warm-bench.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { loadManifest } from "../../lib/manifest.mjs"; -import { loadWrapper } from "./wrapper.mjs"; -import { runWarmAndStreaming } from "./warm-bench.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); - -test("warm + streaming rows are produced with valid stats", async () => { - const api = await loadWrapper(); - try { - const manifest = loadManifest(CORPUS); - const rows = await runWarmAndStreaming(api, manifest); - - const warm = rows.filter((r) => r.metric === "warm"); - const streaming = rows.filter((r) => r.metric === "streaming"); - assert.ok(warm.length >= 1, "expected at least one warm row"); - assert.ok(streaming.length >= 1, "expected at least one streaming row"); - - for (const r of warm) { - assert.equal(r.unit, "ms"); - assert.ok(r.stats.median >= 0); - assert.ok(r.stats.p99 >= r.stats.median); - } - for (const r of streaming) { - assert.equal(r.unit, "MB/s"); - assert.ok(r.stats.median > 0); - } - } finally { - api.cleanup(); - } -}); -``` - -- [ ] **Step 4: Build wrapper + generate input, then run the test** - -Run: -```bash -./gradlew native-lib:buildNodePackage -node benchmarks/corpus/gen-inputs.mjs -node --test benchmarks/runners/node/warm-bench.test.mjs -``` -Expected: PASS (1 test). If it fails with "Node wrapper not built", the gradle build did not produce `native-lib/node/dist/index.js` — re-run the build step. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/node/wrapper.mjs benchmarks/runners/node/warm-bench.mjs benchmarks/runners/node/warm-bench.test.mjs -git commit -m "W-23545283: Add Node warm + streaming benchmark runner" -``` - ---- - -### Task 6: Cold-start + first-run spawn harness - -**Files:** -- Create: `benchmarks/runners/node/coldstart-child.mjs` -- Create: `benchmarks/runners/node/coldstart.mjs` -- Create: `benchmarks/runners/node/coldstart.test.mjs` - -**Prerequisite:** wrapper built (Task 5 prerequisite) and `node benchmarks/corpus/gen-inputs.mjs` run. - -**Interfaces:** -- Consumes: `loadManifest`, `casesForMetric`, `resolveInputs` (Task 3); `computeStats` (Task 2); `loadWrapper` (Task 5). -- Produces: - - `coldstart-child.mjs` — a CLI child: `node coldstart-child.mjs ` initializes a fresh `DataWeave` instance (timing `initMs`), runs the case's script once (timing `firstRunMs`), and prints exactly one JSON line `{"initMs":,"firstRunMs":}` to stdout. - - `runColdStartAndFirstRun(manifest, { samplesOverride? }): Promise` — spawns N fresh child processes per applicable case, aggregates `initMs` into a `cold-start` row (for cases declaring `cold-start`) and `firstRunMs` into a `first-run` row (for cases declaring `first-run`), both unit `ms`. - -- [ ] **Step 1: Write the child process script** - -Create `benchmarks/runners/node/coldstart-child.mjs`: - -```js -// Fresh-process worker. Measures a cold isolate init + a cold (first) compile+exec -// for one case, then prints a single JSON line. Invoked by coldstart.mjs. -import { readFileSync } from "node:fs"; -import { join } from "node:path"; -import { loadManifest, resolveInputs } from "../../lib/manifest.mjs"; -import { loadWrapper } from "./wrapper.mjs"; - -const [, , corpusDir, caseId] = process.argv; -const nowNs = () => process.hrtime.bigint(); -const msSince = (s) => Number(nowNs() - s) / 1e6; - -const manifest = loadManifest(corpusDir); -const c = manifest.cases.find((x) => x.id === caseId); -if (!c) throw new Error(`unknown case: ${caseId}`); -const script = readFileSync(join(corpusDir, c.script), "utf-8"); - -const resolved = resolveInputs(manifest, c); -const inputs = {}; -for (const [name, v] of Object.entries(resolved)) { - inputs[name] = { content: v.buffer, mimeType: v.mimeType, charset: v.charset ?? "utf-8" }; -} - -const api = await loadWrapper(); -const dw = new api.DataWeave(); - -const initStart = nowNs(); -dw.initialize(); -const initMs = msSince(initStart); - -const runStart = nowNs(); -const result = dw.run(script, inputs); -const firstRunMs = msSince(runStart); -if (!result.success) throw new Error(`first run failed: ${result.error}`); - -dw.cleanup(); -process.stdout.write(JSON.stringify({ initMs, firstRunMs }) + "\n"); -``` - -- [ ] **Step 2: Write the spawn harness** - -Create `benchmarks/runners/node/coldstart.mjs`: - -```js -import { execFileSync } from "node:child_process"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { casesForMetric } from "../../lib/manifest.mjs"; -import { computeStats } from "../../lib/stats.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CHILD = join(__dirname, "coldstart-child.mjs"); - -/** Spawn one fresh process and parse its single JSON line. */ -function sampleOnce(corpusDir, caseId) { - const out = execFileSync(process.execPath, [CHILD, corpusDir, caseId], { encoding: "utf-8" }); - const line = out.trim().split("\n").pop(); - return JSON.parse(line); -} - -/** - * @returns {Promise>} - */ -export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}) { - const rows = []; - const ids = new Set([ - ...casesForMetric(manifest, "cold-start").map((c) => c.id), - ...casesForMetric(manifest, "first-run").map((c) => c.id), - ]); - - for (const id of ids) { - const c = manifest.cases.find((x) => x.id === id); - const n = samplesOverride ?? c.iterations?.samples ?? 20; - const inits = []; - const firsts = []; - for (let i = 0; i < n; i++) { - const { initMs, firstRunMs } = sampleOnce(manifest.corpusDir, id); - inits.push(initMs); - firsts.push(firstRunMs); - } - if (c.metrics.includes("cold-start")) { - rows.push({ id, metric: "cold-start", unit: "ms", stats: computeStats(inits), iterations: n }); - } - if (c.metrics.includes("first-run")) { - rows.push({ id, metric: "first-run", unit: "ms", stats: computeStats(firsts), iterations: n }); - } - } - return rows; -} -``` - -- [ ] **Step 3: Write the integration test** - -Create `benchmarks/runners/node/coldstart.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { loadManifest } from "../../lib/manifest.mjs"; -import { runColdStartAndFirstRun } from "./coldstart.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); - -test("cold-start and first-run rows are produced from fresh processes", async () => { - const manifest = loadManifest(CORPUS); - // Keep sample count tiny so the test stays fast (each sample spawns a process). - const rows = await runColdStartAndFirstRun(manifest, { samplesOverride: 3 }); - - const cold = rows.filter((r) => r.metric === "cold-start"); - const first = rows.filter((r) => r.metric === "first-run"); - assert.ok(cold.length >= 1, "expected a cold-start row (trivial declares it)"); - assert.ok(first.length >= 1, "expected first-run rows"); - for (const r of [...cold, ...first]) { - assert.equal(r.unit, "ms"); - assert.ok(r.stats.median > 0); - assert.equal(r.iterations, 3); - } -}); -``` - -- [ ] **Step 4: Run the test** - -Run: -```bash -node --test benchmarks/runners/node/coldstart.test.mjs -``` -Expected: PASS (1 test). Requires the wrapper to be built and `records-large.json` generated (Task 5 prerequisites). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/node/coldstart-child.mjs benchmarks/runners/node/coldstart.mjs benchmarks/runners/node/coldstart.test.mjs -git commit -m "W-23545283: Add cold-start + first-run spawn harness" -``` - ---- - -### Task 7: Emit — orchestrate, validate ids, write result file - -**Files:** -- Create: `benchmarks/runners/node/emit.mjs` -- Create: `benchmarks/runners/node/emit.test.mjs` - -**Prerequisite:** wrapper built; `records-large.json` generated. - -**Interfaces:** -- Consumes: `loadManifest`, `validateResultIds` (Task 3); `gatherEnv` (Task 4); `loadWrapper`, `runWarmAndStreaming` (Task 5); `runColdStartAndFirstRun` (Task 6). -- Produces: - - `buildResult(env, cases): object` — assembles the full schema object (`schemaVersion:"1.0"`, `runner`, `env`, `timestamp`, `cases`). - - `main()` — runs both harnesses, merges rows, calls `validateResultIds` (fail-fast), writes `benchmarks/results/node-.json`, prints the path. Run via `node benchmarks/runners/node/emit.mjs`. - -- [ ] **Step 1: Write the failing unit test (fail-fast + shape, no native needed)** - -Create `benchmarks/runners/node/emit.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; -import { buildResult } from "./emit.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); - -test("buildResult produces a schema-shaped object", () => { - const env = { - runner: "node-wrapper", os: "x", cpu: "y", runtimeVersion: "node vX", - weaveVersion: "2.12.0-x", commit: "abc", dwlibBuildId: "dwlib-x", - }; - const cases = [{ id: "trivial", metric: "warm", unit: "ms", stats: { median: 1 }, iterations: 10 }]; - const r = buildResult(env, cases); - assert.equal(r.schemaVersion, "1.0"); - assert.equal(r.runner, "node-wrapper"); - assert.ok(typeof r.timestamp === "string"); - assert.deepEqual(r.cases, cases); -}); - -test("orphan ids are rejected before writing", () => { - const manifest = loadManifest(CORPUS); - assert.throws( - () => validateResultIds(manifest, [{ id: "totally-made-up" }]), - /orphan id/ - ); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/runners/node/emit.test.mjs` -Expected: FAIL — cannot find module `./emit.mjs`. - -- [ ] **Step 3: Write the implementation** - -Create `benchmarks/runners/node/emit.mjs`: - -```js -import { writeFileSync, mkdirSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; -import { gatherEnv } from "../../lib/env.mjs"; -import { loadWrapper } from "./wrapper.mjs"; -import { runWarmAndStreaming } from "./warm-bench.mjs"; -import { runColdStartAndFirstRun } from "./coldstart.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); -const RESULTS_DIR = join(__dirname, "..", "..", "results"); - -/** Assemble the full schema object. */ -export function buildResult(env, cases) { - return { - schemaVersion: "1.0", - runner: env.runner, - env, - timestamp: new Date().toISOString(), - cases, - }; -} - -export async function main() { - const manifest = loadManifest(CORPUS); - const env = gatherEnv({ runner: "node-wrapper", runtimeVersion: `node ${process.version}` }); - - // Cold-start / first-run first (fresh processes), then warm/streaming in-process. - const coldRows = await runColdStartAndFirstRun(manifest); - const api = await loadWrapper(); - let warmRows; - try { - warmRows = await runWarmAndStreaming(api, manifest); - } finally { - api.cleanup(); - } - - const cases = [...coldRows, ...warmRows]; - validateResultIds(manifest, cases); // fail-fast on any orphan id - - mkdirSync(RESULTS_DIR, { recursive: true }); - const stamp = new Date().toISOString().replace(/[:.]/g, "-"); - const outPath = join(RESULTS_DIR, `node-${stamp}.json`); - writeFileSync(outPath, JSON.stringify(buildResult(env, cases), null, 2)); - console.log(`wrote ${outPath} (${cases.length} rows)`); - return outPath; -} - -// Run when invoked directly. -if (import.meta.url === pathToFileURLSafe(process.argv[1])) { - main().catch((e) => { - console.error(e.message); - process.exit(1); - }); -} - -function pathToFileURLSafe(p) { - try { - return new URL(`file://${p}`).href; - } catch { - return ""; - } -} -``` - -> Note: the direct-invocation guard is more robust using Node's helper. Implement the guard with `import { pathToFileURL } from "node:url";` at the top and `if (import.meta.url === pathToFileURL(process.argv[1]).href)`, and delete the `pathToFileURLSafe` helper. Do not ship the hand-rolled `file://` concatenation. - -- [ ] **Step 4: Run unit test to verify it passes** - -Run: `node --test benchmarks/runners/node/emit.test.mjs` -Expected: PASS (2 tests). - -- [ ] **Step 5: Run the full emit end-to-end (integration)** - -Run: -```bash -node benchmarks/corpus/gen-inputs.mjs -node benchmarks/runners/node/emit.mjs -``` -Expected: prints `wrote .../benchmarks/results/node-.json (N rows)`. Open the file and confirm it has `schemaVersion`, `env.weaveVersion`, and case rows for all four metrics. - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/runners/node/emit.mjs benchmarks/runners/node/emit.test.mjs -git commit -m "W-23545283: Add emit orchestrator writing schema-conformant results" -``` - ---- - -### Task 8: Report script - -**Files:** -- Create: `benchmarks/report/report.mjs` -- Create: `benchmarks/report/report.test.mjs` -- Create: `benchmarks/report/fixtures/node-a.json` -- Create: `benchmarks/report/fixtures/engine-b.json` - -**Interfaces:** -- Consumes: `loadManifest` (Task 3). Result JSON files (schema from Task 1). -- Produces (all unit-testable without native): - - `computeDelta(value: number, baseline: number, unit: string): number` — percent change; sign is raw `(value-baseline)/baseline*100`. Interpretation (lower-better for `ms`, higher-better for `MB/s`) is applied by the caller/formatter. - - `detectSkew(results: Result[]): string[]` — distinct `env.weaveVersion` values across results (length > 1 ⇒ skew). - - `buildTable(manifest, results, baselineRunner): { header, rows }` — one row per `(id, metric)`, columns per runner plus a delta-vs-baseline column. - - `main(argv)` — parses `[files...] [--baseline ] [--emit ]`, prints the skew banner + table; `--emit` throws `not implemented` (documented seam). - -- [ ] **Step 1: Write fixtures** - -Create `benchmarks/report/fixtures/node-a.json`: - -```json -{ - "schemaVersion": "1.0", - "runner": "node-wrapper", - "env": { "os": "darwin-arm64", "cpu": "M1", "runtimeVersion": "node v20", "weaveVersion": "2.12.0-20260413", "commit": "abc", "dwlibBuildId": "dwlib-1" }, - "timestamp": "2026-07-22T12:00:00.000Z", - "cases": [ - { "id": "trivial", "metric": "warm", "unit": "ms", "stats": { "median": 2.0 }, "iterations": 100 }, - { "id": "map-scale", "metric": "streaming", "unit": "MB/s", "stats": { "median": 300 }, "iterations": 10 } - ] -} -``` - -Create `benchmarks/report/fixtures/engine-b.json`: - -```json -{ - "schemaVersion": "1.0", - "runner": "engine", - "env": { "os": "darwin-arm64", "cpu": "M1", "runtimeVersion": "jvm 24", "weaveVersion": "2.13.0-SNAPSHOT", "commit": "def", "dwlibBuildId": "n/a" }, - "timestamp": "2026-07-22T12:05:00.000Z", - "cases": [ - { "id": "trivial", "metric": "warm", "unit": "ms", "stats": { "median": 4.0 }, "iterations": 100 }, - { "id": "map-scale", "metric": "streaming", "unit": "MB/s", "stats": { "median": 150 }, "iterations": 10 } - ] -} -``` - -- [ ] **Step 2: Write the failing test** - -Create `benchmarks/report/report.test.mjs`: - -```js -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { readFileSync } from "node:fs"; -import { loadManifest } from "../lib/manifest.mjs"; -import { computeDelta, detectSkew, buildTable } from "./report.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "corpus"); -const load = (f) => JSON.parse(readFileSync(join(__dirname, "fixtures", f), "utf-8")); - -test("computeDelta is raw percent change", () => { - assert.equal(computeDelta(2, 4, "ms"), -50); // node 2ms vs engine 4ms - assert.equal(computeDelta(300, 150, "MB/s"), 100); -}); - -test("detectSkew finds differing weave versions", () => { - const skew = detectSkew([load("node-a.json"), load("engine-b.json")]); - assert.equal(skew.length, 2); - assert.ok(skew.includes("2.12.0-20260413")); - assert.ok(skew.includes("2.13.0-SNAPSHOT")); -}); - -test("buildTable joins by (id, metric) with a delta vs baseline", () => { - const manifest = loadManifest(CORPUS); - const { rows } = buildTable(manifest, [load("node-a.json"), load("engine-b.json")], "engine"); - const trivialWarm = rows.find((r) => r.id === "trivial" && r.metric === "warm"); - assert.ok(trivialWarm); - assert.equal(trivialWarm.values["node-wrapper"], 2.0); - assert.equal(trivialWarm.values["engine"], 4.0); - assert.equal(trivialWarm.delta, -50); // node is 50% lower (faster) than engine baseline -}); -``` - -- [ ] **Step 3: Run test to verify it fails** - -Run: `node --test benchmarks/report/report.test.mjs` -Expected: FAIL — cannot find module `./report.mjs`. - -- [ ] **Step 4: Write the implementation** - -Create `benchmarks/report/report.mjs`: - -```js -import { readFileSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath, pathToFileURL } from "node:url"; -import { loadManifest } from "../lib/manifest.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "corpus"); - -/** Raw percent change of value vs baseline. Sign interpretation is per-unit (caller decides). */ -export function computeDelta(value, baseline, _unit) { - if (baseline === 0) return NaN; - return ((value - baseline) / baseline) * 100; -} - -/** Distinct weaveVersions across results; length > 1 means the comparison spans versions. */ -export function detectSkew(results) { - return [...new Set(results.map((r) => r.env.weaveVersion))]; -} - -/** True if lower is better for this unit. */ -function lowerIsBetter(unit) { - return unit === "ms"; -} - -/** - * Join result files against the manifest into one row per (id, metric). - * @param baselineRunner runner name whose value anchors the delta column - */ -export function buildTable(manifest, results, baselineRunner) { - const runners = results.map((r) => r.runner); - const cellByRunner = new Map(); // `${runner}|${id}|${metric}` -> {value, unit} - for (const r of results) { - for (const c of r.cases) { - cellByRunner.set(`${r.runner}|${c.id}|${c.metric}`, { value: c.stats.median, unit: c.unit }); - } - } - - const rows = []; - for (const c of manifest.cases) { - for (const metric of c.metrics) { - const values = {}; - let unit = null; - for (const runner of runners) { - const cell = cellByRunner.get(`${runner}|${c.id}|${metric}`); - if (cell) { - values[runner] = cell.value; - unit = cell.unit; - } - } - if (Object.keys(values).length === 0) continue; // metric declared but no runner ran it - const base = values[baselineRunner]; - const other = runners.find((r) => r !== baselineRunner && values[r] !== undefined); - const delta = - base !== undefined && other !== undefined ? computeDelta(values[other], base, unit) : null; - rows.push({ id: c.id, metric, unit, values, delta, lowerIsBetter: lowerIsBetter(unit) }); - } - } - return { header: ["case", "metric", "unit", ...runners, `Δ vs ${baselineRunner}`], rows }; -} - -function fmt(n) { - return n === undefined ? "—" : Number(n).toFixed(2); -} - -export function main(argv) { - const files = []; - let baseline = null; - let emit = null; - for (let i = 0; i < argv.length; i++) { - if (argv[i] === "--baseline") baseline = argv[++i]; - else if (argv[i] === "--emit") emit = argv[++i]; - else files.push(argv[i]); - } - if (emit) throw new Error(`--emit ${emit} not implemented (exporter seam reserved for future history/dashboard)`); - if (files.length === 0) throw new Error("usage: report.mjs [--baseline ]"); - - const results = files.map((f) => JSON.parse(readFileSync(f, "utf-8"))); - const manifest = loadManifest(CORPUS); - const baselineRunner = baseline ?? (results.find((r) => r.runner === "engine")?.runner ?? results[0].runner); - - const skew = detectSkew(results); - if (skew.length > 1) { - console.log(`⚠️ WEAVE VERSION SKEW: comparing across ${skew.join(" vs ")} — deltas are not clean.`); - console.log(""); - } - - const { header, rows } = buildTable(manifest, results, baselineRunner); - console.log("| " + header.join(" | ") + " |"); - console.log("| " + header.map(() => "---").join(" | ") + " |"); - for (const row of rows) { - const runnerCols = header.slice(3, header.length - 1).map((runner) => fmt(row.values[runner])); - const deltaStr = row.delta === null ? "—" : `${row.delta > 0 ? "+" : ""}${row.delta.toFixed(1)}%`; - console.log(`| ${row.id} | ${row.metric} | ${row.unit} | ${runnerCols.join(" | ")} | ${deltaStr} |`); - } -} - -if (import.meta.url === pathToFileURL(process.argv[1]).href) { - try { - main(process.argv.slice(2)); - } catch (e) { - console.error(e.message); - process.exit(1); - } -} -``` - -- [ ] **Step 5: Run test to verify it passes** - -Run: `node --test benchmarks/report/report.test.mjs` -Expected: PASS (3 tests). - -- [ ] **Step 6: Smoke the CLI against fixtures** - -Run: -```bash -node benchmarks/report/report.mjs benchmarks/report/fixtures/node-a.json benchmarks/report/fixtures/engine-b.json --baseline engine -``` -Expected: a skew banner (2.12 vs 2.13) followed by a markdown table with `node-wrapper`, `engine`, and a `Δ vs engine` column. - -- [ ] **Step 7: Commit** - -```bash -git add benchmarks/report/ -git commit -m "W-23545283: Add report script (join, delta, skew banner)" -``` - ---- - -### Task 9: Gradle wiring (`native-lib:benchmark`) - -**Files:** -- Modify: `native-lib/build.gradle` (add a `benchmark` task after the `nodeTest` task, around line 199) - -**Interfaces:** -- Consumes: `buildNodePackage` (existing task — builds wrapper dist + addon + stages dwlib); the Node runner (`emit.mjs`) and report (`report.mjs`). -- Produces: an opt-in `native-lib:benchmark` Gradle task, skipped unless `-Pbenchmark=true`, not wired into `build`/`test`. - -- [ ] **Step 1: Add the task** - -In `native-lib/build.gradle`, immediately after the `nodeTest` task registration (the block ending at line 199), add: - -```groovy -tasks.register('benchmark', Exec) { - // Opt-in only: skipped unless -Pbenchmark=true. Never part of build/test. - onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } - - dependsOn tasks.named('buildNodePackage') - workingDir("${rootDir}/benchmarks") - - def script = 'node corpus/gen-inputs.mjs && node runners/node/emit.mjs && node report/report.mjs results/*.json' - if (System.getProperty('os.name').toLowerCase().contains('windows')) { - commandLine('cmd', '/c', script) - } else { - commandLine('bash', '-c', script) - } -} -``` - -- [ ] **Step 2: Verify the task is registered and skipped by default** - -Run: -```bash -./gradlew native-lib:benchmark --dry-run -``` -Expected: the task list includes `:native-lib:benchmark` (dry-run prints the task graph without executing). Running without `-Pbenchmark=true` and without dry-run shows the task `SKIPPED` due to `onlyIf`. - -- [ ] **Step 3: Verify the full opt-in run (integration)** - -Run: -```bash -./gradlew native-lib:benchmark -Pbenchmark=true -``` -Expected: builds the wrapper, generates inputs, writes a `benchmarks/results/node-*.json`, and prints the comparison table (single runner — no skew banner). This is slow (native build); acceptable for a manual/opt-in benchmark run. - -- [ ] **Step 4: Commit** - -```bash -git add native-lib/build.gradle -git commit -m "W-23545283: Wire opt-in native-lib:benchmark Gradle task" -``` - ---- - -## Follow-up scope (not in this plan) - -- `benchmarks/runners/python/` — Python runner emitting the same schema. -- `benchmarks/runners/engine/` — pointer/README here; the `-Ddw.perf`-gated scalatest harness lives in the `data-weave` repo, reads this corpus, emits this schema. -- Results accumulation / dashboard (the `--emit` seam is reserved but unimplemented). - -## Self-Review - -**Spec coverage:** -- Shared corpus + manifest → Task 3. ✓ -- Common JSON schema (incl. `env.commit`/`dwlibBuildId`, immutable id, `schemaVersion`) → Tasks 1, 4, 7. ✓ -- Four metrics → Tasks 5 (warm, streaming) + 6 (cold-start, first-run). ✓ -- Explicit `metrics[]` + fail-fast orphan id → Task 3 (`loadManifest`, `validateResultIds`) enforced in Task 7. ✓ -- Node reference runner (in-process + spawn harness) → Tasks 5, 6, 7. ✓ -- Report with per-unit delta direction + skew banner + `--baseline` + `--emit` seam → Task 8. ✓ -- Gradle opt-in task, off by default → Task 9. ✓ -- Results local-only / gitignored → Task 1 `.gitignore`. ✓ -- `benchmarks/runners/` extension point for Python/engine → layout in Task 1 README + Follow-up scope. ✓ - -**Placeholder scan:** No "TBD"/"handle edge cases" placeholders. Two ESM-correctness notes (env.mjs, warm-bench.mjs, emit.mjs) explicitly give the corrected code to ship rather than leaving it vague. ✓ - -**Type consistency:** `ResultCase = {id, metric, unit, stats, iterations}` is used identically in Tasks 5, 6, 7, 8. `computeStats` return shape matches the schema `stats` object (Task 1). `loadWrapper()` return `{run, runStreaming, runTransform, DataWeave, cleanup}` matches the wrapper's actual exports and is consumed consistently in Tasks 5–7. `gatherEnv` output fields match the schema `env` required list. ✓ - ---- - -**Deviations flagged for reviewer veto:** -1. **Timing harness:** self-contained `process.hrtime.bigint()` sampling instead of vitest `bench` — so the future engine harness can mirror the exact methodology (the spec named vitest `bench`). -2. **first-run placement:** measured in the spawn harness (fresh process = cold compilation) rather than in-process (spec §4). Yields a real cold-compilation distribution. diff --git a/docs/superpowers/plans/2026-07-23-engine-runner.md b/docs/superpowers/plans/2026-07-23-engine-runner.md deleted file mode 100644 index c2811da..0000000 --- a/docs/superpowers/plans/2026-07-23-engine-runner.md +++ /dev/null @@ -1,1326 +0,0 @@ -# Engine Runner (JVM Benchmark Baseline) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build the engine runner — a Scala/Gradle subproject at `benchmarks/runners/engine/` that drives the bare `DataWeaveScriptingEngine` over the shared benchmark corpus and emits a schema-conformant `results/engine-.json`, so `report/report.mjs` produces the headline native-wrapper-vs-JVM-engine comparison at the same weave version. - -**Architecture:** A new Gradle subproject depends on `org.mule.weave:runtime` (already a build dependency of this repo) so `DataWeaveScriptingEngine` is on the JVM classpath at the *same* `weaveVersion` the native image is built from. The harness mirrors `native-cli`'s `NativeRuntime.scala`: a minimal `EngineShell` compiles a script and writes into an `OutputStream`. Cold-start and first-run are measured by spawning a fresh JVM per sample (`EngineChild`), mirroring the Node `coldstart-child.mjs`; warm and streaming are measured in-process (`WarmBench`) with a JVM JIT warmup floor. `Emit` orchestrates both, computes stats with nearest-rank percentiles identical to `benchmarks/lib/stats.mjs` (guarded by a parity test), stamps env, fail-fast-validates case ids against the manifest, and writes the result JSON. - -**Tech Stack:** Scala 2.12.18, `org.mule.weave:runtime` + `core-modules` (weave engine + data formats), `org.json:json` (JSON I/O), scalatest (`com.github.maiflai.scalatest` plugin + `flexmark` runtime, matching `native-cli`), Gradle `JavaExec`, `System.nanoTime()` timing. Reuses the existing `benchmarks/corpus/`, `benchmarks/schema/`, and `report/report.mjs` unchanged; regenerates large inputs via the existing `corpus/gen-inputs.mjs` (Node). - -## Global Constraints - -- **Same weave version, no skew.** The engine runner uses `api org.mule.weave:runtime:${weaveVersion}` — the *same* `weaveVersion` (`2.12.0-20260413`) the native image is built from. Do not pin a different version. `report.mjs` prints no skew banner in the normal path. -- **Timing methodology (every metric):** measure with `System.nanoTime()`; convert to ms as `(endNs - startNs) / 1e6` (Double). Identical in spirit to the Node runner's `process.hrtime.bigint()` → ms. -- **`id` is the immutable join key.** Emit each case `id` verbatim from the manifest; never invent or rename. Deprecate, never rename. -- **Fail-fast on orphan ids.** `Emit` MUST abort before writing output if any emitted `id` is absent from the manifest. -- **Explicit metrics.** Each manifest case declares `metrics[]` from exactly `["cold-start","first-run","warm","streaming"]`. Run only the metrics a case declares (`casesForMetric`). -- **Correctness guard.** A case whose engine run throws (compile/exec failure) aborts that case with a clear error — never record a bogus fast timing. -- **Schema conformance (mandatory `env` fields).** Every result file validates against `benchmarks/schema/result.schema.json`: top-level `schemaVersion:"1.0"`, `runner`, `env`, `timestamp`, `cases[]`; each case a flat `{id, metric, unit, stats, iterations}` row with no extra keys; `env` includes `os`, `cpu`, `runtimeVersion`, `weaveVersion`, `commit`, `dwlibBuildId` (all required). `dwlibBuildId` has no meaning for a JVM runner → literal `"n/a-engine"`. -- **`runner` is `"engine"`** so `report.mjs:92` auto-selects it as the delta baseline. -- **Units per row:** `ms` for latency metrics (cold-start, first-run, warm), `MB/s` for streaming. -- **JVM warmup floor:** `warm` warmup iterations = `max(manifestWarmup, 2000)` so the JIT reaches steady state; the effective warmup used is logged. -- **Results are local-only.** Output to `benchmarks/results/` (already gitignored via `benchmarks/.gitignore`). Do not commit result files or generated inputs. -- **Deterministic large inputs.** The generated `corpus/inputs/generated/records-large.json` is produced by the existing `corpus/gen-inputs.mjs` (Node) and shared byte-for-byte across runners — the engine consumes that file, never regenerates it differently. - ---- - -### Task 1: Gradle subproject scaffold + settings wiring + smoke test - -**Files:** -- Modify: `settings.gradle` -- Create: `benchmarks/runners/engine/build.gradle` -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/CountingOutputStream.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/SmokeTest.scala` - -**Interfaces:** -- Consumes: nothing. -- Produces: a compilable, testable Gradle subproject at path `:benchmarks-engine`, package `org.mule.weave.benchmark.engine`. `CountingOutputStream` — a `java.io.OutputStream` that discards bytes and exposes `def count(): Long` (used by later tasks as the write sink). - -- [ ] **Step 1: Add the subproject to `settings.gradle`** - -Append to `settings.gradle` (currently three `include` lines): - -```groovy -include 'benchmarks-engine' -project(':benchmarks-engine').projectDir = file('benchmarks/runners/engine') -``` - -- [ ] **Step 2: Create the module `build.gradle`** - -Create `benchmarks/runners/engine/build.gradle`. The root `subprojects {}` block already applies `scala`, `java-library`, `scala-library`, the mule repositories, and the graalvm plugin — this module only adds the scalatest plugin, the weave deps, `org.json`, test deps, and the two tasks: - -```groovy -plugins { - id "com.github.maiflai.scalatest" version "${scalaTestPluginVersion}" -} - -dependencies { - api group: 'org.mule.weave', name: 'runtime', version: weaveVersion - api group: 'org.mule.weave', name: 'core-modules', version: weaveVersion - implementation group: 'org.mule.weave', name: 'parser', version: weaveVersion - implementation group: 'org.mule.weave', name: 'wlang', version: weaveVersion - implementation 'org.json:json:20240303' - - testImplementation group: 'org.scalatest', name: 'scalatest_2.12', version: scalaTestVersion - testRuntimeOnly 'com.vladsch.flexmark:flexmark-all:0.62.2' -} - -def benchmarksDir = "${rootDir}/benchmarks" - -// Regenerate the shared large input via the existing Node generator (idempotent, -// deterministic). Keeps the engine consuming the SAME bytes as other runners. -tasks.register('genBenchInputs', Exec) { - onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } - workingDir(benchmarksDir) - if (System.getProperty('os.name').toLowerCase().contains('windows')) { - commandLine('cmd', '/c', 'node corpus/gen-inputs.mjs') - } else { - commandLine('bash', '-c', 'node corpus/gen-inputs.mjs') - } -} - -// Opt-in only: skipped unless -Pbenchmark=true. Never part of build/test. -tasks.register('benchmarkEngine', JavaExec) { - onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } - dependsOn tasks.named('genBenchInputs'), tasks.named('classes') - classpath = sourceSets.main.runtimeClasspath - mainClass = 'org.mule.weave.benchmark.engine.Emit' - args("${benchmarksDir}/corpus", "${benchmarksDir}/results", "${rootDir}") - jvmArgs = ['-Xmx6G'] -} -``` - -- [ ] **Step 3: Write the failing smoke test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/SmokeTest.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers - -class SmokeTest extends AnyFreeSpec with Matchers { - "CountingOutputStream counts written bytes" in { - val out = new CountingOutputStream() - out.write("hello".getBytes("UTF-8")) - out.write(42) - out.count() shouldBe 6L - } -} -``` - -- [ ] **Step 4: Run the test to verify it fails (does not compile — class missing)** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.SmokeTest"` -Expected: FAIL — compilation error, `CountingOutputStream` not found. - -- [ ] **Step 5: Implement `CountingOutputStream`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/CountingOutputStream.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import java.io.OutputStream - -/** An OutputStream that discards all bytes and only counts how many were written. - * Used as the write sink when benchmarking — we measure the work of producing - * output without paying for allocation/retention of the result. */ -class CountingOutputStream extends OutputStream { - private var written: Long = 0L - - override def write(b: Int): Unit = { written += 1 } - - override def write(b: Array[Byte]): Unit = { written += b.length } - - override def write(b: Array[Byte], off: Int, len: Int): Unit = { written += len } - - def count(): Long = written -} -``` - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.SmokeTest"` -Expected: PASS (1 test). This proves the subproject compiles, resolves the weave deps, and runs scalatest. - -- [ ] **Step 7: Commit** - -```bash -git add settings.gradle benchmarks/runners/engine/build.gradle \ - benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/CountingOutputStream.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/SmokeTest.scala -git commit -m "W-23545283: Scaffold engine-runner Gradle subproject" -``` - ---- - -### Task 2: Stats — nearest-rank percentiles with parity test vs `lib/stats.mjs` - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Stats.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/StatsParityTest.scala` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `Stats.Summary(min: Double, median: Double, p90: Double, p99: Double, mean: Double)` - - `Stats.computeStats(samples: Seq[Double]): Stats.Summary` — nearest-rank percentiles on a sorted copy: `pct(p) = sorted( min(n-1, max(0, ceil(p/100 * n) - 1)) )`; `mean = sum/n`. Throws `IllegalArgumentException` on empty input. Identical math to `benchmarks/lib/stats.mjs:computeStats`. - - `Stats.toMBps(totalBytes: Long, elapsedMs: Double): Double` — `totalBytes / 1e6 / (elapsedMs / 1000)`; throws if `elapsedMs <= 0`. Identical to `lib/stats.mjs:toMBps`. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/StatsParityTest.scala`. These expected values are the exact outputs of `benchmarks/lib/stats.mjs` for the same inputs: - -```scala -package org.mule.weave.benchmark.engine - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers - -class StatsParityTest extends AnyFreeSpec with Matchers { - "computeStats matches lib/stats.mjs nearest-rank on 1..100" in { - val s = Stats.computeStats((1 to 100).map(_.toDouble)) - s.min shouldBe 1.0 - s.median shouldBe 50.0 // ceil(0.5*100)-1 = 49 -> sorted(49) = 50 - s.p90 shouldBe 90.0 // ceil(0.9*100)-1 = 89 -> 90 - s.p99 shouldBe 99.0 // ceil(0.99*100)-1 = 98 -> 99 - s.mean shouldBe 50.5 - } - - "computeStats sorts before ranking" in { - val s = Stats.computeStats(Seq(5.0, 1.0, 3.0, 2.0, 4.0)) - s.min shouldBe 1.0 - s.median shouldBe 3.0 // ceil(0.5*5)-1 = 2 -> sorted(2) = 3 - s.p90 shouldBe 5.0 // ceil(0.9*5)-1 = 4 -> 5 - s.mean shouldBe 3.0 - } - - "computeStats rejects empty input" in { - an [IllegalArgumentException] should be thrownBy Stats.computeStats(Seq.empty) - } - - "toMBps matches decimal-MB convention" in { - Stats.toMBps(1000000L, 1000.0) shouldBe 1.0 - Stats.toMBps(500000L, 250.0) shouldBe 2.0 - an [IllegalArgumentException] should be thrownBy Stats.toMBps(10L, 0.0) - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.StatsParityTest"` -Expected: FAIL — `Stats` not found (compilation error). - -- [ ] **Step 3: Implement `Stats`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Stats.scala`: - -```scala -package org.mule.weave.benchmark.engine - -/** Percentile + throughput math kept byte-for-byte compatible with - * benchmarks/lib/stats.mjs so engine deltas compare cleanly against Node. - * Any divergence is caught by StatsParityTest. */ -object Stats { - - final case class Summary(min: Double, median: Double, p90: Double, p99: Double, mean: Double) - - /** Nearest-rank percentiles on a sorted copy; mean is the arithmetic mean. */ - def computeStats(samples: Seq[Double]): Summary = { - if (samples.isEmpty) { - throw new IllegalArgumentException("computeStats requires a non-empty sequence of numbers") - } - val sorted = samples.sorted.toVector - val n = sorted.length - def pct(p: Double): Double = { - val idx = math.min(n - 1, math.max(0, math.ceil(p / 100.0 * n).toInt - 1)) - sorted(idx) - } - val sum = sorted.sum - Summary(min = sorted.head, median = pct(50), p90 = pct(90), p99 = pct(99), mean = sum / n) - } - - /** Throughput in decimal megabytes per second (1 MB = 1e6 bytes). */ - def toMBps(totalBytes: Long, elapsedMs: Double): Double = { - if (elapsedMs <= 0) throw new IllegalArgumentException("elapsedMs must be > 0") - totalBytes / 1e6 / (elapsedMs / 1000.0) - } -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.StatsParityTest"` -Expected: PASS (4 tests). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Stats.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/StatsParityTest.scala -git commit -m "W-23545283: Add engine-runner stats with lib/stats.mjs parity test" -``` - ---- - -### Task 3: Manifest parser + input resolution - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Manifest.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala` - -**Interfaces:** -- Consumes: `org.json` (parsing), the real corpus at `benchmarks/corpus/`. -- Produces: - - `case class CaseInput(name: String, file: String, mimeType: String, charset: Option[String], generated: Boolean)` - - `case class BenchCase(id: String, script: String, inputs: Seq[CaseInput], metrics: Set[String], iterations: Map[String, Int])` with helpers `warm: Int` (default 100), `warmup: Int` (default 10), `streaming: Int` (default 10), `samples: Int` (default 20). - - `case class ResolvedInput(name: String, bytes: Array[Byte], mimeType: String, charset: Option[String])` - - `class Manifest(val corpusDir: File, val cases: Seq[BenchCase])` with `def ids: Set[String]`. - - `object Manifest`: - - `def load(corpusDir: File): Manifest` — parse + validate `manifest.json` (mirrors `lib/manifest.mjs:loadManifest`: non-empty unique ids, metrics from the allowed set, script file exists, non-generated input files exist). Throws on any violation. - - `def casesForMetric(m: Manifest, metric: String): Seq[BenchCase]` - - `def validateResultIds(m: Manifest, resultIds: Seq[String]): Unit` — throws on any id not in `m.ids`. - - `def resolveScript(m: Manifest, c: BenchCase): String` — read the script file. - - `def resolveInputs(m: Manifest, c: BenchCase): Seq[ResolvedInput]` — read each input file into bytes. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala`. Uses the real corpus (Gradle runs tests with `workingDir = `, i.e. `benchmarks/runners/engine`, so `../../corpus` is `benchmarks/corpus`): - -```scala -package org.mule.weave.benchmark.engine - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers -import java.io.File - -class ManifestTest extends AnyFreeSpec with Matchers { - private val corpus = new File("../../corpus").getCanonicalFile - - "loads all corpus cases with stable ids" in { - val m = Manifest.load(corpus) - m.ids should contain allOf ("trivial", "object-transform", "map-scale", "xml-to-csv", "json-stream", "compile-heavy") - } - - "casesForMetric filters by declared metrics" in { - val m = Manifest.load(corpus) - Manifest.casesForMetric(m, "streaming").map(_.id) should contain allOf ("map-scale", "json-stream") - Manifest.casesForMetric(m, "cold-start").map(_.id) should contain ("trivial") - } - - "resolveScript reads the .dwl source" in { - val m = Manifest.load(corpus) - val trivial = m.cases.find(_.id == "trivial").get - Manifest.resolveScript(m, trivial).trim should include ("2 + 2") - } - - "resolveInputs reads committed input bytes with mime + charset" in { - val m = Manifest.load(corpus) - val xml = m.cases.find(_.id == "xml-to-csv").get - val inputs = Manifest.resolveInputs(m, xml) - inputs should have size 1 - inputs.head.name shouldBe "payload" - inputs.head.mimeType shouldBe "application/xml" - inputs.head.charset shouldBe Some("UTF-16") - inputs.head.bytes.length should be > 0 - } - - "validateResultIds throws on an orphan id" in { - val m = Manifest.load(corpus) - an [RuntimeException] should be thrownBy Manifest.validateResultIds(m, Seq("trivial", "not-a-case")) - noException should be thrownBy Manifest.validateResultIds(m, Seq("trivial")) - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.ManifestTest"` -Expected: FAIL — `Manifest` not found. - -- [ ] **Step 3: Implement `Manifest`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Manifest.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.json.{ JSONArray, JSONObject } - -import java.io.File -import java.nio.charset.StandardCharsets -import java.nio.file.Files -import scala.collection.mutable - -final case class CaseInput(name: String, file: String, mimeType: String, charset: Option[String], generated: Boolean) - -final case class BenchCase( - id: String, - script: String, - inputs: Seq[CaseInput], - metrics: Set[String], - iterations: Map[String, Int]) { - - def warm: Int = iterations.getOrElse("warm", 100) - def warmup: Int = iterations.getOrElse("warmup", 10) - def streaming: Int = iterations.getOrElse("streaming", 10) - def samples: Int = iterations.getOrElse("samples", 20) -} - -final case class ResolvedInput(name: String, bytes: Array[Byte], mimeType: String, charset: Option[String]) - -class Manifest(val corpusDir: File, val cases: Seq[BenchCase]) { - def ids: Set[String] = cases.map(_.id).toSet -} - -object Manifest { - - private val AllowedMetrics = Set("cold-start", "first-run", "warm", "streaming") - - def load(corpusDir: File): Manifest = { - val manifestFile = new File(corpusDir, "manifest.json") - val raw = new String(Files.readAllBytes(manifestFile.toPath), StandardCharsets.UTF_8) - val root = new JSONObject(raw) - val casesArr: JSONArray = root.getJSONArray("cases") - - val seen = mutable.Set[String]() - val cases = (0 until casesArr.length()).map { i => - val obj = casesArr.getJSONObject(i) - val id = obj.getString("id") - if (id.isEmpty) throw new RuntimeException("manifest case is missing an id") - if (seen.contains(id)) throw new RuntimeException(s"duplicate case id: $id") - seen += id - - val metricsArr = obj.getJSONArray("metrics") - if (metricsArr.length() == 0) throw new RuntimeException(s"case $id must declare a non-empty metrics[]") - val metrics = (0 until metricsArr.length()).map(metricsArr.getString).toSet - metrics.foreach(m => if (!AllowedMetrics.contains(m)) throw new RuntimeException(s"case $id has unknown metric: $m")) - - val script = obj.getString("script") - if (!new File(corpusDir, script).exists()) throw new RuntimeException(s"case $id script not found: $script") - - val iterations: Map[String, Int] = - if (obj.has("iterations")) { - val it = obj.getJSONObject("iterations") - it.keySet().toArray.map(_.asInstanceOf[String]).map(k => k -> it.getInt(k)).toMap - } else Map.empty - - val inputs: Seq[CaseInput] = - if (obj.has("inputs")) { - val ins = obj.getJSONObject("inputs") - ins.keySet().toArray.map(_.asInstanceOf[String]).toSeq.map { name => - val io = ins.getJSONObject(name) - val file = io.getString("file") - val generated = io.optBoolean("generated", false) - if (!generated && !new File(corpusDir, file).exists()) { - throw new RuntimeException(s"case $id input '$name' file not found: $file") - } - CaseInput( - name = name, - file = file, - mimeType = io.getString("mimeType"), - charset = if (io.has("charset")) Some(io.getString("charset")) else None, - generated = generated) - } - } else Seq.empty - - BenchCase(id, script, inputs, metrics, iterations) - } - new Manifest(corpusDir, cases) - } - - def casesForMetric(m: Manifest, metric: String): Seq[BenchCase] = - m.cases.filter(_.metrics.contains(metric)) - - def validateResultIds(m: Manifest, resultIds: Seq[String]): Unit = { - resultIds.foreach { id => - if (!m.ids.contains(id)) throw new RuntimeException(s"result contains orphan id not in manifest: $id") - } - } - - def resolveScript(m: Manifest, c: BenchCase): String = - new String(Files.readAllBytes(new File(m.corpusDir, c.script).toPath), StandardCharsets.UTF_8) - - def resolveInputs(m: Manifest, c: BenchCase): Seq[ResolvedInput] = - c.inputs.map { in => - val bytes = Files.readAllBytes(new File(m.corpusDir, in.file).toPath) - ResolvedInput(in.name, bytes, in.mimeType, in.charset) - } -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.ManifestTest"` -Expected: PASS (5 tests). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Manifest.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala -git commit -m "W-23545283: Add engine-runner manifest parser + input resolution" -``` - ---- - -### Task 4: EngineShell — the bare `DataWeaveScriptingEngine` - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineShell.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineShellTest.scala` - -**Interfaces:** -- Consumes: `Manifest.ResolvedInput`, `CountingOutputStream`, the weave runtime API (`DataWeaveScriptingEngine`, `ModuleComponentsFactory`, `ClassLoaderWeaveResourceResolver`, `ParserConfiguration`, `ScriptingBindings`, `BindingValue`, `InputType`, `ServiceManager`, `CharsetProviderService`, `NameIdentifier`). -- Produces: - - `class EngineShell` — constructing it builds a fresh engine (this construction is what `EngineChild` times as `initMs`). - - `EngineShell.run(script: String, name: String, inputs: Seq[ResolvedInput], out: OutputStream): Unit` — compile the script and write output into `out`. Throws on compile/exec failure (the correctness guard). Recompiles on every call (matches the Node wrapper, which recompiles each `run()`), so `warm` measures compile+exec on both sides. - - `EngineShell.safeName(id: String): String` — a NameIdentifier-safe logical name derived from a case id (`"bench_" + id.replaceAll("[^A-Za-z0-9_]", "_")`), so different scripts compiled on one engine never collide on name. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineShellTest.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers -import java.io.File - -class EngineShellTest extends AnyFreeSpec with Matchers { - private val corpus = new File("../../corpus").getCanonicalFile - private val manifest = Manifest.load(corpus) - - private def runCase(id: String): Long = { - val c = manifest.cases.find(_.id == id).get - val shell = new EngineShell() - val out = new CountingOutputStream() - shell.run(Manifest.resolveScript(manifest, c), EngineShell.safeName(id), Manifest.resolveInputs(manifest, c), out) - out.count() - } - - "runs a no-input script (trivial) and writes output" in { - runCase("trivial") should be > 0L - } - - "runs an object transform with a JSON input binding" in { - runCase("object-transform") should be > 0L - } - - "runs the UTF-16 xml-to-csv case (charset path)" in { - runCase("xml-to-csv") should be > 0L - } - - "safeName sanitizes hyphens" in { - EngineShell.safeName("xml-to-csv") shouldBe "bench_xml_to_csv" - } - - "a failing script aborts with an exception" in { - val shell = new EngineShell() - an [Exception] should be thrownBy shell.run("output application/json --- payload.nope + 1", "bench_bad", Seq.empty, new CountingOutputStream()) - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineShellTest"` -Expected: FAIL — `EngineShell` not found. - -- [ ] **Step 3: Implement `EngineShell`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineShell.scala`. This is a minimal reduction of `native-cli`'s `NativeRuntime.scala` — classloader resolver only, a UTF-8 `CharsetProviderService`, and `compileWith` + `write(bindings, sm, Option(out))` exactly as `NativeRuntime.run` drives it: - -```scala -package org.mule.weave.benchmark.engine - -import io.netty.util.internal.PlatformDependent -import org.mule.weave.v2.model.ServiceManager -import org.mule.weave.v2.model.service.CharsetProviderService -import org.mule.weave.v2.parser.ast.variables.NameIdentifier -import org.mule.weave.v2.runtime.{ - BindingValue, - DataWeaveScript, - DataWeaveScriptingEngine, - InputType, - ModuleComponentsFactory, - ParserConfiguration, - ScriptingBindings -} -import org.mule.weave.v2.sdk.ClassLoaderWeaveResourceResolver - -import java.io.OutputStream -import java.nio.charset.{ Charset, StandardCharsets } -import java.util.Properties - -/** A minimal engine harness: builds a bare DataWeaveScriptingEngine (classloader - * resolver only) and compiles+writes a script per run(), mirroring how - * native-cli's NativeRuntime drives the engine. Constructing this class is the - * work EngineChild times as `initMs`. */ -class EngineShell { - - EngineShell.setupEnv() - - private val engine: DataWeaveScriptingEngine = { - val resolver = ClassLoaderWeaveResourceResolver.apply() - new DataWeaveScriptingEngine(ModuleComponentsFactory.apply(resolver), ParserConfiguration(), new Properties()) - } - - // UTF-8 default charset service, matching NativeRuntime.createServiceManager. - // Required so cases that don't pin a charset decode as UTF-8; per-input charsets - // (e.g. the UTF-16 xml-to-csv case) come from the binding itself. - private val serviceManager: ServiceManager = { - val charsetService = new CharsetProviderService { - override def defaultCharset(): Charset = StandardCharsets.UTF_8 - } - val customServices: Map[Class[_], _] = Map(classOf[CharsetProviderService] -> charsetService) - ServiceManager(customServices) - } - - /** Compile `script` and write its output into `out`. Throws on failure. */ - def run(script: String, name: String, inputs: Seq[ResolvedInput], out: OutputStream): Unit = { - val bindings = new ScriptingBindings() - inputs.foreach { in => - val charset = Charset.forName(in.charset.getOrElse("UTF-8")) - val bv = new BindingValue(in.bytes, Some(in.mimeType), Map.empty[String, Any], charset) - bindings.addBinding(in.name, bv) - } - - val config = engine.newConfig() - .withScript(script) - .withNameIdentifier(NameIdentifier(name)) - .withInputs(inputs.map(in => new InputType(in.name, None)).toArray) - .withDefaultOutputType("application/json") - - val compiled: DataWeaveScript = engine.compileWith(config) - // 3-arg write(bindings, serviceManager, target: Option[Any]) writes into `out`, - // exactly as NativeRuntime.run does. A compile/exec failure throws here. - compiled.write(bindings, serviceManager, Option(out)) - } -} - -object EngineShell { - - /** Netty init properties, copied from NativeRuntime.setupEnv. */ - def setupEnv(): Unit = { - System.setProperty("io.netty.processId", Math.abs(PlatformDependent.threadLocalRandom.nextInt).toString) - System.setProperty("io.netty.noUnsafe", true.toString) - } - - /** A NameIdentifier-safe logical name derived from a case id. */ - def safeName(id: String): String = "bench_" + id.replaceAll("[^A-Za-z0-9_]", "_") -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineShellTest"` -Expected: PASS (5 tests). If a data format is unexpectedly missing (runtime "unknown mime type"), confirm `core-modules` is on the classpath (Task 1 dep) — it ships the `META-INF/services` DataFormat/ModuleLoader files. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineShell.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineShellTest.scala -git commit -m "W-23545283: Add EngineShell driving bare DataWeaveScriptingEngine" -``` - ---- - -### Task 5: EngineChild — fresh-JVM cold-start / first-run worker - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineChild.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineChildTest.scala` - -**Interfaces:** -- Consumes: `Manifest`, `EngineShell`, `CountingOutputStream`. -- Produces: `object EngineChild` with `def main(args: Array[String]): Unit`. Args: ` `. Times `initMs` (EngineShell construction) and `firstRunMs` (first compile+write), then prints exactly one JSON line to stdout: `{"initMs":,"firstRunMs":}`. Exits non-zero (uncaught exception) if the case run fails. This is the fresh-JVM analog of `runners/node/coldstart-child.mjs`. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineChildTest.scala`. It spawns a real child JVM using the test's own classpath — proving the spawn path `Emit` will use works end-to-end: - -```scala -package org.mule.weave.benchmark.engine - -import org.json.JSONObject -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers -import java.io.File - -class EngineChildTest extends AnyFreeSpec with Matchers { - private val corpus = new File("../../corpus").getCanonicalFile - - private def spawn(caseId: String): (Int, String) = { - val javaBin = new File(System.getProperty("java.home"), "bin/java").getAbsolutePath - val cp = System.getProperty("java.class.path") - val pb = new ProcessBuilder( - javaBin, "-cp", cp, - "org.mule.weave.benchmark.engine.EngineChild", - corpus.getAbsolutePath, caseId) - val p = pb.start() - val out = scala.io.Source.fromInputStream(p.getInputStream).getLines().toList - val code = p.waitFor() - (code, out.lastOption.getOrElse("")) - } - - "child prints init + first-run timings for a case" in { - val (code, line) = spawn("trivial") - code shouldBe 0 - val obj = new JSONObject(line) - obj.getDouble("initMs") should be > 0.0 - obj.getDouble("firstRunMs") should be > 0.0 - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineChildTest"` -Expected: FAIL — child exits non-zero / no output because `EngineChild` main class does not exist. - -- [ ] **Step 3: Implement `EngineChild`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineChild.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import java.io.File - -/** Fresh-process worker. Measures a cold engine init + a cold (first) compile+exec - * for one case, then prints a single JSON line. Spawned by Emit — the honest JVM - * cold path (process launch + classload + engine init + first compile). */ -object EngineChild { - - private def nowNs(): Long = System.nanoTime() - private def msSince(startNs: Long): Double = (System.nanoTime() - startNs) / 1e6 - - def main(args: Array[String]): Unit = { - val corpusDir = new File(args(0)) - val caseId = args(1) - - val manifest = Manifest.load(corpusDir) - val c = manifest.cases.find(_.id == caseId).getOrElse(sys.error(s"unknown case: $caseId")) - val script = Manifest.resolveScript(manifest, c) - val inputs = Manifest.resolveInputs(manifest, c) - - val initStart = nowNs() - val shell = new EngineShell() - val initMs = msSince(initStart) - - val runStart = nowNs() - shell.run(script, EngineShell.safeName(caseId), inputs, new CountingOutputStream()) - val firstRunMs = msSince(runStart) - - // Single JSON line on stdout; Emit reads the last line. - println(s"""{"initMs":$initMs,"firstRunMs":$firstRunMs}""") - } -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineChildTest"` -Expected: PASS (1 test). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineChild.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineChildTest.scala -git commit -m "W-23545283: Add EngineChild fresh-JVM cold-start/first-run worker" -``` - ---- - -### Task 6: WarmBench — in-process warm (JIT floor) + streaming - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Row.scala` -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/WarmBench.scala` -- Create: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/TestSupport.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/WarmBenchTest.scala` - -**Interfaces:** -- Consumes: `EngineShell`, `Manifest`, `Stats`, `CountingOutputStream`. -- Produces: - - `case class Row(id: String, metric: String, unit: String, stats: Stats.Summary, iterations: Int)` (shared result row; also used by Task 7). - - `object WarmBench`: - - `WARMUP_FLOOR: Int = 2000` - - `runWarm(shell: EngineShell, m: Manifest, warmupCap: Option[Int] = None, iterCap: Option[Int] = None): Seq[Row]` — for each `casesForMetric(m,"warm")`: warmup `warmupCap.getOrElse(max(case.warmup, WARMUP_FLOOR))` iterations, then time `iterCap.getOrElse(case.warm)` iterations of `compile+write`; emit a `warm`/`ms` row. Logs the effective warmup per case. - - `runStreaming(shell: EngineShell, m: Manifest, iterCap: Option[Int] = None): Seq[Row]` — for each `casesForMetric(m,"streaming")`: time `iterCap.getOrElse(case.streaming)` iterations; MB/s = `Stats.toMBps(primaryInput.bytes.length, elapsedMs)` over the first declared input; emit a `streaming`/`MB/s` row. - - `object TestSupport.ensureGeneratedInputs(corpus: File): Unit` — runs `node corpus/gen-inputs.mjs` with `BENCH_LARGE_N=500` if the generated file is missing; used by tests that touch generated-input cases. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/TestSupport.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import java.io.File - -object TestSupport { - /** Ensure the shared large input exists (small N for tests). Best-effort: - * requires Node, which the benchmarks already depend on. */ - def ensureGeneratedInputs(corpus: File): Boolean = { - val gen = new File(corpus, "inputs/generated/records-large.json") - if (gen.exists()) return true - try { - val pb = new ProcessBuilder("node", "corpus/gen-inputs.mjs").directory(corpus.getParentFile) - pb.environment().put("BENCH_LARGE_N", "500") - pb.inheritIO() - pb.start().waitFor() - gen.exists() - } catch { - case _: Throwable => false - } - } -} -``` - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/WarmBenchTest.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers -import java.io.File - -class WarmBenchTest extends AnyFreeSpec with Matchers { - private val corpus = new File("../../corpus").getCanonicalFile - private val manifest = Manifest.load(corpus) - - "warm rows are produced with ms unit and positive median" in { - val shell = new EngineShell() - val rows = WarmBench.runWarm(shell, manifest, warmupCap = Some(2), iterCap = Some(3)) - rows.map(_.id) should contain ("trivial") - all (rows.map(_.metric)) shouldBe "warm" - all (rows.map(_.unit)) shouldBe "ms" - all (rows.map(_.stats.median)) should be > 0.0 - all (rows.map(_.iterations)) shouldBe 3 - } - - "streaming rows are produced with MB/s unit" in { - if (!TestSupport.ensureGeneratedInputs(corpus)) cancel("generated input unavailable (node missing?)") - val shell = new EngineShell() - val rows = WarmBench.runStreaming(shell, manifest, iterCap = Some(2)) - rows.map(_.id) should contain allOf ("map-scale", "json-stream") - all (rows.map(_.metric)) shouldBe "streaming" - all (rows.map(_.unit)) shouldBe "MB/s" - all (rows.map(_.stats.median)) should be > 0.0 - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.WarmBenchTest"` -Expected: FAIL — `Row` / `WarmBench` not found. - -- [ ] **Step 3: Implement `Row` and `WarmBench`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Row.scala`: - -```scala -package org.mule.weave.benchmark.engine - -/** One flat (case, metric) result row — the schema's unit of output. */ -final case class Row(id: String, metric: String, unit: String, stats: Stats.Summary, iterations: Int) -``` - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/WarmBench.scala`: - -```scala -package org.mule.weave.benchmark.engine - -/** In-process metrics: warm steady-state (with a JVM JIT warmup floor) and - * streaming throughput. Timing mirrors the Node runner: System.nanoTime -> ms. */ -object WarmBench { - - val WARMUP_FLOOR: Int = 2000 - - private def nowNs(): Long = System.nanoTime() - private def msSince(startNs: Long): Double = (System.nanoTime() - startNs) / 1e6 - - def runWarm(shell: EngineShell, m: Manifest, warmupCap: Option[Int] = None, iterCap: Option[Int] = None): Seq[Row] = { - Manifest.casesForMetric(m, "warm").map { c => - val script = Manifest.resolveScript(m, c) - val inputs = Manifest.resolveInputs(m, c) - val name = EngineShell.safeName(c.id) - val warmup = warmupCap.getOrElse(math.max(c.warmup, WARMUP_FLOOR)) - val iters = iterCap.getOrElse(c.warm) - - println(s"[warm] ${c.id}: warmup=$warmup iters=$iters") - var i = 0 - while (i < warmup) { shell.run(script, name, inputs, new CountingOutputStream()); i += 1 } - - val samples = new Array[Double](iters) - i = 0 - while (i < iters) { - val start = nowNs() - shell.run(script, name, inputs, new CountingOutputStream()) - samples(i) = msSince(start) - i += 1 - } - Row(c.id, "warm", "ms", Stats.computeStats(samples.toSeq), iters) - } - } - - def runStreaming(shell: EngineShell, m: Manifest, iterCap: Option[Int] = None): Seq[Row] = { - Manifest.casesForMetric(m, "streaming").map { c => - val script = Manifest.resolveScript(m, c) - val inputs = Manifest.resolveInputs(m, c) - val name = EngineShell.safeName(c.id) - val primaryBytes = inputs.head.bytes.length.toLong - val iters = iterCap.getOrElse(c.streaming) - - val mbps = new Array[Double](iters) - var i = 0 - while (i < iters) { - val start = nowNs() - shell.run(script, name, inputs, new CountingOutputStream()) - mbps(i) = Stats.toMBps(primaryBytes, msSince(start)) - i += 1 - } - Row(c.id, "streaming", "MB/s", Stats.computeStats(mbps.toSeq), iters) - } - } -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.WarmBenchTest"` -Expected: PASS (2 tests; streaming test cancels rather than fails if Node is unavailable). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Row.scala \ - benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/WarmBench.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/TestSupport.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/WarmBenchTest.scala -git commit -m "W-23545283: Add WarmBench (warm + streaming) with JIT warmup floor" -``` - ---- - -### Task 7: Env stamp + Result JSON + Emit orchestrator - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EnvStamp.scala` -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Result.scala` -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Emit.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EmitTest.scala` - -**Interfaces:** -- Consumes: `Manifest`, `EngineShell`, `WarmBench`, `Row`, `Stats`, `org.json`. -- Produces: - - `case class Env(runner: String, os: String, cpu: String, runtimeVersion: String, weaveVersion: String, commit: String, dwlibBuildId: String)` - - `object EnvStamp.gather(repoRoot: File): Env` — `runner="engine"`; `os = "-"`; `cpu` best-effort (mac `sysctl -n machdep.cpu.brand_string`, linux `/proc/cpuinfo model name`, else `os.arch`); `runtimeVersion = "jvm " + java.version`; `weaveVersion` from `repoRoot/gradle.properties` (`^weaveVersion=`); `commit` from `git rev-parse --short HEAD` (fallback `"unknown"`); `dwlibBuildId = "n/a-engine"`. - - `object Result.toJson(env: Env, rows: Seq[Row], timestamp: String): String` — schema-conformant JSON (2-space indent). - - `object Emit`: - - `case class Caps(samples: Option[Int] = None, warmup: Option[Int] = None, warm: Option[Int] = None, streaming: Option[Int] = None)` - - `run(corpus: File, resultsDir: File, repoRoot: File, caps: Caps = Caps()): File` — spawn `EngineChild` for cold-start/first-run rows, run `WarmBench` in-process, validate ids (fail-fast), stamp env, write `resultsDir/engine-.json`, return the file. - - `main(args: Array[String]): Unit` — args ` `, no caps. - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EmitTest.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.json.JSONObject -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers -import java.io.File -import java.nio.charset.StandardCharsets -import java.nio.file.Files - -class EmitTest extends AnyFreeSpec with Matchers { - private val corpus = new File("../../corpus").getCanonicalFile - private val repoRoot = new File("../../..").getCanonicalFile - - "emit writes a schema-shaped result file with engine runner and valid ids" in { - if (!TestSupport.ensureGeneratedInputs(corpus)) cancel("generated input unavailable (node missing?)") - val resultsDir = Files.createTempDirectory("engine-emit-test").toFile - val out = Emit.run(corpus, resultsDir, repoRoot, - Emit.Caps(samples = Some(2), warmup = Some(2), warm = Some(2), streaming = Some(2))) - - out.exists() shouldBe true - out.getName should (startWith ("engine-") and endWith (".json")) - - val root = new JSONObject(new String(Files.readAllBytes(out.toPath), StandardCharsets.UTF_8)) - root.getString("schemaVersion") shouldBe "1.0" - root.getString("runner") shouldBe "engine" - - val env = root.getJSONObject("env") - Seq("os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId") - .foreach(k => env.has(k) shouldBe true) - env.getString("dwlibBuildId") shouldBe "n/a-engine" - env.getString("weaveVersion") should not be empty - - val cases = root.getJSONArray("cases") - cases.length() should be > 0 - val manifest = Manifest.load(corpus) - for (i <- 0 until cases.length()) { - val c = cases.getJSONObject(i) - manifest.ids should contain (c.getString("id")) - Seq("cold-start", "first-run", "warm", "streaming") should contain (c.getString("metric")) - Seq("ms", "MB/s") should contain (c.getString("unit")) - c.getJSONObject("stats").getDouble("median") should be >= 0.0 - } - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EmitTest"` -Expected: FAIL — `Emit` / `EnvStamp` / `Result` not found. - -- [ ] **Step 3: Implement `EnvStamp`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EnvStamp.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import java.io.File -import java.nio.charset.StandardCharsets -import java.nio.file.Files -import scala.util.control.NonFatal - -final case class Env( - runner: String, - os: String, - cpu: String, - runtimeVersion: String, - weaveVersion: String, - commit: String, - dwlibBuildId: String) - -object EnvStamp { - - def gather(repoRoot: File): Env = Env( - runner = "engine", - os = s"${System.getProperty("os.name")}-${System.getProperty("os.arch")}", - cpu = cpuModel(), - runtimeVersion = "jvm " + System.getProperty("java.version"), - weaveVersion = readWeaveVersion(repoRoot), - commit = gitCommit(repoRoot), - dwlibBuildId = "n/a-engine") - - private def readWeaveVersion(repoRoot: File): String = { - val txt = new String(Files.readAllBytes(new File(repoRoot, "gradle.properties").toPath), StandardCharsets.UTF_8) - """(?m)^weaveVersion=(.+)$""".r.findFirstMatchIn(txt).map(_.group(1).trim) - .getOrElse(throw new RuntimeException("weaveVersion not found in gradle.properties")) - } - - private def gitCommit(repoRoot: File): String = - exec(Seq("git", "rev-parse", "--short", "HEAD"), repoRoot).getOrElse("unknown") - - private def cpuModel(): String = { - val os = System.getProperty("os.name").toLowerCase - val fromShell = - if (os.contains("mac")) exec(Seq("sysctl", "-n", "machdep.cpu.brand_string"), new File(".")) - else if (os.contains("linux")) - exec(Seq("bash", "-c", "grep -m1 'model name' /proc/cpuinfo | cut -d: -f2"), new File(".")) - else None - fromShell.map(_.trim).filter(_.nonEmpty).getOrElse(System.getProperty("os.arch")) - } - - private def exec(cmd: Seq[String], dir: File): Option[String] = - try { - val p = new ProcessBuilder(cmd: _*).directory(dir).start() - val out = scala.io.Source.fromInputStream(p.getInputStream).mkString.trim - if (p.waitFor() == 0 && out.nonEmpty) Some(out) else None - } catch { case NonFatal(_) => None } -} -``` - -- [ ] **Step 4: Implement `Result`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Result.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.json.{ JSONArray, JSONObject } - -/** Serializes rows + env into the shared benchmark result schema - * (benchmarks/schema/result.schema.json). */ -object Result { - - def toJson(env: Env, rows: Seq[Row], timestamp: String): String = { - val root = new JSONObject() - root.put("schemaVersion", "1.0") - root.put("runner", env.runner) - - val envObj = new JSONObject() - envObj.put("os", env.os) - envObj.put("cpu", env.cpu) - envObj.put("runtimeVersion", env.runtimeVersion) - envObj.put("weaveVersion", env.weaveVersion) - envObj.put("commit", env.commit) - envObj.put("dwlibBuildId", env.dwlibBuildId) - root.put("env", envObj) - - root.put("timestamp", timestamp) - - val casesArr = new JSONArray() - rows.foreach { r => - val c = new JSONObject() - c.put("id", r.id) - c.put("metric", r.metric) - c.put("unit", r.unit) - c.put("iterations", r.iterations) - val s = new JSONObject() - s.put("min", r.stats.min) - s.put("median", r.stats.median) - s.put("p90", r.stats.p90) - s.put("p99", r.stats.p99) - s.put("mean", r.stats.mean) - c.put("stats", s) - casesArr.put(c) - } - root.put("cases", casesArr) - - root.toString(2) - } -} -``` - -- [ ] **Step 5: Implement `Emit`** - -Create `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Emit.scala`: - -```scala -package org.mule.weave.benchmark.engine - -import org.json.JSONObject - -import java.io.File -import java.nio.charset.StandardCharsets -import java.nio.file.Files -import java.time.Instant - -/** Orchestrator: spawns EngineChild for cold-start/first-run, runs WarmBench - * in-process, validates ids, stamps env, and writes the result JSON. */ -object Emit { - - final case class Caps( - samples: Option[Int] = None, - warmup: Option[Int] = None, - warm: Option[Int] = None, - streaming: Option[Int] = None) - - def run(corpus: File, resultsDir: File, repoRoot: File, caps: Caps = Caps()): File = { - val manifest = Manifest.load(corpus) - - val coldRows = spawnColdAndFirstRun(manifest, corpus, caps.samples) - - val shell = new EngineShell() - val warmRows = - WarmBench.runWarm(shell, manifest, caps.warmup, caps.warm) ++ - WarmBench.runStreaming(shell, manifest, caps.streaming) - - val rows = coldRows ++ warmRows - Manifest.validateResultIds(manifest, rows.map(_.id)) // fail-fast on orphan ids - - val env = EnvStamp.gather(repoRoot) - val now = Instant.now().toString - val json = Result.toJson(env, rows, now) - - resultsDir.mkdirs() - val out = new File(resultsDir, s"engine-${now.replaceAll("[:.]", "-")}.json") - Files.write(out.toPath, json.getBytes(StandardCharsets.UTF_8)) - println(s"wrote ${out.getAbsolutePath} (${rows.length} rows)") - out - } - - /** Spawn a fresh JVM per sample; aggregate init/first-run per case. */ - private def spawnColdAndFirstRun(manifest: Manifest, corpus: File, samplesCap: Option[Int]): Seq[Row] = { - val ids = (Manifest.casesForMetric(manifest, "cold-start").map(_.id) ++ - Manifest.casesForMetric(manifest, "first-run").map(_.id)).distinct - - ids.flatMap { id => - val c = manifest.cases.find(_.id == id).get - val n = samplesCap.getOrElse(c.samples) - val inits = new Array[Double](n) - val firsts = new Array[Double](n) - var i = 0 - while (i < n) { - val (initMs, firstMs) = sampleOnce(corpus, id) - inits(i) = initMs - firsts(i) = firstMs - i += 1 - } - val rows = scala.collection.mutable.ArrayBuffer[Row]() - if (c.metrics.contains("cold-start")) - rows += Row(id, "cold-start", "ms", Stats.computeStats(inits.toSeq), n) - if (c.metrics.contains("first-run")) - rows += Row(id, "first-run", "ms", Stats.computeStats(firsts.toSeq), n) - rows.toSeq - } - } - - private def sampleOnce(corpus: File, caseId: String): (Double, Double) = { - val javaBin = new File(System.getProperty("java.home"), "bin/java").getAbsolutePath - val cp = System.getProperty("java.class.path") - val pb = new ProcessBuilder( - javaBin, "-cp", cp, - "org.mule.weave.benchmark.engine.EngineChild", - corpus.getAbsolutePath, caseId) - val p = pb.start() - val lines = scala.io.Source.fromInputStream(p.getInputStream).getLines().toList - val code = p.waitFor() - if (code != 0) throw new RuntimeException(s"EngineChild failed for case '$caseId' (exit $code)") - val obj = new JSONObject(lines.last) - (obj.getDouble("initMs"), obj.getDouble("firstRunMs")) - } - - def main(args: Array[String]): Unit = { - require(args.length >= 3, "usage: Emit ") - run(new File(args(0)), new File(args(1)), new File(args(2))) - } -} -``` - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `./gradlew benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EmitTest"` -Expected: PASS (1 test; cancels if Node unavailable for input generation). - -- [ ] **Step 7: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EnvStamp.scala \ - benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Result.scala \ - benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Emit.scala \ - benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EmitTest.scala -git commit -m "W-23545283: Add Emit orchestrator, env stamp, and result serializer" -``` - ---- - -### Task 8: Wire the Gradle task end-to-end + README fix + report validation - -**Files:** -- Modify: `benchmarks/README.md` -- (Verification only, no source change) `benchmarks/runners/engine/build.gradle` (`benchmarkEngine` task from Task 1), `benchmarks/report/report.mjs`, `benchmarks/schema/schema.test.mjs`. - -**Interfaces:** -- Consumes: `Emit` (Task 7), the existing `report/report.mjs` and `schema/schema.test.mjs`. -- Produces: a runnable `./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true` that writes `benchmarks/results/engine-.json` which `report.mjs` renders as the baseline; corrected README. - -- [ ] **Step 1: Fix the stale README claim** - -In `benchmarks/README.md` the Layout bullet currently spans two lines (12–13): - -``` -- `runners/node/` — the Node reference runner. `runners/python/` and `runners/engine/` - are follow-ups; the engine harness lives in the `data-weave` repo but reads this corpus. -``` - -Replace those two lines (Edit `old_string` must match both verbatim, including the leading `- ` and the two-space continuation indent) with: - -```markdown -- `runners/node/` — the Node reference runner. `runners/engine/` is the JVM baseline - (Scala/Gradle subproject `:benchmarks-engine`, depends on `org.mule.weave:runtime` at - the same `weaveVersion` the native image is built from). `runners/python/` is a follow-up. -``` - -Then add an engine-runner invocation under the "Running" section: - -```markdown -Run the engine (JVM) baseline and let the report pick it up as the comparison anchor: - - ./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true # writes results/engine-.json - node report/report.mjs results/*.json # engine is auto-selected as baseline -``` - -- [ ] **Step 2: Commit the docs fix** - -```bash -git add benchmarks/README.md -git commit -m "W-23545283: Correct README — engine runner lives in this repo" -``` - -- [ ] **Step 3: Run the whole engine test suite** - -Run: `./gradlew benchmarks-engine:test` -Expected: PASS — SmokeTest, StatsParityTest, ManifestTest, EngineShellTest, EngineChildTest, WarmBenchTest, EmitTest all green (streaming/emit tests cancel only if Node is missing). - -- [ ] **Step 4: Run the opt-in benchmark task end-to-end** - -Run: `./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true` -Expected: `genBenchInputs` regenerates `corpus/inputs/generated/records-large.json`, then `Emit` prints `wrote .../benchmarks/results/engine-.json ( rows)`. This is real timing — it takes a few minutes (fresh JVM per cold-start/first-run sample + 2000-iter warmup per warm case). - -- [ ] **Step 5: Verify the emitted result conforms to the schema shape** - -`benchmarks/schema/schema.test.mjs` only asserts the *schema document's* shape — it does not validate result files. So confirm the engine output structurally with a one-off (no new deps; pure Node built-ins), from the repo root: - -Run: -```bash -node -e ' -const {readFileSync,readdirSync}=require("node:fs"); -const dir="benchmarks/results"; -const f=readdirSync(dir).filter(n=>n.startsWith("engine-")&&n.endsWith(".json")).sort().pop(); -const r=JSON.parse(readFileSync(dir+"/"+f,"utf8")); -const need=(o,k)=>{if(!(k in o))throw new Error("missing "+k);}; -["schemaVersion","runner","env","timestamp","cases"].forEach(k=>need(r,k)); -if(r.schemaVersion!=="1.0")throw new Error("schemaVersion"); -if(r.runner!=="engine")throw new Error("runner"); -["os","cpu","runtimeVersion","weaveVersion","commit","dwlibBuildId"].forEach(k=>need(r.env,k)); -for(const c of r.cases){["id","metric","unit","stats","iterations"].forEach(k=>need(c,k)); - if(!["cold-start","first-run","warm","streaming"].includes(c.metric))throw new Error("metric "+c.metric); - if(!["ms","MB/s"].includes(c.unit))throw new Error("unit "+c.unit); - need(c.stats,"median");} -console.log("OK "+f+" ("+r.cases.length+" rows)"); -' -``` -Expected: `OK engine-.json ( rows)`. Any missing key or bad enum throws. - -- [ ] **Step 6: Verify the report renders the engine as baseline with no skew banner** - -Run: `cd benchmarks && node report/report.mjs results/*.json` -Expected: a markdown table grouped by case × metric with an `engine` column and a `Δ vs engine` column; **no** `⚠️ WEAVE VERSION SKEW` banner (engine and any wrapper result share the same `weaveVersion` from `gradle.properties`). If only the engine result exists, it appears as the single baseline column. - -- [ ] **Step 7: Final commit (if any verification-driven tweaks were needed)** - -```bash -git add -A -git commit -m "W-23545283: Verify engine-runner end-to-end (benchmarkEngine + report)" -``` - ---- - -## Self-Review - -**1. Spec coverage** (against `docs/superpowers/specs/2026-07-23-engine-runner-design.md`): -- Bare `DataWeaveScriptingEngine` → Task 4 (`EngineShell`). ✓ -- Scala language, `benchmarks/runners/engine/` placement → Tasks 1–7. ✓ -- Gradle subproject + settings + opt-in task + auto report pickup → Tasks 1, 8. ✓ -- SPI note (core-modules on classpath, no re-materialization) → Task 1 dep + Task 4 Step 4 note. ✓ -- Engine shell specifics (classloader resolver, netty setupEnv, UTF-8 CharsetProviderService, compileWith + write-to-OutputStream) → Task 4. ✓ -- Fresh-JVM cold-start + first-run → Task 5 (`EngineChild`) + Task 7 spawn aggregation. ✓ -- JIT warmup floor `max(manifest, 2000)`, streaming MB/s → Task 6. ✓ -- Self-contained emit: Stats parity test, env, id fail-fast, schema conformance, `dwlibBuildId:"n/a-engine"` → Tasks 2, 7. ✓ -- Docs fix → Task 8. ✓ -- Testing (StatsParityTest, schema conformance, manifest parsing, e2e report) → Tasks 2, 3, 8. ✓ -- Open note resolved: Node wrapper recompiles per `run()`, so engine `warm` also recompiles per iteration (compile+exec both sides) → Task 4 interface + Task 6. ✓ - -**2. Placeholder scan:** No "TBD"/"TODO"/"handle edge cases" — every code step has complete, compilable code. ✓ - -**3. Type consistency:** `Row`, `Stats.Summary`, `Env`, `ResolvedInput`, `BenchCase`, `Emit.Caps`, `EngineShell.run(script, name, inputs, out)`, `EngineShell.safeName`, `Manifest.casesForMetric/validateResultIds/resolveScript/resolveInputs`, `WarmBench.runWarm/runStreaming`, `Result.toJson`, `EnvStamp.gather` are named identically everywhere they appear across tasks. ✓ diff --git a/docs/superpowers/plans/2026-07-23-python-runner.md b/docs/superpowers/plans/2026-07-23-python-runner.md deleted file mode 100644 index f95674f..0000000 --- a/docs/superpowers/plans/2026-07-23-python-runner.md +++ /dev/null @@ -1,1246 +0,0 @@ -# Python Runner (native-lib wrapper benchmark) Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Build the Python benchmark runner — a stdlib-only set of scripts at `benchmarks/runners/python/` that drives the DataWeave Python binding over the shared corpus and emits a schema-conformant `results/python-.json`, auto-registered into the `benchmarkCompare` aggregator, so `report/report.mjs` produces Python-vs-Node-vs-engine deltas at the same weave version. - -**Architecture:** Pure Python (stdlib only), structured to mirror the **Node** runner file-for-file (Python is Node's true peer: both are dynamic-language FFI wrappers over the *same* staged `dwlib`). Timing is hand-rolled `time.perf_counter_ns()` → ms for parity with Node (`process.hrtime.bigint()`) and the engine (`System.nanoTime()`). `stats.py`/`manifest.py`/`env.py` reimplement `benchmarks/lib/*.mjs` self-contained (no runtime `node` except the shared input generator). Cold-start/first-run spawn a fresh `python` per sample (`coldstart_child.py`); warm/streaming run in-process (`warm_bench.py`). A parity test (`test_bench.py`) guards the reimplemented math against drift and is wired always-on into `native-lib:test`. - -**Tech Stack:** Python 3.9+ (stdlib `unittest`, `subprocess`, `hashlib`, `time.perf_counter_ns`, `json`, `pathlib`), the `dataweave` binding at `native-lib/python/src`, Gradle `Exec` tasks invoking `python3` directly (no venv/pip), the existing Node `corpus/gen-inputs.mjs` for the shared large input. Reuses `benchmarks/corpus/`, `benchmarks/schema/`, and `report/report.mjs` unchanged. - -## Global Constraints - -- **Stdlib only.** No third-party Python packages; no venv/pip step. The runner is invoked as `python3 runners/python/.py`. -- **Timing methodology (every metric):** `time.perf_counter_ns()`; convert to ms as `ns / 1e6` (float). Identical in spirit to Node's `process.hrtime.bigint()` → ms and the engine's `System.nanoTime()`. -- **No JIT warmup floor.** CPython has no JIT — `warm` uses `iterations.warmup` verbatim (like the Node runner), NOT the engine's `max(warmup, 2000)` floor. Do not add a floor. -- **Nearest-rank percentiles, byte-compatible with `lib/stats.mjs`:** `pct(p) = sorted[min(n-1, max(0, ceil(p/100 * n) - 1))]`; `mean = sum/n`. Throughput `to_mbps = total_bytes / 1e6 / (elapsed_ms/1000)` (decimal MB). -- **`id` is the immutable join key.** Emit each case `id` verbatim from the manifest; never invent or rename. -- **Fail-fast on orphan ids.** `emit.py` MUST abort before writing output if any emitted `id` is absent from the manifest. -- **Explicit metrics.** Run only the metrics a case declares in `metrics[]` (`cases_for_metric`). -- **Correctness guard.** A case whose `run()` returns `success == False`, or whose streaming `.metadata.success` is false, aborts that case with a clear error — never record a bogus fast timing. -- **Schema conformance (mandatory `env` fields).** Every result validates against `benchmarks/schema/result.schema.json`: top-level `schemaVersion:"1.0"`, `runner`, `env`, `timestamp`, `cases[]`; each case a flat `{id, metric, unit, stats, iterations}` row with no extra keys; `env` includes `os`, `cpu`, `runtimeVersion`, `weaveVersion`, `commit`, `dwlibBuildId` (all required). -- **`runner` is `"python-wrapper"`** — the report column name and dedupe key, symmetric with Node's `"node-wrapper"`. (`report.mjs` still auto-selects `engine` as the delta baseline.) -- **Real `dwlibBuildId`:** `"dwlib-" + sha256(str(size) + first-64KB-of-file).hexdigest()[:8]`, pointed at the Python staging path (`native-lib/python/src/dataweave/native/dwlib.*`), honoring a `DATAWEAVE_NATIVE_LIB` override. Same formula as `lib/env.mjs`. -- **Units per row:** `ms` for latency (cold-start, first-run, warm), `MB/s` for streaming. -- **Results are local-only.** Output to `benchmarks/results/` (already gitignored). Do not commit result files or generated inputs. -- **Deterministic large inputs.** Generated via the existing `corpus/gen-inputs.mjs` (Node), byte-identical to other runners — the Python runner consumes that file, never regenerates it differently. -- **All always-on unit tests are dwlib-free.** The Python binding's `dwlib` requires a multi-minute native build, so every test in `test_bench.py` uses dependency injection (fake api / fake `sample_fn`), synthetic manifests, or temp files. The real dwlib integration is verified only by the opt-in `benchmarkPython` end-to-end run (Task 8). - ---- - -### Task 1: `stats.py` + parity test scaffold - -**Files:** -- Create: `benchmarks/runners/python/stats.py` -- Test: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: nothing. -- Produces: - - `stats.compute_stats(samples) -> {"min","median","p90","p99","mean"}` — nearest-rank percentiles on a sorted copy; raises `ValueError` on empty input. - - `stats.to_mbps(total_bytes, elapsed_ms) -> float` — decimal-MB throughput; raises `ValueError` if `elapsed_ms <= 0`. - - `test_bench.py` — the always-on parity harness (stdlib `unittest`), extended by later tasks. - -- [ ] **Step 1: Write the failing test (creates `test_bench.py`)** - -Create `benchmarks/runners/python/test_bench.py`. These expected values are the exact outputs of `benchmarks/lib/stats.mjs` for the same inputs: - -```python -"""Always-on parity guard for the Python runner: nearest-rank stats vs -lib/stats.mjs, manifest parsing, env stamp, and the pure emit/aggregation -logic. Pure stdlib — no dwlib, no venv. Run: python3 runners/python/test_bench.py""" - -import unittest -from pathlib import Path - -# benchmarks/runners/python -> benchmarks -> corpus -CORPUS = Path(__file__).resolve().parents[2] / "corpus" - -import stats - - -class TestStats(unittest.TestCase): - def test_matches_lib_stats_on_1_to_100(self): - s = stats.compute_stats([float(i) for i in range(1, 101)]) - self.assertEqual(s["min"], 1.0) - self.assertEqual(s["median"], 50.0) # ceil(0.5*100)-1 = 49 -> sorted[49] = 50 - self.assertEqual(s["p90"], 90.0) # ceil(0.9*100)-1 = 89 -> 90 - self.assertEqual(s["p99"], 99.0) # ceil(0.99*100)-1 = 98 -> 99 - self.assertEqual(s["mean"], 50.5) - - def test_sorts_before_ranking(self): - s = stats.compute_stats([5.0, 1.0, 3.0, 2.0, 4.0]) - self.assertEqual(s["min"], 1.0) - self.assertEqual(s["median"], 3.0) # ceil(0.5*5)-1 = 2 -> sorted[2] = 3 - self.assertEqual(s["p90"], 5.0) # ceil(0.9*5)-1 = 4 -> 5 - self.assertEqual(s["mean"], 3.0) - - def test_rejects_empty(self): - with self.assertRaises(ValueError): - stats.compute_stats([]) - - def test_to_mbps(self): - self.assertEqual(stats.to_mbps(1_000_000, 1000.0), 1.0) - self.assertEqual(stats.to_mbps(500_000, 250.0), 2.0) - with self.assertRaises(ValueError): - stats.to_mbps(10, 0.0) - - -if __name__ == "__main__": - unittest.main() -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'stats'` (module not yet created). - -- [ ] **Step 3: Implement `stats.py`** - -Create `benchmarks/runners/python/stats.py`: - -```python -"""Percentile + throughput math kept behavior-compatible with -benchmarks/lib/stats.mjs so the Python runner's deltas compare cleanly against -the Node and engine runners. Any divergence is caught by test_bench.py.""" - -import math - - -def compute_stats(samples): - """Nearest-rank percentiles on a sorted copy; mean is the arithmetic mean. - Mirrors lib/stats.mjs:computeStats.""" - if not samples: - raise ValueError("compute_stats requires a non-empty sequence of numbers") - ordered = sorted(samples) - n = len(ordered) - - def pct(p): - idx = min(n - 1, max(0, math.ceil(p / 100 * n) - 1)) - return ordered[idx] - - return { - "min": ordered[0], - "median": pct(50), - "p90": pct(90), - "p99": pct(99), - "mean": sum(ordered) / n, - } - - -def to_mbps(total_bytes, elapsed_ms): - """Throughput in decimal megabytes per second (1 MB = 1e6 bytes). - Mirrors lib/stats.mjs:toMBps.""" - if elapsed_ms <= 0: - raise ValueError("elapsed_ms must be > 0") - return total_bytes / 1e6 / (elapsed_ms / 1000) -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 4 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/stats.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner stats with lib/stats.mjs parity test" -``` - ---- - -### Task 2: `manifest.py` + manifest parsing tests - -**Files:** -- Create: `benchmarks/runners/python/manifest.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: the real corpus at `benchmarks/corpus/`. -- Produces: - - `manifest.METRICS` — the allowed metric tuple. - - `manifest.load_manifest(corpus_dir) -> {"corpusDir": str, "cases": list, "ids": set}` — parse + validate `manifest.json` (mirrors `lib/manifest.mjs:loadManifest`: non-empty unique ids, metrics from the allowed set, script exists, non-generated input files exist). Raises `ValueError` on any violation. - - `manifest.cases_for_metric(manifest, metric) -> list` - - `manifest.resolve_inputs(manifest, case_obj) -> {name: {"bytes": bytes, "mimeType": str, "charset": str|None}}` - - `manifest.read_script(manifest, case_obj) -> str` - - `manifest.validate_result_ids(manifest, result_ids) -> None` — raises `ValueError` on any id not in `manifest["ids"]`. - -- [ ] **Step 1: Write the failing test** - -Add `import manifest` beneath `import stats` in `test_bench.py`, and insert this class above the `if __name__` guard: - -```python -class TestManifest(unittest.TestCase): - def setUp(self): - self.m = manifest.load_manifest(CORPUS) - - def test_loads_all_corpus_ids(self): - for cid in ("trivial", "object-transform", "map-scale", "xml-to-csv", - "json-stream", "compile-heavy"): - self.assertIn(cid, self.m["ids"]) - - def test_cases_for_metric_filters(self): - streaming = [c["id"] for c in manifest.cases_for_metric(self.m, "streaming")] - self.assertIn("map-scale", streaming) - self.assertIn("json-stream", streaming) - cold = [c["id"] for c in manifest.cases_for_metric(self.m, "cold-start")] - self.assertIn("trivial", cold) - - def test_read_script(self): - trivial = next(c for c in self.m["cases"] if c["id"] == "trivial") - self.assertIn("2 + 2", manifest.read_script(self.m, trivial)) - - def test_resolve_inputs_reads_bytes_mime_charset(self): - xml = next(c for c in self.m["cases"] if c["id"] == "xml-to-csv") - resolved = manifest.resolve_inputs(self.m, xml) - self.assertEqual(list(resolved.keys()), ["payload"]) - self.assertEqual(resolved["payload"]["mimeType"], "application/xml") - self.assertEqual(resolved["payload"]["charset"], "UTF-16") - self.assertGreater(len(resolved["payload"]["bytes"]), 0) - - def test_validate_result_ids_rejects_orphan(self): - with self.assertRaises(ValueError): - manifest.validate_result_ids(self.m, ["trivial", "not-a-case"]) - manifest.validate_result_ids(self.m, ["trivial"]) # no raise -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'manifest'`. - -- [ ] **Step 3: Implement `manifest.py`** - -Create `benchmarks/runners/python/manifest.py`: - -```python -"""Parse and validate corpus/manifest.json, resolve scripts + inputs. -Mirrors benchmarks/lib/manifest.mjs.""" - -import json -from pathlib import Path - -# The only metrics a case may declare. -METRICS = ("cold-start", "first-run", "warm", "streaming") - - -def load_manifest(corpus_dir): - corpus_dir = Path(corpus_dir) - raw = json.loads((corpus_dir / "manifest.json").read_text(encoding="utf-8")) - cases = raw.get("cases") - if not isinstance(cases, list): - raise ValueError("manifest.cases must be an array") - ids = set() - for c in cases: - cid = c.get("id") - if not cid: - raise ValueError("manifest case is missing an id") - if cid in ids: - raise ValueError(f"duplicate case id: {cid}") - ids.add(cid) - metrics = c.get("metrics") - if not isinstance(metrics, list) or not metrics: - raise ValueError(f"case {cid} must declare a non-empty metrics[]") - for m in metrics: - if m not in METRICS: - raise ValueError(f"case {cid} has unknown metric: {m}") - script = c.get("script") - if not script or not (corpus_dir / script).exists(): - raise ValueError(f"case {cid} script not found: {script}") - for name, inp in (c.get("inputs") or {}).items(): - if inp.get("file") and not inp.get("generated") \ - and not (corpus_dir / inp["file"]).exists(): - raise ValueError(f"case {cid} input '{name}' file not found: {inp['file']}") - return {"corpusDir": str(corpus_dir), "cases": cases, "ids": ids} - - -def cases_for_metric(manifest, metric): - return [c for c in manifest["cases"] if metric in c["metrics"]] - - -def resolve_inputs(manifest, case_obj): - """Read a case's declared inputs into bytes.""" - corpus_dir = Path(manifest["corpusDir"]) - out = {} - for name, inp in (case_obj.get("inputs") or {}).items(): - data = (corpus_dir / inp["file"]).read_bytes() - out[name] = {"bytes": data, "mimeType": inp["mimeType"], "charset": inp.get("charset")} - return out - - -def read_script(manifest, case_obj): - return (Path(manifest["corpusDir"]) / case_obj["script"]).read_text(encoding="utf-8") - - -def validate_result_ids(manifest, result_ids): - """Fail-fast: raise if any result id is not present in the manifest.""" - for rid in result_ids: - if rid not in manifest["ids"]: - raise ValueError(f"result contains orphan id not in manifest: {rid}") -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 9 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/manifest.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner manifest parser + input resolution" -``` - ---- - -### Task 3: `env.py` — env stamp with real `dwlibBuildId` - -**Files:** -- Create: `benchmarks/runners/python/env.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: `gradle.properties` (weaveVersion), `git` (commit), the staged/overridden `dwlib` (build id). -- Produces: - - `env.gather_env() -> {"runner","os","cpu","runtimeVersion","weaveVersion","commit","dwlibBuildId"}` — `runner="python-wrapper"`; `os="-"`; `cpu` best-effort; `runtimeVersion="python "`; `weaveVersion` from `gradle.properties`; `commit` from git (fallback `"unknown"`); `dwlibBuildId` real (sha256 formula) or `"unknown"` if no lib found. - -- [ ] **Step 1: Write the failing test** - -Add `import env as envmod` beneath `import manifest` in `test_bench.py`, add `import hashlib`, `import os`, `import tempfile` beneath the existing stdlib imports, and insert this class above the `if __name__` guard: - -```python -class TestEnv(unittest.TestCase): - def test_gather_env_has_required_keys(self): - e = envmod.gather_env() - for k in ("os", "cpu", "runtimeVersion", "weaveVersion", "commit", "dwlibBuildId"): - self.assertIn(k, e) - self.assertEqual(e["runner"], "python-wrapper") - self.assertTrue(e["runtimeVersion"].startswith("python ")) - self.assertTrue(e["weaveVersion"]) # non-empty (read from gradle.properties) - - def test_dwlib_build_id_formula(self): - data = b"hello world" * 10 - with tempfile.NamedTemporaryFile(suffix=".dylib", delete=False) as f: - f.write(data) - path = f.name - os.environ["DATAWEAVE_NATIVE_LIB"] = path - try: - e = envmod.gather_env() - size = os.path.getsize(path) - h = hashlib.sha256() - h.update(str(size).encode()) - h.update(data[:65536]) - self.assertEqual(e["dwlibBuildId"], "dwlib-" + h.hexdigest()[:8]) - finally: - del os.environ["DATAWEAVE_NATIVE_LIB"] - os.unlink(path) -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'env'`. - -- [ ] **Step 3: Implement `env.py`** - -Create `benchmarks/runners/python/env.py`: - -```python -"""Env stamp for the Python runner, mirroring benchmarks/lib/env.mjs. Includes a -REAL dwlibBuildId — the Python runner wraps the same staged dwlib as Node, so an -unchanged lib yields the same id in both runners' result files.""" - -import hashlib -import os -import platform -import re -import subprocess -import sys -from pathlib import Path - -# benchmarks/runners/python -> repo root -_REPO_ROOT = Path(__file__).resolve().parents[3] -_ENV_NATIVE_LIB = "DATAWEAVE_NATIVE_LIB" - - -def _read_weave_version(): - txt = (_REPO_ROOT / "gradle.properties").read_text(encoding="utf-8") - m = re.search(r"^weaveVersion=(.+)$", txt, re.MULTILINE) - if not m: - raise RuntimeError("weaveVersion not found in gradle.properties") - return m.group(1).strip() - - -def _read_commit(): - try: - return subprocess.check_output( - ["git", "rev-parse", "--short", "HEAD"], - cwd=_REPO_ROOT, stderr=subprocess.DEVNULL, - ).decode().strip() - except Exception: - return "unknown" - - -def _normalize_arch(machine): - # Match Node's process.arch vocabulary so labels read identically across runners. - return {"x86_64": "x64", "amd64": "x64"}.get(machine, machine) - - -def _cpu_model(): - try: - if sys.platform == "darwin": - return subprocess.check_output( - ["sysctl", "-n", "machdep.cpu.brand_string"], stderr=subprocess.DEVNULL, - ).decode().strip() - if sys.platform.startswith("linux"): - for line in Path("/proc/cpuinfo").read_text().splitlines(): - if line.startswith("model name"): - return line.split(":", 1)[1].strip() - except Exception: - pass - return platform.machine() or "unknown" - - -def _dwlib_path(): - override = (os.environ.get(_ENV_NATIVE_LIB) or "").strip() - if override: - return Path(override) - base = _REPO_ROOT / "native-lib" / "python" / "src" / "dataweave" / "native" - for ext in (".dylib", ".so", ".dll"): - p = base / f"dwlib{ext}" - if p.exists(): - return p - return None - - -def _read_dwlib_build_id(): - # sha256 over (size + first 64KB), same formula as lib/env.mjs. - p = _dwlib_path() - if p and p.exists(): - size = p.stat().st_size - head = p.read_bytes()[:65536] - h = hashlib.sha256() - h.update(str(size).encode()) - h.update(head) - return "dwlib-" + h.hexdigest()[:8] - return "unknown" - - -def gather_env(): - return { - "runner": "python-wrapper", - "os": f"{sys.platform}-{_normalize_arch(platform.machine())}", - "cpu": _cpu_model(), - "runtimeVersion": f"python {platform.python_version()}", - "weaveVersion": _read_weave_version(), - "commit": _read_commit(), - "dwlibBuildId": _read_dwlib_build_id(), - } -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 11 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/env.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner env stamp with real dwlibBuildId" -``` - ---- - -### Task 4: `wrapper.py` — locate + import the binding - -**Files:** -- Create: `benchmarks/runners/python/wrapper.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: the `dataweave` binding at `native-lib/python/src`. -- Produces: - - `wrapper.load_wrapper(src=None) -> module` — put `native-lib/python/src` (or `src` override) on `sys.path` and `import dataweave`; raise `RuntimeError` with a build hint if it cannot be imported. Importing the binding is dwlib-free (the lib loads lazily on `DataWeave().initialize()`). - -- [ ] **Step 1: Write the failing test** - -Add `import wrapper` beneath `import env as envmod` in `test_bench.py`, and insert this class above the `if __name__` guard: - -```python -class TestWrapper(unittest.TestCase): - def test_load_wrapper_exposes_api(self): - api = wrapper.load_wrapper() - for attr in ("DataWeave", "run", "run_transform", "run_streaming"): - self.assertTrue(hasattr(api, attr), f"binding missing {attr}") -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'wrapper'`. - -- [ ] **Step 3: Implement `wrapper.py`** - -Create `benchmarks/runners/python/wrapper.py`: - -```python -"""Locate and import the DataWeave Python binding (native-lib/python). The -binding loads the staged dwlib lazily on DataWeave().initialize(), so importing -the module itself is dwlib-free.""" - -import sys -from pathlib import Path - -# benchmarks/runners/python -> repo root -> native-lib/python/src -_SRC = Path(__file__).resolve().parents[3] / "native-lib" / "python" / "src" - - -def load_wrapper(src=None): - src = Path(src) if src is not None else _SRC - if str(src) not in sys.path: - sys.path.insert(0, str(src)) - try: - import dataweave - except ImportError as e: - raise RuntimeError( - f"DataWeave Python binding not importable from {src}. " - f"Run: ./gradlew native-lib:stagePythonNativeLib ({e})" - ) - return dataweave -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 12 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/wrapper.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner binding loader" -``` - ---- - -### Task 5: `coldstart_child.py` + `coldstart.py` — fresh-process spawn harness - -**Files:** -- Create: `benchmarks/runners/python/coldstart_child.py` -- Create: `benchmarks/runners/python/coldstart.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: `manifest` (`load_manifest`, `cases_for_metric`, `read_script`, `resolve_inputs`), `wrapper` (child only), `stats.compute_stats`. -- Produces: - - `coldstart_child.py` — a script (`python3 coldstart_child.py `) that times `DataWeave()` construction + `.initialize()` (`initMs`) and the first `.run()` (`firstRunMs`), exits non-zero on failure, and prints one JSON line `{"initMs":..,"firstRunMs":..}`. dwlib-dependent — never imported by tests, only spawned in Task 8. - - `coldstart.run_cold_start_and_first_run(manifest, sample_fn=, samples_override=None) -> list[row]` — for each case declaring cold-start and/or first-run: call `sample_fn(corpusDir, caseId)` N times (N = `samples_override` or `iterations.samples` or 20), aggregate into `cold-start`/`first-run` rows (only the metrics the case declares). `sample_fn` is injectable for dwlib-free testing. - -- [ ] **Step 1: Write the failing test** - -Add `import coldstart` beneath `import wrapper` in `test_bench.py`, and insert this class above the `if __name__` guard. It injects a fake `sample_fn`, so it needs no dwlib and no input files: - -```python -class TestColdstartAggregation(unittest.TestCase): - def _manifest(self): - return { - "corpusDir": str(CORPUS), - "cases": [ - {"id": "trivial", "script": "scripts/trivial.dwl", - "metrics": ["cold-start", "first-run"], "iterations": {"samples": 5}}, - {"id": "object-transform", "script": "scripts/object-transform.dwl", - "metrics": ["first-run"], "iterations": {"samples": 5}}, - ], - "ids": {"trivial", "object-transform"}, - } - - def test_aggregates_injected_samples(self): - calls = [] - - def fake_sample(corpus_dir, case_id): - calls.append(case_id) - return (1.0, 2.0) # (initMs, firstRunMs) - - rows = coldstart.run_cold_start_and_first_run( - self._manifest(), sample_fn=fake_sample, samples_override=3) - - by = {(r["id"], r["metric"]): r for r in rows} - self.assertIn(("trivial", "cold-start"), by) - self.assertIn(("trivial", "first-run"), by) - self.assertIn(("object-transform", "first-run"), by) - self.assertNotIn(("object-transform", "cold-start"), by) - - self.assertEqual(by[("trivial", "cold-start")]["unit"], "ms") - self.assertEqual(by[("trivial", "cold-start")]["stats"]["median"], 1.0) - self.assertEqual(by[("trivial", "first-run")]["stats"]["median"], 2.0) - self.assertEqual(by[("trivial", "cold-start")]["iterations"], 3) - # 2 cases sampled 3 times each. - self.assertEqual(len(calls), 6) -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'coldstart'`. - -- [ ] **Step 3: Implement `coldstart_child.py`** - -Create `benchmarks/runners/python/coldstart_child.py`: - -```python -"""Fresh-process worker. Measures a cold engine init + a cold (first) compile+exec -for one case, then prints a single JSON line. Spawned by coldstart.py — the honest -cold path (process launch + interpreter start + dlopen(dwlib) + isolate creation + -first compile+exec).""" - -import json -import sys -import time - -from manifest import load_manifest, read_script, resolve_inputs -from wrapper import load_wrapper - - -def _to_inputs(resolved): - inputs = {} - for name, v in resolved.items(): - inputs[name] = { - "content": v["bytes"], - "mimeType": v["mimeType"], - "charset": v["charset"] or "utf-8", - } - return inputs - - -def main(argv): - corpus_dir, case_id = argv[1], argv[2] - manifest = load_manifest(corpus_dir) - c = next((x for x in manifest["cases"] if x["id"] == case_id), None) - if c is None: - raise SystemExit(f"unknown case: {case_id}") - script = read_script(manifest, c) - inputs = _to_inputs(resolve_inputs(manifest, c)) - - api = load_wrapper() - dw = api.DataWeave() - - init_start = time.perf_counter_ns() - dw.initialize() - init_ms = (time.perf_counter_ns() - init_start) / 1e6 - - run_start = time.perf_counter_ns() - result = dw.run(script, inputs) - first_run_ms = (time.perf_counter_ns() - run_start) / 1e6 - if not result.success: - raise SystemExit(f"first run failed: {result.error}") - - dw.cleanup() - sys.stdout.write(json.dumps({"initMs": init_ms, "firstRunMs": first_run_ms}) + "\n") - - -if __name__ == "__main__": - main(sys.argv) -``` - -- [ ] **Step 4: Implement `coldstart.py`** - -Create `benchmarks/runners/python/coldstart.py`: - -```python -"""Spawn orchestrator: N fresh processes per case, aggregate cold-start / -first-run rows. Mirrors runners/node/coldstart.mjs.""" - -import json -import subprocess -import sys -from pathlib import Path - -from manifest import cases_for_metric -from stats import compute_stats - -_CHILD = str(Path(__file__).resolve().parent / "coldstart_child.py") - - -def _spawn_sample(corpus_dir, case_id): - """Spawn one fresh process and parse its single JSON line.""" - out = subprocess.check_output([sys.executable, _CHILD, str(corpus_dir), case_id], text=True) - line = out.strip().split("\n")[-1] - obj = json.loads(line) - return obj["initMs"], obj["firstRunMs"] - - -def run_cold_start_and_first_run(manifest, sample_fn=_spawn_sample, samples_override=None): - # Unique case ids that declare cold-start and/or first-run, preserving order. - ids = [] - for c in cases_for_metric(manifest, "cold-start") + cases_for_metric(manifest, "first-run"): - if c["id"] not in ids: - ids.append(c["id"]) - - rows = [] - for cid in ids: - c = next(x for x in manifest["cases"] if x["id"] == cid) - n = samples_override if samples_override is not None \ - else c.get("iterations", {}).get("samples", 20) - inits, firsts = [], [] - for _ in range(n): - init_ms, first_ms = sample_fn(manifest["corpusDir"], cid) - inits.append(init_ms) - firsts.append(first_ms) - if "cold-start" in c["metrics"]: - rows.append({"id": cid, "metric": "cold-start", "unit": "ms", - "stats": compute_stats(inits), "iterations": n}) - if "first-run" in c["metrics"]: - rows.append({"id": cid, "metric": "first-run", "unit": "ms", - "stats": compute_stats(firsts), "iterations": n}) - return rows -``` - -- [ ] **Step 5: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 13 tests ... OK`. - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/runners/python/coldstart_child.py benchmarks/runners/python/coldstart.py \ - benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner fresh-process cold-start/first-run harness" -``` - ---- - -### Task 6: `warm_bench.py` — in-process warm + streaming - -**Files:** -- Create: `benchmarks/runners/python/warm_bench.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: `manifest` (`cases_for_metric`, `read_script`, `resolve_inputs`), `stats` (`compute_stats`, `to_mbps`), and an injected `api` object exposing `run(script, inputs)` and `run_transform(script, input_stream, input_name=, input_mime_type=, input_charset=)`. -- Produces: - - `warm_bench.run_warm_and_streaming(api, manifest, warmup_cap=None, warm_cap=None, streaming_cap=None) -> list[row]` — for each `warm` case: `warmup` runs (uncapped default `iterations.warmup` or 10 — NO JIT floor), then `iters` timed `api.run` reps → a `warm`/`ms` row. For each `streaming` case: `iters` reps of `api.run_transform` over the 64KB-chunked primary input, drained + `.metadata.success`-guarded → a `streaming`/`MB/s` row via `to_mbps(len(primary_bytes), elapsed_ms)`. Raises `RuntimeError` on any non-success result (correctness guard). - -- [ ] **Step 1: Write the failing test** - -Add `import warm_bench` beneath `import coldstart` in `test_bench.py`, and insert this class above the `if __name__` guard. It injects a fake `api` and uses a synthetic manifest pointing at the committed `person-record.json`, so it needs no dwlib and no generated inputs: - -```python -class _FakeResult: - def __init__(self, success=True, error=None): - self.success, self.error = success, error - - -class _FakeMeta: - def __init__(self, success=True, error=None): - self.success, self.error = success, error - - -class _FakeStream: - def __init__(self, chunks, meta): - self._chunks = iter(chunks) - self.metadata = meta - - def __iter__(self): - return self - - def __next__(self): - return next(self._chunks) - - -class _FakeApi: - def __init__(self, run_ok=True): - self.run_ok = run_ok - - def run(self, script, inputs=None): - return _FakeResult(self.run_ok, None if self.run_ok else "boom") - - def run_transform(self, script, input_stream, input_name="payload", - input_mime_type="application/json", input_charset=None): - for _ in input_stream: # consume, mimicking the real read side - pass - return _FakeStream([b"out-chunk"], _FakeMeta(True)) - - -class TestWarmBench(unittest.TestCase): - def _warm_manifest(self): - return { - "corpusDir": str(CORPUS), - "cases": [{"id": "trivial", "script": "scripts/trivial.dwl", - "metrics": ["warm"], "iterations": {}}], - "ids": {"trivial"}, - } - - def _streaming_manifest(self): - return { - "corpusDir": str(CORPUS), - "cases": [{"id": "object-transform", "script": "scripts/object-transform.dwl", - "inputs": {"payload": {"file": "inputs/person-record.json", - "mimeType": "application/json"}}, - "metrics": ["streaming"], "iterations": {}}], - "ids": {"object-transform"}, - } - - def test_warm_rows(self): - rows = warm_bench.run_warm_and_streaming( - _FakeApi(), self._warm_manifest(), warmup_cap=1, warm_cap=3) - self.assertEqual(len(rows), 1) - self.assertEqual(rows[0]["metric"], "warm") - self.assertEqual(rows[0]["unit"], "ms") - self.assertEqual(rows[0]["iterations"], 3) - self.assertGreaterEqual(rows[0]["stats"]["median"], 0.0) - - def test_streaming_rows(self): - rows = warm_bench.run_warm_and_streaming( - _FakeApi(), self._streaming_manifest(), streaming_cap=2) - self.assertEqual(len(rows), 1) - self.assertEqual(rows[0]["metric"], "streaming") - self.assertEqual(rows[0]["unit"], "MB/s") - self.assertGreater(rows[0]["stats"]["median"], 0.0) - - def test_warm_guard_raises_on_failure(self): - with self.assertRaises(RuntimeError): - warm_bench.run_warm_and_streaming( - _FakeApi(run_ok=False), self._warm_manifest(), warmup_cap=1, warm_cap=1) -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'warm_bench'`. - -- [ ] **Step 3: Implement `warm_bench.py`** - -Create `benchmarks/runners/python/warm_bench.py`: - -```python -"""In-process metrics: warm steady-state and streaming throughput. Timing mirrors -the Node runner: perf_counter_ns -> ms. NO JIT warmup floor (CPython has no JIT), -matching Node — unlike the engine's 2000-iter floor.""" - -import time - -from manifest import cases_for_metric, read_script, resolve_inputs -from stats import compute_stats, to_mbps - - -def _to_inputs(resolved): - inputs = {} - for name, v in resolved.items(): - inputs[name] = {"content": v["bytes"], "mimeType": v["mimeType"], - "charset": v["charset"] or "utf-8"} - return inputs - - -def _chunked(data, size=65536): - for i in range(0, len(data), size): - yield data[i:i + size] - - -def _assert_ok(result): - if not result.success: - raise RuntimeError(f"run failed: {result.error}") - - -def run_warm_and_streaming(api, manifest, warmup_cap=None, warm_cap=None, streaming_cap=None): - rows = [] - - for c in cases_for_metric(manifest, "warm"): - script = read_script(manifest, c) - inputs = _to_inputs(resolve_inputs(manifest, c)) - warmup = warmup_cap if warmup_cap is not None \ - else c.get("iterations", {}).get("warmup", 10) - iters = warm_cap if warm_cap is not None \ - else c.get("iterations", {}).get("warm", 100) - - for _ in range(warmup): - _assert_ok(api.run(script, inputs)) - samples = [] - for _ in range(iters): - start = time.perf_counter_ns() - _assert_ok(api.run(script, inputs)) - samples.append((time.perf_counter_ns() - start) / 1e6) - rows.append({"id": c["id"], "metric": "warm", "unit": "ms", - "stats": compute_stats(samples), "iterations": iters}) - - for c in cases_for_metric(manifest, "streaming"): - script = read_script(manifest, c) - resolved = resolve_inputs(manifest, c) - primary_name, primary = next(iter(resolved.items())) - iters = streaming_cap if streaming_cap is not None \ - else c.get("iterations", {}).get("streaming", 10) - - mbps = [] - for _ in range(iters): - start = time.perf_counter_ns() - stream = api.run_transform( - script, _chunked(primary["bytes"]), - input_name=primary_name, input_mime_type=primary["mimeType"], - input_charset=primary["charset"], - ) - for _chunk in stream: - pass - meta = stream.metadata - if meta is None or not meta.success: - raise RuntimeError(f"stream failed: {getattr(meta, 'error', 'no metadata')}") - elapsed_ms = (time.perf_counter_ns() - start) / 1e6 - mbps.append(to_mbps(len(primary["bytes"]), elapsed_ms)) - rows.append({"id": c["id"], "metric": "streaming", "unit": "MB/s", - "stats": compute_stats(mbps), "iterations": iters}) - - return rows -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 16 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/warm_bench.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner warm + streaming (no JIT floor)" -``` - ---- - -### Task 7: `emit.py` — collector + result serializer - -**Files:** -- Create: `benchmarks/runners/python/emit.py` -- Modify: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: `coldstart.run_cold_start_and_first_run`, `warm_bench.run_warm_and_streaming`, `env.gather_env`, `manifest` (`load_manifest`, `validate_result_ids`), `wrapper.load_wrapper`. -- Produces: - - `emit.build_result(env, cases) -> dict` — pure assembler: `{schemaVersion:"1.0", runner, env, timestamp, cases}`. `timestamp` is ISO-8601 UTC millis with `Z` suffix. - - `emit.main() -> Path` — run cold-start/first-run (fresh processes) + warm/streaming (in-process, one initialized `DataWeave`), fail-fast id-validate, stamp env, write `results/python-.json`. dwlib-dependent; exercised in Task 8. - - `emit.CORPUS`, `emit.RESULTS_DIR` — resolved from `__file__`. - -- [ ] **Step 1: Write the failing test** - -Add `import emit` beneath `import warm_bench` in `test_bench.py`, and insert this class above the `if __name__` guard. It tests only the pure `build_result` (like Node's `emit.test.mjs`), so no dwlib: - -```python -class TestEmit(unittest.TestCase): - def test_build_result_shape(self): - env = {"runner": "python-wrapper", "os": "darwin-arm64", "cpu": "x", - "runtimeVersion": "python 3.9.6", "weaveVersion": "2.12.0-x", - "commit": "abc", "dwlibBuildId": "dwlib-x"} - cases = [{"id": "trivial", "metric": "warm", "unit": "ms", - "stats": {"median": 1.0}, "iterations": 3}] - r = emit.build_result(env, cases) - self.assertEqual(r["schemaVersion"], "1.0") - self.assertEqual(r["runner"], "python-wrapper") - self.assertEqual(r["env"], env) - self.assertEqual(r["cases"], cases) - self.assertTrue(r["timestamp"].endswith("Z")) -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: FAIL — `ModuleNotFoundError: No module named 'emit'`. - -- [ ] **Step 3: Implement `emit.py`** - -Create `benchmarks/runners/python/emit.py`: - -```python -"""Collector/main: run cold-start/first-run (fresh processes) + warm/streaming -(in-process), validate ids, stamp env, write results/python-.json. Emit-only — -does NOT render the report (root :benchmarkCompare renders once over all runners).""" - -import json -from datetime import datetime, timezone -from pathlib import Path - -from coldstart import run_cold_start_and_first_run -from env import gather_env -from manifest import load_manifest, validate_result_ids -from warm_bench import run_warm_and_streaming -from wrapper import load_wrapper - -# benchmarks/runners/python -> benchmarks -_BENCH_DIR = Path(__file__).resolve().parents[2] -CORPUS = _BENCH_DIR / "corpus" -RESULTS_DIR = _BENCH_DIR / "results" - - -def build_result(env, cases): - ts = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%S.%f")[:-3] + "Z" - return { - "schemaVersion": "1.0", - "runner": env["runner"], - "env": env, - "timestamp": ts, - "cases": cases, - } - - -def main(): - manifest = load_manifest(CORPUS) - - # Cold-start / first-run first (fresh processes), then warm/streaming in-process. - cold_rows = run_cold_start_and_first_run(manifest) - - api = load_wrapper() - dw = api.DataWeave() - dw.initialize() - try: - warm_rows = run_warm_and_streaming(dw, manifest) - finally: - dw.cleanup() - - cases = cold_rows + warm_rows - validate_result_ids(manifest, [c["id"] for c in cases]) # fail-fast on orphan ids - - env = gather_env() - result = build_result(env, cases) - - RESULTS_DIR.mkdir(parents=True, exist_ok=True) - stamp = result["timestamp"].replace(":", "-").replace(".", "-") - out = RESULTS_DIR / f"python-{stamp}.json" - out.write_text(json.dumps(result, indent=2), encoding="utf-8") - print(f"wrote {out} ({len(cases)} rows)") - return out - - -if __name__ == "__main__": - main() -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `cd benchmarks && python3 runners/python/test_bench.py` -Expected: PASS — `Ran 17 tests ... OK`. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/emit.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add Python runner emit collector + result serializer" -``` - ---- - -### Task 8: Gradle wiring (3 tasks) + README + end-to-end verification - -**Files:** -- Modify: `native-lib/build.gradle` (add three tasks after `benchmarkNode`; extend the `test` block) -- Modify: `benchmarks/README.md` -- (Verification only) `benchmarks/runners/python/emit.py`, `report/report.mjs`, `schema/result.schema.json` - -**Interfaces:** -- Consumes: `emit.py` (Task 7), the existing `report/report.mjs` and `schema/result.schema.json`, the existing `pythonExe` def (`native-lib/build.gradle:116`) and `stagePythonNativeLib` task. -- Produces: `native-lib:benchmarkPython` (tagged `ext.benchmarkRunner=true`, opt-in, emit-only), `native-lib:benchmarkPythonStatsTest` (always-on), `native-lib:benchmarkJsUnitTest` (always-on), both test tasks wired into `native-lib:test`; corrected README. - -- [ ] **Step 1: Add the three tasks to `native-lib/build.gradle`** - -Insert the following immediately after the `benchmarkNode` task (which ends at the closing `}` on line 235, before `tasks.named('test')`). `pythonExe` is the script-level `def` from line 116 and is in scope here: - -```groovy -// --- Python runner benchmark tasks --- - -// The Python runner as an aggregator-registered runner: emits its result file -// but does NOT render the report (root :benchmarkCompare renders once over all -// runners). Tagged `benchmarkRunner` so :benchmarkCompare discovers it. -tasks.register('benchmarkPython', Exec) { - onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } - ext.benchmarkRunner = true - - dependsOn tasks.named('stagePythonNativeLib') - workingDir("${rootDir}/benchmarks") - - // Generate the shared input via the Node generator (idempotent, deterministic, - // byte-identical to other runners) then emit; no report here. - def script = 'node corpus/gen-inputs.mjs && ' + pythonExe + ' runners/python/emit.py' - if (System.getProperty('os.name').toLowerCase().contains('windows')) { - commandLine('cmd', '/c', script) - } else { - commandLine('bash', '-c', script) - } -} - -// Always-on parity guard: pure-stdlib stats + manifest + env tests (no dwlib, no -// venv). Wired into `test` so drift vs lib/stats.mjs fails the normal build. -tasks.register('benchmarkPythonStatsTest', Exec) { - if (project.findProperty('skipPythonTests')?.toString()?.toBoolean() == true) { - enabled = false - } - workingDir("${rootDir}/benchmarks") - commandLine(pythonExe, 'runners/python/test_bench.py') -} - -// Always-on parity guard for the dwlib-free JS harness tests (shared lib + report -// + node emit builder). Excludes the dwlib-dependent warm-bench/coldstart tests. -tasks.register('benchmarkJsUnitTest', Exec) { - if (project.findProperty('skipNodeTests')?.toString()?.toBoolean() == true) { - enabled = false - } - workingDir("${rootDir}/benchmarks") - def files = 'lib/stats.test.mjs lib/manifest.test.mjs lib/env.test.mjs ' + - 'report/report.test.mjs runners/node/emit.test.mjs' - def script = 'node --test ' + files - if (System.getProperty('os.name').toLowerCase().contains('windows')) { - commandLine('cmd', '/c', script) - } else { - commandLine('bash', '-c', script) - } -} -``` - -- [ ] **Step 2: Wire both always-on tests into `native-lib:test`** - -In `native-lib/build.gradle`, replace the existing `test` block (lines 237–240): - -```groovy -tasks.named('test') { - dependsOn tasks.named('pythonTest') - dependsOn tasks.named('nodeTest') -} -``` - -with: - -```groovy -tasks.named('test') { - dependsOn tasks.named('pythonTest') - dependsOn tasks.named('nodeTest') - dependsOn tasks.named('benchmarkPythonStatsTest') - dependsOn tasks.named('benchmarkJsUnitTest') -} -``` - -- [ ] **Step 3: Verify the always-on tests run under `native-lib:test`'s new deps** - -Run: `./gradlew native-lib:benchmarkPythonStatsTest native-lib:benchmarkJsUnitTest` -Expected: both PASS — the Python parity suite prints `Ran 17 tests ... OK`; `node --test` reports all 5 JS files passing with a non-zero test count and exit 0. - -- [ ] **Step 4: Fix the README Layout claim** - -In `benchmarks/README.md`, the Layout bullet currently reads (lines 12–14): - -``` -- `runners/node/` — the Node reference runner. `runners/engine/` is the JVM baseline - (Scala/Gradle subproject `:benchmarks-engine`, depends on `org.mule.weave:runtime` at - the same `weaveVersion` the native image is built from). `runners/python/` is a follow-up. -``` - -Replace those three lines verbatim with: - -``` -- `runners/node/` — the Node reference runner. `runners/engine/` is the JVM baseline - (Scala/Gradle subproject `:benchmarks-engine`, depends on `org.mule.weave:runtime` at - the same `weaveVersion` the native image is built from). `runners/python/` is the - Python runner (stdlib scripts under `native-lib`, wrapping the same staged `dwlib` as Node). -``` - -- [ ] **Step 5: Add the Python invocation to the README Running section** - -In `benchmarks/README.md`, under "Single-runner options", the block currently reads: - -``` - ./gradlew native-lib:benchmark -Pbenchmark=true # Node only: build wrapper, run, report - ./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true # engine (JVM) only: writes results/engine-.json -``` - -Replace it verbatim with: - -``` - ./gradlew native-lib:benchmark -Pbenchmark=true # Node only: build wrapper, run, report - ./gradlew benchmarks-engine:benchmarkEngine -Pbenchmark=true # engine (JVM) only: writes results/engine-.json - ./gradlew native-lib:benchmarkPython -Pbenchmark=true # Python only: writes results/python-.json -``` - -- [ ] **Step 6: Commit the wiring + docs** - -```bash -git add native-lib/build.gradle benchmarks/README.md -git commit -m "W-23545283: Wire Python runner + always-on parity tests into Gradle" -``` - -- [ ] **Step 7: Set the GraalVM env for the native build** - -The Python runner's end-to-end run needs the staged `dwlib`, which needs a native build under GraalVM. On this machine: - -Run: -```bash -sdk use java 21.0.11-graal -export GRAALVM_HOME="$JAVA_HOME" -``` -Expected: `java -version` reports GraalVM; `$GRAALVM_HOME` and `$JAVA_HOME` both point at the graal candidate. - -- [ ] **Step 8: Run the Python runner end-to-end (opt-in, dwlib-dependent)** - -Run: `./gradlew native-lib:benchmarkPython -Pbenchmark=true` -Expected: `stagePythonNativeLib` stages `dwlib.*` into `native-lib/python/src/dataweave/native/`, `gen-inputs.mjs` writes `corpus/inputs/generated/records-large.json`, then `emit.py` prints `wrote .../benchmarks/results/python-.json ( rows)`. This is real timing — it takes a few minutes (native build + fresh Python process per cold-start/first-run sample). If `native-lib:nativeCompile` has not run yet, this triggers it first (longer). - -- [ ] **Step 9: Verify the emitted result conforms to the schema shape** - -From the repo root (pure Node built-ins, no new deps): - -Run: -```bash -node -e ' -const {readFileSync,readdirSync}=require("node:fs"); -const dir="benchmarks/results"; -const f=readdirSync(dir).filter(n=>n.startsWith("python-")&&n.endsWith(".json")).sort().pop(); -const r=JSON.parse(readFileSync(dir+"/"+f,"utf8")); -const need=(o,k)=>{if(!(k in o))throw new Error("missing "+k);}; -["schemaVersion","runner","env","timestamp","cases"].forEach(k=>need(r,k)); -if(r.schemaVersion!=="1.0")throw new Error("schemaVersion"); -if(r.runner!=="python-wrapper")throw new Error("runner "+r.runner); -["os","cpu","runtimeVersion","weaveVersion","commit","dwlibBuildId"].forEach(k=>need(r.env,k)); -if(!r.env.dwlibBuildId.startsWith("dwlib-"))throw new Error("dwlibBuildId "+r.env.dwlibBuildId); -for(const c of r.cases){["id","metric","unit","stats","iterations"].forEach(k=>need(c,k)); - if(!["cold-start","first-run","warm","streaming"].includes(c.metric))throw new Error("metric "+c.metric); - if(!["ms","MB/s"].includes(c.unit))throw new Error("unit "+c.unit); - need(c.stats,"median");} -console.log("OK "+f+" ("+r.cases.length+" rows)"); -' -``` -Expected: `OK python-.json ( rows)`. Any missing key or bad enum throws. Note the real `dwlibBuildId` (starts with `dwlib-`), unlike the engine's `n/a-engine`. - -- [ ] **Step 10: Verify the report renders the Python column** - -Run: `cd benchmarks && node report/report.mjs results/*.json` -Expected: a markdown table grouped by case × metric with a `python-wrapper` column (alongside any `node-wrapper`/`engine` results present) and a `Δ vs engine` column when the engine result exists; **no** `⚠️ WEAVE VERSION SKEW` banner (all runners share the same `weaveVersion` from `gradle.properties`). - -- [ ] **Step 11: (Optional) Verify aggregator auto-discovery** - -Run: `./gradlew benchmarkCompare -Pbenchmark=true` -Expected: the root aggregator discovers `benchmarkPython` (via its `ext.benchmarkRunner = true` tag) alongside `benchmarkNode`, runs both, and renders the report once. Confirms the runner self-registered with no edit to the `benchmarkCompare` task. (Long-running: builds the wrapper and runs both runners.) - -- [ ] **Step 12: Final commit (if any verification-driven tweaks were needed)** - -```bash -git add -A -git commit -m "W-23545283: Verify Python runner end-to-end (benchmarkPython + report + aggregator)" -``` - ---- - -## Self-Review - -**1. Spec coverage** (against `docs/superpowers/specs/2026-07-23-python-runner-design.md`): -- Pure stdlib, `benchmarks/runners/python/`, Node-mirroring structure → Tasks 1–7. ✓ -- `perf_counter_ns()` → ms timing parity → coldstart_child (Task 5), warm_bench (Task 6). ✓ -- **No JIT warmup floor** (verbatim `iterations.warmup`) → Task 6 `warm_bench.run_warm_and_streaming` + Global Constraints. ✓ -- Metric methodology table (cold-start/first-run fresh process; warm/streaming in-process) → Tasks 5, 6. ✓ -- Correctness guard (run/stream non-success aborts) → Task 5 child `SystemExit`, Task 6 `_assert_ok`/metadata guard + test. ✓ -- Explicit-dict input form + UTF-16 charset path → `_to_inputs` in Tasks 5/6; `resolve_inputs` charset in Task 2. ✓ -- Env: `runner:"python-wrapper"`, os `-` (x86_64→x64), cpu, `python `, weaveVersion, commit, **real dwlibBuildId** at the Python staging path w/ env override → Task 3 + parity test. ✓ -- Nearest-rank stats + to_mbps reimpl of lib/stats.mjs → Task 1 + parity test. ✓ -- Fail-fast id validation + schema conformance + `python-.json` → Task 2 `validate_result_ids`, Task 7 `emit`. ✓ -- Three Gradle tasks (`benchmarkPython` tagged/opt-in/emit-only; `benchmarkPythonStatsTest` always-on; `benchmarkJsUnitTest` always-on dwlib-free JS) + `test` wiring → Task 8. ✓ -- Testing: always-on parity (stats + manifest + env + wrapper + pure aggregation/emit), dwlib integration via end-to-end run → Tasks 1–7 (`test_bench.py`), Task 8 Steps 8–11. ✓ -- Docs (README Layout + Running) → Task 8 Steps 4–5. ✓ -- Out of scope honored: no corpus/schema/report.mjs/engine/Node-source changes; no `settings.gradle` change (scripts under existing module); dwlib-dependent Node tests left unwired (explicit file list in `benchmarkJsUnitTest`). ✓ - -**2. Placeholder scan:** No "TBD"/"TODO"/"handle edge cases" — every code step has complete, runnable code; every command has expected output. ✓ - -**3. Type/name consistency across tasks:** `compute_stats`/`to_mbps` (Task 1) used in Tasks 5/6; `load_manifest`/`cases_for_metric`/`resolve_inputs`/`read_script`/`validate_result_ids` (Task 2) used in Tasks 5/6/7; `gather_env` (Task 3) used in Task 7; `load_wrapper` (Task 4) used in Tasks 5/7; `run_cold_start_and_first_run(manifest, sample_fn=, samples_override=)` (Task 5) called in Task 7; `run_warm_and_streaming(api, manifest, warmup_cap=, warm_cap=, streaming_cap=)` (Task 6) called in Task 7; `build_result(env, cases)` (Task 7) used by `main` + test. Row shape `{id, metric, unit, stats, iterations}` identical in coldstart/warm_bench/emit. Result key `corpusDir` consistent between `load_manifest` output and all consumers. ✓ diff --git a/docs/superpowers/plans/2026-07-24-streaming-alignment.md b/docs/superpowers/plans/2026-07-24-streaming-alignment.md deleted file mode 100644 index cc575ca..0000000 --- a/docs/superpowers/plans/2026-07-24-streaming-alignment.md +++ /dev/null @@ -1,944 +0,0 @@ -# Streaming Methodology Alignment Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Make the `streaming` benchmark metric measure the same operation on all three runners (chunked input + concurrent deferred output), so its cross-runner delta is meaningful, then remove the report's `n/a` suppression. - -**Architecture:** Add `deferred=true` streaming-script variants to the corpus, selected only for the `streaming` metric via a new manifest `streamingScript` field. The engine runner gains a streaming path that binds a chunk-paced `InputStream` (lazy input) and drains the deferred `PipedInputStream` result. native-lib runners already stream input and consume a deferred `InputStream` result, so they only switch to the deferred script. The report stops suppressing the streaming delta. - -**Tech Stack:** Scala 2.12 (engine runner, scalatest `AnyFreeSpec`/`Matchers`, org.json), Node ESM (`node:test`), Python 3 (`unittest`), DataWeave runtime `DataWeaveScriptingEngine`. - -## Global Constraints - -- **Scala version is 2.12.18** — `scala.util.Using` does NOT exist; use explicit `try/finally` for resource cleanup. -- **Do not touch the warm / first-run / cold-start code paths.** Only the streaming path changes. Warm/first-run must keep resolving the base `script`, not the variant. -- **Throughput formula is unchanged:** MB/s = `bytes / 1e6 / (ms / 1000)` over the primary input byte count (`Stats.toMBps` / `to_mbps` / `toMBps`). Denominator stays the primary *input* bytes, never the output/drained bytes. -- **Engine streaming input chunk size = 65536 bytes (64KB)**, matching Node's `chunked(buffer, 65536)`. -- **The engine runner uses no security manager**, so the default `NoSecurityManagerService` grants the `DEFERRED` privilege — no privilege plumbing is needed for `deferred=true`. -- **Never hand-fabricate benchmark numbers.** `RESULTS.md` streaming rows are only updated from a real run (final task); if no run is done, leave the committed report as-is and note it. -- Commit after each task with the `W-23545283:` subject prefix used throughout the branch. - ---- - -## File Structure - -- `benchmarks/corpus/scripts/map-scale.stream.dwl` — **new**, deferred variant of map-scale. -- `benchmarks/corpus/scripts/json-stream.stream.dwl` — **new**, deferred variant of json-stream. -- `benchmarks/corpus/manifest.json` — **modify**, add `streamingScript` to the two streaming cases. -- `benchmarks/lib/manifest.mjs` — **modify**, add `resolveStreamingScript` + validate the field. -- `benchmarks/runners/python/manifest.py` — **modify**, add `resolve_streaming_script` + validate. -- `benchmarks/runners/engine/.../Manifest.scala` — **modify**, add `streamingScript` to `BenchCase` + `resolveStreamingScript`. -- `benchmarks/runners/node/warm-bench.mjs` — **modify**, streaming loop resolves the variant script. -- `benchmarks/runners/python/warm_bench.py` — **modify**, streaming loop resolves the variant script. -- `benchmarks/runners/engine/.../EngineShell.scala` — **modify**, add `runStreaming` + a `ChunkedInputStream` helper (new file). -- `benchmarks/runners/engine/.../ChunkedInputStream.scala` — **new**, 64KB-paced InputStream over a byte array. -- `benchmarks/runners/engine/.../WarmBench.scala` — **modify**, streaming uses `runStreaming` over a chunked stream + variant script. -- `benchmarks/report/report.mjs` — **modify**, remove `streaming` from non-comparable set + drop footnote. -- Tests: `manifest.test.mjs`, `test_bench.py`, `ManifestTest.scala`, `EngineShellTest.scala`, `WarmBenchTest.scala`, `report.test.mjs`. - ---- - -## Task 1: Add corpus streaming-script variants (deferred output) - -**Files:** -- Create: `benchmarks/corpus/scripts/map-scale.stream.dwl` -- Create: `benchmarks/corpus/scripts/json-stream.stream.dwl` - -**Interfaces:** -- Produces: two `.dwl` files with `deferred=true` output, identical transform bodies to their base scripts. Referenced by Task 2's manifest field. - -- [ ] **Step 1: Create `map-scale.stream.dwl`** - -Body is identical to `scripts/map-scale.dwl` except the output directive adds `deferred=true`: - -``` -output application/json deferred=true ---- -payload map (item) -> { id: item.id, doubled: item.value * 2, label: "item_" ++ item.id } -``` - -- [ ] **Step 2: Create `json-stream.stream.dwl`** - -``` -output application/json deferred=true ---- -payload map (item) -> { id: item.id, name: item.name } -``` - -- [ ] **Step 3: Sanity-check the scripts run deferred through the prebuilt dwlib** - -Run (dylib already built, exports `run_script_input_output_callback`): - -```bash -cd /Users/aradunsky/Documents/public/data-weave-cli -DATAWEAVE_NATIVE_LIB="$PWD/native-lib/python/src/dataweave/native/dwlib.dylib" \ -python3 - <<'PY' -import sys; sys.path.insert(0, "native-lib/python/src") -from dataweave import DataWeave -inp = open("benchmarks/corpus/inputs/generated/records-large.json","rb").read() -def chunked(d, s=65536): - for i in range(0, len(d), s): yield d[i:i+s] -dw = DataWeave(); dw.initialize() -for f in ["benchmarks/corpus/scripts/map-scale.stream.dwl", "benchmarks/corpus/scripts/json-stream.stream.dwl"]: - script = open(f).read() - gen = dw.run_transform(script, chunked(inp), input_name="payload", input_mime_type="application/json", input_charset="utf-8") - total = sum(len(c) for c in gen) - print(f, "success=", gen.metadata.success, "out_bytes=", total) -dw.cleanup() -PY -``` - -Expected: both lines print `success= True out_bytes=` with a positive byte count (~34103 for map-scale). If `records-large.json` is absent, generate it first: `node benchmarks/corpus/gen-inputs.mjs`. - -- [ ] **Step 4: Commit** - -```bash -git add benchmarks/corpus/scripts/map-scale.stream.dwl benchmarks/corpus/scripts/json-stream.stream.dwl -git commit -m "W-23545283: Add deferred=true streaming-script variants for map-scale + json-stream" -``` - ---- - -## Task 2: Add `streamingScript` to the manifest - -**Files:** -- Modify: `benchmarks/corpus/manifest.json` - -**Interfaces:** -- Produces: `streamingScript` field on the `map-scale` and `json-stream` cases. Consumed by the manifest resolvers in Tasks 3–5. - -- [ ] **Step 1: Add `streamingScript` to the `map-scale` case** - -In `benchmarks/corpus/manifest.json`, the `map-scale` case currently has `"script": "scripts/map-scale.dwl"`. Add a sibling field so the object reads: - -```json - { - "id": "map-scale", - "script": "scripts/map-scale.dwl", - "streamingScript": "scripts/map-scale.stream.dwl", - "inputs": { - "payload": { "file": "inputs/generated/records-large.json", "mimeType": "application/json", "generated": true } - }, - "metrics": ["first-run", "warm", "streaming"], - "iterations": { "warm": 30, "warmup": 3, "streaming": 10, "samples": 15 } - }, -``` - -- [ ] **Step 2: Add `streamingScript` to the `json-stream` case** - -```json - { - "id": "json-stream", - "script": "scripts/json-stream.dwl", - "streamingScript": "scripts/json-stream.stream.dwl", - "inputs": { - "payload": { "file": "inputs/generated/records-large.json", "mimeType": "application/json", "generated": true } - }, - "metrics": ["first-run", "warm", "streaming"], - "iterations": { "warm": 30, "warmup": 3, "streaming": 10, "samples": 15 } - }, -``` - -- [ ] **Step 3: Verify JSON is valid** - -Run: `node -e "JSON.parse(require('fs').readFileSync('benchmarks/corpus/manifest.json','utf8')); console.log('ok')"` -Expected: `ok` - -- [ ] **Step 4: Commit** - -```bash -git add benchmarks/corpus/manifest.json -git commit -m "W-23545283: Point streaming cases at deferred streamingScript variants" -``` - ---- - -## Task 3: Node manifest — `resolveStreamingScript` + validation - -**Files:** -- Modify: `benchmarks/lib/manifest.mjs` -- Test: `benchmarks/lib/manifest.test.mjs` - -**Interfaces:** -- Produces: `resolveStreamingScript(manifest, caseObj)` returning the absolute-read script text for the streaming metric — reads `streamingScript` if present, else falls back to `script`. Consumed by Task 8 (`warm-bench.mjs`). -- Consumes: existing `loadManifest`, `resolveInputs` from this module. - -- [ ] **Step 1: Write the failing test** - -Add to `benchmarks/lib/manifest.test.mjs` (import `resolveStreamingScript` in the existing import block from `./manifest.mjs`): - -```javascript -test("resolveStreamingScript prefers streamingScript, falls back to script", () => { - const m = loadManifest(CORPUS); - const mapScale = m.cases.find((c) => c.id === "map-scale"); - const streamText = resolveStreamingScript(m, mapScale); - assert.ok(streamText.includes("deferred=true"), "streaming variant declares deferred=true"); - - const objTransform = m.cases.find((c) => c.id === "object-transform"); // no streamingScript - const fallback = resolveStreamingScript(m, objTransform); - const base = readFileSync(join(CORPUS, objTransform.script), "utf-8"); - assert.equal(fallback, base, "falls back to the base script when no streamingScript"); -}); -``` - -Add these imports at the top of the test file if not already present: - -```javascript -import { readFileSync } from "node:fs"; -import { join } from "node:path"; -``` - -(The file already imports `loadManifest`, `casesForMetric`, `METRICS`; add `resolveStreamingScript` to that import list. `CORPUS` is already defined in the file.) - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/lib/manifest.test.mjs` -Expected: FAIL — `resolveStreamingScript is not a function` (or `not exported`). - -- [ ] **Step 3: Implement `resolveStreamingScript` and validate the field in `loadManifest`** - -In `benchmarks/lib/manifest.mjs`, inside the `loadManifest` per-case loop, right after the existing `script` existence check (the block that throws `case ${c.id} script not found`), add validation that a declared `streamingScript` exists: - -```javascript - if (c.streamingScript && !existsSync(join(corpusDir, c.streamingScript))) { - throw new Error(`case ${c.id} streamingScript not found: ${c.streamingScript}`); - } -``` - -Then add the exported resolver near `resolveInputs`: - -```javascript -/** - * Read the script used for the streaming metric: the `streamingScript` variant - * (e.g. a deferred=true output) if declared, else the base `script`. Warm and - * first-run always use `script`, never this. - */ -export function resolveStreamingScript(manifest, caseObj) { - const rel = caseObj.streamingScript ?? caseObj.script; - return readFileSync(join(manifest.corpusDir, rel), "utf-8"); -} -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `node --test benchmarks/lib/manifest.test.mjs` -Expected: PASS (all tests, including the existing ones). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/lib/manifest.mjs benchmarks/lib/manifest.test.mjs -git commit -m "W-23545283: Add resolveStreamingScript + streamingScript validation (node lib)" -``` - ---- - -## Task 4: Python manifest — `resolve_streaming_script` + validation - -**Files:** -- Modify: `benchmarks/runners/python/manifest.py` -- Test: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Produces: `resolve_streaming_script(manifest, case_obj)` returning script text — `streamingScript` if present, else `script`. Consumed by Task 9 (`warm_bench.py`). - -- [ ] **Step 1: Write the failing test** - -In `benchmarks/runners/python/test_bench.py`, find the `TestManifest` class (it contains `test_read_script`). Add: - -```python - def test_resolve_streaming_script_prefers_variant(self): - map_scale = next(c for c in self.m["cases"] if c["id"] == "map-scale") - text = manifest.resolve_streaming_script(self.m, map_scale) - self.assertIn("deferred=true", text) - - def test_resolve_streaming_script_falls_back_to_base(self): - obj = next(c for c in self.m["cases"] if c["id"] == "object-transform") - self.assertEqual( - manifest.resolve_streaming_script(self.m, obj), - manifest.read_script(self.m, obj), - ) -``` - -(`self.m` is the loaded manifest fixture already used by `TestManifest`; confirm by reading the class `setUp`. If the fixture is named differently, use that name.) - -- [ ] **Step 2: Run test to verify it fails** - -Run: `cd benchmarks/runners/python && python3 -m unittest test_bench.TestManifest -v` -Expected: FAIL — `AttributeError: module 'manifest' has no attribute 'resolve_streaming_script'`. - -- [ ] **Step 3: Implement resolver + validation** - -In `benchmarks/runners/python/manifest.py`, inside `load_manifest`'s per-case loop, right after the existing `script` existence check (the `raise ValueError(f"case {cid} script not found` block), add: - -```python - streaming_script = c.get("streamingScript") - if streaming_script and not (corpus_dir / streaming_script).exists(): - raise ValueError(f"case {cid} streamingScript not found: {streaming_script}") -``` - -Then add, next to `read_script`: - -```python -def resolve_streaming_script(manifest, case_obj): - """Script for the streaming metric: the streamingScript variant if declared, - else the base script. Warm/first-run always use read_script.""" - rel = case_obj.get("streamingScript") or case_obj["script"] - return (Path(manifest["corpusDir"]) / rel).read_text(encoding="utf-8") -``` - -- [ ] **Step 4: Run test to verify it passes** - -Run: `cd benchmarks/runners/python && python3 -m unittest test_bench.TestManifest -v` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/manifest.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Add resolve_streaming_script + streamingScript validation (python)" -``` - ---- - -## Task 5: Scala manifest — `streamingScript` field + `resolveStreamingScript` - -**Files:** -- Modify: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Manifest.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala` - -**Interfaces:** -- Produces: `BenchCase.streamingScript: Option[String]` and `Manifest.resolveStreamingScript(m, c): String`. Consumed by Task 7 (`WarmBench.scala`). -- Consumes: existing `Manifest.load`, `resolveScript`, `resolveInputs`. - -- [ ] **Step 1: Write the failing test** - -Append to `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala` (inside the existing `AnyFreeSpec` body — check the class name/structure first and match it): - -```scala - "resolveStreamingScript prefers the streamingScript variant" in { - val corpus = new File("../../corpus").getCanonicalFile - val m = Manifest.load(corpus) - val mapScale = m.cases.find(_.id == "map-scale").get - Manifest.resolveStreamingScript(m, mapScale) should include ("deferred=true") - } - - "resolveStreamingScript falls back to the base script" in { - val corpus = new File("../../corpus").getCanonicalFile - val m = Manifest.load(corpus) - val obj = m.cases.find(_.id == "object-transform").get // no streamingScript - Manifest.resolveStreamingScript(m, obj) shouldBe Manifest.resolveScript(m, obj) - } -``` - -Ensure `import java.io.File` is present in the test file. - -- [ ] **Step 2: Run test to verify it fails** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.ManifestTest"` -Expected: FAIL — compilation error (`streamingScript`/`resolveStreamingScript` not found) or missing member. - -- [ ] **Step 3: Add the field to `BenchCase` and parse it** - -In `Manifest.scala`, add `streamingScript` to the case class: - -```scala -final case class BenchCase( - id: String, - script: String, - streamingScript: Option[String], - inputs: Seq[CaseInput], - metrics: Set[String], - iterations: Map[String, Int]) { -``` - -In `load`, after the existing `script` existence check, parse + validate the optional variant: - -```scala - val streamingScript: Option[String] = - if (obj.has("streamingScript")) { - val ss = obj.getString("streamingScript") - if (!new File(corpusDir, ss).exists()) throw new RuntimeException(s"case $id streamingScript not found: $ss") - Some(ss) - } else None -``` - -Update the `BenchCase(...)` construction at the end of the loop to pass it: - -```scala - BenchCase(id, script, streamingScript, inputs, metrics, iterations) -``` - -Add the resolver in the `Manifest` object, next to `resolveScript`: - -```scala - def resolveStreamingScript(m: Manifest, c: BenchCase): String = { - val rel = c.streamingScript.getOrElse(c.script) - new String(Files.readAllBytes(new File(m.corpusDir, rel).toPath), StandardCharsets.UTF_8) - } -``` - -- [ ] **Step 4: Fix any other `BenchCase(...)` constructions** - -The new field is a positional param. Search for other constructions and add `None` (or the value) as needed: - -Run: `rg -n 'BenchCase\(' benchmarks/runners/engine/src` -For each hit outside `Manifest.scala` (e.g. tests building a `BenchCase` directly), insert `streamingScript = None` in the correct position. If there are none, proceed. - -- [ ] **Step 5: Run test to verify it passes** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.ManifestTest"` -Expected: PASS. - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/Manifest.scala benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/ManifestTest.scala -git commit -m "W-23545283: Add streamingScript field + resolveStreamingScript (engine manifest)" -``` - ---- - -## Task 6: Engine `ChunkedInputStream` helper + `EngineShell.runStreaming` - -**Files:** -- Create: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/ChunkedInputStream.scala` -- Modify: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineShell.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineShellTest.scala` - -**Interfaces:** -- Produces: - - `class ChunkedInputStream(data: Array[Byte], chunkSize: Int) extends InputStream` — an InputStream over `data` whose `read(buf, off, len)` returns at most `chunkSize` bytes per call (paces input into the runtime). - - `EngineShell.runStreaming(script: String, name: String, input: InputStream, inMime: String, inCharset: Option[String]): Long` — binds `input` as a lazy `BindingValue`, compiles the (deferred) script, drains the deferred `InputStream` result, returns the drained byte count. -- Consumes: existing `EngineShell` engine + `serviceManager`, `Manifest.ResolvedInput` (indirectly via caller). - -- [ ] **Step 1: Write the failing test** - -Add to `EngineShellTest.scala`: - -```scala - "ChunkedInputStream returns at most chunkSize bytes per read" in { - val data = Array.tabulate[Byte](10)(_.toByte) - val in = new ChunkedInputStream(data, 4) - val buf = new Array[Byte](8) - in.read(buf, 0, 8) shouldBe 4 // capped at chunkSize - in.read(buf, 0, 8) shouldBe 4 - in.read(buf, 0, 8) shouldBe 2 // remainder - in.read(buf, 0, 8) shouldBe -1 // EOF - } - - "runStreaming binds a lazy InputStream and drains deferred output" in { - val c = manifest.cases.find(_.id == "map-scale").get - val bytes = Manifest.resolveInputs(manifest, c).head.bytes - val shell = new EngineShell() - val drained = shell.runStreaming( - Manifest.resolveStreamingScript(manifest, c), - EngineShell.safeName(c.id), - new ChunkedInputStream(bytes, 65536), - "application/json", - None) - drained should be > 0L - } -``` - -The test references `manifest` (already a field in `EngineShellTest`). It needs the generated input; guard it: - -```scala - // at the top of the second test body, before use: - if (!TestSupport.ensureGeneratedInputs(corpus)) cancel("generated input unavailable (node missing?)") -``` - -Add `import java.io.InputStream` if needed (only if referenced directly; `ChunkedInputStream` hides it). - -- [ ] **Step 2: Run test to verify it fails** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineShellTest"` -Expected: FAIL — `ChunkedInputStream`/`runStreaming` not found (compile error). - -- [ ] **Step 3: Create `ChunkedInputStream.scala`** - -```scala -package org.mule.weave.benchmark.engine - -import java.io.InputStream - -/** An InputStream over an in-memory byte array that returns at most `chunkSize` - * bytes per read() call, pacing input into the DataWeave runtime the way a - * network/pipe feeder would. The bytes are already resident, so this models - * "runtime reads input lazily" without a real producer thread. Mirrors the - * native-lib runners feeding fixed-size chunks. */ -class ChunkedInputStream(data: Array[Byte], chunkSize: Int) extends InputStream { - private var pos: Int = 0 - - override def read(): Int = { - if (pos >= data.length) -1 - else { - val b = data(pos) & 0xff - pos += 1 - b - } - } - - override def read(b: Array[Byte], off: Int, len: Int): Int = { - if (pos >= data.length) -1 - else { - val n = math.min(math.min(len, chunkSize), data.length - pos) - System.arraycopy(data, pos, b, off, n) - pos += n - n - } - } - - override def available(): Int = data.length - pos -} -``` - -- [ ] **Step 4: Add `runStreaming` to `EngineShell`** - -In `EngineShell.scala`, add these imports if not present: `import java.io.InputStream`. Add the method inside the `class EngineShell` body, after the existing `run(...)`: - -```scala - /** Streaming variant of `run`: binds `input` as a lazy InputStream (so the - * runtime reads it incrementally), compiles the deferred script, and drains - * the deferred PipedInputStream result in a read loop. Returns the number of - * output bytes drained. Throws on compile/exec failure or a non-InputStream - * (non-deferred) result, so a script that forgot `deferred=true` fails loudly. */ - def runStreaming(script: String, name: String, input: InputStream, inMime: String, inCharset: Option[String]): Long = { - val bindings = new ScriptingBindings() - val charset = Charset.forName(inCharset.getOrElse("UTF-8")) - val bv = new BindingValue(input, Some(inMime), Map.empty[String, Any], charset) - bindings.addBinding(name, bv) - - val config = engine.newConfig() - .withScript(script) - .withNameIdentifier(NameIdentifier(name)) - .withInputs(Array(new InputType(name, None))) - .withDefaultOutputType("application/json") - - val compiled: DataWeaveScript = engine.compileWith(config) - val result: DataWeaveResult = compiled.write(bindings, serviceManager, "application/json", Option.empty[Any]) - result.getContent match { - case is: InputStream => - try { - val buf = new Array[Byte](65536) - var total = 0L - var n = is.read(buf) - while (n > 0) { total += n; n = is.read(buf) } - total - } finally { - is.close() - } - case other => - throw new RuntimeException( - s"streaming result is not an InputStream (did the script declare deferred=true?): ${other.getClass.getName}") - } - } -``` - -Add these imports to `EngineShell.scala` if missing: `DataWeaveResult` (from `org.mule.weave.v2.runtime`). Update the existing runtime import block: - -```scala -import org.mule.weave.v2.runtime.{ - BindingValue, - DataWeaveResult, - DataWeaveScript, - DataWeaveScriptingEngine, - InputType, - ModuleComponentsFactory, - ParserConfiguration, - ScriptingBindings -} -``` - -Note on the `write` overload: `write(bindings, serviceManager, outputMimeType, target: Option[Any])` exists on `DataWeaveScript` (verified in `DataWeaveScriptingEngine.scala:984`). Passing `"application/json"` + `Option.empty[Any]` gives a writer whose deferred setting comes from the script's `deferred=true` directive; the result content is the `PipedInputStream`. - -- [ ] **Step 5: Run test to verify it passes** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.EngineShellTest"` -Expected: PASS (existing `run` tests still green; new streaming tests pass, or the second `cancel`s only if generated input is unavailable — acceptable). - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/ChunkedInputStream.scala benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/EngineShell.scala benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/EngineShellTest.scala -git commit -m "W-23545283: Add engine streaming path (chunked InputStream + deferred drain)" -``` - ---- - -## Task 7: Wire `WarmBench.runStreaming` to the streaming path - -**Files:** -- Modify: `benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/WarmBench.scala` -- Test: `benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/WarmBenchTest.scala` - -**Interfaces:** -- Consumes: `EngineShell.runStreaming`, `ChunkedInputStream`, `Manifest.resolveStreamingScript`, `Stats.toMBps`. -- Produces: `WarmBench.runStreaming` now measures chunked-input + deferred-output throughput. Same `Row(id, "streaming", "MB/s", ...)` output shape (consumed by `Emit`). - -- [ ] **Step 1: Update `WarmBenchTest` streaming assertion** - -The existing test `"streaming rows are produced with MB/s unit"` stays valid (same output shape). Strengthen it to prove the streaming variant is used — replace that test body with: - -```scala - "streaming rows are produced with MB/s unit via the deferred variant" in { - if (!TestSupport.ensureGeneratedInputs(corpus)) cancel("generated input unavailable (node missing?)") - val shell = new EngineShell() - val rows = WarmBench.runStreaming(shell, manifest, iterCap = Some(2)) - rows.map(_.id) should contain allOf ("map-scale", "json-stream") - all (rows.map(_.metric)) shouldBe "streaming" - all (rows.map(_.unit)) shouldBe "MB/s" - all (rows.map(_.stats.median)) should be > 0.0 - } -``` - -- [ ] **Step 2: Run test to verify current state** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.WarmBenchTest"` -Expected: The streaming test still passes against the OLD batch implementation (output shape unchanged). This step confirms the test is green before the refactor; the behavior change is validated by it staying green after. - -- [ ] **Step 3: Rewrite `runStreaming` to use the streaming path** - -In `WarmBench.scala`, replace the body of `runStreaming`'s per-case loop. The new version resolves the streaming variant script, wraps the primary input bytes in a `ChunkedInputStream`, and times `shell.runStreaming`. Replace the method with: - -```scala - def runStreaming(shell: EngineShell, m: Manifest, iterCap: Option[Int] = None): Seq[Row] = { - Manifest.casesForMetric(m, "streaming").map { c => - val script = Manifest.resolveStreamingScript(m, c) - val inputs = Manifest.resolveInputs(m, c) - val name = EngineShell.safeName(c.id) - val primary = inputs.head - val primaryBytes = primary.bytes.length.toLong - val iters = iterCap.getOrElse(c.streaming) - - // Guard: if this case does NOT declare warm, warm up first so streaming isn't JIT-cold. - // Warmup uses the deferred streaming path too, over a fresh ChunkedInputStream each time. - if (!c.metrics.contains("warm")) { - println(s"[streaming] ${c.id}: streaming-only, warming up first ($WARMUP_FLOOR iters)") - var w = 0 - while (w < WARMUP_FLOOR) { - shell.runStreaming(script, name, new ChunkedInputStream(primary.bytes, 65536), primary.mimeType, primary.charset) - w += 1 - } - } - - val mbps = new Array[Double](iters) - var i = 0 - while (i < iters) { - val in = new ChunkedInputStream(primary.bytes, 65536) - val start = nowNs() - shell.runStreaming(script, name, in, primary.mimeType, primary.charset) - mbps(i) = Stats.toMBps(primaryBytes, msSince(start)) - i += 1 - } - Row(c.id, "streaming", "MB/s", Stats.computeStats(mbps.toSeq), iters) - } - } -``` - -Update the scaladoc comment above `runStreaming` (currently describing the batch/asymmetry) to: - -```scala - /** Streaming throughput: feeds the primary input in 64KB chunks via a - * ChunkedInputStream (mirroring the native-lib runners) and drains the - * deferred output InputStream per iteration. MB/s is over the primary input - * bytes. The streaming-script variant (deferred=true) is resolved via - * Manifest.resolveStreamingScript so warm/first-run keep the base script. - */ -``` - -Delete the old "Methodology asymmetry" paragraph in that comment — it no longer applies. - -A `ChunkedInputStream` is single-use (position advances to EOF), so a fresh instance is created per iteration and per warmup pass — this is intentional and cheap (no byte copy; wraps the shared array). - -- [ ] **Step 4: Run test to verify it passes** - -Run: `./gradlew :benchmarks-engine:test --tests "org.mule.weave.benchmark.engine.WarmBenchTest"` -Expected: PASS. - -- [ ] **Step 5: Run the full engine suite to confirm no regressions** - -Run: `./gradlew :benchmarks-engine:test` -Expected: BUILD SUCCESSFUL, all tests pass (Emit, EngineChild, EngineShell, Manifest, Smoke, StatsParity, WarmBench). - -- [ ] **Step 6: Commit** - -```bash -git add benchmarks/runners/engine/src/main/scala/org/mule/weave/benchmark/engine/WarmBench.scala benchmarks/runners/engine/src/test/scala/org/mule/weave/benchmark/engine/WarmBenchTest.scala -git commit -m "W-23545283: Engine streaming measures chunked-input + deferred-output throughput" -``` - ---- - -## Task 8: Node streaming loop resolves the deferred variant - -**Files:** -- Modify: `benchmarks/runners/node/warm-bench.mjs` -- Test: `benchmarks/runners/node/warm-bench.test.mjs` - -**Interfaces:** -- Consumes: `resolveStreamingScript` from `../../lib/manifest.mjs` (Task 3). -- The `runTransform` call, chunk size (64KB), and MB/s formula are unchanged — only the *script* the streaming loop runs changes. - -- [ ] **Step 1: Strengthen the test to assert the deferred script is used** - -The existing test `"warm + streaming rows are produced with valid stats"` covers the output shape. Add a focused test to `warm-bench.test.mjs` that the streaming loop runs a `deferred=true` script by asserting the resolver picks it (behavioral proof that doesn't require intercepting the wrapper): - -```javascript -import { resolveStreamingScript } from "../../lib/manifest.mjs"; - -test("streaming uses the deferred=true script variant", () => { - const manifest = loadManifest(CORPUS); - const mapScale = manifest.cases.find((c) => c.id === "map-scale"); - assert.ok(resolveStreamingScript(manifest, mapScale).includes("deferred=true")); -}); -``` - -(Reuses `loadManifest` and `CORPUS` already imported/defined in the file.) - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/runners/node/warm-bench.test.mjs` -Expected: FAIL — `resolveStreamingScript is not exported` if Task 3 not merged; if Task 3 is merged, this passes immediately at the resolver level but the *loop* still reads the base script. To make the test meaningful, also verify the loop change in Step 3. - -- [ ] **Step 3: Update the streaming loop to resolve the variant** - -In `warm-bench.mjs`, the file currently imports `{ casesForMetric, resolveInputs }` from `../../lib/manifest.mjs`. Add `resolveStreamingScript`: - -```javascript -import { casesForMetric, resolveInputs, resolveStreamingScript } from "../../lib/manifest.mjs"; -``` - -In the streaming loop (the `for (const c of casesForMetric(manifest, "streaming"))` block), replace: - -```javascript - const script = readScript(manifest, c); -``` - -with: - -```javascript - const script = resolveStreamingScript(manifest, c); -``` - -Leave the `warm` loop's `readScript(manifest, c)` untouched. `readScript` is still used by the warm loop, so keep the helper. - -- [ ] **Step 4: Run the tests** - -Run: `node --test benchmarks/runners/node/warm-bench.test.mjs` -Expected: PASS. (This spawns the real wrapper; if the native lib is unavailable the pre-existing test would already fail — that's an environment issue, not this change.) - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/node/warm-bench.mjs benchmarks/runners/node/warm-bench.test.mjs -git commit -m "W-23545283: Node streaming runs the deferred=true script variant" -``` - ---- - -## Task 9: Python streaming loop resolves the deferred variant - -**Files:** -- Modify: `benchmarks/runners/python/warm_bench.py` -- Test: `benchmarks/runners/python/test_bench.py` - -**Interfaces:** -- Consumes: `resolve_streaming_script` from `manifest` (Task 4). -- `run_transform` call, 8KB chunking, MB/s formula unchanged — only the script changes. - -- [ ] **Step 1: Write the failing test** - -The `TestWarmBench` streaming test uses a `_FakeApi` and a synthetic `_streaming_manifest`, so it does not exercise the real script resolution. Add a manifest-level assertion to `TestManifest` (or `TestWarmBench`) instead: - -```python - def test_streaming_uses_deferred_variant(self): - map_scale = next(c for c in self.m["cases"] if c["id"] == "map-scale") - self.assertIn("deferred=true", manifest.resolve_streaming_script(self.m, map_scale)) -``` - -(Place in `TestManifest`, which has `self.m`. This overlaps Task 4's resolver test but asserts the corpus wiring specifically; keep it — it documents intent at the runner boundary.) - -If you want the loop itself covered: update `_FakeApi.run_transform` is not necessary; the loop change is verified by reading the diff + the resolver test. Do not over-engineer a fake that intercepts the script string. - -- [ ] **Step 2: Run test to verify it fails or passes** - -Run: `cd benchmarks/runners/python && python3 -m unittest test_bench.TestManifest -v` -Expected: PASS if Task 4 merged (resolver exists + corpus has the variant). If it fails with `deferred=true` not found, Task 1/2 are incomplete. - -- [ ] **Step 3: Update the streaming loop** - -In `warm_bench.py`, the imports line is: - -```python -from manifest import cases_for_metric, read_script, resolve_inputs -``` - -Add `resolve_streaming_script`: - -```python -from manifest import cases_for_metric, read_script, resolve_inputs, resolve_streaming_script -``` - -In the streaming loop (`for c in cases_for_metric(manifest, "streaming"):`), replace: - -```python - script = read_script(manifest, c) -``` - -with: - -```python - script = resolve_streaming_script(manifest, c) -``` - -Leave the `warm` loop's `read_script(manifest, c)` untouched. - -- [ ] **Step 4: Run the tests** - -Run: `cd benchmarks/runners/python && python3 -m unittest test_bench -v` -Expected: PASS (all classes). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/python/warm_bench.py benchmarks/runners/python/test_bench.py -git commit -m "W-23545283: Python streaming runs the deferred=true script variant" -``` - ---- - -## Task 10: Un-suppress the streaming delta in the report - -**Files:** -- Modify: `benchmarks/report/report.mjs` -- Test: `benchmarks/report/report.test.mjs` - -**Interfaces:** -- `computeDelta`, `buildTable`, `formatDelta` keep their signatures. `NON_COMPARABLE_METRICS` becomes empty; `streaming` rows now carry a real delta. - -- [ ] **Step 1: Update the report tests to expect a real streaming delta** - -In `report.test.mjs`, replace the two tests added for finding #2 — `"streaming metric is non-comparable — no cross-runner delta is printed"` and `"renderMarkdown footnotes the n/a delta when a non-comparable metric is present"` — with: - -```javascript -test("streaming metric now carries a real cross-runner delta", () => { - const manifest = loadManifest(CORPUS); - const results = [load("node-a.json"), load("engine-b.json")]; - const { rows } = buildTable(manifest, results, "engine"); - - // Fixtures: map-scale streaming node=300 vs engine=150 MB/s. With aligned - // methodology this is a real +100% delta, no longer suppressed. - const streaming = rows.find((r) => r.id === "map-scale" && r.metric === "streaming"); - assert.ok(streaming, "fixture should exercise a streaming row"); - assert.equal(streaming.comparable, true); - assert.equal(streaming.delta, 100); - assert.equal(formatDelta(streaming), "+100.0%"); -}); - -test("renderMarkdown emits no streaming non-comparable footnote", () => { - const manifest = loadManifest(CORPUS); - const results = [load("node-a.json"), load("engine-b.json")]; - const table = buildTable(manifest, results, "engine"); - const md = renderMarkdown(table, results, { - baselineRunner: "engine", - stamp: { commit: "abc1234", date: "2026-07-24T14:33:03Z" }, - }); - assert.ok(md.includes("| map-scale | streaming | MB/s |"), "streaming row is present"); - assert.ok(!md.includes("not like-for-like across runners"), "footnote removed"); -}); -``` - -- [ ] **Step 2: Run test to verify it fails** - -Run: `node --test benchmarks/report/report.test.mjs` -Expected: FAIL — the old behavior still sets `comparable=false`/`n/a` and emits the footnote. - -- [ ] **Step 3: Remove the suppression** - -In `report.mjs`: - -Change the non-comparable set to empty and update its comment: - -```javascript -/** - * Metrics whose cross-runner numbers are not like-for-like. Empty since the - * streaming methodology was aligned across runners (chunked input + deferred - * output) — the streaming delta is meaningful again. Kept as a seam for any - * future non-comparable metric. - */ -const NON_COMPARABLE_METRICS = new Set(); -``` - -Leave `formatDelta`, `buildTable`'s `comparable` computation, and the `n/a` branch exactly as-is — with an empty set, `comparable` is always `true` and `n/a` never renders, but the mechanism stays for the future. - -Remove the footnote block in `renderMarkdown` (the `if (table.rows.some((r) => !r.comparable)) { out.push("> \`n/a\` deltas mark ... ", "") }` block) and the console footnote block in `main` (the `if (rows.some((r) => !r.comparable)) { console.log(...) }` block). Since `comparable` is now always true, these are dead; delete both for cleanliness. - -- [ ] **Step 4: Run test to verify it passes** - -Run: `node --test benchmarks/report/report.test.mjs` -Expected: PASS. - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/report/report.mjs benchmarks/report/report.test.mjs -git commit -m "W-23545283: Un-suppress streaming delta now that methodology is aligned" -``` - ---- - -## Task 11: Full verification + regenerate RESULTS.md from a real run - -**Files:** -- Modify: `benchmarks/report/RESULTS.md` (only from a real run) - -**Interfaces:** none — end-to-end validation. - -- [ ] **Step 1: Run all benchmark unit tests across runners** - -```bash -cd /Users/aradunsky/Documents/public/data-weave-cli -node --test benchmarks/lib/manifest.test.mjs benchmarks/report/report.test.mjs benchmarks/runners/node/warm-bench.test.mjs -cd benchmarks/runners/python && python3 -m unittest test_bench -v && cd - -./gradlew :benchmarks-engine:test -``` - -Expected: all green. - -- [ ] **Step 2: Decide on RESULTS.md regeneration** - -The committed `RESULTS.md` streaming rows currently show `n/a`. They must be regenerated from a real benchmark run, not hand-edited. If a full run across all three runners is feasible in this environment: - -```bash -# Produces per-runner result JSONs, then joins into RESULTS.md. -./gradlew native-lib:benchmark # node + python runners (opt-in task) -./gradlew :benchmarks-engine:run --args=" " # engine runner (confirm the actual run task/args in build.gradle) -node benchmarks/report/report.mjs --baseline engine --markdown benchmarks/report/RESULTS.md -``` - -Verify the regenerated `RESULTS.md`: the `map-scale`/`json-stream` streaming rows now show a signed `%` delta (not `n/a`), and the non-comparable footnote is gone. - -If a full run is NOT feasible here, do NOT edit `RESULTS.md` by hand. Instead leave it and note in the PR that RESULTS.md must be regenerated on a machine that can run the full benchmark. (The report *code* is already correct and tested; only the committed sample output lags.) - -- [ ] **Step 3: Commit (only if RESULTS.md was regenerated from a real run)** - -```bash -git add benchmarks/report/RESULTS.md -git commit -m "W-23545283: Regenerate RESULTS.md with aligned streaming deltas" -``` - -- [ ] **Step 4: Push the branch** - -```bash -git push -``` - ---- - -## Self-Review notes - -- **Spec coverage:** Corpus variants (Task 1), manifest field (Task 2), three manifest resolvers (Tasks 3–5), engine streaming rework (Tasks 6–7), native-lib script switch (Tasks 8–9), report un-suppression (Task 10), verification + RESULTS.md (Task 11). All spec sections mapped. -- **Type consistency:** `resolveStreamingScript` (JS/Scala) / `resolve_streaming_script` (Python) return script *text*, matching existing `resolveScript`/`read_script`. `EngineShell.runStreaming(script, name, input: InputStream, inMime, inCharset: Option[String]): Long` is referenced identically in Tasks 6 and 7. `ChunkedInputStream(data, chunkSize)` constructor consistent across Tasks 6–7. `BenchCase.streamingScript: Option[String]` positional field added in Task 5 and consumed via `resolveStreamingScript` (not by direct field access in the runner), so no other call sites break beyond the constructor (Task 5 Step 4 sweeps for those). -- **Placeholders:** none — every code step shows full code; the only conditional is Task 11 Step 2 (real-run gate), which is a deliberate guard against fabricating numbers, not a TODO. -- **Risk carried from spec:** deferred errors surface via logging, not exceptions; the engine `runStreaming` drains and returns a byte count, and `WarmBenchTest` asserts `median > 0` — a fully-empty (silently failed) stream would yield 0 bytes and fail the test. diff --git a/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md b/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md deleted file mode 100644 index 396ce2b..0000000 --- a/docs/superpowers/plans/2026-07-27-cli-benchmark-runner.md +++ /dev/null @@ -1,1108 +0,0 @@ -# CLI Benchmark Runner Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Add a fourth benchmark runner that measures the shipped `dw` native CLI over the shared corpus, emitting `cold-start`, `first-run`, and `warm` metrics into the existing comparison harness. - -**Architecture:** A build-gated benchmark harness inside `native-cli` (compiled into `dw` only under `-Pbenchmark=true`, tree-shaken out of production) prints a `READY` marker after constructing one `NativeRuntime`, then runs timed work — mirroring the Node/Python/engine child protocol. A Node parent under `benchmarks/runners/cli/` spawns that binary per case, stamps cold-start at spawn→READY, and reads back in-process timings, reusing the shared `lib/` modules. - -**Tech Stack:** Java (picocli entrypoint + generated constant), Scala 2.12 (benchmark harness on `NativeRuntime`), GraalVM native-image, Node.js (parent orchestrator, ESM), Gradle, scalatest, `node --test`. - -## Global Constraints - -- **Weave runtime version:** pinned by `weaveVersion` in `gradle.properties` — never hardcode; read it (parent already does via `lib/env.mjs`). -- **Benchmark tasks are opt-in only:** every Gradle benchmark task guards with `onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true }`. Never part of normal `build`/`test`/CI. -- **Production `dw` must not contain benchmark code:** gated by a generated `BenchmarkMode.ENABLED` constant that is `false` unless `-Pbenchmark=true`; native-image folds the unreachable branch away. -- **Result schema is frozen:** output must conform to `benchmarks/schema/result.schema.json` (`schemaVersion: "1.0"`; metrics ∈ `cold-start|first-run|warm|streaming`; units ∈ `ms|MB/s`). Do NOT edit the schema, `report.mjs`, `benchmarkCompare`, or `corpus/manifest.json`. -- **Runner column name:** the result's `runner` field is `"cli"` (the report's column + dedupe key). -- **Runner registration contract:** a runner integrates by (1) writing `benchmarks/results/-.json` and (2) tagging its Gradle task `ext.benchmarkRunner = true`. `benchmarkCompare` auto-discovers it — do NOT edit `benchmarkCompare`. -- **Child stdout discipline:** the only stdout the harness emits is the line `READY` (flushed) followed by exactly one JSON line. Transformation output goes to a discarding stream, never stdout. -- **`dwlibBuildId` env field:** `"n/a-cli"` (the CLI is a binary, not the staged `dwlib`), following the engine runner's `"n/a-engine"` convention. -- **Cross-platform:** Gradle `Exec` tasks branch on `os.name` containing `windows` (`cmd /c` vs `bash -c`), matching existing tasks. Binary name is `dw` (`dw.exe` on Windows). - ---- - -## File Structure - -**Created:** -- `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala` — the corpus-agnostic in-binary harness (arg parse, `coldfirst`/`warm` modes, READY + JSON output). -- `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` — scalatest for the harness + a guard that `BenchmarkMode.ENABLED` is `false` in a normal build. -- `benchmarks/runners/cli/locate.mjs` — resolves the bench-enabled `dw` binary. -- `benchmarks/runners/cli/coldstart.mjs` — spawns `coldfirst` per sample; cold-start + first-run rows. -- `benchmarks/runners/cli/warm.mjs` — spawns `warm` once per warm case; warm rows. -- `benchmarks/runners/cli/emit.mjs` — assembles env + rows, writes `results/cli-.json`. -- `benchmarks/runners/cli/locate.test.mjs` — dwlib/binary-free unit test for `locate.mjs`. -- `benchmarks/runners/cli/emit.test.mjs` — dwlib/binary-free unit test for the result builder. - -**Modified:** -- `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java` — dispatch to the harness before picocli when gated + env set. -- `native-cli/build.gradle` — `genBenchmarkMode` task (generates `BenchmarkMode.java`), wire onto compile, `benchmarkCli` runner task. -- `native-lib/build.gradle` — add `runners/cli/*.test.mjs` to the always-on `benchmarkJsUnitTest` file list. -- `benchmarks/README.md` — document the CLI runner. - ---- - -## Task 1: Generate the `BenchmarkMode.ENABLED` build gate - -**Files:** -- Modify: `native-cli/build.gradle` (add `genBenchmarkMode` task near `genVersions` at line ~51; wire into `compileScala`/`compileJava` deps like `genVersions`) -- Verify against: `native-cli/build.gradle:48-74` (the `genVersions` pattern generates into `build/genresource`, which is already a source dir per `native-cli/build.gradle:7-13`) - -**Interfaces:** -- Produces: a generated Java class `org.mule.weave.cli.BenchmarkMode` with `public static final boolean ENABLED` — `true` only when the Gradle property `benchmark` is truthy, else `false`. Consumed by Task 2 (`DWCLI`) and Task 3's guard test. - -Rationale: a **Java** constant (not Scala) so `DWCLI.java` reads it with no cross-language friction; `build/genresource` is already on the Scala srcDir but `javac` also compiles generated Java there via the existing `compileJava` classpath wiring (`native-cli/build.gradle:153-154`). Generate into a Java-compiled location: use a dedicated `build/genjava` dir added to the java sourceSet to keep it unambiguous. - -- [ ] **Step 1: Add a generated-Java source dir to the java sourceSet** - -In `native-cli/build.gradle`, extend the `sourceSets` block (currently lines 7-13) to add a java srcDir: - -```groovy -sourceSets { - main { - scala { - srcDirs = ['src/main/scala', 'build/genresource'] - } - java { - srcDirs += 'build/genjava' - } - } -} -``` - -- [ ] **Step 2: Add the `genBenchmarkMode` task** - -Immediately after the `genVersions` task (after line 67 in `native-cli/build.gradle`), add: - -```groovy -def genJavaDirectory = new File("$project.buildDir/genjava") - -task genBenchmarkMode() { - def enabled = project.findProperty('benchmark')?.toString()?.toBoolean() == true - def benchmarkMode = new File(genJavaDirectory, "org/mule/weave/cli/BenchmarkMode.java") - def parentFile = benchmarkMode.getParentFile() - if (!parentFile.exists()) { - parentFile.mkdirs() - } - final PrintWriter outputPrinter = new PrintWriter(new FileWriter(benchmarkMode)) - outputPrinter.println("package org.mule.weave.cli;") - outputPrinter.println() - outputPrinter.println("// GENERATED by genBenchmarkMode — do not edit.") - outputPrinter.println("// ENABLED is true only when built with -Pbenchmark=true; native-image") - outputPrinter.println("// folds the benchmark branch away as dead code when this is false.") - outputPrinter.println("public final class BenchmarkMode {") - outputPrinter.println(" private BenchmarkMode() {}") - outputPrinter.println(" public static final boolean ENABLED = " + enabled + ";") - outputPrinter.println("}") - outputPrinter.close() -} -``` - -- [ ] **Step 3: Wire it into compilation** - -Update the existing `compileScala` block (lines 72-74) and add a `compileJava` dependency so the constant exists before either compiles: - -```groovy -defaultTasks += genVersions - -compileScala { - dependsOn genVersions - dependsOn genBenchmarkMode -} - -compileJava { - dependsOn genBenchmarkMode -} -``` - -- [ ] **Step 4: Verify normal build generates `ENABLED = false`** - -Run: `./gradlew native-cli:genBenchmarkMode && cat native-cli/build/genjava/org/mule/weave/cli/BenchmarkMode.java` -Expected: file contains `public static final boolean ENABLED = false;` - -- [ ] **Step 5: Verify benchmark build generates `ENABLED = true`** - -Run: `./gradlew native-cli:genBenchmarkMode -Pbenchmark=true && cat native-cli/build/genjava/org/mule/weave/cli/BenchmarkMode.java` -Expected: file contains `public static final boolean ENABLED = true;` - -- [ ] **Step 6: Commit** - -```bash -git add native-cli/build.gradle -git commit -m "build: generate BenchmarkMode.ENABLED gate for native-cli" -``` - ---- - -## Task 2: `BenchmarkHarness` — the in-binary corpus-agnostic harness - -**Files:** -- Create: `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala` -- Create: `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` - -**Interfaces:** -- Consumes: `org.mule.weave.dwnative.NativeRuntime` (constructor `new NativeRuntime(libDir: File, path: Array[File], console: Console, maybeLanguageLevel: Option[DataWeaveVersion])`; method `run(script: String, nameIdentifier: String, inputs: ScriptingBindings, out: OutputStream, defaultOutputMimeType: String, maybePrivileges: Option[Seq[String]]): WeaveExecutionResult` where `WeaveExecutionResult.success(): Boolean` and `.result(): String`); `org.mule.weave.dwnative.utils.DataWeaveUtils#getLibPathHome(): File`; `org.mule.weave.dwnative.cli.DefaultConsole`; `org.mule.weave.v2.runtime.ScriptingBindings#addBinding(name, value: BindingValue)`; `org.mule.weave.v2.runtime.BindingValue(bytes: Array[Byte], mimeType: Option[String], props: Map[String,Any], charset: Charset)`. -- Produces: `object BenchmarkHarness { def main(args: Array[String]): Unit }` and (for tests) `def parseArgs(args: Array[String]): BenchArgs`, `case class BenchArgs(mode: String, scriptFile: String, inputs: Seq[BenchInput], warmup: Int, iters: Int)`, `case class BenchInput(name: String, file: String, mimeType: String, charset: String)`, and `def runColdFirst(args, out: java.io.PrintStream, sink: OutputStream): Unit` / `def runWarm(args, out: java.io.PrintStream, sink: OutputStream): Unit` (out = where READY/JSON go; sink = discard stream for transform output). `main` calls these with `System.out` and a fresh `CountingOutputStream`. - -Reuse a discarding stream identical in behavior to the engine runner's `CountingOutputStream`; define a small private one here rather than depend on the `benchmarks-engine` module (no such dependency exists from `native-cli`). - -Arg format from the parent (one `--input` per binding): -``` ---bench-mode=coldfirst|warm ---script= ---input==:: ---warmup= (warm mode only; default 0) ---iters= (warm mode only; default 100) -``` - -- [ ] **Step 1: Write the failing test for arg parsing** - -Create `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala`: - -```scala -package org.mule.weave.dwnative.benchmark - -import org.scalatest.freespec.AnyFreeSpec -import org.scalatest.matchers.should.Matchers - -class BenchmarkHarnessTest extends AnyFreeSpec with Matchers { - - "parseArgs" - { - "parses coldfirst mode with one input" in { - val a = BenchmarkHarness.parseArgs(Array( - "--bench-mode=coldfirst", - "--script=/tmp/x.dwl", - "--input=payload=/tmp/p.json:application/json:utf-8")) - a.mode shouldBe "coldfirst" - a.scriptFile shouldBe "/tmp/x.dwl" - a.inputs should have size 1 - a.inputs.head shouldBe BenchInput("payload", "/tmp/p.json", "application/json", "utf-8") - } - - "parses warm mode with warmup and iters" in { - val a = BenchmarkHarness.parseArgs(Array( - "--bench-mode=warm", "--script=/tmp/x.dwl", "--warmup=5", "--iters=30")) - a.mode shouldBe "warm" - a.warmup shouldBe 5 - a.iters shouldBe 30 - a.inputs shouldBe empty - } - - "handles a mimeType-only input (charset defaults to utf-8)" in { - val a = BenchmarkHarness.parseArgs(Array( - "--bench-mode=coldfirst", "--script=/tmp/x.dwl", - "--input=payload=/tmp/p.json:application/json")) - a.inputs.head.charset shouldBe "utf-8" - } - } -} -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` -Expected: FAIL — `BenchmarkHarness` / `BenchInput` not found (compilation error). - -- [ ] **Step 3: Implement `BenchmarkHarness` with parsing + modes** - -Create `native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala`: - -```scala -package org.mule.weave.dwnative.benchmark - -import org.mule.weave.dwnative.NativeRuntime -import org.mule.weave.dwnative.WeaveExecutionResult -import org.mule.weave.dwnative.cli.DefaultConsole -import org.mule.weave.dwnative.utils.DataWeaveUtils -import org.mule.weave.v2.runtime.BindingValue -import org.mule.weave.v2.runtime.ScriptingBindings - -import java.io.{ File, OutputStream, PrintStream } -import java.nio.charset.Charset -import java.nio.file.Files - -final case class BenchInput(name: String, file: String, mimeType: String, charset: String) -final case class BenchArgs(mode: String, scriptFile: String, inputs: Seq[BenchInput], warmup: Int, iters: Int) - -/** Corpus-agnostic in-binary benchmark harness. Reachable only in a build made with - * -Pbenchmark=true (guarded by BenchmarkMode.ENABLED in DWCLI); native-image folds it - * out of a production dw. Prints "READY" the instant one NativeRuntime is constructed, - * then a single JSON line of timings. Parent (benchmarks/runners/cli) measures cold-start - * as spawn->READY wall-clock. */ -object BenchmarkHarness { - - /** Discards bytes; used as the transform write sink so we never touch real stdout. */ - private final class DiscardStream extends OutputStream { - override def write(b: Int): Unit = () - override def write(b: Array[Byte]): Unit = () - override def write(b: Array[Byte], off: Int, len: Int): Unit = () - } - - private def nowNs(): Long = System.nanoTime() - private def msSince(startNs: Long): Double = (System.nanoTime() - startNs) / 1e6 - - def parseArgs(args: Array[String]): BenchArgs = { - var mode = "" - var script = "" - val inputs = scala.collection.mutable.ArrayBuffer[BenchInput]() - var warmup = 0 - var iters = 100 - args.foreach { arg => - val eq = arg.indexOf('=') - val key = if (eq >= 0) arg.substring(0, eq) else arg - val value = if (eq >= 0) arg.substring(eq + 1) else "" - key match { - case "--bench-mode" => mode = value - case "--script" => script = value - case "--warmup" => warmup = value.toInt - case "--iters" => iters = value.toInt - case "--input" => - // value = =:[:] - val nameSep = value.indexOf('=') - val name = value.substring(0, nameSep) - val rest = value.substring(nameSep + 1) - val parts = rest.split(":", 3) - val file = parts(0) - val mimeType = parts(1) - val charset = if (parts.length > 2 && parts(2).nonEmpty) parts(2) else "utf-8" - inputs += BenchInput(name, file, mimeType, charset) - case _ => throw new RuntimeException(s"unknown bench arg: $arg") - } - } - if (mode.isEmpty) throw new RuntimeException("--bench-mode is required") - if (script.isEmpty) throw new RuntimeException("--script is required") - BenchArgs(mode, script, inputs.toSeq, warmup, iters) - } - - private def newRuntime(): NativeRuntime = { - val console = DefaultConsole.enableSilent() - val utils = new DataWeaveUtils(console) - new NativeRuntime(utils.getLibPathHome(), Array.empty[File], console, None) - } - - private def readScript(a: BenchArgs): String = - new String(Files.readAllBytes(new File(a.scriptFile).toPath), java.nio.charset.StandardCharsets.UTF_8) - - private def bindings(a: BenchArgs): ScriptingBindings = { - val b = new ScriptingBindings() - a.inputs.foreach { in => - val bytes = Files.readAllBytes(new File(in.file).toPath) - val bv = new BindingValue(bytes, Some(in.mimeType), Map.empty[String, Any], Charset.forName(in.charset)) - b.addBinding(in.name, bv) - } - b - } - - private def assertOk(r: WeaveExecutionResult): Unit = - if (!r.success()) throw new RuntimeException("run failed: " + r.result()) - - def runColdFirst(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { - val script = readScript(a) - val b = bindings(a) - val rt = newRuntime() // engine init — measured externally as cold-start - out.println("READY"); out.flush() - val start = nowNs() - assertOk(rt.run(script, "bench", b, sink, "application/json", None)) - val firstRunMs = msSince(start) - out.println("{\"firstRunMs\":" + firstRunMs + "}") - } - - def runWarm(a: BenchArgs, out: PrintStream, sink: OutputStream): Unit = { - val script = readScript(a) - val b = bindings(a) - val rt = newRuntime() - out.println("READY"); out.flush() - var i = 0 - while (i < a.warmup) { assertOk(rt.run(script, "bench", b, sink, "application/json", None)); i += 1 } - val samples = new Array[Double](a.iters) - i = 0 - while (i < a.iters) { - val start = nowNs() - assertOk(rt.run(script, "bench", b, sink, "application/json", None)) - samples(i) = msSince(start) - i += 1 - } - out.println("{\"warmMs\":[" + samples.mkString(",") + "]}") - } - - def main(args: Array[String]): Unit = { - val a = parseArgs(args) - val sink = new DiscardStream() - a.mode match { - case "coldfirst" => runColdFirst(a, System.out, sink) - case "warm" => runWarm(a, System.out, sink) - case other => throw new RuntimeException(s"unknown --bench-mode: $other") - } - } -} -``` - -- [ ] **Step 4: Run the parsing test to verify it passes** - -Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` -Expected: PASS (3 parsing tests). - -- [ ] **Step 5: Add behavioral tests (READY + JSON discipline, warm array, failure)** - -Append to `BenchmarkHarnessTest.scala` inside the class, using a temp script/input and capturing an in-memory `PrintStream`: - -```scala - import java.io.{ ByteArrayOutputStream, File, PrintStream } - import java.nio.charset.StandardCharsets - import java.nio.file.Files - - private def tmp(suffix: String, content: String): File = { - val f = File.createTempFile("bench", suffix) - f.deleteOnExit() - Files.write(f.toPath, content.getBytes(StandardCharsets.UTF_8)) - f - } - - private def capture(fn: PrintStream => Unit): String = { - val buf = new ByteArrayOutputStream() - val ps = new PrintStream(buf, true, "UTF-8") - fn(ps) - new String(buf.toByteArray, StandardCharsets.UTF_8) - } - - "runColdFirst" - { - "emits READY then a single firstRunMs JSON line, output not on the stream" in { - val script = tmp(".dwl", "output application/json --- payload.a + 1") - val input = tmp(".json", "{\"a\": 41}") - val a = BenchArgs("coldfirst", script.getAbsolutePath, - Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) - val sink = new ByteArrayOutputStream() - val stdout = capture(ps => BenchmarkHarness.runColdFirst(a, ps, sink)) - val lines = stdout.split("\n").filter(_.nonEmpty) - lines.head shouldBe "READY" - lines.last should include ("firstRunMs") - lines.count(_.contains("firstRunMs")) shouldBe 1 - // The transformed "42" went to the sink, NOT to stdout. - new String(sink.toByteArray, StandardCharsets.UTF_8).trim shouldBe "42" - } - } - - "runWarm" - { - "emits READY then a warmMs array of length iters" in { - val script = tmp(".dwl", "output application/json --- payload.a + 1") - val input = tmp(".json", "{\"a\": 41}") - val a = BenchArgs("warm", script.getAbsolutePath, - Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 1, 3) - val stdout = capture(ps => BenchmarkHarness.runWarm(a, ps, new ByteArrayOutputStream())) - val json = stdout.split("\n").filter(_.contains("warmMs")).head - json should include ("warmMs") - // 3 comma-separated samples -> 2 commas inside the array - json.count(_ == ',') shouldBe 2 - } - } - - "a failing script throws (non-zero exit path)" in { - val script = tmp(".dwl", "output application/json --- payload.missing.deep.path()") - val input = tmp(".json", "{}") - val a = BenchArgs("coldfirst", script.getAbsolutePath, - Seq(BenchInput("payload", input.getAbsolutePath, "application/json", "utf-8")), 0, 0) - an [RuntimeException] should be thrownBy - BenchmarkHarness.runColdFirst(a, capturePs(), new ByteArrayOutputStream()) - } - - private def capturePs(): PrintStream = new PrintStream(new ByteArrayOutputStream(), true, "UTF-8") -``` - -- [ ] **Step 6: Run all harness tests to verify they pass** - -Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` -Expected: PASS (parsing + coldfirst + warm + failure). - -Note: if the `.dwl` script for the failure case does not actually throw, replace its body with one that reliably fails, e.g. `output application/json --- 1 / 0` — the intent is only that a failed run raises. - -- [ ] **Step 7: Commit** - -```bash -git add native-cli/src/main/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarness.scala \ - native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala -git commit -m "feat: add in-binary BenchmarkHarness for native-cli" -``` - ---- - -## Task 3: Gate the harness behind `BenchmarkMode.ENABLED` in `DWCLI` - -**Files:** -- Modify: `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java:32-34` (the `main` method) -- Modify: `native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala` (add the `ENABLED == false` guard) - -**Interfaces:** -- Consumes: `org.mule.weave.cli.BenchmarkMode.ENABLED` (Task 1), `org.mule.weave.dwnative.benchmark.BenchmarkHarness.main` (Task 2). -- Produces: no new public API; behavior — when `BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null`, `dw` dispatches to `BenchmarkHarness.main(args)` before picocli. Otherwise unchanged. - -The env-var name is `DW_BENCH` (specific, collision-unlikely). `ENABLED` is the real gate: in production it is a compile-time `false`, so `BenchmarkHarness` is unreachable and native-image drops it. - -- [ ] **Step 1: Write the guard test that production has ENABLED=false** - -Append to `BenchmarkHarnessTest.scala`: - -```scala - "BenchmarkMode.ENABLED" - { - "is false in a normal (non -Pbenchmark) build" in { - // Tests run without -Pbenchmark, so the generated constant must be false — - // proving the harness is dead code / stripped from a production image. - org.mule.weave.cli.BenchmarkMode.ENABLED shouldBe false - } - } -``` - -- [ ] **Step 2: Run to verify it fails to compile (constant not generated for test yet)** - -Run: `./gradlew native-cli:test --tests "org.mule.weave.dwnative.benchmark.BenchmarkHarnessTest"` -Expected: FAIL — `BenchmarkMode` symbol not found *unless* `genBenchmarkMode` ran. If it fails on the symbol, run `./gradlew native-cli:genBenchmarkMode` once, then re-run. Expected after generation: PASS for this guard (normal build → `false`). - -Note: `compileScala`/`compileJava` already `dependsOn genBenchmarkMode` (Task 1 Step 3), so the test compile generates it. If the IDE/test invocation skips it, the explicit `genBenchmarkMode` run resolves it. - -- [ ] **Step 3: Modify `DWCLI.main` to dispatch when gated** - -In `native-cli/src/main/java/org/mule/weave/cli/DWCLI.java`, replace the `main` method (lines 32-34): - -```java - public static void main(String[] args) { - // Benchmark dispatch: only reachable in a build made with -Pbenchmark=true - // (BenchmarkMode.ENABLED is a compile-time false in production, so native-image - // folds this branch and BenchmarkHarness away). DW_BENCH selects the mode. - if (BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null) { - org.mule.weave.dwnative.benchmark.BenchmarkHarness.main(args); - return; - } - new DWCLI().run(args, DefaultConsole$.MODULE$); - } -``` - -- [ ] **Step 4: Run the full native-cli test suite to verify nothing regressed** - -Run: `./gradlew native-cli:test` -Expected: PASS, including the `ENABLED shouldBe false` guard. - -- [ ] **Step 5: Commit** - -```bash -git add native-cli/src/main/java/org/mule/weave/cli/DWCLI.java \ - native-cli/src/test/scala/org/mule/weave/dwnative/benchmark/BenchmarkHarnessTest.scala -git commit -m "feat: gate BenchmarkHarness dispatch behind BenchmarkMode.ENABLED + DW_BENCH" -``` - ---- - -## Task 4: Parent `locate.mjs` — resolve the bench-enabled `dw` binary - -**Files:** -- Create: `benchmarks/runners/cli/locate.mjs` -- Create: `benchmarks/runners/cli/locate.test.mjs` - -**Interfaces:** -- Produces: `export function locateBinary(): string` — returns an absolute path to the `dw` binary. Resolution order: `process.env.DW_BENCH_BIN` if set (used as-is), else `/native-cli/build/native/nativeCompile/dw` (`dw.exe` on Windows). Throws with a build hint if the resolved path does not exist. Consumed by Tasks 5 & 6. - -Mirror `benchmarks/runners/node/wrapper.mjs` (repo-root computation via `import.meta.url`, `existsSync` check, actionable error). - -- [ ] **Step 1: Write the failing test** - -Create `benchmarks/runners/cli/locate.test.mjs`: - -```javascript -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { locateBinary } from "./locate.mjs"; - -test("DW_BENCH_BIN override is returned as-is when it exists", () => { - // Point at a file guaranteed to exist: this test file itself. - const self = new URL(import.meta.url).pathname; - process.env.DW_BENCH_BIN = self; - try { - assert.equal(locateBinary(), self); - } finally { - delete process.env.DW_BENCH_BIN; - } -}); - -test("throws an actionable error when the binary is absent", () => { - process.env.DW_BENCH_BIN = "/nonexistent/dw-binary-xyz"; - try { - assert.throws(() => locateBinary(), /nativeCompile|not found|build/i); - } finally { - delete process.env.DW_BENCH_BIN; - } -}); -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `node --test benchmarks/runners/cli/locate.test.mjs` -Expected: FAIL — cannot find module `./locate.mjs`. - -- [ ] **Step 3: Implement `locate.mjs`** - -Create `benchmarks/runners/cli/locate.mjs`: - -```javascript -import { existsSync } from "node:fs"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -// benchmarks/runners/cli -> benchmarks/runners -> benchmarks -> repo root -const REPO_ROOT = join(__dirname, "..", "..", ".."); -const BIN_NAME = process.platform === "win32" ? "dw.exe" : "dw"; -const DEFAULT_BIN = join(REPO_ROOT, "native-cli", "build", "native", "nativeCompile", BIN_NAME); - -/** - * Resolve the benchmark-enabled `dw` native binary. Honors DW_BENCH_BIN (absolute - * path to a bench-built dw); otherwise the default nativeCompile output. The binary - * must be built with -Pbenchmark=true so BenchmarkHarness is reachable. - */ -export function locateBinary() { - const candidate = process.env.DW_BENCH_BIN || DEFAULT_BIN; - if (!existsSync(candidate)) { - throw new Error( - `dw benchmark binary not found at ${candidate}. ` + - `Build it with: ./gradlew native-cli:nativeCompile -Pbenchmark=true ` + - `(or set DW_BENCH_BIN to a bench-enabled dw).` - ); - } - return candidate; -} -``` - -- [ ] **Step 4: Run to verify it passes** - -Run: `node --test benchmarks/runners/cli/locate.test.mjs` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/cli/locate.mjs benchmarks/runners/cli/locate.test.mjs -git commit -m "feat: add cli runner binary locator" -``` - ---- - -## Task 5: Parent `coldstart.mjs` — cold-start + first-run rows - -**Files:** -- Create: `benchmarks/runners/cli/coldstart.mjs` - -**Interfaces:** -- Consumes: `locateBinary` (Task 4); shared libs `casesForMetric`, `resolveInputs` from `../../lib/manifest.mjs`; `computeStats` from `../../lib/stats.mjs`. -- Produces: `export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}): Promise>` — for each case declaring `cold-start` or `first-run`, spawns `dw` in `coldfirst` mode `n` times, stamps cold-start at spawn→`READY`, parses `firstRunMs`. Emits a `cold-start` row (unit `ms`) only for cases that declare it, and a `first-run` row only for cases that declare it. Consumed by Task 7. - -This is `benchmarks/runners/node/coldstart.mjs` with the child command swapped to the `dw` binary. Build the per-input arg `--input==::` using absolute corpus file paths. - -- [ ] **Step 1: Implement `coldstart.mjs`** - -Create `benchmarks/runners/cli/coldstart.mjs`: - -```javascript -import { spawn } from "node:child_process"; -import { join } from "node:path"; -import { casesForMetric } from "../../lib/manifest.mjs"; -import { computeStats } from "../../lib/stats.mjs"; -import { locateBinary } from "./locate.mjs"; - -/** Build `--input=name=file:mime:charset` args for a case (absolute paths). */ -function inputArgs(manifest, c) { - const args = []; - for (const [name, inp] of Object.entries(c.inputs ?? {})) { - const file = join(manifest.corpusDir, inp.file); - const charset = inp.charset ?? "utf-8"; - args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); - } - return args; -} - -/** - * Spawn one fresh dw process in coldfirst mode. Cold-start = wall-clock from just - * before spawn to the child's "READY" marker (process launch + native image load + - * NativeRuntime init). first-run is timed in-process by the child. Rejects on a - * non-zero exit or a missing READY/JSON line so a failed sample never records a - * bogus timing. - */ -function sampleOnce(bin, manifest, c) { - const scriptPath = join(manifest.corpusDir, c.script); - const args = ["--bench-mode=coldfirst", `--script=${scriptPath}`, ...inputArgs(manifest, c)]; - return new Promise((resolve, reject) => { - const t0 = process.hrtime.bigint(); - const child = spawn(bin, args, { - stdio: ["ignore", "pipe", "pipe"], - env: { ...process.env, DW_BENCH: "1" }, - }); - let coldStartMs; - let stdout = ""; - let stderr = ""; - child.stdout.setEncoding("utf-8"); - child.stdout.on("data", (chunk) => { - stdout += chunk; - if (coldStartMs === undefined && stdout.includes("READY\n")) { - coldStartMs = Number(process.hrtime.bigint() - t0) / 1e6; - } - }); - child.stderr.setEncoding("utf-8"); - child.stderr.on("data", (chunk) => (stderr += chunk)); - child.on("error", reject); - child.on("close", (code) => { - if (code !== 0) { - reject(new Error(`cli coldfirst failed for '${c.id}' (exit ${code})\n${stderr}`)); - return; - } - if (coldStartMs === undefined) { - reject(new Error(`cli coldfirst for '${c.id}' never printed READY\n${stderr}`)); - return; - } - const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); - if (!jsonLine) { - reject(new Error(`cli coldfirst for '${c.id}' printed no result line\n${stderr}`)); - return; - } - const { firstRunMs } = JSON.parse(jsonLine); - resolve({ coldStartMs, firstRunMs }); - }); - }); -} - -/** @returns {Promise>} */ -export async function runColdStartAndFirstRun(manifest, { samplesOverride } = {}) { - const bin = locateBinary(); - const rows = []; - const ids = new Set([ - ...casesForMetric(manifest, "cold-start").map((c) => c.id), - ...casesForMetric(manifest, "first-run").map((c) => c.id), - ]); - - for (const id of ids) { - const c = manifest.cases.find((x) => x.id === id); - const n = samplesOverride ?? c.iterations?.samples ?? 20; - const colds = []; - const firsts = []; - for (let i = 0; i < n; i++) { - const { coldStartMs, firstRunMs } = await sampleOnce(bin, manifest, c); - colds.push(coldStartMs); - firsts.push(firstRunMs); - } - if (c.metrics.includes("cold-start")) { - rows.push({ id, metric: "cold-start", unit: "ms", stats: computeStats(colds), iterations: n }); - } - if (c.metrics.includes("first-run")) { - rows.push({ id, metric: "first-run", unit: "ms", stats: computeStats(firsts), iterations: n }); - } - } - return rows; -} -``` - -- [ ] **Step 2: Sanity-check syntax (no binary needed)** - -Run: `node --check benchmarks/runners/cli/coldstart.mjs` -Expected: no output, exit 0. - -Note: an end-to-end run of this file requires a bench-built `dw` and is exercised by the smoke test in Task 8; there is no dwlib-free unit test for it (it spawns the binary), matching how `runners/node/coldstart.test.mjs` is excluded from the always-on JS parity set. - -- [ ] **Step 3: Commit** - -```bash -git add benchmarks/runners/cli/coldstart.mjs -git commit -m "feat: add cli runner cold-start + first-run sampler" -``` - ---- - -## Task 6: Parent `warm.mjs` — warm rows - -**Files:** -- Create: `benchmarks/runners/cli/warm.mjs` - -**Interfaces:** -- Consumes: `locateBinary` (Task 4); `casesForMetric` from `../../lib/manifest.mjs`; `computeStats` from `../../lib/stats.mjs`. -- Produces: `export async function runWarm(manifest): Promise>` — for each case declaring `warm`, spawns `dw` once in `warm` mode with `--warmup`/`--iters` from the case's `iterations`, reads back the `warmMs[]` array, and produces a `warm` row (unit `ms`). Consumed by Task 7. - -Reuse the same `inputArgs` shape as Task 5 (duplicated as a small local helper — the two samplers are independent and each is small; a shared module is not warranted by YAGNI, matching how the Node runner keeps `coldstart.mjs` and `warm-bench.mjs` separate). - -- [ ] **Step 1: Implement `warm.mjs`** - -Create `benchmarks/runners/cli/warm.mjs`: - -```javascript -import { spawn } from "node:child_process"; -import { join } from "node:path"; -import { casesForMetric } from "../../lib/manifest.mjs"; -import { computeStats } from "../../lib/stats.mjs"; -import { locateBinary } from "./locate.mjs"; - -function inputArgs(manifest, c) { - const args = []; - for (const [name, inp] of Object.entries(c.inputs ?? {})) { - const file = join(manifest.corpusDir, inp.file); - const charset = inp.charset ?? "utf-8"; - args.push(`--input=${name}=${file}:${inp.mimeType}:${charset}`); - } - return args; -} - -/** Spawn dw once in warm mode; resolve the parsed warmMs[] sample array. */ -function warmSamples(bin, manifest, c) { - const scriptPath = join(manifest.corpusDir, c.script); - const warmup = c.iterations?.warmup ?? 10; - const iters = c.iterations?.warm ?? 100; - const args = [ - "--bench-mode=warm", - `--script=${scriptPath}`, - `--warmup=${warmup}`, - `--iters=${iters}`, - ...inputArgs(manifest, c), - ]; - return new Promise((resolve, reject) => { - const child = spawn(bin, args, { - stdio: ["ignore", "pipe", "pipe"], - env: { ...process.env, DW_BENCH: "1" }, - }); - let stdout = ""; - let stderr = ""; - child.stdout.setEncoding("utf-8"); - child.stdout.on("data", (chunk) => (stdout += chunk)); - child.stderr.setEncoding("utf-8"); - child.stderr.on("data", (chunk) => (stderr += chunk)); - child.on("error", reject); - child.on("close", (code) => { - if (code !== 0) { - reject(new Error(`cli warm failed for '${c.id}' (exit ${code})\n${stderr}`)); - return; - } - const jsonLine = stdout.split("\n").filter((l) => l && l !== "READY").pop(); - if (!jsonLine) { - reject(new Error(`cli warm for '${c.id}' printed no result line\n${stderr}`)); - return; - } - const { warmMs } = JSON.parse(jsonLine); - if (!Array.isArray(warmMs) || warmMs.length === 0) { - reject(new Error(`cli warm for '${c.id}' returned no samples\n${stderr}`)); - return; - } - resolve({ warmMs, iters }); - }); - }); -} - -/** @returns {Promise>} */ -export async function runWarm(manifest) { - const bin = locateBinary(); - const rows = []; - for (const c of casesForMetric(manifest, "warm")) { - const { warmMs, iters } = await warmSamples(bin, manifest, c); - rows.push({ id: c.id, metric: "warm", unit: "ms", stats: computeStats(warmMs), iterations: iters }); - } - return rows; -} -``` - -- [ ] **Step 2: Sanity-check syntax** - -Run: `node --check benchmarks/runners/cli/warm.mjs` -Expected: no output, exit 0. - -- [ ] **Step 3: Commit** - -```bash -git add benchmarks/runners/cli/warm.mjs -git commit -m "feat: add cli runner warm sampler" -``` - ---- - -## Task 7: Parent `emit.mjs` — assemble and write the result file - -**Files:** -- Create: `benchmarks/runners/cli/emit.mjs` -- Create: `benchmarks/runners/cli/emit.test.mjs` - -**Interfaces:** -- Consumes: `loadManifest`, `validateResultIds` from `../../lib/manifest.mjs`; `gatherEnv` from `../../lib/env.mjs`; `runColdStartAndFirstRun` (Task 5); `runWarm` (Task 6); `locateBinary` (Task 4, for the version probe). -- Produces: `export function buildResult(env, cases)` (schema-shaped object, identical contract to the Node runner's) and `export async function main(): Promise` (writes `results/cli-.json`, returns its path). Runner name `"cli"`. - -`runtimeVersion`: probe `dw --version` synchronously; take the first line, or fall back to `"dw"` if the probe fails. `dwlibBuildId` comes from `gatherEnv` but the CLI is not the staged dwlib — override it to `"n/a-cli"` after gathering. - -- [ ] **Step 1: Write the failing test for `buildResult`** - -Create `benchmarks/runners/cli/emit.test.mjs`: - -```javascript -import { test } from "node:test"; -import assert from "node:assert/strict"; -import { join, dirname } from "node:path"; -import { fileURLToPath } from "node:url"; -import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; -import { buildResult } from "./emit.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); - -test("buildResult produces a schema-shaped object with runner 'cli'", () => { - const env = { - runner: "cli", os: "x", cpu: "y", runtimeVersion: "dw vX", - weaveVersion: "2.12.0-x", commit: "abc", dwlibBuildId: "n/a-cli", - }; - const cases = [{ id: "trivial", metric: "cold-start", unit: "ms", stats: { median: 1 }, iterations: 10 }]; - const r = buildResult(env, cases); - assert.equal(r.schemaVersion, "1.0"); - assert.equal(r.runner, "cli"); - assert.ok(typeof r.timestamp === "string"); - assert.deepEqual(r.cases, cases); -}); - -test("orphan ids are rejected before writing", () => { - const manifest = loadManifest(CORPUS); - assert.throws(() => validateResultIds(manifest, [{ id: "totally-made-up" }]), /orphan id/); -}); -``` - -- [ ] **Step 2: Run to verify it fails** - -Run: `node --test benchmarks/runners/cli/emit.test.mjs` -Expected: FAIL — cannot find module `./emit.mjs`. - -- [ ] **Step 3: Implement `emit.mjs`** - -Create `benchmarks/runners/cli/emit.mjs`: - -```javascript -import { writeFileSync, mkdirSync } from "node:fs"; -import { execFileSync } from "node:child_process"; -import { join, dirname } from "node:path"; -import { fileURLToPath, pathToFileURL } from "node:url"; -import { loadManifest, validateResultIds } from "../../lib/manifest.mjs"; -import { gatherEnv } from "../../lib/env.mjs"; -import { locateBinary } from "./locate.mjs"; -import { runColdStartAndFirstRun } from "./coldstart.mjs"; -import { runWarm } from "./warm.mjs"; - -const __dirname = dirname(fileURLToPath(import.meta.url)); -const CORPUS = join(__dirname, "..", "..", "corpus"); -const RESULTS_DIR = join(__dirname, "..", "..", "results"); - -/** Assemble the full schema object (identical contract to the Node runner). */ -export function buildResult(env, cases) { - return { - schemaVersion: "1.0", - runner: env.runner, - env, - timestamp: new Date().toISOString(), - cases, - }; -} - -/** Best-effort `dw --version` first line; falls back to "dw". */ -function probeVersion(bin) { - try { - const out = execFileSync(bin, ["--version"], { encoding: "utf-8" }); - const line = out.split("\n").map((l) => l.trim()).filter(Boolean)[0]; - return line ? `dw ${line}` : "dw"; - } catch { - return "dw"; - } -} - -export async function main() { - const manifest = loadManifest(CORPUS); - const bin = locateBinary(); - const env = gatherEnv({ runner: "cli", runtimeVersion: probeVersion(bin) }); - // The CLI is a native binary, not the staged dwlib — override the lib fingerprint. - env.dwlibBuildId = "n/a-cli"; - - const coldRows = await runColdStartAndFirstRun(manifest); - const warmRows = await runWarm(manifest); - - const cases = [...coldRows, ...warmRows]; - validateResultIds(manifest, cases); - - mkdirSync(RESULTS_DIR, { recursive: true }); - const stamp = new Date().toISOString().replace(/[:.]/g, "-"); - const outPath = join(RESULTS_DIR, `cli-${stamp}.json`); - writeFileSync(outPath, JSON.stringify(buildResult(env, cases), null, 2)); - console.log(`wrote ${outPath} (${cases.length} rows)`); - return outPath; -} - -if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) { - main().catch((e) => { - console.error(e.message); - process.exit(1); - }); -} -``` - -- [ ] **Step 4: Run to verify the test passes** - -Run: `node --test benchmarks/runners/cli/emit.test.mjs` -Expected: PASS (2 tests). - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/runners/cli/emit.mjs benchmarks/runners/cli/emit.test.mjs -git commit -m "feat: add cli runner emit entrypoint" -``` - ---- - -## Task 8: Gradle `benchmarkCli` runner task + JS test wiring - -**Files:** -- Modify: `native-cli/build.gradle` (add `benchmarkCli` task) -- Modify: `native-lib/build.gradle:307-320` (add cli test files to `benchmarkJsUnitTest`) - -**Interfaces:** -- Consumes: `native-cli:nativeCompile` (must be invoked with `-Pbenchmark=true` so the harness is present); the shared `node corpus/gen-inputs.mjs`; `benchmarks/runners/cli/emit.mjs` (Task 7). -- Produces: a Gradle task `benchmarkCli` tagged `ext.benchmarkRunner = true`, discovered by the root `benchmarkCompare`. Writes `benchmarks/results/cli-.json`; does NOT render the report. - -- [ ] **Step 1: Add the `benchmarkCli` task to `native-cli/build.gradle`** - -Append to `native-cli/build.gradle`: - -```groovy -// The CLI runner as an aggregator-registered runner: emits its result file but -// does NOT render the report (the root :benchmarkCompare renders once over all -// runners). Tagged `benchmarkRunner` so :benchmarkCompare discovers it automatically. -// Requires the bench-enabled binary — nativeCompile must run with -Pbenchmark=true so -// BenchmarkMode.ENABLED is true and BenchmarkHarness is reachable in dw. -tasks.register('benchmarkCli', Exec) { - onlyIf { project.findProperty('benchmark')?.toString()?.toBoolean() == true } - ext.benchmarkRunner = true - - dependsOn tasks.named('nativeCompile') - workingDir("${rootDir}/benchmarks") - - def script = 'node corpus/gen-inputs.mjs && node runners/cli/emit.mjs' - if (System.getProperty('os.name').toLowerCase().contains('windows')) { - commandLine('cmd', '/c', script) - } else { - commandLine('bash', '-c', script) - } -} -``` - -- [ ] **Step 2: Add the cli JS tests to the always-on parity set** - -In `native-lib/build.gradle`, in `benchmarkJsUnitTest` (lines 307-320), extend the `files` string to include the cli runner's dwlib-free tests: - -```groovy - def files = 'lib/stats.test.mjs lib/manifest.test.mjs lib/env.test.mjs ' + - 'report/report.test.mjs runners/node/emit.test.mjs ' + - 'runners/cli/locate.test.mjs runners/cli/emit.test.mjs' -``` - -- [ ] **Step 3: Verify the JS parity tests pass (no binary needed)** - -Run: `./gradlew native-lib:benchmarkJsUnitTest` -Expected: PASS — includes `runners/cli/locate.test.mjs` and `runners/cli/emit.test.mjs`. - -- [ ] **Step 4: Verify `benchmarkCli` is discovered but skipped without the opt-in** - -Run: `./gradlew native-cli:benchmarkCli` -Expected: task is SKIPPED (the `onlyIf` is false without `-Pbenchmark=true`), build succeeds. - -- [ ] **Step 5: Commit** - -```bash -git add native-cli/build.gradle native-lib/build.gradle -git commit -m "build: register benchmarkCli runner + wire cli JS parity tests" -``` - ---- - -## Task 9: End-to-end smoke test + README - -**Files:** -- Modify: `benchmarks/README.md` - -**Interfaces:** -- Consumes: everything above. This task validates the full path once against a real bench-built binary, then documents the runner. - -The end-to-end run needs a GraalVM toolchain (per the repo README/CLAUDE.md). It is a manual verification gate, not an automated test in `build`. - -- [ ] **Step 1: Build the bench-enabled binary** - -Run: `./gradlew native-cli:nativeCompile -Pbenchmark=true` -Expected: produces `native-cli/build/native/nativeCompile/dw`. (Several minutes; needs `GRAALVM_HOME`/`JAVA_HOME` set to a GraalVM with `native-image`, per CLAUDE.md.) - -- [ ] **Step 2: Smoke-run one cold-start sample directly against the binary** - -Run: -```bash -node -e ' -import("./benchmarks/runners/cli/coldstart.mjs").then(async (m) => { - const { loadManifest } = await import("./benchmarks/lib/manifest.mjs"); - const manifest = loadManifest("./benchmarks/corpus"); - const rows = await m.runColdStartAndFirstRun(manifest, { samplesOverride: 2 }); - const cold = rows.filter(r => r.metric === "cold-start"); - const first = rows.filter(r => r.metric === "first-run"); - if (cold.length < 1 || first.length < 1) { console.error("missing rows"); process.exit(1); } - for (const r of [...cold, ...first]) { - if (!(r.stats.median > 0)) { console.error("non-positive median", r); process.exit(1); } - } - console.log("smoke OK:", cold.length, "cold,", first.length, "first rows"); -}); -' -``` -Expected: prints `smoke OK: N cold, M first rows`; a positive `cold-start` (spawn→READY) and `firstRunMs` for each sampled case. (Uses `samplesOverride: 2` to stay fast.) - -- [ ] **Step 3: Run the full cross-runner comparison including cli** - -Run: `./gradlew benchmarkCompare -Pbenchmark=true` -Expected: the printed table includes a `cli` column with `cold-start`, `first-run`, and `warm` rows populated, `streaming` rows blank (`—`) for the cli column, and a `Δ cli vs ` column. - -- [ ] **Step 4: Document the runner in `benchmarks/README.md`** - -In `benchmarks/README.md`: - -Under **Layout** (after the `runners/python/` sentence, ~line 15), add: -``` - `runners/cli/` is the CLI runner: a Node parent that spawns the `dw` native - binary (built with `-Pbenchmark=true`, which compiles in an in-binary - benchmark harness gated by `BenchmarkMode.ENABLED` and dispatched via the - `DW_BENCH` env var — the shipped `dw` contains none of it). It emits - `cold-start`, `first-run`, and `warm`; it does **not** emit `streaming` - (the `dw run` path has no chunked-input FFI like the library's). -``` - -Under **Single-runner options** (~line 52), add: -``` - ./gradlew native-cli:benchmarkCli -Pbenchmark=true # CLI only: writes results/cli-.json -``` -and note its prerequisite: -``` -The **CLI runner** requires the bench-enabled binary -(`./gradlew native-cli:nativeCompile -Pbenchmark=true`); set `DW_BENCH_BIN` to -point at a prebuilt one. Like the library runners it needs the GraalVM toolchain. -``` - -- [ ] **Step 5: Commit** - -```bash -git add benchmarks/README.md -git commit -m "docs: document the CLI benchmark runner" -``` - ---- - -## Self-Review - -**Spec coverage:** -- Native binary measured, not JVM entrypoint → Tasks 2, 8 (spawns `dw`). ✓ -- READY-marker protocol, cold-start + first-run + warm → Tasks 2 (harness), 5 (cold/first), 6 (warm). ✓ -- Build-time gate, stripped from production → Tasks 1 (`BenchmarkMode`), 3 (`ENABLED &&` dispatch + guard test). ✓ -- Corpus only, no streaming → no manifest edit; cli emits only cold/first/warm (Tasks 5–7); README documents the gap (Task 9). ✓ -- Corpus-agnostic harness (no manifest knowledge in native-cli) → Task 2 takes file-path args; parent resolves corpus (Tasks 5–7). ✓ -- Runner registration contract (result file + `ext.benchmarkRunner`, no `benchmarkCompare` edit) → Task 8. ✓ -- `runner: "cli"`, `dwlibBuildId: "n/a-cli"`, `runtimeVersion` from `dw --version` → Task 7. ✓ -- Error handling (non-zero exit / missing READY / missing JSON / failed run) → Tasks 2 (harness throws), 5 & 6 (parent rejects). ✓ -- Output discipline (only READY + one JSON line; output to discard stream) → Task 2 + its behavioral test. ✓ -- Testing: Scala harness + ENABLED guard (Tasks 2, 3); dwlib-free JS in parity set (Tasks 4, 7, 8); smoke (Task 9). ✓ -- README update → Task 9. ✓ - -**Placeholder scan:** No TBD/TODO/"handle edge cases"; every code step shows full code; every command has expected output. ✓ - -**Type consistency:** `BenchArgs`/`BenchInput`/`parseArgs`/`runColdFirst`/`runWarm`/`main` (Task 2) are used consistently in Task 3's dispatch and Task 2's tests. `runColdStartAndFirstRun(manifest, {samplesOverride})` (Task 5) and `runWarm(manifest)` (Task 6) match their calls in `emit.mjs` (Task 7) and the smoke test (Task 9). `locateBinary()` (Task 4) is imported by Tasks 5, 6, 7. `buildResult(env, cases)` (Task 7) matches its test. `--input=name=file:mime:charset` arg format is identical between the parser (Task 2) and both parent samplers (Tasks 5, 6). `BenchmarkMode.ENABLED` (Task 1) matches its use in Task 3 and the guard test. ✓ From 90bf8210d8f889b2a8da59fc50adf1924fc61ad1 Mon Sep 17 00:00:00 2001 From: andres-rad Date: Tue, 28 Jul 2026 09:55:52 -0300 Subject: [PATCH 14/15] fix: address benchmark runner review feedback --- .gitignore | 1 + benchmarks/runners/cli/coldstart.mjs | 8 +++++++- benchmarks/runners/cli/warm.mjs | 8 +++++++- benchmarks/runners/node/coldstart.mjs | 8 +++++++- .../src/main/java/org/mule/weave/cli/DWCLI.java | 13 +++++++------ 5 files changed, 29 insertions(+), 9 deletions(-) diff --git a/.gitignore b/.gitignore index d6ea206..f976a29 100644 --- a/.gitignore +++ b/.gitignore @@ -25,4 +25,5 @@ grimoires/ .windsurf/ .claude/ +.vscode/ docs/superpowers/plans/ diff --git a/benchmarks/runners/cli/coldstart.mjs b/benchmarks/runners/cli/coldstart.mjs index 39429ac..caca5d6 100644 --- a/benchmarks/runners/cli/coldstart.mjs +++ b/benchmarks/runners/cli/coldstart.mjs @@ -58,7 +58,13 @@ function sampleOnce(bin, manifest, c) { reject(new Error(`cli coldfirst for '${c.id}' printed no result line\n${stderr}`)); return; } - const { firstRunMs } = JSON.parse(jsonLine); + let firstRunMs; + try { + ({ firstRunMs } = JSON.parse(jsonLine)); + } catch (error) { + reject(new Error(`cli coldfirst for '${c.id}' printed invalid JSON: ${jsonLine}`, { cause: error })); + return; + } resolve({ coldStartMs, firstRunMs }); }); }); diff --git a/benchmarks/runners/cli/warm.mjs b/benchmarks/runners/cli/warm.mjs index 3bd84c7..7de1af9 100644 --- a/benchmarks/runners/cli/warm.mjs +++ b/benchmarks/runners/cli/warm.mjs @@ -48,7 +48,13 @@ function warmSamples(bin, manifest, c) { reject(new Error(`cli warm for '${c.id}' printed no result line\n${stderr}`)); return; } - const { warmMs } = JSON.parse(jsonLine); + let warmMs; + try { + ({ warmMs } = JSON.parse(jsonLine)); + } catch (error) { + reject(new Error(`cli warm for '${c.id}' printed invalid JSON: ${jsonLine}`, { cause: error })); + return; + } if (!Array.isArray(warmMs) || warmMs.length === 0) { reject(new Error(`cli warm for '${c.id}' returned no samples\n${stderr}`)); return; diff --git a/benchmarks/runners/node/coldstart.mjs b/benchmarks/runners/node/coldstart.mjs index f6f7354..9e1a28c 100644 --- a/benchmarks/runners/node/coldstart.mjs +++ b/benchmarks/runners/node/coldstart.mjs @@ -49,7 +49,13 @@ function sampleOnce(corpusDir, caseId) { reject(new Error(`coldstart child for '${caseId}' printed no result line\n${stderr}`)); return; } - const { firstRunMs } = JSON.parse(jsonLine); + let firstRunMs; + try { + ({ firstRunMs } = JSON.parse(jsonLine)); + } catch (error) { + reject(new Error(`coldstart child for '${caseId}' printed invalid JSON: ${jsonLine}`, { cause: error })); + return; + } resolve({ coldStartMs, firstRunMs }); }); }); diff --git a/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java b/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java index 8387aad..008e8a6 100644 --- a/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java +++ b/native-cli/src/main/java/org/mule/weave/cli/DWCLI.java @@ -30,12 +30,13 @@ public class DWCLI { public static void main(String[] args) { - // Benchmark dispatch: only reachable in a build made with -Pbenchmark=true - // (BenchmarkMode.ENABLED is a compile-time false in production, so native-image - // folds this branch and BenchmarkHarness away). DW_BENCH selects the mode. - if (BenchmarkMode.ENABLED && System.getenv("DW_BENCH") != null) { - org.mule.weave.dwnative.benchmark.BenchmarkHarness.main(args); - return; + // Benchmark dispatch: only reachable in a build made with -Pbenchmark=true. + // The outer compile-time constant lets javac remove this block from production. + if (BenchmarkMode.ENABLED) { + if (System.getenv("DW_BENCH") != null) { + org.mule.weave.dwnative.benchmark.BenchmarkHarness.main(args); + return; + } } new DWCLI().run(args, DefaultConsole$.MODULE$); } From 3900e03ad6c97b0bbe48fb754815861ce927986d Mon Sep 17 00:00:00 2001 From: andres-rad Date: Tue, 28 Jul 2026 11:17:43 -0300 Subject: [PATCH 15/15] docs: add repository guidance for coding agents --- AGENTS.md | 185 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 185 insertions(+) create mode 100644 AGENTS.md diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..5232632 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,185 @@ +# AGENTS.md + +## Purpose + +This repository packages the MuleSoft DataWeave runtime as GraalVM native artifacts. +The language/runtime itself is supplied by pinned `org.mule.weave:*` dependencies; +language implementation changes generally belong in the upstream DataWeave repository. + +- `native-cli`: native `dw` executable. Java/picocli parses arguments; Scala implements commands. +- `native-lib`: native `dwlib` shared library, C ABI, Python binding, and Node binding. +- `native-cli-integration-tests`: downloaded suites executed against the compiled native CLI. +- `benchmarks/`: shared corpus/schema and CLI, JVM, Node, and Python benchmark runners. + +Key entry points are `DWCLI.java`, `NativeRuntime.scala`, `NativeLib.java`, and +`ScriptRuntime.java`. Read the nearest module README before broad changes. + +## Toolchain + +- Use the checked-in `./gradlew` wrapper. +- Native builds require GraalVM Community Java 24 with `native-image`. +- Set `GRAALVM_HOME` and `JAVA_HOME`; native-image tasks need about 6 GB heap. +- Node bindings require Node.js 18+ and a C compiler/node-gyp toolchain. +- Python bindings support Python 3.9+. +- Java source/target compatibility is 17; Scala is 2.12. + +## Build Commands + +Run from the repository root unless a command says otherwise. + +```bash +./gradlew native-cli:test +./gradlew native-lib:test -PskipNodeTests=true -PskipPythonTests=true +./gradlew native-cli:nativeCompile +./gradlew native-lib:nativeCompile +./gradlew build -PskipNodeTests=true +./gradlew native-cli:distro +./gradlew native-lib:buildPythonWheel +./gradlew native-lib:buildNodePackage +./gradlew clean +``` + +Native outputs are `native-cli/build/native/nativeCompile/dw` and +`native-lib/build/native/nativeCompile/dwlib.{dylib,so,dll}`. Useful flags: +`-PskipNodeTests=true`, `-PskipPythonTests=true`, `-PskipStripDebug=true`, and +`-PpythonExe=/path/to/python`. + +## Test Commands + +```bash +./gradlew native-cli:test +# One ScalaTest suite (preferred single-test granularity) +./gradlew native-cli:test --tests "org.mule.weave.dwnative.cli.DataWeaveCLITest" +# One native-lib JUnit class or method; skip unrelated binding lanes +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeTest" \ + -PskipNodeTests=true -PskipPythonTests=true +./gradlew native-lib:test --tests "org.mule.weave.lib.ScriptRuntimeTest.runSimpleScript" \ + -PskipNodeTests=true -PskipPythonTests=true +# Native CLI integration/TCK suite (builds and runs the native binary) +./gradlew -PweaveTestSuiteVersion=2.10.0 -DweaveSuiteVersion=2.10.0 \ + native-cli-integration-tests:test +./gradlew native-cli-integration-tests:test \ + --tests "org.mule.weave.clinative.NativeCliTest" +# Dependency-free benchmark JavaScript tests +./gradlew native-lib:benchmarkJsUnitTest +cd benchmarks && node --test lib/stats.test.mjs +cd benchmarks && node --test --test-name-pattern="computeStats returns" lib/stats.test.mjs +``` + +Node/Vitest commands (run after native staging/build when using integration tests): + +```bash +cd native-lib/node +npm install +npm run build +npm test +npm run test:unit +npm run test:integration +npm test -- tests/unit/result.test.ts +npm test -- tests/integration/dataweave.test.ts -t "basic arithmetic" +``` + +Python binding tests are a custom executable script, not a configured pytest suite: + +```bash +./gradlew native-lib:pythonTest +cd native-lib/python && python3 tests/test_dataweave_module.py +cd benchmarks/runners/python +python3 -m unittest test_bench.TestStats.test_matches_lib_stats_on_1_to_100 +``` + +Node TCK is separate from normal `nodeTest`: run `./gradlew native-lib:stageTckSuites`, +then `cd native-lib/node && npm run test:tck`. + +## Lint and Formatting + +There is no repository-wide lint or formatter command: no Scalafmt, Checkstyle, +Spotless, ESLint, Prettier, Black, or Ruff configuration is checked in. Do not claim a +lint step exists or introduce formatting churn. Match neighboring code. For TypeScript, +`cd native-lib/node && npm run build:ts` is the configured strict type check. + +No `.cursorrules`, `.cursor/rules/`, or `.github/copilot-instructions.md` rules exist. + +## Code Style + +- Preserve local import grouping; use explicit imports and avoid new wildcards. +- Java: four spaces, braces on the declaration line, braced control flow, explicit + types, `PascalCase` classes, `camelCase` members, `UPPER_SNAKE_CASE` constants. +- Scala: two spaces, prefer `val`, idiomatic `foreach { x => ... }`, explicit public + return types, case classes/sealed traits for domain variants, no semicolons. +- TypeScript/ESM: two spaces, double quotes, semicolons, trailing commas in multiline + constructs, `camelCase` values, `PascalCase` types/classes. +- TypeScript is strict and emits CommonJS. Use `node:` built-in imports and `import type`; + local `.ts` imports omit extensions. +- Benchmark `.mjs` is ESM: include `.mjs` extensions and use `import.meta.url`, never + `require` or CommonJS-only assumptions. +- Python: PEP 8 naming, four spaces, type hints on public/boundary APIs, dataclasses for + result/input models, and snake_case public names translated to camelCase wire fields. +- Gradle formatting varies by file; preserve the file's indentation and quote style. +- Add comments/Javadoc/TSDoc for public APIs, ABI contracts, ownership, concurrency, or + non-obvious algorithms. Avoid comments that merely restate code. + +## Types and Errors + +- Model domain outcomes explicitly: Scala case classes/traits, TypeScript interfaces, + Python dataclasses. Use loose JSON/maps only at serialization or FFI boundaries. +- Preserve wire fields exactly: `success`, `result`, `error`, `mimeType`, `charset`, + `binary`, `streamHandle`, `content`, and `properties`. +- Script failures normally return an unsuccessful result envelope. Binding/lifecycle + failures throw `DataWeaveError`; opt-in modes may promote script failures to + `DataWeaveScriptError`. CLI failures need context and a nonzero exit code. +- Include operation context in errors, avoid null-only messages, and escape every string + crossing a manually constructed JSON boundary. +- Use `try/finally` for native allocations, stream registrations, isolate attachment, + and cleanup. Suppress cleanup errors only when they must not mask an earlier failure. +- Never let JavaScript or Python exceptions unwind across C callbacks; translate them to + the documented callback status (`0` success, nonzero/-1 error, `0` read means EOF). + +## Native and FFI Rules + +- Treat exported C names, argument order, callback semantics, nullability, ownership, + and result JSON as a stable ABI. Update Java, C addon, TypeScript, Python, tests, and + documentation together when changing it. +- Every OS thread calling Graal must attach its own isolate thread and detach afterward. + Never reuse an `IsolateThread` from another OS thread. +- Do not capture Graal `Word`/pointer values in Java lambdas; convert to raw addresses and + reconstruct pointers inside explicit workers. +- Copy callback/native buffers before returning; never retain borrowed pointers or use a + Graal-owned C string after `free_cstring`. +- Preserve bounded queues/backpressure and chunk remainders; chunks can exceed the native + 8 KiB callback buffer and do not align with logical records. +- Node worker threads must use N-API thread-safe functions; do not call V8 directly. +- Native-image `buildArgs` are load-bearing; do not simplify initialization, charset, + locale, HTTP, reflection, or resources without native build/runtime coverage. +- Preserve SPI descriptors for `DataFormat` and `ModuleLoader`; `native-lib` deliberately + re-materializes them when creating its fat JAR. Missing entries often appear as + runtime "unknown mime type" failures, not build errors. + +## Tests and Generated Files + +- Scala CLI tests use `AnyFreeSpec`/`Matchers`; exercise picocli parsing and assert exit + code, stdout, and stderr. +- Java library tests use JUnit 5. Node uses Vitest projects (`unit`, `integration`, `tck`). + Benchmark JavaScript uses `node:test`; its Gradle task enumerates files explicitly, so + wire new test files into `native-lib/build.gradle` when they belong in normal testing. +- Streaming changes need chunk integrity, terminal metadata, large-chunk, failure, and + cleanup coverage. Shared benchmark math/schema changes need JS/Python/Scala parity. +- Never edit generated `build/genresource/.../ComponentVersion.scala`, + `build/genjava/.../BenchmarkMode.java`, Graal headers, staged `dwlib.*`, downloaded TCK + suites, benchmark results, or generated benchmark inputs. Change generators/sources. +- Gradle stages native libraries into Python and Node packages; do not commit staged or + packaged artifacts (`node_modules`, `dist`, `native`, wheels, tarballs, results). + +## Benchmarks and CI + +Benchmarks are opt-in: `./gradlew benchmarkCompare -Pbenchmark=true`. Runner tasks carry +`ext.benchmarkRunner = true`; the root aggregator auto-discovers them. New runners must +use the shared corpus/schema and write `benchmarks/results/-.json`. +Do not add benchmark execution to normal `build` or `test`. The CLI benchmark requires a +binary built with `-Pbenchmark=true`; `DW_BENCH_BIN` may select a prebuilt one. + +CI builds Ubuntu and Windows with GraalVM 24. Native CLI regression suites and Node TCK +run only on `master`. Run the smallest relevant suite, then the nearest module test; use +a native build for FFI, SPI, resources, initialization, packaging, or native-image changes. +Keep PRs focused, add regression tests, minimize dependencies, and report vulnerabilities +through the Salesforce portal named in `SECURITY.md`, never through public issues.