Render the live pulse client-side in the Run Inspector (pulseplot v1)

[kateebonner]

Important

Problem

The Run Inspector's live pulse view is per-iteration PNGs rendered inside the solve process's callback: plot richness is coupled to solve speed, frames are static ~97KB images (no theming, no interactivity), and the write path is prone to mid-write read races (the watcher-side completeness gate now guards this, but the writer — now inside Piccolo's live-plot callback — remains non-atomic and outside the template's control).

Approach

Emit the raw pulse data per iterate (drive values + timestep) as additive lines in the run log, and render the pulse natively in the inspector webview with a new plot component. The emitter is a small user-defined callback on the same solver-agnostic (primal, iter) hook the live PNG callback rides — no package changes. During the prototype phase the native plot renders side-by-side with the PNG preview to validate visual fidelity; Piccolo's plot_pulse stays canonical for publication-grade and archival plots. Key reason: the pulse data is ~1KB/iter, so rendering becomes free at any cadence and interactivity becomes possible later.

Approaches Considered

• Render on a background thread in the solve process — still Julia-side, needs per-script concurrency boilerplate, GC still couples it to the solver. Rejected.
• Separate render process queueing artifacts — full second Julia + Makie load (~1 min startup, double memory). Rejected as heavyweight.
• Client-side render from emitted data (chosen) — smallest data, solver-agnostic emitter, unlocks interactivity later.
• Keep PNGs as-is — mid-write races are now gated watcher-side, but the coupling and static-image limits remain. Rejected as the long-term state.

Scope

In: the webview plot component (stacked per-drive, matching plot_pulse's layout); prototype-grade pulse-line emission in the solve template (user-defined callback); parsing + forwarding on both delivery paths (replay ingest and live tail); host-side buffering for late panel materialization; a standalone dev harness; a re-recorded bundled demo run carrying pulse lines; side-by-side compare in the panel.
Out: hover/scrub interactivity — deferred until the compare phase validates the direction. Other panels (populations, fidelity) — pulse only for v1. Free/per-knot timesteps — v1 targets uniform timesteps (what the template pins); the format leaves room to extend. Run-dir contract docs pass — the contract is frozen for β; this issue is additive-only and GA emission waits on coordination. PNG-path removal — rides the later change that makes the native plot primary. Upstream atomic-save fix in Piccolo's live-plot callback — separate engineer-lane item.

Assumptions / Open Qs

• Additive run-log lines don't violate the β freeze: the log is a stdout tee already carrying arbitrary output, and both existing consumers act only on anchored regex matches. Being flagged in the team thread regardless.
• Fidelity bar for the compare phase is "recognizably the same plot" (layout, bounds, shapes) — pixel parity with Makie is not required.
• Merge-ordering prerequisite: the inspector view reorg (PR refactor(extension): TS-composed inspector view + brand/layout token split #65) must land first — the component builds on its atoms/components/view system.

---

Acceptance Criteria

1. A run whose log carries pulse lines shows the native plot beside the PNG preview; runs without them render exactly as today (PNG only, no empty plot slot). The new-run warming transition clears/hides the pulse plot, so switching from a pulse-carrying run to a pulse-less one leaves no stale plot.
2. The native plot matches plot_pulse's stacked layout: one panel per drive, step (stairs) rendering, bounds band with dashed limits, per-drive labels, shared time axis with the label on the bottom panel only.
3. The plot is theme-aware: colors, spacing, and typography come from the design tokens (adding per-series tokens to the brand sheet as needed) and render correctly in light and dark themes.
4. The pulse-line parser is unit-tested against the pinned grammar, including: meta and per-iter lines parse; Inf/NaN values handled as the stats parser does; malformed or unknown lines ignored; a pulse line before any meta line is dropped; a repeated meta line wins and resets state (the tailer re-reads the file from offset 0 on truncation, so duplicate meta is expected); records whose drive count or knot count disagree with the current meta are ignored; a meta line whose label or bounds count disagrees with its own drive count is treated as malformed.
5. The staged bundled demo run (re-recorded with pulse lines) exercises replay ingest end-to-end: meta plus the newest pulse record are forwarded and the native plot renders beside the PNGs.
6. Live-tail routing is tested at the line-handler seam: the tailer forwards the meta line and each pulse record in order as they land (on a live run the tailer is the only carrier of meta — replay ingest sees an empty log at run start).
7. Pulse records survive late panel materialization: meta + newest record are buffered host-side and replayed when the webview resolves, before any terminal-state message (mirrors the existing buffered-image idiom).
8. Rendering is bounded: pulse refreshes reuse the host's existing 5 Hz throttle policy, and a component update at fixture scale (2 drives × 50 knots) completes within a 16 ms frame budget, measured in the dev harness.
9. No changes to the frozen contract artifacts or their validators; the view-seam contract test is updated for the new message types (pulsemeta, pulse).

Key Decisions

• Wire grammar (pinned; candidate GA format). Two anchored line shapes, numeric fields drawn from the same numeric domain as the stats line (decimal/scientific, Inf/-Inf/NaN):
  • AMICODE_PULSE_META drives=<int> knots=<int> labels=<comma-separated double-quoted strings; labels restricted to characters excluding double quotes and commas> bounds=<comma-separated lo:hi pairs, one per drive>
  • AMICODE_PULSE iter=<int> dt=<num> a=<per-drive value lists: values comma-separated, drives semicolon-separated>
    Values at ~6 significant figures (≈1KB/iter at fixture scale). dt is a scalar because v1 targets uniform timesteps; a per-knot array is the documented GA extension if free-time problems need it.
• Emitter placement: a user-defined callback in the solve template implementing the solver-agnostic intermediate-callback interface (primal + iteration), reconstructing the pulse the same way the live PNG callback does, printing the two line shapes to stdout. Prototype-grade until contract coordination lands.
• Delivery paths: the pure run-dir reader parses pulse lines during replay ingest and forwards meta + the newest record only (a finished run's full history would burst thousands of identical-end-state messages at the webview); the live tailer forwards the meta line and each record as they land. Both route through a new method on the run-sink interface.
• Host buffering: the inspector buffers pulse meta + the newest record until the webview materializes, replayed in the existing buffered-replay order (before the terminal-state message).
• Rendering: hand-rolled SVG — the series are tens of points at fixture scale; above 512 knots per drive the component decimates by stride before rendering (explicit envelope for the no-chart-library decision; min/max-preserving decimation is the GA refinement — stride can alias single-knot spikes on a stairs plot).
• Component API mirrors the existing component idiom: a factory returning an element plus meta/update/clear methods, styles via the shared style registry, values from the brand tokens.
• Staging is harness-first: build the component + parser pure and iterate visually in a standalone dev page (dev-only directory with an npm script; excluded from the packaged VSIX) against the recorded fixture; only then add the extension wiring.
• Fixture: re-record the bundled demo run with the emitting template so the stock demo carries pulse lines; refresh its PNGs together and keep the packaging required-assets list satisfied (the run log is gitignore-adjacent — verify it ships).

Constraints & Invariants

• β contract freeze: nothing added or changed in the manifest, result, or completion artifacts or their validators; run-log additions must be ignorable by consumers that don't know them.
• Publication-grade and archival plotting stays Piccolo-side (plot_pulse); this component is the live view only.
• The PNG preview remains fully functional throughout this issue — the compare phase depends on it.
• The dev harness must not ship in the VSIX.

Prior Art / Patterns

• The stats-line pipeline (per-iteration stats line → replay ingest + log tailer → typed webview message → metric cards) is the exact pattern to mirror, including its field-renaming at the view seam.
• The buffered-replay idiom in the inspector host (image / completion / warming buffered until the webview resolves) is the pattern for pulse buffering.
• The TS-composed view system (atoms → components → view, constructable-stylesheet registry, brand/layout token split) is introduced by the open inspector-view reorg PR (refactor(extension): TS-composed inspector view + brand/layout token split #65) — a merge prerequisite, not landed prior art.
• The demo-replay command stages the bundled demo run as a contract-identical fresh run — the re-recorded fixture rides it unchanged.
• Piccolo's plot_pulse stacked layout (stairs for zero-order pulses, bounds band, per-drive panels) is the visual reference; the live PNG callback (landed via the LivePulsePlotCallback template change) is the emitter's placement reference.

Source

• Team thread agreement: plot_pulse stays for publication-grade plotting; the viz gets a separate plotter (Jack, gennadi, Kate — Slack, 2026-07-02).
• Prerequisite: inspector view reorg (PR refactor(extension): TS-composed inspector view + brand/layout token split #65). Related: the pending watcher frame-race fix (completeness gate), which remains necessary while PNGs are displayed.

Notes

• The emitter rides the same solver-agnostic hook as the live PNG callback, so it survives the MadNLP/Altissimo migration; the stats line does not (its objective/infeasibility fields exist only on the raw Ipopt callback) — that gap is a separate concern to file independently.
• The upstream non-atomic PNG save inside Piccolo's live-plot callback is a separate engineer-lane fix candidate; the watcher's completeness gate covers it from the consumer side meanwhile.

[kateebonner]

Scope amendment (design-lane owner decision, 2026-07-02): the side-by-side compare phase is skipped — the native pulseplot fully replaces PNG display in the inspector (implemented in PR #67, commit b9622cb).

Amended acceptance criteria:

• AC1 (side-by-side + no-empty-slot) → the native plot is the only pulse view; pulse-less runs (old runs, non-emitting scripts) show a hint to re-run with the current template; the warming transition still clears the plot.
• AC5/AC6 (side-by-side rendering clauses) → replay + live-tail delivery unchanged, rendering solely native.
• PNG-path removal, previously out of scope, is now in scope display-side only: iter_<N>.png remains a run-dir/archival artifact per the team-thread agreement (plot_pulse stays publication-grade); the inspector no longer reads it.

All other ACs (grammar, stream policy, buffering, throttle, theming, contract freeze) stand as written.

[kateebonner]

Closing per design-lane owner: implementation complete in PR #67, which now carries the full design context (wire format, delivery policy, scope amendment to full PNG-display replacement, verification). This issue remains the design history — grilling decisions, review-loop record, and the amendment thread.