0.2a — Wire AbstractIntermediateCallback into the Ipopt + MadNLP backends [DirectTrajOpt.jl]

[jack-champagne]

Phase: 0-prime — Contracts / foundation · Plan task: 0.2 · Est: ~1 pd (scope is Ipopt-only; the type, the MadNLP wiring, and the DTO release already exist) · Suggested role: Dev A (Julia solver path)

Important

Decision surface

DirectTrajOpt.AbstractIntermediateCallback already exists, is exported, and is released (v0.9.5/v0.9.6), and the MadNLP backend already wraps and fires it per-iteration (auto-couple + fixed_variable_treatment = RelaxBound + termination, with passing tests). The Ipopt backend does not: its solve path takes a raw callback::Union{Nothing,Function} closure and wires it through Ipopt.CallbackFunction(), and IpoptOptions exposes no intermediate_callback field. So a LivePulsePlotCallback (which subtypes the agnostic type) works under MadNLP but cannot be installed under Ipopt. This slice closes that gap — and only that gap. No DTO bump/registration is needed (the symbol is already on a released, pinned version); no MadNLP work is needed (already wired).

What to build

Add an intermediate_callback::Any = nothing option to the Ipopt path (IpoptOptions) and wire it per-iteration so that an AbstractIntermediateCallback instance is invoked with the full primal vector and the iteration index each iteration, with its Bool return driving termination (true continue, false stop early → user-requested termination). Reconstruct/surface the full primal from Ipopt's optimizer state so the same callback object that runs under MadNLP runs unchanged under Ipopt. Then add the matching Piccolo Ipopt install path so LivePulsePlotCallback can be attached to an Ipopt solve the same way it is to MadNLP today (live_callbacks.jl currently documents only MadNLPOptions(intermediate_callback=cb)).

Leave the existing raw callback::Union{Nothing,Function} Ipopt path intact (the text-telemetry / checkpointing closure path) — the new option is additive and opt-in.

Acceptance criteria

• An AbstractIntermediateCallback subtype installed via the Ipopt path is invoked once per IPM iteration, receiving the current full NLP primal vector and the iteration index. (S4)
• The primal passed under Ipopt reconstructs the same trajectory shape the MadNLP path produces for the same problem — the identical callback object runs under both backends with no backend-specific branching in the callback. (S4)
• An Ipopt-installed callback returning false halts the solve as a user-requested termination and the run still yields a valid terminal result (no loss of the last solver progress); returning true lets it run to its normal stop. (S5)
• A solve with no intermediate_callback set is byte-for-byte unaffected, and the existing raw-Function Ipopt callback path keeps working. (S4)
• Piccolo can attach LivePulsePlotCallback to an Ipopt solve via an IpoptOptions(intermediate_callback=cb) install path mirroring the MadNLP one, emitting per-iter PNGs end-to-end. (S4/S5)

Constraints & invariants

• The callback contract is backend-neutral: same signature (primal::AbstractVector, iter::Integer) -> Bool, same cadence (main IPM loop only), and same termination semantics as the MadNLP path.
• The abstract type, its docstring contract, and the MadNLP wiring are not modified — Ipopt is brought up to the existing contract, not the contract changed.

Prior art / patterns

• AbstractIntermediateCallback (the released, exported agnostic per-iter hook) and its docstring contract.
• The MadNLP extension's adapter that wraps the agnostic callback, auto-couples the full primal (RelaxBound), and maps the Bool return to termination — the reference behavior the Ipopt path must match.
• The Ipopt extension's existing Callbacks module / IpoptOptimizerState per-iter fields and its raw-Function callback path (text telemetry / checkpointing) — left in place; the new option sits alongside it.
• Piccolo's LivePulsePlotCallback and its documented MadNLPOptions(intermediate_callback=cb) install path — the template for the new Ipopt install path.
• The inline-@testitem / TestItemRunner convention used by both the MadNLP ext tests (_Stopper, auto-couple probes) and the Piccolo Makie ext tests.

Supersedes from β

• β.5's hand-rolled per-iter PNG path and β's methods(solve!) reach-through to a raw Ipopt callback: post-this, the live plot flows through AbstractIntermediateCallback on both backends, satisfying the Phase 0-prime DoD clause "β's live plot now flows through AbstractIntermediateCallback."

Out of scope

• DTO version bump / JuliaRegistrator release — not needed: the symbol is already on released v0.9.5/v0.9.6 and Piccolo already pins DirectTrajOpt = "0.9.5".
• Any MadNLP-side change — the MadNLP backend already wraps, auto-couples, and fires the callback per-iter with Bool termination (tested via _Stopper).
• Modifying the abstract type or its contract — Ipopt conforms to the existing contract.
• HDF5 pulse export and the catalog store/consumer — Phase 3, not the callback contract.
• amico-run / TS inspector changes — the TS side consumes emitted PNGs, unchanged here.

Testing

Extend DTO's Ipopt-ext inline @testitem suite with per-iter-invocation, full-primal-parity-with-MadNLP, and false-return-terminates behaviors (RED→GREEN); the MadNLP per-iter behavior is already green and can serve as a parity oracle. Add a Piccolo Ipopt-install smoke test in the Makie-ext suite. No DTO release gate — runs against the already-pinned released version.

Source

• Phase plan §3 Phase 0-prime task 0.2 ([plan-20260603-124231-amicode-phased-build]); DoD: "β's live plot now flows through AbstractIntermediateCallback."
• PRD S4 (Run Inspector refreshes per-iter pulse plot) and S5 (live iter / f / inf_pr / inf_du stats row) — the live inspector this hook feeds.
• Decision D10 (capability lives in the LLM-authored script via Piccolo public API). The audit-era "Piccolo references a DTO symbol absent from DTO main" item is resolved (symbol released + pinned) and is intentionally not a deliverable here.
• Related: β.5 — Minimal single-run inspector (stats row + live plot) #5 (β.5 single-run inspector — its hand-rolled PNG path is superseded here).

Notes

• Verified state on DirectTrajOpt.jl origin/main and tags: AbstractIntermediateCallback exported (src/solvers/_solvers.jl), released in v0.9.5/v0.9.6; MadNLP ext fully wires it (ext/MadNLPSolverExt/); the Ipopt path is the sole backend lacking it. Piccolo origin/main pins DirectTrajOpt = "0.9.5".
• One vertical slice; do not decompose. The lone residual (Ipopt option + wiring) plus its Piccolo install path ship together because the install path is untestable until the option exists.

[jack-champagne]

Done — AbstractIntermediateCallback for the Ipopt backend shipped in DirectTrajOpt 0.9.7 (released); consumed by #43. (MadNLP path was already present.)