feat(instr): INVOKEDYNAMIC handler isolation with detach-safe dispatch

[jbachorik]

Summary

Replaces INVOKESTATIC handler copying with INVOKEDYNAMIC dispatch. Probe handler methods live in the probe class — defined in a per-probe ClassLoader on JDK 8/9-14 or as a hidden class on JDK 15+ (no longer pinned to the bootstrap CL) — and are resolved via MethodHandles.publicLookup().findStatic(); no bytecode is copied into target classes. Every dispatch site is a MutableCallSite, so when a probe is unregistered the live call sites can be relinked to a noop — preventing crashes in the instrumented application if a probe handler would otherwise run against torn-down BTraceRuntime state. This restores the safety invariant the older cushion bytecode-rewrite provided, at the dispatch layer instead of via redefineClasses. Works on Java 8+.

Because probe classes now live in per-probe residency, probe detach actually releases the class for unloading (previously every attach/detach cycle leaked one probe class into Metaspace for the JVM lifetime). Handler MethodHandle caching moved to a per-probe instance field that dies with the probe object, and Client.cleanupTransformers wires BTraceRuntimes.removeRuntime so the runtime registry entry is released too.

Cross-classloader wiring

The dispatcher lives in the bootstrap CL (it has to — it's the BSM for INDY sites emitted from anywhere). The repository implementation lives in the agent CL (it needs BTraceRuntime, SLF4J, ASM). A single volatile HandlerRepository repository field on IndyDispatcher bridges the two, populated reflectively from HandlerRepositoryImpl's static init. Probe classes themselves sit in a per-probe CL / hidden class, reachable from the dispatcher only via the MethodHandle returned by resolution — no name-based lookup is required from either loader.

flowchart LR
  subgraph BootCL["Bootstrap ClassLoader"]
    IND["IndyDispatcher<br/><br/>• bootstrap BSM<br/>• invalidateProbe<br/>• LIVE_SITES registry<br/>• repository bridge field"]
    HR["HandlerRepository<br/>(interface)"]
  end
  subgraph AgentCL["Agent ClassLoader"]
    HRI["HandlerRepositoryImpl<br/><br/>• probeMap (by name)<br/>• resolveHandler<br/>• register / unregisterProbe"]
    RT["BTraceRuntime<br/>ASM · SLF4J"]
  end
  HRI -. "implements via method ref" .-> HR
  HRI -- "reflective Field.set in static{}" --> IND
  IND -- "repository.resolveHandler" --> HRI
  HRI --> RT

Loading

Dispatch chain — happy path

Hot path after JIT warm-up: the MutableCallSite target is the resolved handle, HotSpot treats it as @Stable, inlines directly.

sequenceDiagram
  autonumber
  participant IM as Instrumented method
  participant JVM as JVM linker
  participant IND as IndyDispatcher (boot CL)
  participant HRI as HandlerRepositoryImpl (agent CL)
  participant P as Probe handler (per-probe CL / hidden)

  IM->>JVM: invokedynamic (first hit)
  JVM->>IND: bootstrap(caller, name, type, probeClassName)
  IND->>HRI: repository.resolveHandler(probe, name, type)
  HRI->>HRI: probeMap.get(probe), read probe.getProbeClass()
  HRI->>P: publicLookup().findStatic(probeClass, short, type)
  HRI-->>IND: MethodHandle
  IND->>IND: mcs = new MutableCallSite(type), setTarget(mh), register weakRef
  IND-->>JVM: MutableCallSite
  Note over IM,P: subsequent invocations — JIT inlines mcs.target directly
  IM->>P: probeHandler(args)

Loading

Dispatch chain — unresolved path (self-relinking trampoline)

When bootstrap runs before the probe is registered (or the repository bridge isn't wired yet), we install a trampoline. Each invocation retries resolution; on success the MCS target is updated and the trampoline is bypassed from then on.

sequenceDiagram
  autonumber
  participant IM as Instrumented method
  participant MCS as MutableCallSite
  participant TR as Trampoline handle
  participant IND as IndyDispatcher.relink
  participant HRI as HandlerRepositoryImpl

  Note over IM,HRI: Invocation N — probe not yet registered
  IM->>MCS: invoke(args)
  MCS->>TR: dispatch to current target
  TR->>IND: relink(mcs, probe, name, type)
  IND->>HRI: tryResolve(...)
  HRI-->>IND: null
  IND-->>TR: noop(type)
  TR->>TR: invoke noop(args)
  TR-->>IM: default value / void

  Note over IM,HRI: registerProbe(probe) happens here

  Note over IM,HRI: Invocation N+1
  IM->>MCS: invoke(args)
  MCS->>TR: dispatch
  TR->>IND: relink(mcs, probe, name, type)
  IND->>HRI: tryResolve(...)
  HRI-->>IND: MethodHandle
  IND->>MCS: setTarget(mh), MutableCallSite.syncAll
  IND-->>TR: mh
  TR->>TR: invoke mh(args)
  TR-->>IM: probe handler result

  Note over IM,MCS: Invocation N+2 onward — JIT re-optimises, inlines through mcs.target

Loading

Detach — invalidate-live-sites flow

The safety property: after unregisterProbe, no call site may enter probe handler code. This is what the old cushion bytecode rewrite ensured; we now do it at the dispatch layer.

sequenceDiagram
  autonumber
  participant CL as Client (detach)
  participant PN as BTraceProbeNode.unregister
  participant HRI as HandlerRepositoryImpl
  participant IND as IndyDispatcher.invalidateProbe
  participant MCS1 as live MCS #1
  participant MCS2 as live MCS #2

  CL->>PN: unregister()
  PN->>HRI: unregisterProbe(probe)
  HRI->>HRI: probeMap.remove(name)
  HRI->>HRI: handlerCache.removeIf(key startsWith name)
  HRI->>IND: invalidateProbe(name)
  IND->>IND: q = LIVE_SITES[name]
  loop each weakRef in q
    IND->>IND: site = ref.get(), drop if null
    IND->>MCS1: setTarget(noop(type))
    IND->>MCS2: setTarget(noop(type))
  end
  IND->>IND: MutableCallSite.syncAll(collected)
  Note over MCS1,MCS2: next invocation lands on noop — JIT sites deopt on syncAll

Loading

Call-site state machine

stateDiagram-v2
  [*] --> Trampoline: bootstrap — probe not yet registered
  [*] --> Resolved: bootstrap — probe registered (setTarget MH)
  Trampoline --> Resolved: registerProbe then invoke (relink + syncAll)
  Trampoline --> Trampoline: invoke while unregistered (noop this call)
  Resolved --> Noop: unregisterProbe (invalidateProbe + syncAll)
  Trampoline --> Noop: unregisterProbe
  Noop --> [*]: call site eligible for GC (weakRef collected)

Loading

Data layout

IndyDispatcher.LIVE_SITES (bootstrap CL)

Per-probe weak registry of dispatch sites, used only by invalidateProbe.

ConcurrentMap<String, ConcurrentLinkedQueue<WeakReference<MutableCallSite>>>
    │
    ├── "com/example/MyTrace"      → [weakRef(mcs-A), weakRef(mcs-B), ...]
    ├── "com/example/OtherTrace"   → [weakRef(mcs-C)]
    └── ...

• Key: probe class internal name (slashes).
• Value: queue of weak references so unreachable call sites (e.g. unloaded instrumented classes) don't leak. invalidateProbe prunes nulls opportunistically.
• Entries added once per successful bootstrap(...) call; never removed except by weak-reference collection.

HandlerRepositoryImpl.probeMap + per-probe MH cache (agent CL)

probeMap  : ConcurrentMap<String probeName, BTraceProbe>

# On each BTraceProbeSupport instance:
handlerCache  : ConcurrentMap<HandlerSubKey, MethodHandle>
HandlerSubKey : (String handlerName, MethodType type) with precomputed hash

• probeMap is the source of truth for "is probe X registered?"
• The handler MH cache is per-probe — it lives on BTraceProbeSupport and is reached via default interface methods BTraceProbe.getCachedHandler / cacheHandler. Because each cache instance is owned by one probe, unregister is probeMap.remove(name) (no cross-probe scan) and the cache evicts naturally when the probe object is collected. MH references into the defined probe class are also dropped when BTraceProbeSupport.clearProbeClass() runs, so they don't retain the class beyond detach.
• Cache is positive-only: a null return from resolveHandler means "try again on next invocation". A negative entry would defeat the MutableCallSite self-heal (next bootstrap would get a stale null and install a trampoline that immediately sees the cached null again).

Bootstrap static arguments (what the JVM passes)

bootstrap(
  MethodHandles.Lookup caller,   // lookup context of the instrumented class
  String               name,     // probe-prefixed handler name, e.g. "MyTrace$onEntry"
  MethodType           type,     // call-site signature
  String               probeClassName  // BSM extra const — probe class internal name
)

The fourth argument is emitted as a CONSTANT_String BSM argument by Instrumentor.invokeBTraceAction. name is stripped to its short form inside resolveHandler ("MyTrace$onEntry" → "onEntry") before the findStatic call.

New files

• IndyDispatcher — Java 8+ invokedynamic bootstrap class in boot CL. Resolves the probe handler, installs a MutableCallSite, tracks live sites, and supports invalidateProbe(name) for detach-time relink.
• HandlerRepositoryImpl (rewritten) — per-probe positive-only cache (delegated to BTraceProbeSupport), no sentinel, no eviction-on-register. unregisterProbe drops the probeMap entry and propagates to IndyDispatcher.invalidateProbe; no cross-probe keyset scan.
• ProbeAnchor — shared helper (JDK 8 source set, visible to both the java9 and java11 source sets) that generates a per-probe anchor class in a fresh unnamed ClassLoader so the probe class lands in its own loader (JDK 9-14 path).
• ProbeClassUnloadingTest — weak-reachability test covering the new residency. Verifies the probe Class<?> and its CL become weakly reachable after detach + GC, and that the same probe name can be defined twice in one JVM (previously a LinkageError on JDK 8).
• DispatchBenchmark — JMH benchmark: baseline direct call, ConstantCallSite dispatch, MutableCallSite dispatch.

Deleted

• Indy.java — JDK 15-only defineHiddenClass bootstrap, no longer needed.
• CopyingVisitor.java — handler bytecode copying, no longer needed.
• static/ golden files (~198) — replaced by unified dynamic/ files.

Refactored

• Instrumentor.invokeBTraceAction — always emits INVOKEDYNAMIC; removed useHiddenClasses dual-mode gate; BSM owner: Indy → IndyDispatcher.
• Probe lifecycle symmetry: registerProbe moved to BTraceProbeNode/BTraceProbePersisted.register() after defineClass; unregisterProbe added to both unregister() methods; removed premature registration in BTraceProbeFactory and redundant cleanup in Client.onExit. unregisterProbe now also relinks live call sites via IndyDispatcher.invalidateProbe.
• BTraceRuntime.Impl.defineClass — signature reduced from (byte[], boolean) to (byte[]). The mustBeBootstrap parameter became dead once every impl created a per-probe residency regardless; dropped from the interface and all three impls. Sole production caller BTraceProbeSupport.defineClass updated.
• BTraceRuntimeImpl_8.defineClass — always uses new ClassLoader(null){} (previously bootstrap for transforming probes).
• BTraceRuntimeImpl_9.defineClass — per-probe anchor class in a fresh unnamed loader, then privateLookupIn(anchor, …).defineClass(bytes) so the probe lands in the per-probe loader (not Auxiliary's bootstrap loader).
• BTraceRuntimeImpl_11.defineClass — defineHiddenClass(bytes, true) on JDK 15+ (reflective; no ClassOption — STRONG would pin lifetime to the defining loader), per-probe anchor on JDK 11-14.
• BTraceRuntimeImpl_9/_11 — StackWalker frames filtered for org.openjdk.btrace.runtime.auxiliary.* in getCallerClassLoader() / getCallerClass().
• HandlerRepositoryImpl.resolveHandler — reads the probe Class<?> via probe.getProbeClass() (cached by BTraceProbeSupport.defineClass) instead of Class.forName(probeName). Decouples resolution from bootstrap-CL visibility.
• HandlerRepositoryImpl.handlerCache — removed. Cache is now a per-probe ConcurrentMap<HandlerSubKey, MethodHandle> on BTraceProbeSupport; accessed via BTraceProbe.getCachedHandler / cacheHandler default interface methods. Dies with the probe object.
• BTraceRuntimes.removeRuntime(String) — new public primitive; Client.cleanupTransformers calls it after probe.unregister() so the static runtime registry no longer leaks a BTraceRuntime.Impl (and its probe class) per attach/detach cycle.
• Cushion-method infrastructure removed from BTraceClassWriter, BTraceTransformer, and Instrumentor. The crash-safety contract now lives at the dispatch layer.

JDK 8 reflection-inflation cascade fix

testTraceAll() failed on JDK 8 only with StackOverflowError during agent init. Root cause: agent's reflective Method.invoke() crosses the JDK 8 inflation threshold (~15 calls/Method), JVM defines sun/reflect/GeneratedMethodAccessorN via DelegatingClassLoader, fires BTraceTransformer.transform() for the synthetic class, transformer instruments it → ASM frame computation issues more reflective calls → another inflation → another transform callback → recursion → SOE.

Prior commit 4d53e21 added a ThreadLocal re-entrancy guard in isBootstrapClass(); the guard had a documented timing hole and silently mis-reported bootstrap-class status during the inflation window — and the cascade still formed via OTHER reflective calls in ASM's getCommonSuperClass() path. Reverted.

Structural fix: BTraceTransformer.transform() early-exits for JVM-synthesized accessor classes (sun/reflect/Generated*, jdk/internal/reflect/Generated*) regardless of class loader, BEFORE any reflection-driven ASM analysis runs on them. sun/reflect/ also added to ClassFilter.SENSITIVE_CLASSES for defense-in-depth.

Dispatch-cost measurement

From DispatchBenchmark on JDK 21 (local run):

variant	ns/op	vs CCS
baseline (direct static call)	0.40	0.11×
instrumented (ConstantCallSite)	3.72	1.00×
instrumentedMutable (MutableCallSite, stable target)	3.63	0.97×

MutableCallSite is within noise of ConstantCallSite once the JIT treats the target as stable — dropping CCS has no measurable dispatch-throughput cost, and is what makes detach-time relink-to-noop possible at all.

Test plan

• ./gradlew :btrace-instr:test — 663 tests pass (incl. new ClassFilterSensitiveTest, BTraceTransformerEarlyExitTest; HandlerRepositoryImplTest expanded from 3 to 10 tests).
• ./gradlew :btrace-instr:test -PupdateTestData — 382 dynamic golden files regenerated.
• ./gradlew :benchmarks:runtime-benchmarks:jmh — numbers above.
• ./gradlew :integration-tests:test -Pintegration — JDK 8 path requires CI (no local JDK 8).

Coverage added to HandlerRepositoryImplTest

#	Test	What it guards
1	testRegisterAndResolveReturnsNullForUnknownHandler	No crash when probe class isn't loadable
2	testUnregisterClearsCacheForProbe	Cache is scoped to probe name prefix
3	testUnregisterDoesNotAffectOtherProbes	Per-probe eviction isolation
4	testResolvesRealHandlerFromLoadedClass	Happy-path publicLookup().findStatic resolution produces an invocable handle
5	testBootstrapReturnsMutableCallSiteOnImmediateResolution	Every call site is an MCS, not a CCS (so it remains relinkable)
6	testBootstrapBeforeRegistrationHealsViaMutableCallSite	Trampoline self-heals once probe is registered
7	testBootstrapWithNullRepositoryDoesNotCrash	Null-repository path returns a no-crash MCS
8	testUnregisterRelinksLiveCallSiteToNoop	Core crash-safety: handler does NOT fire after unregister
9	testUnregisterProbeADoesNotInvalidateProbeB	invalidateProbe is properly scoped
10	testUnregisterRelinksAllLiveSitesForProbe	Multi-site relink — no off-by-one in the registry iteration
11	testResolveHandlerUsesProbeClassAccessorNotClassForName	Resolution path goes through probe.getProbeClass() — no Class.forName dependency
12	testGetProbeClassExposesClassOnStubProbe	Accessor plumbing

Coverage added in ProbeClassUnloadingTest

#	Test	What it guards
1	probeClassWeaklyReachableAfterDefine	Probe Class<?> + its ClassLoader are weakly reachable after detach + removeRuntime
2	sameProbeNameCanBeDefinedTwice	Defining the same internal probe name twice in one JVM no longer throws LinkageError — each attach has its own residency

Run locally on JDK 8 via JAVA_TEST_HOME=$(sdk home java 8.0.482-librca) ./gradlew :btrace-instr:test and on JDK 21 via the default build.

Acceptance criteria

• IndyDispatcher works from Java 8+.
• Every dispatch site is a MutableCallSite (including the immediate-success path) so unregister can relink to noop.
• IndyDispatcher.invalidateProbe(name) relinks every live call site for that probe to a type-correct noop handle, and calls MutableCallSite.syncAll to drop optimised JIT targets promptly.
• HandlerRepositoryImpl resolves handlers via publicLookup().findStatic() with a positive-only ConcurrentHashMap cache; negative results are not cached (MCS trampoline handles retry).
• HandlerRepositoryImpl.unregisterProbe clears the cache for the probe and calls IndyDispatcher.invalidateProbe.
• ~400 redundant static/ golden files replaced with unified dynamic/ files.
• HandlerRepositoryImplTest covers the 10 scenarios above.
• DispatchBenchmark covers baseline + ConstantCallSite + MutableCallSite.
• All instrumentor tests pass.
• testTraceAll() passes on JDK 8 — pending CI run.
• Probe classes are defined in a per-probe ClassLoader (JDK 8/9-14) or hidden class (JDK 15+), never bootstrap CL.
• After probe.unregister() + BTraceRuntimes.removeRuntime, the probe Class<?> and its ClassLoader become weakly reachable (ProbeClassUnloadingTest.probeClassWeaklyReachableAfterDefine).
• Same probe name can be defined twice in one JVM (ProbeClassUnloadingTest.sameProbeNameCanBeDefinedTwice).
• HandlerRepositoryImpl holds no module-level handler cache — per-probe only.
• BTraceRuntime.Impl.defineClass signature is (byte[]); mustBeBootstrap removed everywhere.
• Client.cleanupTransformers calls BTraceRuntimes.removeRuntime after probe.unregister().
• Implementation plan committed at docs/plans/2026-04-20-probe-class-unloading.md.

🤖 Generated with Claude Code

---

This change is