fix(json,json-cppagent): cache JsonSerializerOptions singletons — plug DynamicMethod LCG leak (+3.2-3.8 MB/h RSS)

[ottobolyos]

Summary

Cache JsonSerializerOptions as static readonly singletons in both MTConnect.NET-JSON and MTConnect.NET-JSON-cppagent JsonFunctions.cs, plus every sibling site that re-allocated the option object per call. Prevents unbounded loader-heap growth from LCG-emitted property accessors, observed on two DIME-connector production hosts as a +3.2–3.8 MB/h RSS climb with a flat managed heap.

Root cause

A fresh JsonSerializerOptions instance owns its own serialization-metadata cache. Building that cache emits reflection-based property accessors (DynamicMethod in the runtime's LCG heap) for every property in the reachable type graph. Allocating one on every serialization call therefore re-emits those accessors on every call, and the emitted code accumulates in the runtime's loader heaps where the GC cannot reclaim it.

Peer diagnosis (tempco-investigation-server, 2026-08-21) on two hosts:

• tempco-001 (amd64, server-GC) and tim-001 (arm64, workstation-GC) — completely different source/sink shapes, 28× Lua/Scriban usage delta between them. The only shared serialization surface was MTConnect JSON.
• dotnet-counters: GC heap flat (76–87 MB sawtooth); working set climbed (445 → 460 MB); ~370 MB outside the managed heap.
• Number of Methods Jitted: 17,651,641 in 71 h on tempco-001 (+67/s forever) — healthy service is typically 10 k–50 k lifetime.
• IL Bytes Jitted: +67,083 B / 90 s = 2.7 MB/h, matching the 3.3 MB/h RSS climb.
• dotnet-gcdump: System.Text.Json.Serialization.Metadata.ReflectionEmitCachingMemberAccessor+Cache+CacheEntry present.
• dotnet-trace LCG DynamicMethods: 8946 hits on dynamicClass, 2465 on IObservationOutput accessor, 1371 on JsonEvents, 1161 on JsonSamples.

Why both JsonFunctions.cs files ship in one PR. DIME's MQTT sink at DIME/Connectors/MtConnectMqtt/Sink.cs:49 sets documentFormat: "JSON-cppagent-mqtt", so per-observation serialization on peer's production hosts flowed through MTConnect.NET-JSON-cppagent.JsonFunctions, not the plain one. An earlier peer hot-swap that patched only the plain-JSON assembly did not reduce JIT churn, because DIME's real workload never touched it. MTConnect.NET-JSON-cppagent.dll also defines 25 Streams.Json classes vs 10 in the plain assembly, so the LCG cost per serialization is proportionally larger. Both assemblies carry independent copies of the anti-pattern (five new JsonSerializerOptions sites each in DefaultOptions, IndentOptions, Convert, ConvertBytes, ConvertStream); both must ship together in this same PR, rather than split into a follow-up, to close the leak for cppagent-format consumers.

Fix

Both JsonFunctions.cs files gain:

private static readonly JsonSerializerOptions _defaultOptions = CreateOptions(false);
private static readonly JsonSerializerOptions _indentOptions  = CreateOptions(true);

private static JsonSerializerOptions CreateOptions(bool indented) => new JsonSerializerOptions { ... WriteIndented = indented, ... };

private static JsonSerializerOptions GetOptions(JsonConverter converter, bool indented)
{
    if (converter == null) return indented ? _indentOptions : _defaultOptions;
    var options = CreateOptions(indented);
    options.Converters.Add(converter);
    return options;
}

public static JsonSerializerOptions DefaultOptions => _defaultOptions;
public static JsonSerializerOptions IndentOptions  => _indentOptions;

Convert/ConvertBytes/ConvertStream now delegate through GetOptions(converter, indented). Every in-tree caller (verified via grep across libraries/, agent/, tests/) passes converter == null and does not mutate the returned instance, so every in-tree call reuses the singleton. The cold-path branch preserves the public API contract for external consumers that supply a per-call converter.

Sibling sweep (same static-readonly pattern, folded in atomically here rather than deferred to a follow-up PR):

• libraries/MTConnect.NET-Common/Configurations/AgentConfiguration.cs — _readOptions (ReadCommentHandling = Skip), 2 call sites.
• libraries/MTConnect.NET-Common/Configurations/AdapterApplicationConfiguration.cs — _readOptions, 2 call sites.
• libraries/MTConnect.NET-Common/Agents/MTConnectAgentInformation.cs — _saveOptions (WriteIndented = true), 1 call site.
• libraries/MTConnect.NET-Common/Clients/MTConnectClientInformation.cs — _saveOptions, 1 call site.
• libraries/MTConnect.NET-Common/Buffers/MTConnectAssetFileBuffer.cs — _writeOptions, 1 call site.
• libraries/MTConnect.NET-MQTT/MTConnectMqttMessage.cs — _agentInformationOptions, 1 call site.

libraries/MTConnect.NET-Common/Buffers/MTConnectObservationFileBuffer.cs was in the peer's sweep list but already calls JsonSerializer.Serialize with no options argument — no anti-pattern present.

Regression tests

tests/MTConnect.NET-JSON-Tests/Regressions/JsonSerializerOptionsSingletonTests.cs and its cppagent mirror pin the singleton contract:

• DefaultOptions_returns_the_same_instance_on_repeat_access — ReferenceEquals(JsonFunctions.DefaultOptions, JsonFunctions.DefaultOptions) == true.
• IndentOptions_returns_the_same_instance_on_repeat_access — same for the indented preset.
• DefaultOptions_and_IndentOptions_are_distinct_instances.
• DefaultOptions_WriteIndented_is_false and IndentOptions_WriteIndented_is_true — behavior guards.
• JsonFunctions_holds_a_static_readonly_options_field — reflection guard that the shared field exists (blocks the anti-pattern regressing under any future refactor).

Verified RED on upstream/master (587a126): 3 fails + 3 passes per project (6 fails total across both suites). Verified GREEN after fix: 6 pass + 6 pass. Full-suite GREEN on bluefin/net8.0:

Project	Pass / Total
MTConnect.NET-JSON-Tests	69 / 69
MTConnect.NET-JSON-cppagent-Tests	369 / 369
MTConnect.NET-Common-Tests	3987 / 3987

Total 4425 tests, zero failures.

Depends on

None. Standalone fix branched off upstream/master.

Efficacy validation

JIT / emit churn eliminated in production; end-to-end RSS-floor confirmation received 2026-08-21 from the same peer that filed the original diagnosis (tempco-investigation-server session, ≥ 22 h post-deploy on tim-001, matched-window controlled comparison against unpatched tempco-001). IL bytes / method transitioned from 22 (emitted accessor stubs) to 108 (ordinary real methods); working set 292–300 MB stable.

Same-host matched-process-age comparison on tim-001 (both DLLs patched, deployed 2026-08-20T23:05:21Z):

Metric	Pre-fix (cppagent unpatched)	Post-fix (both patched)
Methods/s at ~3 min uptime	198.2	14.2
Methods/s at ~6–10 min	160–205 (at 17–21 min)	0.6–1.3
Total jitted at 3 min	56,254	27,815
Total jitted at 10–20 min	231,709 at 20 min	28,669 at 10 min (plateaued)
Post-fix delta over 5 min	—	+192 methods (28,477 → 28,669)

~225× reduction; curve is FLAT, not merely lower.

Supporting signals:

• IL bytes / method: 22 (emitted accessor stubs) → 108 (ordinary real methods) — remaining churn is tiered-compilation warm-up, not emit churn.
• dotnet-trace LCG DynamicMethod count over a 20 s window: 5 862 → 1 276 (post-fix sample still warming when captured; has since flatlined).
• GC heap unchanged in character (26–28 MB); working set 292–300 MB stable across the window.

Assembly hygiene

Rebuilt cppagent DLL reports .ver 7:0:0:0 identical to the shipped assembly; informational version unchanged; no strong-name change. comm on the .assembly extern sets between pre-fix and post-fix returns empty — no new external references introduced on either assembly. Drop-in binary-compatible.

End-to-end RSS-floor confirmation (2026-08-21)

Controlled comparison, both hosts, identical wall-clock window (08:00–17:00 CDT, 13:00–22:00Z Chicago manufacturing shift), trough-based least squares on 15-min post-GC minima:

	tim-001 (patched)	tempco-001 (control, unpatched)
Working-day window (13–22Z)	+0.96 ± 0.52 MB/h	+4.47 ± 0.47 MB/h
Last 6 h	+2.44 ± 0.29 MB/h	+4.21 ± 0.82 MB/h
Full run	+1.14 ± 0.13 MB/h (22.5 h)	+3.19 ± 0.05 MB/h (95 h)

Working-day confidence intervals do not overlap. The control host resumed climbing when the shift started; the patched host did not.

Same-host before/after on tim-001 (cleanest measure — same machine, same config, same workload conditions):

Window	RSS-floor growth
PRE-FIX (34 h, cppagent unpatched)	+3.5 ± 0.4 MB/h
POST-FIX (22.5 h, both patched)	+1.14 ± 0.13 MB/h

67 % reduction in RSS-floor growth. Container-limit cap projection moves from ~22 days to ~61 days (against the 2 GB container ceiling).

JIT side-by-side, workload-independent, taken minutes apart:

Metric	tim-001 patched (11.4 h up)	tempco-001 control (84 h up)
Methods jitted (total)	31 523	20 458 426
Rate	0.3 /s	30.4 /s
IL bytes jitted	0.06 MB/h	1.43 MB/h
GC heap	39 MB	124 MB
Working set	330 MB	597 MB

649× difference in total methods compiled. The patched host's total is essentially just its own startup JIT and has been flat for a day; the control added ~2.8 M methods in the 24 h between measurements.

Residual grower caveat. tim-001's last-6h window reads +2.44 ± 0.29 MB/h — above its own 22-hour average, and JIT cannot account for it (0.3 methods/s = 0.06 MB/h). The residual is entirely anonymous memory (+1.10 MB/h anon out of +1.11 MB/h rss, decomposed against /proc/self/status + smaps; file/shmem/threads/fds/sockets all flat over the 22.5 h window). With the managed GC heap at 39 MB and methods-jitted at 0.3/s, loader/code heap is not growing either — so the residual mechanism is distinct from the reflection-emit class this PR closes. Every sibling JsonSerializerOptions construction site on MTConnect.NET-Common was independently verified as already using the static readonly singleton pattern (either landed by this PR or pre-existing), consistent with the peer's finding. The residual is most likely native-side (glibc arena fragmentation, native allocator growth, or a native-side buffer) and is being characterised in a separate investigation. This PR eliminates the dominant reflection-emit leak (67 % reduction in RSS-floor growth, 649× reduction in cumulative JIT); the residual is out of scope + tracked separately.

Measurement discipline: RestartCount=0 throughout, no memory.events max/oom on either host, no container hit its limit, and tim-001 published normally the whole time (30 mtconnectMqttSink writes/60 s).

Dime review cycles

Cycle 1 — baseline pass on b2703730

• code-review: [BLOCKER] H1 — the per-call new JsonSerializerOptions on every Convert / ConvertBytes / ConvertStream invocation re-emits LCG DynamicMethod property accessors on every serialization, matching the peer's tempco-001 / tim-001 heap-outside-GC diagnosis (+3.2–3.8 MB/h RSS climb). Fix: cache the options as static readonly singletons. → Fixed @ f1565222 (the initial cache-singletons commit).
• code-review: [FINDING] M1 — the shared singleton must be frozen (MakeReadOnly on net8+) so a caller's stray Converters.Add fails fast rather than silently polluting the process-wide instance. → Fixed @ adfed47d.
• security-audit: NO FINDINGS.
• simplification: NO FINDINGS.
• improvement: NO FINDINGS.
• documentation-audit: NO FINDINGS.
• test-coverage-audit: [TEST] FLOOR gap — thread-safety, cold-path converter path, and Convert overload parity lacked regression pins. → Fixed @ b2703730 (coverage-FLOOR pin batch, TDD RED-first before the H1 fix).

Cycle 2 — verify freeze + warm

• code-review / security-audit / simplification / improvement: NO FINDINGS.
• documentation-audit: [DOCS] L1 — the new CreateOptions / GetOptions helpers lacked XML /// blocks explaining the hot-path / cold-path invariant. → Fixed @ c212f6fc.
• test-coverage-audit: NO FINDINGS.

Cycle 3 — verify L1 docs

• code-review / security-audit / simplification / improvement / documentation-audit: NO FINDINGS.
• test-coverage-audit: NO FINDINGS.
• documentation-audit: [DOCS] L2 — AmE spelling drift in the freshly-added doc comments (serialise, initialise, synchronise). → Fixed @ 38cbeb07.

Cycle 4 — deeper warm-up + AmE re-sweep

• improvement: [IMPROVE] M1-C3 — the initial warm-up used Serialize<object>(null, options), which only bootstrapped STJ's shared reflection resolver but never named a concrete MTConnect type; the cold LCG emit for the four top-level response envelopes still fell on the first user-facing request. → Fixed @ 432be963 (warm-up traverses MTConnect response graph via typed Serialize calls on JsonStreamsDocument / JsonAssetsDocument(null) / JsonDevicesDocument).
• documentation-audit: [DOCS] M2-C3 — repo-wide grep surfaced additional AmE-token drift (amortise, synchronise, flavour). → Fixed @ e55badca.
• test-coverage-audit: [TEST] FLOOR gap — no regression pin for the warm-up-before-MakeReadOnly static-ctor ordering invariant (a reorder would surface as NotSupportedException on the first real serialize). → Fixed @ 86b60a66.
• Cycle-4 leftovers surfaced by verification pass: [IMPROVE] F-IMP-001 MEDIUM (WarmReachableGraph omits ErrorResponseDocument) + LOW (DefaultOptions / IndentOptions <remarks> state the invariant but not the WHY / LCG loader-heap link) — deferred to cycle 5.

Cycle 5 — cycle-4 leftovers + Error warm-up correctness

• code-review / security-audit / documentation-audit: NO FINDINGS.
• improvement: [IMPROVE] F-IMP-001 M-C4 (Error warm-up missing) → Fixed @ 014347b7; [IMPROVE] L-C4 (WHY-LCG remarks) → Fixed @ 014347b7.
• improvement: [IMPROVE] F-IMP-C5-001 MEDIUM — cycle-5 Error warm-up populated only new ErrorResponseDocument() (null Header / Errors / Version), leaving the concrete MTConnectErrorHeader (9 accessors), Error (2), and System.Version (6) cold on the first real error response. → Fixed @ 6b081eac (populates Header = new MTConnectErrorHeader(), Errors = new[] { new Error() }, Version = new Version(2, 5) so STJ walks the runtime types the production /probe-error and parse-failure paths actually serialize).
• improvement: [IMPROVE] F-IMP-C5-002 LOW — WarmReachableGraph depends on the null-tolerance of the JsonAssetsDocument(IAssetsResponseDocument) ctor; a future null-guard would turn first assembly load into TypeInitializationException. → Fixed @ 6b081eac (adds an <remarks> block on the ctor documenting the null-tolerance contract, the dependent warm-up site, and the requirement to update both atomically in the same commit before changing the ctor contract).
• simplification: [SIMPLIFY] F-SIMP-001 LOW — the first <remarks> paragraph on IndentOptions is byte-identical to the paragraph on DefaultOptions in both files. → Closed with rationale: IntelliSense hover shows only the current member's remarks and does not chase <see cref/>; a consumer inspecting IndentOptions in isolation must see the invariant text self-contained, not as a pointer. Self-contained per-member docs are the maintainer preference.
• test-coverage-audit: [TEST] MEDIUM — cycle-5's initial Frozen_singleton_can_serialize_ErrorResponseDocument_proving_Error_warm_up_ran pin was tautological (mutation-verified: removing the warm-up line left the pin green; STJ's DefaultJsonTypeInfoResolver lazily populates JsonTypeInfo on frozen options for arbitrary types once TypeInfoResolver is set). → Fixed @ 5a196588 (replaced with an IL-inspection walker WarmReachableGraph_IL_contains_newobj_for_ErrorResponseDocument_ctor + companion IL pins for the three Json* top-level surrogates per fixture). [TEST] LOW — the five sibling per-call → static-readonly refactors (AgentConfiguration, AdapterApplicationConfiguration, MTConnectAgentInformation, MTConnectClientInformation, MTConnectAssetFileBuffer, MTConnectMqttMessage) had no test-time guard. → Fixed @ 5a196588 (JsonSerializerOptionsSiblingSingletonTests + MTConnectMqttMessageSingletonTests structural pins, sibling-mutation-verified).

Cycle 6 — verify cycle-5 fixes + walker match against populated shape

• security-audit / simplification / improvement / documentation-audit: NO FINDINGS.
• code-review: [FINDING] F-CR-C6-001 LOW — the populated Error warm-up hard-codes new Version(2, 5) where the repo already exposes MTConnectVersions.Version25; reads scan-and-recognize vs. wonder why "2.5" appears amid a 2.7-max library. → Fixed @ ec86e338 (swap to the canonical constant; unused using System; removed from cppagent JsonFunctions.cs).
• test-coverage-audit: [TEST] F-COV-C6-001 MEDIUM — cycle-5's IL walker only pinned the newobj for ErrorResponseDocument; a revert to new ErrorResponseDocument() (naked) would silently pass the pin while re-opening the exact LCG-emit-on-first-error class F-IMP-C5-001 closed. → Fixed @ e19674ec (adds WarmReachableGraph_IL_contains_newobj_for_concrete_Error_envelope_fields pin per fixture, asserting newobj producers for MTConnectErrorHeader + Errors.Error + System.Version; mutation-verified on bluefin).

Cycle 6 — chase-fix for F-CR-C6-001 side-effect

• All six agents: NO cycle-7-eligible findings, one same-cycle fix landed:
  • [TEST] chase-fix @ 31668585 — the F-CR-C6-001 swap to MTConnectVersions.Version25 eliminated the newobj System.Version in WarmReachableGraph (an ldsfld now loads the canonical constant), which would have false-positived the cycle-6 walker on System.Version. Widened the IL walker helper AssertWarmReachableGraphNewsUp in both fixtures to accept EITHER newobj T::.ctor OR ldsfld <static-field-of-type-T> — both producers are semantically equivalent for warm-up because STJ walks the runtime type of the value fed into Serialize regardless of source. <summary> blocks updated in both fixtures to name the widened contract and cross-reference F-CR-C6-001. Mutation-verified: deleting Version = MTConnectVersions.Version25 fires the pin with the exact System.Version failure message; restoring returns to green.

Cycle 7 — full-diff re-sweep at HEAD 31668585

• code-review: NO FINDINGS. Verified: no circular init risk from JsonFunctions.cctor referencing MTConnectVersions.Version25 (leaf static class, no MTConnect deps); widened walker's ldsfld branch endianness sound on every .NET-supported target; sibling singleton adopters follow the pattern with cross-referencing comments back to JsonFunctions.cs.
• security-audit: NO FINDINGS. Verified: Module.ResolveField(token) operates on the test project's own compiled assembly (no untrusted metadata); no secret exposure in walker error strings; no dependency-manifest change.
• simplification: NO FINDINGS. Verified: comment blocks on both WarmReachableGraph methods (~28 lines each) trace to distinct closed-finding rationales; the dual-opcode walker branch is clearer as-is than split into two helpers.
• improvement: NO FINDINGS. Verified: MTConnectVersions.Version25 field-initializer chain has no cycle; widened walker adds zero production-side overhead; using System; correctly retained in plain-JSON (needed for DateTime / TimeSpan in GetTimestamp), correctly removed from cppagent (no unqualified System.* reference remains).
• documentation-audit: NO FINDINGS. Verified: no docs page references the removed magic new Version(2, 5) literal; both updated <summary> blocks accurately name the widened walker contract and cross-reference F-CR-C6-001; no AmE-token drift in the cycle-6/7 prose.
• test-coverage-audit: NO FINDINGS. Verified: mutation-tests confirm the widened walker catches every intended regression class (RED on delete of the Version producer, GREEN on restore); nested-field-of-same-type analysis returns no false-positive class.

(Zero unfixed findings — Ready-eligible.)

Test results at HEAD 31668585 (bluefin 2026-08-21, dotnet test per project):

Project	Pass / Total
MTConnect.NET-JSON-Tests	82 / 82
MTConnect.NET-JSON-cppagent-Tests	382 / 382
MTConnect.NET-Common-Tests	4085 / 4085
MTConnect.NET-AgentModule-MqttRelay-Tests	63 / 63

Total 4612 tests, zero failures. Net +11 pins over cycle 1 (Error warm-up IL walker + concrete-field pin + sibling singleton pins + MQTT singleton pin + widened walker <summary> updates). All 14 commits upstream/master..HEAD signed (%G? = G).