Mu2eG4MT: suppress spurious fatal Cache001 at end of job

[oksuzian]

Summary

Mu2eG4MT jobs abort stochastically (~1-of-2 at 100 events / 2 schedules with ShieldingM_EMZ) after the output file has closed, with:

*** ExceptionHandler is not defined ***
*** G4Exception : Cache001
      issued by : G4CacheReference<V>::Destroy
Internal fatal error. Invalid G4Cache size (requested id: 460 but cache has size: 459 ...

The art file on disk is complete and correct — only the exit status lies, which is fatal for grid production (good jobs marked failed). This is the failure discussed on the mu2e lists in Aug 2025 (Genser/Brown/Culbertson), present since Geant4 ~11.2, and one of the reasons MT was retired from production fcls (Production#449 context).

This PR makes the job exit cleanly by installing, at Mu2eG4MT::endJob, a G4VExceptionHandler that suppresses exactly Cache001 and keeps default fatal behavior for everything else.

Root cause (gdb backtrace)

The crash is not in Mu2e's run-manager teardown. It happens on the main art thread inside __run_exit_handlers():

#5  G4CacheReference<G4ParticleHPAngular::toBeCached>::Destroy  (G4CacheDetails.hh:170)
#6  G4Cache<G4ParticleHPAngular::toBeCached>::~G4Cache
#7  G4ParticleHPAngular::~G4ParticleHPAngular
#8  G4ParticleHPInelasticCompFS::~G4ParticleHPInelasticCompFS
...
#16 G4HadronicInteractionRegistry::Clean
#18 G4ThreadLocalSingleton<G4HadronicInteractionRegistry>::Clear
#20 __run_exit_handlers

ParticleHP model instances are owned by per-thread G4HadronicInteractionRegistry singletons, destroyed only by a G4ThreadLocalSingleton static destructor at process exit — on whichever thread calls exit(). Their G4Cache<V> members were created on G4 worker threads; G4Cache backing stores are thread-local vectors grown lazily, so the main thread's store (size 459 here) is shorter than the worker-created ids (460–3260) and Destroy raises the fatal.

• Native Geant4 MT never sees this: there the main thread is the G4 master, and its store is fully grown. It bites frameworks (art) that run all G4 work on secondary threads — matching K. Genser's observation that it is Mu2e-specific.
• The ~50% rate: TBB sometimes executes one schedule's G4 work on the main thread, which grows the main store.

Why suppression is correct (not error-hiding)

~G4Cache running on thread X can only ever free X's own slot; the values in other threads' stores are unreachable by construction. An id beyond X's store means X has no slot — there is nothing to free and nothing to corrupt. When a G4VExceptionHandler consumes the exception, Destroy takes Geant4's own written handled-exception path: return; without touching the store. At process exit that is exactly the right action (the OS reclaims the per-thread values). Geant4 half-acknowledges the scenario: the exception text guesses it, and the 11.5.0.beta rewrite of this file already made the sibling case (cache() == nullptr) silently tolerated.

Scoping: the handler is installed only at endJob, so a genuine mid-run Cache001 (real API misuse) still aborts under the default handler. Only Cache001 is swallowed; all other fatal exceptions still abort. The handler object is deliberately leaked so it survives into exit(). Logs the first suppressed occurrence once, then stays silent.

Validation (v13_12_10 + Run1Bak backing, ceSimReco 100 events, MT 2 threads x 2 schedules)

check	result
14 MT + 2 ST runs, exit status	16/16 exit 0, "Art has completed"
Cache001 raised (and suppressed)	5/14 MT runs; one run showed 2801 contiguous ids (460–3260) — the historical abort-on-first hid 2800 further would-be fatals
output identity, suppressed-vs-clean MT pairs	bit-identical (g4 StepPointMC and StrawDigi hashes, all events)
ST pairs	bit-identical, handler never fires

The real fix is 3 lines in Geant4 — recipe for Mu2e's spack-built G4

Mu2e compiles its own Geant4 through spack (the p094 envset links spackages/.../geant4-11.3.2-<hash>), so the G4-level fix can be deployed centrally without waiting for CERN. The patch makes the size-mismatch branch of G4CacheReference<V>::Destroy (both specializations) a silent no-op instead of a FatalException — the identical post-state to this PR's handler, so the validation above covers it:

g4cachedetails-teardown-tolerant.patch (applies cleanly to 11.3.2 and 11.4.0 — the file is byte-identical in both; 11.5.0.beta rewrote the file and needs a trivial rebase to its three specializations)

--- a/source/global/management/include/G4CacheDetails.hh	2026-08-26 20:24:11.346603006 -0500
+++ b/source/global/management/include/G4CacheDetails.hh	2026-08-26 21:06:53.338193968 -0500
@@ -162,13 +162,12 @@
 #endif
     if(cache()->size() < id)
     {
-      G4ExceptionDescription msg;
-      msg << "Internal fatal error. Invalid G4Cache size (requested id: " << id
-          << " but cache has size: " << cache()->size();
-      msg << " Possibly client created G4Cache object in a thread and"
-          << " tried to delete it from another thread!";
-      G4Exception("G4CacheReference<V>::Destroy", "Cache001", FatalException,
-                  msg);
+      // This cache object is being destroyed on a thread whose thread-local
+      // store never grew a slot for it (e.g. exit-time destruction, via
+      // G4ThreadLocalSingleton, of objects created on worker threads).
+      // There is nothing to release on this thread; values in other threads'
+      // stores are unreachable here by construction and are reclaimed at
+      // process exit.
       return;
     }
     if(cache()->size() > id && (*cache())[id] != nullptr)
@@ -234,13 +233,12 @@
 #endif
     if(cache()->size() < id)
     {
-      G4ExceptionDescription msg;
-      msg << "Internal fatal error. Invalid G4Cache size (requested id: " << id
-          << " but cache has size: " << cache()->size();
-      msg << " Possibly client created G4Cache object in a thread and"
-          << " tried to delete it from another thread!";
-      G4Exception("G4CacheReference<V*>::Destroy", "Cache001", FatalException,
-                  msg);
+      // This cache object is being destroyed on a thread whose thread-local
+      // store never grew a slot for it (e.g. exit-time destruction, via
+      // G4ThreadLocalSingleton, of objects created on worker threads).
+      // There is nothing to release on this thread; values in other threads'
+      // stores are unreachable here by construction and are reclaimed at
+      // process exit.
       return;
     }
     if(cache()->size() > id && (*cache())[id] != nullptr)

Recipe (for the spack maintainer):

1. Drop the patch into the geant4 package dir of the spack repo that builds Mu2e's G4: $SPACK_ROOT/var/spack/repos/builtin/packages/geant4/
2. Register it in package.py: patch('g4cachedetails-teardown-tolerant.patch', when='@11.2:11.4')
3. Rebuild with the same spec/variants and republish the muse env (muse-al9-prof-e29-p094-style) — note the header is template code compiled into libG4processes, so editing the installed header alone does nothing; the full geant4 rebuild is required.
4. This PR's handler is complementary: it simply never fires under a patched G4.

Upstream status

• Not fixed upstream: 11.4.0's G4CacheDetails.hh is byte-identical to 11.3.2; the 11.5.0.beta rewrite tolerates only the empty-store case and keeps the size-mismatch fatal.
• Only tracker entry is Bugzilla #2424 (GPS flavor of the same defect, ASSIGNED, stale since 2021); also geant4-forum 6045. A fresh report with this reproducer + backtrace is planned.

Relation to #1951

Independent of the MT reproducibility fix (#1951): that one makes MT output deterministic; this one makes MT jobs exit cleanly. Both are needed to make Mu2eG4MT usable in production; neither changes single-threaded behavior.

🤖 Generated with Claude Code

https://claude.ai/code/session_01R5HnfdYMwXXrJGAkYVE48c

[FNALbuild]

Hi @oksuzian,
You have proposed changes to files in these packages:

• Mu2eG4

which require these tests: build.

@Mu2e/write, @Mu2e/fnalbuild-users have access to CI actions on main.

⌛ The following tests have been triggered for 3f2f82e: build (Build queue - API unavailable)

About FNALbuild. Code review on Mu2e/Offline.

[FNALbuild]

☀️ The build tests passed at 3f2f82e.

Test	Result	Details
test with	✅	Command did not list any other PRs to include
merge	✅	Merged 3f2f82e at 8af4faf
build (prof)	✅	Log file. Build time: 08 min 53 sec
ceSimReco	✅	Log file.
g4test_03MT	✅	Log file.
transportOnly	✅	Log file.
POT	✅	Log file.
g4study	✅	Log file.
cosmicSimReco	✅	Log file.
cosmicOffSpill	✅	Log file.
ceSteps	✅	Log file.
ceDigi	✅	Log file.
muDauSteps	✅	Log file.
ceMix	✅	Log file.
rootOverlaps	✅	Log file.
g4surfaceCheck	✅	Log file.
trigger	❓	Log file. Return Code 1.
check_cmake	✅	Log file.
FIXME, TODO	✅	TODO (0) FIXME (0) in 1 files
clang-tidy	➡️	2 errors 13 warnings
whitespace check	✅	no whitespace errors found

N.B. These results were obtained from a build of this Pull Request at 3f2f82e after being merged into the base branch at 8af4faf.

For more information, please check the job page here.
Build artifacts are deleted after 5 days. If this is not desired, select Keep this build forever on the job page.