What happened?
Every successful region teardown logs a batch of alarming WARNs that read like crashes, scaling with worker count:
| log line |
where |
when it fires on a healthy teardown |
actor X is not reachable anymore, it might have crashed. old ref = ... |
PekkoActorRefMappingService.scala:76-81 (removeActorRef) |
once per worker — RegionExecutionManager.scala:198 calls removeActorRef unconditionally for every worker of every completed region |
| same WARN again |
via the dead-letter branch WorkflowActor.scala (MessageBecomesDeadLetter → removeActorRef(dest)) |
again from every peer that still had an in-flight message to the stopped worker |
unknown identifier: ... |
PekkoActorRefMappingService.scala:122 |
on every send to a worker that was already removed (e.g. resent messages, late stats queries) |
So an N-worker region emits ≥N "might have crashed" WARNs on the normal completion path — before any real problem exists. This trains operators (and CI log readers) to ignore WARNs, and it buries genuine crash evidence. It is the largest remaining contributor to teardown log noise now that the EndWorker retry storm is fixed (#6916).
Note the logger names here are Amber-style (<actorId>] [<Class>), which never match package-prefix logback filters like org.apache=WARN, so this cannot be silenced by logback configuration — the call sites themselves need to distinguish "removed deliberately during teardown" from "vanished unexpectedly" (e.g. a removeActorRef(id, expected = true) variant logged at debug/info).
How to reproduce?
Run any workflow to completion and grep the coordinator log for might have crashed — one WARN per worker per region despite a fully successful run. LoopIntegrationSpec on CI shows this repeatedly (once per loop iteration per region).
Version/Branch
main (observed at 429be11; related to the log-noise follow-ups in #6796 and the fix in #6916).
What happened?
Every successful region teardown logs a batch of alarming WARNs that read like crashes, scaling with worker count:
actor X is not reachable anymore, it might have crashed. old ref = ...PekkoActorRefMappingService.scala:76-81(removeActorRef)RegionExecutionManager.scala:198callsremoveActorRefunconditionally for every worker of every completed regionWorkflowActor.scala(MessageBecomesDeadLetter→removeActorRef(dest))unknown identifier: ...PekkoActorRefMappingService.scala:122So an N-worker region emits ≥N "might have crashed" WARNs on the normal completion path — before any real problem exists. This trains operators (and CI log readers) to ignore WARNs, and it buries genuine crash evidence. It is the largest remaining contributor to teardown log noise now that the EndWorker retry storm is fixed (#6916).
Note the logger names here are Amber-style (
<actorId>] [<Class>), which never match package-prefix logback filters likeorg.apache=WARN, so this cannot be silenced by logback configuration — the call sites themselves need to distinguish "removed deliberately during teardown" from "vanished unexpectedly" (e.g. aremoveActorRef(id, expected = true)variant logged at debug/info).How to reproduce?
Run any workflow to completion and grep the coordinator log for
might have crashed— one WARN per worker per region despite a fully successful run.LoopIntegrationSpecon CI shows this repeatedly (once per loop iteration per region).Version/Branch
main (observed at 429be11; related to the log-noise follow-ups in #6796 and the fix in #6916).