chore(amber): reduce engine log verbosity and pin the CI log-level backstop#6797
chore(amber): reduce engine log verbosity and pin the CI log-level backstop#6797aglinxinyuan wants to merge 6 commits into
Conversation
…ine logs
Amber engine loggers are named "<actorId>] [<ClassName>" by AmberLogging so
the pattern renders both the actor id and the class. A side effect is that
these logger names start with the actor id, not "org.apache", so the
`org.apache=WARN` rule in logback.xml never matches them and every one of
these statements emits at the root INFO level. In loop/integration tests
workers are recreated each iteration, so per-ECM, per-worker, per-region
lines fire dozens of times and flood the CI console.
Demote the chatty per-message / per-worker-lifecycle / per-region logs to
DEBUG so they no longer print at the default INFO level (CI, local, and
prod), and demote three misleading WARNs that fire on normal operation:
* receive/process/send ECM (DataProcessor.scala, main_loop.py)
* register->Actor / "unknown identifier" / "might have crashed"
(PekkoActorRefMappingService.scala)
* replay-log conf, DP thread start/exit, "Starting the worker",
"Region N successfully terminated", operator completion
* ClientActor "cannot handle" fallthrough
Genuine fault paths are untouched (DP-thread ERROR, "Failed to fetch
actorRef" WARN, "Error when terminating region" WARN). As a CI backstop,
pin TEXERA_SERVICE_LOG_LEVEL and UDF_PYTHON_LOG_STREAMHANDLER_LEVEL to WARN
on the amber unit and integration test steps.
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Reduces CI console log volume by demoting high-frequency Amber engine logs (Scala + Python) from INFO/WARN to DEBUG and pinning CI log levels to WARN as a backstop.
Changes:
- Demoted hot-path per-worker/per-message engine logs from INFO/WARN to DEBUG across Amber Scala and Python runtime paths.
- Demoted selected WARN logs that occur during normal teardown/races to DEBUG to avoid noisy CI output.
- Added CI workflow environment overrides to force JVM/Python worker logs to WARN in unit/integration steps.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| amber/src/main/scala/org/apache/texera/amber/engine/common/client/ClientActor.scala | Demotes “cannot handle” log from WARN to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/promisehandlers/StartHandler.scala | Demotes worker start log to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/DataProcessor.scala | Demotes ECM receive/process/send and completion summary logs to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/worker/DPThread.scala | Demotes DP thread lifecycle logs to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/scheduling/RegionExecutionManager.scala | Demotes “successfully terminated” log to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/common/WorkflowActor.scala | Demotes replay-log config log to DEBUG. |
| amber/src/main/scala/org/apache/texera/amber/engine/architecture/common/PekkoActorRefMappingService.scala | Demotes registration / normal teardown race logs to DEBUG. |
| amber/src/main/python/core/runnables/main_loop.py | Demotes ECM receive/process/send logs to DEBUG. |
| .github/workflows/build.yml | Pins CI log level env vars to WARN for unit/integration steps. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Automated Reviewer SuggestionsBased on the
|
Codecov Report❌ Patch coverage is ❌ Your patch status has failed because the patch coverage (36.36%) is below the target coverage (60.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #6797 +/- ##
============================================
+ Coverage 75.54% 77.27% +1.73%
- Complexity 3452 3522 +70
============================================
Files 1161 1161
Lines 45921 45927 +6
Branches 5101 5101
============================================
+ Hits 34690 35491 +801
+ Misses 9613 8851 -762
+ Partials 1618 1585 -33
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
✅ No material benchmark regressions detected🟢 7 better · 🔴 0 worse · ⚪ 8 noise (<±5%) · 0 without baseline
Baseline detailsLatest main
Raw CSVconfig_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,490.27,200,128000,408,0.249,23149.05,35536.30,35536.30
1,100,10,64,20,2499.93,2000,1280000,800,0.488,124747.81,148107.36,148107.36
2,1000,10,64,20,21391.71,20000,12800000,935,0.571,1073256.47,1117888.49,1117888.49 |
- main_loop.py: switch the three per-ECM debug logs from eager f-strings
to loguru's lazy `{}`-placeholder form. loguru's _log short-circuits on
core.min_level before formatting, so the ChannelIdentity/command string
building is now skipped entirely when DEBUG is disabled (the default).
- PekkoActorRefMappingService: reword the demoted teardown log to a neutral
"removed actor ref for <id>" — removeActorRef also runs on normal graceful
region teardown, so the old "might have crashed" wording was misleading.
The Scala interpolated debug logs are left as-is: com.typesafe.scala-logging
3.9.6 Logger methods are macros that guard the message with isDebugEnabled,
so `s"..."` is not built when the level is disabled.
|
Let me review this. I may need some time |
loguru has no WARN level (only WARNING), so UDF_PYTHON_LOG_STREAMHANDLER_LEVEL=WARN made every spawned Python UDF worker raise `ValueError: Level 'WARN' does not exist` at startup, before it connected back to the JVM. PythonProxyClient then blocked forever on the port promise (Await.result, no timeout), so the amber-integration job hung until GitHub's 6h cap. logback (TEXERA_SERVICE_LOG_LEVEL) keeps WARN, which is its correct spelling. Also add a 40-min timeout to the amber-integration job so a worker-startup failure fails fast instead of burning a 6h runner.
aglinxinyuan
changed the title
|
Pushed 35203dc to fix a self-inflicted hang in this PR's own CI backstop. The Fix:
|
- build.yml: the WARN/WARNING pins on the two amber test steps now flip to DEBUG when the run has "Enable debug logging" set (runner.debug == '1'), so the demoted engine logs come back on demand without editing the workflow. - build.yml: lower the amber-integration job timeout from 40 to 25 minutes; the last 38 green runs on main took 9.0-16.4 min (avg 12.1), so 25 keeps cold-cache headroom while still failing ~14x faster than the 6h cap. - PekkoActorRefMappingService: reword the teardown log to "actor <id> is not reachable anymore. old ref = <ref>" per review - the mapping service cannot tell a graceful stop from a crash, so the message now infers neither.
The CI backstop sets TEXERA_SERVICE_LOG_LEVEL=WARN, which cluster.conf forwards into pekko.loglevel. Pekko only accepts OFF/ERROR/WARNING/INFO/ DEBUG (Logging.levelFor), so every ActorSystem creation printed "unknown pekko.loglevel WARN" with a LoggerException stack trace and fell back to ERROR - one per spec in the amber unit and integration jobs, the very noise this PR is meant to remove. The env knob cannot simply switch to WARNING: the same variable drives the logback root level in logback.xml, and logback silently falls back to DEBUG on names it does not know - flooding instead of quieting. Normalize at the single choke point instead: PekkoConfig.pekkoConfig (which every ActorSystem, prod and test, receives via AmberRuntime.pekkoConfig) now rewrites pekko.loglevel through normalizePekkoLogLevel (WARN -> WARNING, TRACE/ALL -> DEBUG), so one env knob drives both systems in either spelling. Covered by a new invariant test - the resolved loglevel must be pekko-valid whatever the env holds, red on WARN before this fix - plus unit tests for the mapping.
|
Pushed b67bf3a to fix a noise regression the WARN backstop itself introduced — the CI logs were full of this on every Root cause: Why not just set Fix: normalize at the single choke point. Test-first: added an invariant test to |
What changes were proposed in this PR?
Root cause.
AmberLoggingnames each logger"<actorId>] [<ClassName>"so the logback pattern[%logger]renders both the actor id and the class. The side effect is that these logger names start with the actor id (WF6-…,COORDINATOR), notorg.apache…, so the<logger name="org.apache" level="WARN"/>rule inamber/src/main/resources/logback.xmlnever matches the engine classes. They all fall through to the root logger and emit atINFO. In loop/integration tests, workers are recreated every iteration, so per-ECM / per-worker / per-region lines fire dozens of times and flood the CI console.This PR demotes those statements to
DEBUGso they no longer print at the defaultINFOlevel (CI, local, and prod), and demotes threeWARNs that fire on normal operation:receive/process/send ECM(2–3× per control message, fullChannelIdentitydump)DataProcessor.scala,main_loop.pyregister <id> -> Actor[…]PekkoActorRefMappingService.scala…is not reachable anymore, it might have crashed— fires on graceful teardownPekkoActorRefMappingService.scalaunknown identifier— registration racePekkoActorRefMappingService.scalaworker replay log writing conf,DP thread started/exits,Starting the worker.WorkflowActor,DPThread,StartHandlerRegion N successfully terminated.RegionExecutionManager.scala…completed, # of input ports…DataProcessor.scalaclient actor cannot handle …—//skipfallthroughClientActor.scalaGenuine fault paths are untouched:
DP Thread exists unexpectedly(ERROR),Failed to fetch actorRef(WARN), andError when terminating region(WARN, the failure branch — distinct from the demotedsuccessfully terminatedsuccess branch).As a CI backstop, the amber unit and integration test steps in
build.ymlpin the log level so any remainingINFOchatter is suppressed there even before the source demotions take effect. The two logging stacks spell the level differently, which is a trap worth calling out: logback takesTEXERA_SERVICE_LOG_LEVEL=WARN, but the spawned Python UDF worker uses loguru, whose only warning level isWARNING. Feeding loguruWARNraisesValueError: Level 'WARN' does not existat worker startup — before the worker hands its Flight port back to the JVM — and the JVM then blocks on the proxy handshake with no internal timeout, hangingamber-integrationuntil GitHub's 6 h runner cap. So the Python var is pinned toUDF_PYTHON_LOG_STREAMHANDLER_LEVEL=WARNING, and, symmetrically, logback keepsWARN(it falls back toDEBUGon an unknown level likeWARNING, which would flood logs instead).amber-integrationadditionally getstimeout-minutes: 40so any future worker-startup deadlock fails fast instead of burning a full runner.Net effect: the bulk of the per-iteration output from loop/integration specs disappears from the default-level log, while
DEBUGkeeps it available on demand.Any related issues, documentation, discussions?
Closes #6796
How was this PR tested?
The source demotions are a log-level-only change — no behavior changes — so they rely on the existing test suites run in CI. Verified additionally by inspection and by the CI backstop's own behavior:
ReplayLogGeneratorSpecmatches"cannot handle", but against an unrelatedRuntimeExceptionfromReplayLogGenerator, not theClientActorlog.)WARN/WARNINGenv does not break config specs.UdfConfigSpec's andPekkoConfigSpec's default-value assertions are guarded (ifUnset/!sys.env.contains(...)), andpekko.stdout-loglevelis hardcodedINFOincluster.conf(onlypekko.loglevelis env-driven), so its unguarded assertion still passes.logger.add(sys.stderr, level="WARN")raisesValueError: Level 'WARN' does not existwhilelevel="WARNING"succeeds (loguru's levels areTRACE/DEBUG/INFO/SUCCESS/WARNING/ERROR/CRITICAL); logback acceptsWARN. WithWARNING,amber-integrationruns green again — the ubuntu leg finishes the integration step in ~8.5 min, versus the earlierWARNruns that sat in that step ~3 h until cancelled.scalafixAll --checkis unaffected); the longest changed Scala line is 96 columns (< the 100maxColumn), soscalafmtCheckAllstays green.A follow-up sweep for other hot-path
INFO/WARNlogs (e.g.AsyncRPCClientnull control-reply, the range-shuffle partitioner,Tuplecast warnings) is captured in #6796 rather than expanded here.Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8 [1M context])