Skip to content

chore: configure Logger overload handling - #3404

Open
thecristen wants to merge 4 commits into
mainfrom
cbj/logs
Open

chore: configure Logger overload handling#3404
thecristen wants to merge 4 commits into
mainfrom
cbj/logs

Conversation

@thecristen

Copy link
Copy Markdown
Collaborator

High load is already being handled by Logger to some extent -- whenever we see the following types of messages, the logging mechanism is adjusting its approach (and messages do get dropped in extreme cases):

[notice] node=dotcom@ip-10-1-17-70 Handler :default switched from :sync to :drop mode
[notice] node=dotcom@ip-10-1-79-229 Handler :default switched from :async to :drop mode�

This PR explores the question "what if the system is overwhelmed by too many logs?"

Implementation

  • Lower the thresholds required to shift between async, sync, and drop modes
  • Enable terminating overloaded handlers
    • Uses default value for memory of 3mb
    • Uses low value of 500 for queue length. Dotcom never sees a queue length that big outside of extreme events.

Outside of the Logger configuration itself,

  • I realized we were using Logster to format our logs, nice! But that also means we could disable Phoenix's default logging. I realized we could upgrade here -- despite it being a pre-major version change I felt like it would be safe... Screens already uses it in prod and the Logster version has been in place for years (a very slowly-developing library, it seems)
    • During this update I realized its default formatter worked the same as our custom one, so I have removed our custom one

Screenshots

No change!

How to test

We probably should deploy this to a staging server and make sure it logs what we expect.

@thecristen
thecristen requested a review from a team as a code owner August 4, 2026 21:50
@thecristen
thecristen requested a review from lvachon1 August 4, 2026 21:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant