Skip to content

Fix Prometheus latency histograms mis-scaled by 1000x#693

Open
wbarnha wants to merge 1 commit into
masterfrom
claude/fix-260-prometheus-histogram-scale
Open

Fix Prometheus latency histograms mis-scaled by 1000x#693
wbarnha wants to merge 1 commit into
masterfrom
claude/fix-260-prometheus-histogram-scale

Conversation

@wbarnha

@wbarnha wbarnha commented Jul 19, 2026

Copy link
Copy Markdown
Member

Note: Before submitting this pull request, please review our contributing guidelines.

Description

Fixes #260.

Every latency Histogram in FaustMetrics (events_runtime_latency, producer_send_latency, producer_error_send_latency, assign_latency, rebalance_done_consumer_latency, rebalance_done_latency, http_latency, consumer_commit_latency) was created with no explicit buckets=, so prometheus_client defaulted to its second-scale DEFAULT_BUCKETS (0.00510.0).

Every one of these histograms is fed millisecond values via Monitor.ms_since() (see the various PrometheusMonitor.on_* methods). Real latency observations — typically tens to hundreds of milliseconds — mostly land in the +Inf overflow bucket instead of any meaningful bucket, silently mis-scaling every latency dashboard/alert built on Faust's Prometheus sensor by 1000x.

Fix

Add MS_LATENCY_BUCKETS, a millisecond-scale bucket tuple with the same shape as Histogram.DEFAULT_BUCKETS scaled x1000, and pass it explicitly to all 8 latency histograms.

Tests

  • Parametrized test asserting every latency histogram's bucket boundaries equal MS_LATENCY_BUCKETS.
  • A direct test that a 250ms observation lands in the 250ms bucket rather than the overflow bucket.
  • Verified both fail without the fix (ImportError on the missing MS_LATENCY_BUCKETS symbol; KeyError on the 250.0 bucket boundary, since the old default buckets only had a 0.25 boundary) — confirming the tests aren't vacuous.
  • flake8, black --check, isort --check-only clean.
  • Full tests/unit/sensors suite passes (118 passed), excluding the pre-existing test_datadog.py errors caused by the datadog package not being installed in this sandbox — unrelated to this change.

🤖 Generated with Claude Code


Generated by Claude Code

Every latency Histogram in FaustMetrics (events_runtime_latency,
producer_send_latency, producer_error_send_latency, assign_latency,
rebalance_done_consumer_latency, rebalance_done_latency, http_latency,
consumer_commit_latency) was created with no explicit `buckets=`, so
prometheus_client defaulted to its second-scale DEFAULT_BUCKETS
(0.005-10.0). Every one of these histograms is fed millisecond values via
Monitor.ms_since() (see PrometheusMonitor.on_*), so real latency
observations (typically tens to hundreds of milliseconds) mostly land in
the `+Inf` overflow bucket instead of a meaningful bucket -- silently
mis-scaling every latency dashboard/alert built on Faust's Prometheus
sensor by 1000x.

Add MS_LATENCY_BUCKETS, a millisecond-scale bucket tuple with the same
shape as Histogram.DEFAULT_BUCKETS (scaled x1000), and pass it explicitly
to all 8 latency histograms.

Fixes #260.

Tests: parametrized check that every latency histogram's bucket boundaries
equal MS_LATENCY_BUCKETS, plus a direct check that a 250ms observation
lands in the 250ms bucket rather than the overflow bucket. Verified both
fail without the fix (missing MS_LATENCY_BUCKETS import; KeyError on the
250.0 bucket, since the old default buckets only had a 0.25 boundary).
flake8/black/isort clean; full tests/unit/sensors suite passes (excluding
the pre-existing, unrelated test_datadog.py errors caused by the datadog
package not being installed in this sandbox).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HHPL4VFWQRQPpjR1gXSKyL
@codecov

codecov Bot commented Jul 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 94.15%. Comparing base (3073eb9) to head (5877547).

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #693   +/-   ##
=======================================
  Coverage   94.14%   94.15%           
=======================================
  Files         104      104           
  Lines       11136    11137    +1     
  Branches     1201     1201           
=======================================
+ Hits        10484    10486    +2     
+ Misses        551      550    -1     
  Partials      101      101           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prometheus histogram buckets are in seconds, but values are milliseconds

1 participant