From e06e865549cb07f26020097e330738b17235df7b Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 8 Jul 2026 16:52:14 -0400 Subject: [PATCH 01/10] DOC-226: Document per-entity client quota metrics (26.2) Adds a "Track quota use per entity" subsection to the Monitor client throughput section, documenting ENG-37 (redpanda PR #30832): - kafka_per_entity_quota_metrics cluster property (opt-in, no restart) - The two new per-entity counters on /public_metrics and /metrics - Entity/quota-type label scheme and cardinality bounding behavior - Utilization calculation guidance with a PromQL example Builds on the existing aggregate-metrics content rather than duplicating it. Gated to Self-Managed pending confirmation of Cloud support for the new cluster property. DRAFT-VERIFY comments mark items to confirm against a 26.2 build. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../manage-throughput.adoc | 57 +++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index ce1e6da001..421e6a4767 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -727,6 +727,63 @@ TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context. ---- endif::[] +// DRAFT-VERIFY (DOC-226): This subsection documents ENG-37 (redpanda PR #30832, merged to +// dev 2026-06-24, targets v26.2). Verify all flagged items against a 26.2 build before +// publishing. SME: Mateusz Najda (@mnajda-redpanda). +// DRAFT-VERIFY (DOC-226): Confirm whether Redpanda Cloud exposes the +// kafka_per_entity_quota_metrics cluster property. Gated to Self-Managed until confirmed. +ifndef::env-cloud[] +==== Track quota use per entity + +The quota metrics above aggregate throttling and throughput by quota rule and quota type, so they cannot tell you which specific client is approaching its limit. Per-entity quota metrics label each series with the throttled entity's identity, so you can measure how much of an enforced quota each client actually uses, identify throttled clients by name, and alert on a workload before it saturates its quota. + +Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: + +[,bash] +---- +rpk cluster config set kafka_per_entity_quota_metrics true +---- + +The change takes effect without a broker restart. + +When enabled, Redpanda exposes two additional counters: + +* Total per-entity quota throttling delay, in milliseconds: +** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throttle_time_ms_by_entity` +** `/metrics` - `vectorized_kafka_quotas_client_quota_throttle_time_ms_by_entity` +* Per-entity quota throughput (bytes for produce and fetch quotas, partition mutations for partition mutation quotas): +** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` +** `/metrics` - `vectorized_kafka_quotas_client_quota_throughput_by_entity` + +// DRAFT-VERIFY (DOC-226): Confirm the exposed internal metric names on a 26.2 build, then +// replace the plain code spans above with xrefs to the regenerated metrics reference pages +// (anchors do not exist until /gen-metrics runs for 26.2). + +Each series is labeled with the entity identity and the quota type: + +* `redpanda_quota_type`: Always present. One of `produce_quota`, `fetch_quota`, or `partition_mutation_quota`. +* `redpanda_quota_user`: The user principal, for user-based quotas. +* `redpanda_quota_client_id`: The client ID, for client ID-based quotas. +* `redpanda_quota_group_name`: The client ID prefix, for quotas that apply to a <>. +// DRAFT-VERIFY (DOC-226): Confirm the redpanda_quota_group_name label corresponds to +// client-id-prefix quotas (not consumer groups) on a live build. + +Combined quotas, such as a user with a specific client ID, include each matching label on the same series. + +To keep metric cardinality bounded, Redpanda registers a per-entity series only while an entity is actively being throttled, and removes the series after the entity has been idle. An entity appears in these metrics only after it has been throttled at least once. From that point, Redpanda records the entity's throughput on every request, not only on throttled requests, so a Prometheus `rate()` query reflects the entity's actual throughput. + +To measure how much of its quota an entity is using, divide the entity's throughput rate by its configured limit. For example, the following query returns the produce throughput rate for each throttled client ID: + +[,promql] +---- +sum by (redpanda_quota_client_id) ( + rate(redpanda_kafka_quotas_client_quota_throughput_by_entity{redpanda_quota_type="produce_quota"}[5m]) +) +---- + +Compare the result with the entity's configured limit from xref:reference:rpk/rpk-cluster/rpk-cluster-quotas-describe.adoc[`rpk cluster quotas describe`]. A ratio close to 1 means the client is saturating its quota and its requests are being delayed. +endif::[] + == See also - xref:manage:cluster-maintenance/about-throughput-quotas.adoc[] From 2d65e6bf5e4be800a5bcf6c9bd5906c2930c03ed Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Wed, 8 Jul 2026 17:46:56 -0400 Subject: [PATCH 02/10] DOC-226: Rework per-entity metrics intro to contextualize the feature Intro now follows the situation / what-it-does / benefit / when-to-use formula instead of leading with the metrics mechanics. Co-Authored-By: Claude Opus 4.8 (1M context) --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 421e6a4767..c2587ad027 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -735,7 +735,7 @@ endif::[] ifndef::env-cloud[] ==== Track quota use per entity -The quota metrics above aggregate throttling and throughput by quota rule and quota type, so they cannot tell you which specific client is approaching its limit. Per-entity quota metrics label each series with the throttled entity's identity, so you can measure how much of an enforced quota each client actually uses, identify throttled clients by name, and alert on a workload before it saturates its quota. +When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions: Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name, alert on a workload before it saturates its quota, and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: From a2438ee2a1d6a832159c2585197b185a55bdf8be Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:13:27 -0500 Subject: [PATCH 03/10] Update modules/manage/pages/cluster-maintenance/manage-throughput.adoc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Gellért Peresztegi-Nagy --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index c2587ad027..f26651eec8 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -776,7 +776,7 @@ To measure how much of its quota an entity is using, divide the entity's through [,promql] ---- -sum by (redpanda_quota_client_id) ( +sum by (redpanda_quota_client_id, pod) ( rate(redpanda_kafka_quotas_client_quota_throughput_by_entity{redpanda_quota_type="produce_quota"}[5m]) ) ---- From dfe8eac3b9875825ab2abe9a911de347899e768b Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:21:24 -0500 Subject: [PATCH 04/10] Apply suggestion from @Feediver1 --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index f26651eec8..c6101b469d 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -735,7 +735,7 @@ endif::[] ifndef::env-cloud[] ==== Track quota use per entity -When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions: Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name, alert on a workload before it saturates its quota, and right-size quota values based on observed usage. +When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions: Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: From 1c728bc85b82f7d68221fdb6652a1523879154a8 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:50:26 -0500 Subject: [PATCH 05/10] Apply suggestion from @Feediver1 --- .../manage/pages/cluster-maintenance/manage-throughput.adoc | 3 --- 1 file changed, 3 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index c6101b469d..a7a900fee2 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -727,9 +727,6 @@ TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context. ---- endif::[] -// DRAFT-VERIFY (DOC-226): This subsection documents ENG-37 (redpanda PR #30832, merged to -// dev 2026-06-24, targets v26.2). Verify all flagged items against a 26.2 build before -// publishing. SME: Mateusz Najda (@mnajda-redpanda). // DRAFT-VERIFY (DOC-226): Confirm whether Redpanda Cloud exposes the // kafka_per_entity_quota_metrics cluster property. Gated to Self-Managed until confirmed. ifndef::env-cloud[] From 10934f0c5529e815dad7eee70d0f5de186a280eb Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:52:06 -0500 Subject: [PATCH 06/10] Apply suggestion from @Feediver1 --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index a7a900fee2..8d60eb5e0b 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -732,7 +732,7 @@ endif::[] ifndef::env-cloud[] ==== Track quota use per entity -When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions: Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name and right-size quota values based on observed usage. +When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions. Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name and right-size quota values based on observed usage. Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: From 08c515ed85a44acf0824b56f1346c92638f6a802 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:53:35 -0500 Subject: [PATCH 07/10] Apply suggestion from @Feediver1 --- .../manage/pages/cluster-maintenance/manage-throughput.adoc | 3 --- 1 file changed, 3 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 8d60eb5e0b..1f75912391 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -752,9 +752,6 @@ When enabled, Redpanda exposes two additional counters: ** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` ** `/metrics` - `vectorized_kafka_quotas_client_quota_throughput_by_entity` -// DRAFT-VERIFY (DOC-226): Confirm the exposed internal metric names on a 26.2 build, then -// replace the plain code spans above with xrefs to the regenerated metrics reference pages -// (anchors do not exist until /gen-metrics runs for 26.2). Each series is labeled with the entity identity and the quota type: From 85d38b34532bb60639b47e2ef8450aea5933f10e Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Fri, 10 Jul 2026 10:54:16 -0500 Subject: [PATCH 08/10] Apply suggestion from @Feediver1 --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 -- 1 file changed, 2 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 1f75912391..710053d561 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -759,8 +759,6 @@ Each series is labeled with the entity identity and the quota type: * `redpanda_quota_user`: The user principal, for user-based quotas. * `redpanda_quota_client_id`: The client ID, for client ID-based quotas. * `redpanda_quota_group_name`: The client ID prefix, for quotas that apply to a <>. -// DRAFT-VERIFY (DOC-226): Confirm the redpanda_quota_group_name label corresponds to -// client-id-prefix quotas (not consumer groups) on a live build. Combined quotas, such as a user with a specific client ID, include each matching label on the same series. From c5373b09c23a4ee07641ccb9c7576f6df131a1fa Mon Sep 17 00:00:00 2001 From: Joyce Fee Date: Thu, 16 Jul 2026 19:29:24 -0400 Subject: [PATCH 09/10] Un-gate per-entity quota metrics for Cloud (SME confirmed) Gellert confirmed on PR #1779 that per-entity quota metrics are exposed in Redpanda Cloud. Remove the DRAFT-VERIFY and the Self-Managed-only gating around the section, keeping the /metrics (internal) endpoints and the rpk enablement command Self-Managed-only; Cloud points to the cluster-properties configuration page instead. Also state the per-broker rationale for the pod aggregation in the example query, per the applied review suggestion. Co-Authored-By: Claude Fable 5 --- .../manage-throughput.adoc | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 710053d561..5d872ab78d 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -727,19 +727,23 @@ TRACE 2024-06-14 15:37:59,195 [shard 2:main] kafka_quotas - connection_context. ---- endif::[] -// DRAFT-VERIFY (DOC-226): Confirm whether Redpanda Cloud exposes the -// kafka_per_entity_quota_metrics cluster property. Gated to Self-Managed until confirmed. -ifndef::env-cloud[] ==== Track quota use per entity When a workload slows down because a client hits its throughput quota, the aggregate quota metrics can confirm that throttling is happening, but they cannot tell you which client is being throttled or how close each client is to its limit. Per-entity quota metrics answer these questions. Redpanda labels each throttle-time and throughput series with the identity of the throttled entity, so you can measure how much of an enforced quota each client actually uses. Use these metrics to identify throttled clients by name and right-size quota values based on observed usage. -Per-entity quota metrics are disabled by default because each throttled entity adds metric series. To enable them: +Per-entity quota metrics are disabled by default because each throttled entity adds metric series. + +ifndef::env-cloud[] +To enable them: [,bash] ---- rpk cluster config set kafka_per_entity_quota_metrics true ---- +endif::[] +ifdef::env-cloud[] +To enable them, set the `kafka_per_entity_quota_metrics` cluster property. See xref:manage:cluster-maintenance/config-cluster.adoc[]. +endif::[] The change takes effect without a broker restart. @@ -747,10 +751,14 @@ When enabled, Redpanda exposes two additional counters: * Total per-entity quota throttling delay, in milliseconds: ** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throttle_time_ms_by_entity` +ifndef::env-cloud[] ** `/metrics` - `vectorized_kafka_quotas_client_quota_throttle_time_ms_by_entity` +endif::[] * Per-entity quota throughput (bytes for produce and fetch quotas, partition mutations for partition mutation quotas): ** `/public_metrics` - `redpanda_kafka_quotas_client_quota_throughput_by_entity` +ifndef::env-cloud[] ** `/metrics` - `vectorized_kafka_quotas_client_quota_throughput_by_entity` +endif::[] Each series is labeled with the entity identity and the quota type: @@ -764,7 +772,7 @@ Combined quotas, such as a user with a specific client ID, include each matching To keep metric cardinality bounded, Redpanda registers a per-entity series only while an entity is actively being throttled, and removes the series after the entity has been idle. An entity appears in these metrics only after it has been throttled at least once. From that point, Redpanda records the entity's throughput on every request, not only on throttled requests, so a Prometheus `rate()` query reflects the entity's actual throughput. -To measure how much of its quota an entity is using, divide the entity's throughput rate by its configured limit. For example, the following query returns the produce throughput rate for each throttled client ID: +To measure how much of its quota an entity is using, divide the entity's throughput rate by its configured limit. Quota limits apply per broker, so aggregate by broker as well as by entity to keep the results comparable with the configured limits. For example, the following query returns the produce throughput rate for each throttled client ID on each broker: [,promql] ---- @@ -774,7 +782,6 @@ sum by (redpanda_quota_client_id, pod) ( ---- Compare the result with the entity's configured limit from xref:reference:rpk/rpk-cluster/rpk-cluster-quotas-describe.adoc[`rpk cluster quotas describe`]. A ratio close to 1 means the client is saturating its quota and its requests are being delayed. -endif::[] == See also From eb3a4029b8875655a21a6911de2bf8b2f678e307 Mon Sep 17 00:00:00 2001 From: Joyce Fee <102751339+Feediver1@users.noreply.github.com> Date: Mon, 20 Jul 2026 11:57:26 -0500 Subject: [PATCH 10/10] Update modules/manage/pages/cluster-maintenance/manage-throughput.adoc Co-authored-by: Michele Cyran --- modules/manage/pages/cluster-maintenance/manage-throughput.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc index 5d872ab78d..6b059ed831 100644 --- a/modules/manage/pages/cluster-maintenance/manage-throughput.adoc +++ b/modules/manage/pages/cluster-maintenance/manage-throughput.adoc @@ -734,7 +734,7 @@ When a workload slows down because a client hits its throughput quota, the aggre Per-entity quota metrics are disabled by default because each throttled entity adds metric series. ifndef::env-cloud[] -To enable them: +To enable them, run: [,bash] ----