Design: managed metrics collection by dennis-upbound · Pull Request #363 · modelplaneai/modelplane

dennis-upbound · 2026-07-21T17:51:56Z

Description of your changes

A design doc, not code yet — opening as a draft for direction before implementation.

Collecting a deployment's metrics is hand-wired today (#269): vLLM exposes /metrics on its serving port, the serving stack runs a Prometheus per workload cluster with open PodMonitor discovery, and an operator writes a PodMonitor by hand, keeps it in sync with the serving shape (leader/worker, prefill/decode), and removes it on teardown (the #264 example).

This proposes having Modelplane compose the collection, per source it owns, on by default with an opt-out (spec.template.spec.metrics.enabled):

Engine — compose-model-replica composes a PodMonitor selecting the replica's serving pods, scraping the engine's /metrics by a named port. Naming the port fixes a latent prefill/decode gap: the decode engine serves on 8001 (the pd-sidecar takes 8000), so the manual targetPort: 8000 scrapes the sidecar, not the engine.
Endpoint picker — wherever an EPP is composed (multi-pod Unified, prefill/decode), collect its llm_d_epp_* metrics too. Since we own the EPP Deployment (llm-d-router-endpoint-picker:v0.9.0), we set --metrics-endpoint-auth=false and scrape :9090 with a plain PodMonitor — no ClusterRole, token, or TLS.

Managed lifecycle and shape-syncing fall out of the existing serving label and open Prometheus discovery.

Where I'd like judgment (in the doc): the metrics field shape and default-on, the engine port name (http vs metrics), and disabling EPP metrics auth vs composing RBAC. Two open questions are flagged.

design/metrics.md only; no code.

I have:

Read and followed Modelplane's contribution process.
~~Run nix flake check (or ./nix.sh flake check) and made sure it passes.~~ (design doc only, no code paths)
~~Added or updated tests covering any composition function changes.~~ (design doc only)
Signed off every commit with git commit -s.

Collecting a deployment's metrics is hand-wired: an operator writes a PodMonitor per deployment, keeps it in sync with the serving shape (leader/worker, prefill/decode), and removes it on teardown. This design has Modelplane compose the collection instead, per source it owns, the engine and the endpoint picker where present, on by default with an opt-out. It covers modelplaneai#269, with a section per source and a diagram. Signed-off-by: Dennis Ramdass <dennis@upbound.io>

haarchri · 2026-07-21T18:51:50Z

+
+## Interaction with #264
+
+The [#264](https://github.com/modelplaneai/modelplane/issues/264) example


In Release notes we need to add a line telling existing users to delete their hand-written podmonitor.yaml, or they get a duplicate scrape job against the same pods after upgrade.

Yes, without deleting it they'd get a duplicate scrape of the same pods after upgrade. Called it out as a release note in the #264 interaction section. Pushed.

haarchri · 2026-07-21T18:52:50Z

+
+- **Port name.** `http` (the serving port that also serves `/metrics`) versus
+  `metrics`. Leaning `http`, since it's the one serving port.
+- **Configurability.** `interval` and `path` are fixed (30s, `/metrics`) for now.


wonder if 30s is too Long If it comes to KV Cache utilization e.g.

For routing this interval doesn't matter: the EPP scrapes engine /metrics on its own fast internal loop (~100ms) to make routing decisions, so routing isn't gated on it. This PodMonitor feeds Prometheus for dashboards and alerting, where 30s is a normal default. If finer KV series are wanted, interval is a knob on the metrics object (open question). Made that distinction explicit in What gets scraped.

haarchri · 2026-07-21T19:01:10Z

+example). Nothing owns that wiring. The operator builds it by hand and keeps it in
+sync with the deployment's shape. They delete it on teardown.
+
+Instead, Modelplane composes the collection, per source:


is Monitoring for Cache / PVC here out of scope?

Out of scope here, yes, this doc is serving metrics (engine + EPP). ModelCache PVC and hydration observability is a separate concern that belongs with the ModelCache work. Added a scope line to the summary.

haarchri · 2026-07-21T19:04:30Z

+
+### What gets scraped
+
+The `PodMonitor` ingests everything the engine exposes on `/metrics`. What the


what are the expected series-per-pod ? is the Prometheus Stack configured with rolling / retention of the storage ?

Order of dozens of series per engine pod (request/latency histograms, KV-cache, throughput), plus the llm_d_epp_* set wherever an EPP runs. Retention and storage sizing are the serving stack's Prometheus config in compose-serving-stack, not this composition, worth a sane default there but out of scope for this doc. Noted it in What gets scraped.

haarchri · 2026-07-21T19:06:17Z

+  replicas: 1
+  template:
+    spec:
+      metrics:


consider observability/monitoring as the object name if it's going to grow past scraping... later

Agree, and it lines up with #77 (ModelService.observability.traces). Renamed to spec.template.spec.observability.metrics.enabled so traces and logs can slot beside metrics later. (Metrics live on the deployment since we scrape its pods; #77's traces live on ModelService, different resource, same observability umbrella.) Pushed.

Superseded by the reframe: after Nic's review the doc drops the per-deployment opt-out field entirely. Collection is now always-on at every layer (engine, EPP, serving-stack) with no opt-in or opt-out, feeding a central Modelplane Prometheus. So there's no longer a field to name under observability. Leaving this for context; the observability grouping question may come back if we add a deployment-level knob later.

haarchri · 2026-07-21T19:07:46Z

+    matchLabels:
+      modelplane.ai/serving: <replica-name>
+  podMetricsEndpoints:
+  - port: http


is http the right name here?

It's an open question in the doc (http vs metrics). Leaning http since it's the one serving port that also serves /metrics, but happy to go metrics if that reads clearer.

Align the opt-out field with modelplaneai#77 by nesting it at observability.metrics.enabled, so traces and logs can join it later. Clarify that the 30s scrape is an observability default and not a routing input (the EPP scrapes engines on its own fast loop), that cache/PVC observability is out of scope, and that an existing hand-written PodMonitor must be deleted on upgrade. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

negz · 2026-07-22T06:30:32Z

@dennis-upbound I think we need to step back and think about what we want to help folks monitor.

There's a few things you could monitor in Modelplane:

Inference signal (data plane). vLLM /metrics, EPP llm_d_epp_*, Envoy. TTFT, tokens/sec, queue depth, KV-cache utilization, request/error rates per model. "Is my model serving well, and is it saturated?"
Substrate health (the stack Modelplane installs on each workload cluster). Gateway up? cert-manager, LWS controller, NVIDIA DRA driver healthy? GPUs allocatable? "Is the machinery on this cluster working?"
Control plane health (Modelplane itself). Crossplane reconcile rates/errors, function latency and panics, provider health, the fleet scheduler actually placing replicas, XR Ready/Synced. "Is the thing I operate working?"
Fleet rollup. Across all clusters/deployments: total capacity, GPU utilization, how many deployments degraded, cost.

I think MD authors will care about 1. The rest feel more like platform team concerns.

This proposal helps with 1, but it feels incomplete. If I understand correctly, it configures the Prometheus instance we already deploy to each cluster to scrape ModelReplica (i.e. vLLM, EPP) metrics, but what then? How does the MD author see and consume those metrics? If the platform team needs to setup plumbing to actually expose the metrics somewhere the MD author can use them (e.g. a dashboard), why ask the MD author to opt-in (or out)?

My hunch is Modelplane should do something like:

Deploy a Prom per IC
Always setup PodMonitors for all MD bits (engines, EPPs, etc) - no opt-in or opt-out
Always setup PodMonitors for all serving stack components
Deploy a centralized Modelplane Prom instance
Have the per-IC instances feed into the centralized one, which also scrapes Crossplane metrics for Modelplane control plane health
Setup recording rules as appropriate

This'd give you one central Prom instance you can scrape to get your entire Modelplane deployment's metrics. Metrics you'd presumably feed onwards into your monitoring and alerting system of choice.

Adopt Nic's direction: step back to what's worth monitoring (data plane, substrate, control plane, fleet roll-up), make PodMonitor collection always-on at every layer instead of a per-deployment opt-out, and add a central Modelplane Prometheus that the per-cluster instances feed and that also scrapes the control plane. Drops the observability.metrics.enabled toggle. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

dennis-upbound · 2026-07-22T16:29:37Z

Completely agree, reframed the doc around this. It now opens with the four things to monitor (data plane / substrate / control plane / fleet roll-up), makes PodMonitor collection always-on at every layer (engine, EPP, serving-stack components) with no opt-in or opt-out, and adds a central Modelplane Prometheus that the per-IC instances feed (remote-write or federation) and that also scrapes Crossplane for control-plane health, with recording rules for the fleet roll-up.

The per-deployment opt-out field is dropped (now the first rejected alternative). Open questions left: remote-write vs federation, central retention sizing, and phasing (data-plane + substrate first, control-plane + roll-up after).

negz · 2026-07-25T00:34:56Z

reframed the doc around this

@dennis-upbound did you forget to push? 🤔 Doc looks unchanged to me.

Pull the design back from a central aggregated Prometheus to what this layer should own: always-on PodMonitors inside each InferenceCluster and a Prometheus URL on the cluster status, so the platform team can scrape or federate without reaching into Modelplane internals. Leave cross-cluster aggregation and control-plane monitoring to the platform team, as the main open question. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

dennis-upbound · 2026-07-27T15:59:13Z

Agree — pulled the scope to exactly this. The doc now composes collection always-on inside each IC (engines, EPPs, serving-stack), no MD-author opt-in/out, and exposes the cluster's Prometheus URL on the InferenceCluster status (status.metrics.prometheusURL) so the platform team can scrape or federate without reaching into internals. Cross-cluster aggregation and Crossplane monitoring stay with the platform team. I left the one-central-Prom-for-the-whole-deployment out of this proposal and flagged it as the main open question, so we can decide separately how far Modelplane takes it.

(Also: the doc genuinely hadn't updated when I said it had — I'd pushed to the wrong remote. Fixed now, the diff reflects the change.)

negz · 2026-07-28T05:06:52Z

+An MD author cares about (1), and the platform team owns the rest. A per-deployment
+opt-in would cover only (1), and even then the platform team has to plumb the metrics
+somewhere the author can read them. So collection is always on inside each cluster,
+with no toggle for an MD author to manage. Aggregating across clusters and monitoring
+the control plane stay with the platform team, and the end of this document returns to
+them.


Nit: the mention of per-deployment opt-in feels like it's a reaction to the earlier iteration of the design. You already cover it in alts considered. I think you can drop it here.

Dropped it — the opt-in only lives in Alternatives now.

negz · 2026-07-28T05:08:15Z

+### One cluster-wide PodMonitor from the serving stack
+
+`compose-serving-stack` could install a single `PodMonitor` selecting all serving pods
+(`modelplane.ai/serving` Exists). It would have no per-deployment lifecycle: it wouldn't
+come and go with a deployment. Composing the engine monitor per replica ties that
+collection to the thing it observes, while the serving-stack monitors, which are
+per-cluster, are composed once with the stack.


Would this work? i.e. Just matching stuff to monitor as it comes and goes?

If so, what's the benefit of the current proposal over this alternative?

Agreed, and switched to it. Now that collection is always-on with no opt-out, the per-replica composition earns nothing over one cluster-wide monitor — so the serving stack composes a single PodMonitor matching modelplane.ai/serving (Exists) for engines, plus one for EPPs and one for the substrate, composed once with the stack. The per-replica version is now the first rejected alternative.

negz · 2026-07-28T05:09:21Z

+```yaml
+status:
+  metrics:
+    prometheusURL: http://prometheus-operated.modelplane-system.svc:9090


How will we expose Prometheus so it can be scraped from outside the cluster?

Made it explicit: Modelplane only publishes the in-cluster URL on the InferenceCluster status — it doesn't expose Prometheus outside the cluster. Reaching it across clusters stays the platform team's job through whatever cross-cluster path they already run (remote-write, a federation scrape, a peered network), the same way they reach any in-cluster service. Exposing it publicly would be a security surface we don't want to own; publishing where it is gets them what they need.

Agree the per-replica composition earns nothing once collection is always-on, so compose one cluster-wide PodMonitor per source in the serving stack instead. State the proposal and what approving it covers up front, answer how the per-cluster Prometheus is reached from outside (the platform team's existing cross-cluster path; Modelplane only publishes the URL), and drop the opt-in mention that was reacting to the earlier draft. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Dennis Ramdass <dennis@upbound.io>

vercel Bot deployed to Preview – modelplane-docs July 21, 2026 17:53 View deployment

haarchri reviewed Jul 21, 2026

View reviewed changes

vercel Bot deployed to Preview – modelplane-docs July 27, 2026 15:57 View deployment

negz reviewed Jul 28, 2026

View reviewed changes

vercel Bot deployed to Preview – modelplane-docs July 28, 2026 13:28 View deployment

modelplaneai deleted a comment from dramdass Jul 28, 2026


		## Interaction with #264

		The [#264](https://github.com/modelplaneai/modelplane/issues/264) example


		### What gets scraped

		The `PodMonitor` ingests everything the engine exposes on `/metrics`. What the

Uh oh!

Conversation

dennis-upbound commented Jul 21, 2026

Description of your changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

negz commented Jul 22, 2026

Uh oh!

dennis-upbound commented Jul 22, 2026

Uh oh!

negz commented Jul 25, 2026

Uh oh!

dennis-upbound commented Jul 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants