docs: refactor service registration brief to HTTP traffic metering by JoseSzycho · Pull Request #181 · datum-cloud/network-services-operator

JoseSzycho · 2026-06-10T15:01:27Z

Warning

In pause till #187 is on place
#181 (comment)

This PR refactors and expands the initial service catalog registration design brief into a structured, standard enhancement proposal focusing on HTTP Traffic Metering for Network Services.

It addresses and resolves all open questions (OD-1 through OD-8) carried over from the previous design brief.

Related to:

docs: design brief for service catalog registration #156 (comment)

Working implementation using OTLP Transport:

Feat/envoy logs #188

…e diagrams and Taskfile automation This changes replaces the service-catalog-registration with documentation more focused in the architecture design and implementation

scotwells

Directionally the architecture looks good. Let's just clean up some of the implementation details. We want to keep this document product / consumer focused.

scotwells · 2026-06-10T18:35:34Z

+
+### Goals
+
+- Define a standard `Service` and `ServiceConfiguration` to register Network Services under the service domain `networking.datumapis.com`.


This is more of an implementation detail. I'd keep goals high level / consumer focused.

scotwells · 2026-06-10T18:36:56Z

+This enhancement defines the architecture, data structures, and roadmap to bring HTTP traffic metering and catalog registration to Network Services. The work is split into two phases:
+- **Phase 1 (Catalog & Metadata):** Declare a `Service` and a companion `ServiceConfiguration` resource (`services.miloapis.com/v1alpha1`) carrying the monitored-resource and meter declarations inline. This is a YAML-only delivery packaged in the `config/services/` bundle.
+- **Phase 2 (Emission & Integration):** Configure Envoy Gateway proxy logging and deploy a custom Vector Agent to scrape access logs, parse billing signals into CloudEvents, and forward them to the local `billing-usage-collector-vector` DaemonSet.


I don't think it's important to highlight the phases in the enhancement. How the work breaks down and is sequenced is better for a project management tool (e.g. GitHub issues), not an enhancement document.

scotwells · 2026-06-10T18:38:39Z

+
+Network Services is a core utility that incurs direct infrastructure costs. Capturing consumption signals is necessary for platform billing and cost-attribution. 
+
+Because `MeterDefinition` fields (such as `meterName` and `measurement.unit`) are immutable once published, establishing correct definitions in the `Draft`/`Provisional` phase is critical. Doing so avoids costly SDK upgrades, meter deprecation cycles, and data migrations.


This feels like internal / implementation detail. The motivation should be more product / consumer focused.

scotwells · 2026-06-10T18:40:32Z

+- Altering the `MeterDefinition` schema or billing pipeline contract.
+- Implementation of the core Billing SDK (owned by the Billing Team).
+- Shared-infrastructure cost attribution or cross-project billing logic.
+- **Deploying or modifying the Billing System or `billing-usage-collector-vector` DaemonSet.** These components are pre-existing, shared platform infrastructure. Our work is limited to deploying a custom Vector Agent (Log Parser) to parse logs and forward them to this existing collector.


I don't think this should be a non-goal, it's more of an implementation detail.

scotwells · 2026-06-10T18:43:41Z

+
+- The **Monitored Resource** is the Kubernetes Gateway API `HTTPRoute` resource, representing the customer-facing HTTP endpoint.
+- **Phase 1** registers the service with the service catalog via declarative YAML configurations. The service catalog fan-out controller automatically creates `MonitoredResourceType` and `MeterDefinition` resources in the billing namespace.
+- **Phase 2** instruments the Envoy Gateway instances to write structured JSON access logs to stdout. A node-level Vector Agent (Log Parser) tails these logs, parses and maps the raw logs into CloudEvents, and forwards them locally via HTTP to the `billing-usage-collector-vector` DaemonSet. The billing collector then handles local disk buffering and reliably forwards them to the Billing System.


We should measure the resource impact this would have on the worker node. It may make sense to have the existing vector agent handle collection from envoy and transforming the results into billable usage events.

scotwells · 2026-06-10T18:46:44Z

+
+---
+
+### Service and ServiceConfiguration Definitions


I don't think it's important to have the actual service configuration file here. That's an implementation detail of registering the metering definitions.

… environment

This reverts commit 1bb2460.

…ster dev environment" This reverts commit 4bcc370.

scotwells · 2026-06-16T16:39:04Z

Following on from the pipeline shape — the piece worth nailing down is how the log line gets the metadata embedded in the first place, since the project isn't something Envoy knows on its own. Project identity lives on the namespace (resourcemanager.miloapis.com/project-name, e.g. p-abc), so it has to be put onto the route before the access log can emit it.

The natural place to do that is the extension server proposed in #187. It already injects per-route filter_metadata for the WAF and Connector, already watches namespaces in a warm cache, and already maps the downstream ns-<uid> back to the upstream namespace — so adding the project name to route metadata is a small, additive change that stays out of the request path. Once it's on the route, the access log format references it with %METADATA(ROUTE:...)% and the line comes out with project (and whatever else we want) already embedded, ready for Vector to map to subject: projects/{name}.

So rather than enrich anywhere downstream, we piggyback on #187 to stamp the metadata at build time and let it fall through into the log line.

…tions for HTTP traffic metering

JoseSzycho · 2026-06-17T14:15:08Z

@scotwells I just updated the enhancement, and I have a working implementation here:

Feat/envoy logs #188

Could you please review the Design Details I introduced in the previous commit?

I propose to different alternatives.

Will wait to hear from you in order to continue with the implementation.

scotwells · 2026-06-17T14:21:48Z

+1. **Network Services Operator (controller).** When the operator reconciles a
+   customer `HTTPRoute` into its downstream representation, it injects the
+   project name as a request header (`x-datum-project-name`) via a
+   `RequestHeaderModifier` filter on each route rule. The project name is read
+   from the upstream cluster identity (the Milo project name) that the
+   operator already holds while mapping upstream → downstream resources. Routes
+   that already define a `RequestHeaderModifier` are merged into rather than
+   duplicated, since Gateway API permits at most one such filter per rule.


While this approach works, I believe the approach in #181 (comment) would be better since it doesn't require exposing billing metadata through headers seen by end users.

With the header approach, we would have to prevent the user from ever specifying this header in their request since we use it for billing attribution. Otherwise we could potentially bill the wrong project for an HTTP route.

thanks for the fast review. Will set this PR on draft and wait till the extension server to be on place for doing the final tweaks into the documentation.

docs: add http traffic metering enhancement proposal with architectur…

832ed5b

…e diagrams and Taskfile automation This changes replaces the service-catalog-registration with documentation more focused in the architecture design and implementation

JoseSzycho changed the base branch from main to feat/service-catalog-registration-design June 10, 2026 15:03

JoseSzycho requested a review from scotwells June 10, 2026 15:06

scotwells requested changes Jun 10, 2026

View reviewed changes

JoseSzycho added 6 commits June 12, 2026 10:27

docs: simplify architecture to use single shared Vector agent

21bb040

feat: add Taskfile.dev.yaml for bootstrapping local multi-cluster dev…

4bcc370

… environment

feat: add test task to Taskfile.dev.yaml

1bb2460

Revert "feat: add test task to Taskfile.dev.yaml"

bc0835e

This reverts commit 1bb2460.

Revert "feat: add Taskfile.dev.yaml for bootstrapping local multi-clu…

64480a2

…ster dev environment" This reverts commit 4bcc370.

chore: auto generate images

3400477

JoseSzycho mentioned this pull request Jun 17, 2026

Feat/envoy logs #188

Draft

docs: document the project name dimension and access log transport op…

5b7a0ad

…tions for HTTP traffic metering

scotwells requested changes Jun 17, 2026

View reviewed changes

JoseSzycho marked this pull request as draft June 17, 2026 15:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: refactor service registration brief to HTTP traffic metering#181

docs: refactor service registration brief to HTTP traffic metering#181
JoseSzycho wants to merge 8 commits into
feat/service-catalog-registration-designfrom
feat/http-traffic-metering

JoseSzycho commented Jun 10, 2026 •

edited

Loading

Uh oh!

scotwells left a comment

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells Jun 10, 2026

Uh oh!

scotwells commented Jun 16, 2026

Uh oh!

JoseSzycho commented Jun 17, 2026

Uh oh!

scotwells Jun 17, 2026

Uh oh!

JoseSzycho Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		### Goals

		- Define a standard `Service` and `ServiceConfiguration` to register Network Services under the service domain `networking.datumapis.com`.


		Network Services is a core utility that incurs direct infrastructure costs. Capturing consumption signals is necessary for platform billing and cost-attribution.

		Because `MeterDefinition` fields (such as `meterName` and `measurement.unit`) are immutable once published, establishing correct definitions in the `Draft`/`Provisional` phase is critical. Doing so avoids costly SDK upgrades, meter deprecation cycles, and data migrations.

Conversation

JoseSzycho commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

scotwells left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

scotwells commented Jun 16, 2026

Uh oh!

JoseSzycho commented Jun 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

JoseSzycho commented Jun 10, 2026 •

edited

Loading