Skip to content

Feat/envoy logs#188

Draft
JoseSzycho wants to merge 2 commits into
feat/http-traffic-meteringfrom
feat/envoy-logs
Draft

Feat/envoy logs#188
JoseSzycho wants to merge 2 commits into
feat/http-traffic-meteringfrom
feat/envoy-logs

Conversation

@JoseSzycho

@JoseSzycho JoseSzycho commented Jun 17, 2026

Copy link
Copy Markdown

Warning

This PR should not be merged. It has been created to show how we can capture the HTTPRoute usage signals.
Once the corresponding enhamcenete has been approved and merged into main, continue with the work on this PR.

Related to:

Summary

  • Controller: inject x-datum-project-name request header into every downstream HTTPRoute rule via a RequestHeaderModifier filter, sourcing the value from the upstream cluster name (Milo project name). Merges into existing modifiers rather than appending a second one.
  • Envoy access log: extend the JSON format with project_name: "%REQ(X-DATUM-PROJECT-NAME)%" and add an OpenTelemetry sink alongside the existing File sink, pushing access logs directly to the billing-usage-collector-vector OTLP receiver.
  • Vector billing collector: add an opentelemetry source (gRPC :4317, HTTP :4318), a normalization shim that collapses OTLP .attributes.* and http_server top-level fields to a common shape, and project_name extraction + emission as a dimension on all four billing CloudEvents.
  • E2E test: switch from manual curl log injection to the real OTLP push path; assert the project_name dimension in the CloudEvent.

Why

The billing pipeline previously could only attribute usage to a project by its control-plane namespace UID (ns-<uid>). This PR surfaces the human-readable project name as a first-class dimension on every billing event, enabling clearer cost attribution in the billing portal without any post-processing joins.

Notes

  • %REQ()% is used instead of %METADATA(ROUTE:...)% because Envoy Gateway's JSON access log formatter does not register the metadata formatter in v1.5.1.
  • Both sinks are kept in parallel: File for existing log scraping / debugging, OTLP for the real push path. This makes the OTLP transport topology-independent (works whether Vector runs as a DaemonSet or Stateless-Aggregator).
  • In ProviderSingle mode (dev/e2e), the upstream cluster name is the literal string "single". In ProviderMilo mode it is the real Milo project name.

@JoseSzycho JoseSzycho changed the base branch from main to feat/http-traffic-metering June 17, 2026 14:05
@JoseSzycho JoseSzycho marked this pull request as draft June 17, 2026 14:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant