fix(feature-flags): add safe agentless EVP fallback

[leoromanovsky]

Motivation

Java already selects a local EVP route from Agent /info and, for agentless Feature Flags, falls back to direct intake for both exposures and flag evaluations. The remaining direct-intake transport has four gaps:

• Direct requests do not honor DD_PROXY_HTTPS and DD_PROXY_NO_PROXY.
• An ambiguous local transport failure leaves subsequent batches on the failed local route instead of switching them to direct intake.
• Direct requests follow redirects while carrying DD-API-KEY.
• The endpoint derived from DD_SITE is not restricted to the expected HTTPS Event Platform intake origin.

Changes

flowchart LR
  Signals["Exposures + flag evaluations"] --> Discover["Startup /info discovery"]
  Discover -->|"v4 advertised"| V4["/evp_proxy/v4"]
  Discover -->|"v2 only"| V2["/evp_proxy/v2"]
  Discover -->|"No compatible route"| Direct["Direct intake + DD-API-KEY"]
  V4 --> Intake["Event Platform intake"]
  V2 --> Intake
  V4 -->|"Definitive failure"| Direct
  V2 -->|"Definitive failure"| Direct
  Direct --> Intake

Loading

• Select the local EVP v4 or v2 route from Agent /info during startup.
• Use direct intake when no supported local route is available.
• Replay definitive local-route failures through direct intake and keep the direct selection sticky.
• Switch future batches after ambiguous local transport failures without replaying the current batch.
• Route direct intake through configured HTTPS proxy settings with NO_PROXY support.
• Reject HTTP and HTTPS redirects from direct Feature Flags intake.
• Add focused route, failure-classification, proxy, exposure, flag-evaluation, and redirect tests.

Decisions

• Replay the current batch only after a definitive refusal or HTTP 403, 404, or 405.
• Keep reset, broken pipe, and timeout failures future-only to avoid duplicate delivery.
• Keep HTTP 429 and 5xx responses on the selected local route.
• Keep TLS verification enabled and use raw DD-API-KEY authentication only for direct intake.
• Do not follow direct Feature Flags intake redirects.

Validation

The Java system tests are defined and enabled in DataDog/system-tests#7601.

Ran the Java spring-boot system-test stack against the branch artifact in these scenarios:

• FEATURE_FLAGGING_AND_EXPERIMENTATION
• FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_DIRECT
• FEATURE_FLAGGING_AND_EXPERIMENTATION_AGENTLESS_SERVERLESS

In each topology, the tests exercised both /api/v2/exposures and /api/v2/flagevaluation, route-specific headers and authentication, aggregate evaluation counts, and zero events on the unused route.

Local validation:

• ./gradlew :communication:test --tests datadog.communication.BackendApiFactoryTest
• ./gradlew :products:feature-flagging:feature-flagging-lib:test --tests com.datadog.featureflag.FeatureFlagBackendApiFactoryTest --tests com.datadog.featureflag.ExposureWriterTests --tests com.datadog.featureflag.FlagEvaluationWriterImplTest
• ./gradlew :communication:spotlessCheck :communication:forbiddenApisMain :products:feature-flagging:feature-flagging-lib:spotlessCheck :products:feature-flagging:feature-flagging-lib:forbiddenApisMain

[datadog-datadog-us1-prod]

🎯 Code Coverage (details)
• Patch Coverage: 42.99%
• Overall Coverage: 58.76% (-0.04%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 489706c | Docs | View more details | Give us feedback!}

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.95 s	[-0.3%; +1.0%] (no difference)
startup:insecure-bank:tracing:Agent	12.91 s	13.06 s	[-1.9%; -0.4%] (maybe better)
startup:petclinic:appsec:Agent	17.55 s	17.22 s	[+1.1%; +2.7%] (significantly worse)
startup:petclinic:iast:Agent	17.46 s	17.61 s	[-1.5%; -0.1%] (maybe better)
startup:petclinic:profiling:Agent	17.48 s	17.51 s	[-1.3%; +0.9%] (no difference)
startup:petclinic:sca:Agent	17.44 s	17.38 s	[-0.7%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.73 s	16.69 s	[-0.7%; +1.2%] (no difference)

Commit: 489706ce · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[linear-code]

FFL-1482

FFL-1484

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 489706ce96

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Preserve single-character no-proxy entries

When DD_PROXY_NO_PROXY, NO_PROXY, or no_proxy contains a one-character entry, this parser silently omits it because parseStringIntoSetOfNonEmptyStrings only adds tokens whose length is greater than one. In particular, NO_PROXY=* produces an empty set, so the explicit wildcard handling in IntakeProxySelector.shouldBypassProxy is unreachable and direct feature-flag traffic still uses HTTPS_PROXY; single-character internal hostnames also regress from the previous getSpacedList behavior. Parse these values with a helper that retains all nonempty tokens.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Match bare NO_PROXY domains against subdomains

When a standard domain entry such as NO_PROXY=datadoghq.com is used with HTTPS_PROXY, this matcher bypasses only the exact host because suffix matching is restricted to entries starting with a dot. Consequently, event-platform-intake.datadoghq.com is still sent through the proxy, even though bare domain entries in no-proxy lists are expected to cover that domain and its subdomains. Apply boundary-aware suffix matching to bare domain entries as well.

Useful? React with 👍 / 👎.

[chatgpt-codex-connector]

Redact credentials from proxy configuration telemetry

When DD_PROXY_HTTPS contains supported userinfo such as http://user:password@proxy:8080, reading it through ConfigProvider.getString records the complete URL in ConfigCollector; ConfigSetting does not classify proxy.https as sensitive, so the telemetry configuration payload serializes the proxy username and password verbatim. Read this setting without collecting the raw value or add explicit redaction for the proxy URL before telemetry emission.

Useful? React with 👍 / 👎.

[datadog-datadog-us1-prod]

NO_PROXY=* does not bypass the new intake proxy. The parser removes the wildcard before the proxy selector reads it.

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 489706c · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[datadog-datadog-us1-prod]

Keep the NO_PROXY wildcard

Feature Flag requests with an API key can go through a proxy that the user explicitly disables.

Assertion details

• Input: Set an HTTPS proxy and set DD_PROXY_NO_PROXY=*, NO_PROXY=*, or no_proxy=*.
• Expected: The wildcard must bypass the proxy for all direct intake hosts.
• Actual: The parser removes the one-character * value. The proxy selector then uses the configured HTTPS proxy.

Was this helpful? React 👍 or 👎
_{🤖 Datadog Autotest · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}