Skip to content

Deflake Envoy AllowPort spec's allowed-port leg - #6077

Closed
JAORMX wants to merge 1 commit into
mainfrom
deflake-envoy-allowport
Closed

Deflake Envoy AllowPort spec's allowed-port leg#6077
JAORMX wants to merge 1 commit into
mainfrom
deflake-envoy-allowport

Conversation

@JAORMX

@JAORMX JAORMX commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

E2E Tests Core (network-isolation) failed on the pre-merge heads of both #6050 (1588e6a4) and #6051 (5c901a9d) — the latter containing zero production code — and passed on main minutes later. Both failures were the same spec, same assertion, within the same minute: the AllowPort spec's allowed leg, https://example.com must be allowed (port 443 is in AllowPort).

Root cause: the allowed leg fetched https://example.com exactly once, immediately after workload readiness. It needs a working DNS → TLS → upstream chain through Envoy's dynamic_forward_proxy, and the first egress request after startup can race the isolation stack's own warm-up (DFP DNS cache, the per-workload DNS container), failing instantly with a local error. The observed failures returned ~40ms and ~190ms after readiness — far too fast for genuine external TLS trouble — and the boolean-only assertion discarded the fetch tool's error text, so the log could not say why.

What changed:

  • The allowed leg retries via Eventually (60s budget, 2s poll) and records the last fetch error text, which is printed on failure.

Why this is fixing an incidental assertion, not weakening a real one:

  • The property this spec pins (Envoy backend: translate AllowPort into egress policy #5915 parity: Envoy must honour AllowPort end-to-end) is steady-state reachability of an allowed port. First-request-after-boot success is not something Envoy's DFP path promises, so a single-shot first fetch asserts warm-up behaviour the product never guaranteed.
  • A genuinely broken AllowPort still fails — after the retry budget, and now with the fetch error text surfaced, which is the diagnostic that distinguishes a warm-up race from a real enforcement break on the next occurrence.
  • The blocked port-80 leg is untouched: denial is enforced locally and deterministically once the stack is warm, which the preceding allowed leg has just proven.

No timeout was raised; the change addresses the mechanism (first-request warm-up) and improves the failure's diagnosability.

Type of change

  • Bug fix
  • New feature
  • Refactoring (no behavior change)
  • Dependency update
  • Documentation
  • Other (describe):

Test plan

  • Linting (task lint; go vet ./... run separately)
  • Manual testing (describe below)

Test-only change to a suite that requires a container runtime, which the dev environment lacks — the retry's effect on the flake is only observable in CI. Verified by compilation (go vet ./test/e2e/), lint, and by checking the failure-time evidence (both CI logs) against the retry semantics: the observed failure mode (instant IsError right after readiness) is exactly what the retry absorbs, and the assertion's contract is unchanged.

Does this introduce a user-facing change?

No.

Special notes for reviewers

The warm-up diagnosis is the best fit for the evidence (instant local failure, correlated timing across two runners, immediate pass on main) but could not be pinned to the exact Envoy error because the old assertion discarded the error text. If this spec ever fails again, the surfaced text settles it.

Generated with Claude Code

The AllowPort spec fetched https://example.com exactly once,
immediately after workload readiness. The allowed leg needs a working
DNS -> TLS -> upstream chain through Envoy's dynamic_forward_proxy, and
the first egress request after startup can race the isolation stack's
own warm-up, failing instantly with a local error before anything
reaches the network. Observed twice on 2026-07-28 (PR heads 1588e6a
and 5c901a9, within the same minute): IsError ~40ms and ~190ms after
readiness — far too fast for genuine external TLS trouble — while the
same spec passed on main minutes later.

The property this spec pins is steady-state reachability of an allowed
port; first-request-after-boot success is not something the DFP path
promises. Retry the allowed leg until the deadline, and surface the
fetch tool's error text on failure — the diagnostic that distinguishes
a warm-up race from a genuinely broken AllowPort, which the old
boolean-only assertion discarded. A broken AllowPort still fails, now
with evidence. The blocked port-80 leg is unchanged: denial is
enforced locally and deterministically once the stack is warm, which
the preceding allowed leg has just proven.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GvBj7nNWqDQ3rKYNeY6aam
@github-actions github-actions Bot added the size/XS Extra small PR: < 100 lines changed label Jul 28, 2026
@JAORMX

JAORMX commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator Author

Superseded by #6076: the maintainer asked for the statuses fix and this deflake to land as a single PR, so this change (identical commit) is now the second commit there. Closing.

@JAORMX JAORMX closed this Jul 28, 2026
@JAORMX
JAORMX deleted the deflake-envoy-allowport branch July 28, 2026 07:53
@codecov

codecov Bot commented Jul 28, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.30%. Comparing base (9b0d83a) to head (700b771).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #6077      +/-   ##
==========================================
+ Coverage   72.25%   72.30%   +0.04%     
==========================================
  Files         724      725       +1     
  Lines       75348    75358      +10     
==========================================
+ Hits        54445    54486      +41     
+ Misses      17022    16971      -51     
- Partials     3881     3901      +20     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size/XS Extra small PR: < 100 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants