test: keep admission available during e2e runs - #364
Draft
scotwells wants to merge 3 commits into
Draft
Conversation
A single manager pod served every admission request in the e2e environment, so one restart took all admission down. Scenarios then failed with a refused connection or a webhook timeout rather than an assertion, and the failures moved between scenarios run to run. The readiness gate also probed only the domain webhook, so a gateway create could still race a webhook that was not yet serving. Key changes: - Run two manager replicas, closer to the three production runs - Probe the gateway admission path before the suite starts - Wait for the full rollout so no replica is still starting
The manager serves admission from the same process that runs every controller. The e2e environment raised its memory to the production base but left cpu at 500m, so under load the process was throttled, the liveness probe missed, and the leader-election lease lapsed. Losing either restarts the process and takes admission with it. Key changes: - Raise the e2e manager cpu alongside the memory already raised
The manager's e2e resources were patched with kubectl after apply, so they were invisible to anyone reading the overlay, unvalidated by the kustomize checks, and swallowed failures behind a trailing true. Key changes: - Set the e2e manager replicas and resources in the e2e overlay - Drop the inline kubectl patching from the environment task
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The end to end suite fails intermittently, and the failure moves between scenarios from one run to the next. Both symptoms seen so far are a refused connection or a webhook timeout, never a failed assertion, so a green or red result does not always reflect the change under test.
One manager pod served every admission request in that environment. A single restart took all admission down for the whole cluster. Production runs three.
This runs two replicas, waits for the full rollout, and probes the gateway admission path before the suite starts. The gate previously covered only the domain webhook, so a gateway create could still race a webhook that was not yet serving.
Related to #344