Poll the cascade settle step and fix its PENDING diagnostics - #2480
Draft
delthas wants to merge 3 commits into
Draft
Poll the cascade settle step and fix its PENDING diagnostics#2480delthas wants to merge 3 commits into
delthas wants to merge 3 commits into
Conversation
Contributor
Hello delthas,My role is to assist you with the merge of this Available options
Available commands
Status report is not available. |
Contributor
Waiting for approvalThe following approvals are needed before I can proceed with the merge:
|
The step waited a fixed 15 seconds and then asserted once. In one census run, CR churn stretched the cascade hops to 94s and 168s and location C became REPLICA 21s after the assertion had already failed. A healthy cascade settles in about 20s and the sibling steps in this file get 300s, so this was never a capacity problem: it was a one-shot assertion in a suite where every comparable step polls. Take the budget from the scenario, like those siblings, and poll until settled. The four blind 15-second sleeps go away with it. Issue: ZENKO-5340
Both polling steps declared a cucumber timeout of exactly the budget the
feature passes them: { timeout: 300_000 } against "within 300 seconds".
Their own assert.fail therefore always lost the race against the
framework timeout, so on a real timeout the failure was reported as the
generic "function timed out" and the step's own diagnostic never printed.
Give them 330s so the assertion inside the step is the one that fires.
Issue: ZENKO-5340
…sing On timeout the step failed with "object ... not found". In the run investigated, cloudserver had returned 200 to 184 HEAD requests on the object it called missing: it was present and PENDING. That message sent an investigation down the wrong path for most of a day. Track the last status the poll actually saw and report it, keeping the "not found" wording only for the case where no HEAD ever succeeded. Issue: ZENKO-5340
delthas
force-pushed
the
improvement/ZENKO-5340/cascade-settle-poll-and-diagnostics
branch
from
August 5, 2026 09:56
ff7267d to
b91090b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two defects in
steps/crrCascade.tsfound by the recent CTST flakiness census. They ship together because they are in the same file and both come from the same misreading of whatPENDINGmeans for a cascade destination.The settle step did not poll (1 of 30 runs). It waited a fixed
When I wait 15 secondsand then asserted once. In the failing run, CR churn stretched the cascade hops to 94 s and 168 s, and location C becameREPLICA21 s after the assertion had already failed.This was never a capacity problem — a healthy cascade settles in about 20 s and the sibling "replicate to location" steps get 300 s, roughly 15x headroom. It was simply a one-shot assertion in a suite where every comparable step polls. The step now reads
should be settled within {int} seconds, matching how every other step in this file is written, takes 120 s from the scenario (6x the healthy case), and polls until settled. The four blind 15 s sleeps go away with it, reclaiming a minute of pure sleep per run.The failure message was wrong. On timeout the replicate step said
object ... not found. In the run investigated, cloudserver had returned 200 to 184 HEAD requests on the object it called missing: the object was present andPENDING. That message sent an investigation down the wrong path for most of a day, which is why it is worth fixing even though it changes no pass/fail outcome. It now reports the last observedReplicationStatus, and still saysnot foundwhen the HEAD genuinely never succeeded.While checking that, a related problem showed up in a sibling: both the replicate step and the tag step declared a cucumber timeout of exactly the budget the feature passes them (
{ timeout: 300_000 }againstwithin 300 seconds), so their ownassert.failalways lost the race against the framework timeout and never printed. Both now get 330 s. This is slightly beyond the ticket's letter, but it is the same defect in the same file and leaving it would keep one of the two diagnostics unprintable.Verified locally:
yarn build:cucumberclean,yarn lintclean (38 pre-existing warnings, 0 errors), and a--dry-runover the cascade tags resolves the renamed step with no undefined steps across all 6 scenarios.The product cause of dropped replication status entries — service-account key rotation on every overlay write, and a transient 403 turned into permanent loss — is tracked in the issues linked from ZENKO-5340 and is out of scope here.
Issue: ZENKO-5340