Fix the Days=0 expiration flake by publishing backlog metrics every 5s in CI

[delthas]

Supersedes two earlier versions of this PR. The first raised the scenario budget 180 s → 420 s; @DarkIsDude and @SylvainSenechal objected that such a budget hides performance regressions, and they were right. This version changes no budget at all — it fixes the cause, and the scenario simply gets fast.

What the 320 s was actually spent on

Not the platform being slow, and not the conductor cron — CI already runs it every 5 s (*/5 * * * * *, six fields). From a CI conductor log:

20:59:55.170  starting new lifecycle batch
20:59:56.442  finished pushing lifecycle batch      <- 1.3 s of actual work
21:00:00.086  skipping lifecycle batch due to previous operation still in progress
21:00:05.108  skipping ...                           (~23 consecutive 5 s ticks)
   ...
21:01:55.03   next batch starts                      <- ~115 s of waiting

LifecycleConductor._controlBacklog won't start a batch until three checks all report zero lag. It can't read Kafka directly — it reads offsets published to ZooKeeper every backlogMetrics.intervalS, and discards nodes older than 2× that interval. At the operator's default of 60 s the conductor is blind to an idle cluster for 60–120 s per batch. Days=0 on a versioned bucket needs three sequential batches, so it spent 319.6 s doing about 4 s of work.

The fix, and what it measured

spec.backbeat.backlogMetricsIntervalSeconds: 5 bounds each gap at 5–10 s. Measured across a 30-run census on this change:

example	before	after
Non versioned	96.9 s	27.3 s
Suspended	177.4 s	36.8 s
Versioned	319.6 s	38.9 s

Flake rate went from 7/30 (23 %) to 2/20 (10 %).

The budget stays at 180 s deliberately. It still catches the regression that matters — if the interval ever stopped applying, the scenario would take ~320 s and fail — while leaving room for the residual below. Lowering it further would have bought nothing and risked false alarms.

The residual 2 failures are a different bug

Investigated in the census dumps: they are not conductor stalls. The lifecycle pipeline loses its Kafka partition assignment entirely for 50–90 s, because on each rollout the outgoing backbeat pod never leaves the consumer group and the coordinator elects an already-SIGTERMed process as leader, so no assignment arrives until session.timeout.ms evicts it. Those runs needed 135–145 s end to end — under the 180 s this PR keeps. The rollouts themselves are the operator re-rendering every backbeat deployment on each location CRUD (ZKOP-568).

Sequencing, and the WIP commit

The CR field is new (ZKOP-569 / scality/zenko-operator#626). Kubernetes silently prunes unknown CR fields, so against operator v1.8.14 this would be a no-op. The first commit therefore points deps.yaml at the operator build for #626's head, and must be replaced with the release tag before this merges.

Issue: ZENKO-5337

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[DarkIsDude]

#2461 (comment) 180 to 420 seems huge and can hide some performance change or bugs. Do we need to be that high ? If yes, why ? can 300 can be enough (one gated batche) ? How did you choose 420 and validated it ? Did you try lower value ?

[DarkIsDude]

See my comment above

[SylvainSenechal]

#2461 (comment) 180 to 420 seems huge and can hide some performance change or bugs. Do we need to be that high ? If yes, why ? can 300 can be enough (one gated batche) ? How did you choose 420 and validated it ? Did you try lower value ?

I was gonna say something similar :

The CI platform is not overloaded at all (last kafka cleaner test says we have ~1000 messages generated during the whole cucumber ci, for all kafka topics). Considering the traffic is so low on the test platform, I think it's better to question any test that takes more than a few seconds and review our backbeat code if it takes more than that 🤔

[claude]

Tag must be a concrete released tag, not a commit SHA (c05e662b...). The PR description notes this must be replaced before merge — flagging so it doesn't slip through. Update to the zenko-operator release tag once scality/zenko-operator#626 is merged and released.

[delthas]

Do not merge yet — this now has a hard dependency on a backbeat fix.

A follow-up 30-run census turned up something that makes this change harmful in the wrong order. The interval fix is verified and does exactly what it should, but by making lifecycle batches fire sooner it moves the cold-status message produce from a median of 99.5 s after the workflow step to 24 s — which lands it inside the window where a backbeat rolling update has left the consumer group with no member. The cold-status consumers never pin fromOffset, so auto.offset.reset falls back to latest and anything produced in that window is discarded permanently: the object is archived in the cold backend and its metadata never records it.

Measured across both censuses:

	July	August (this change)
cold-status batches produced but never consumed	34/222 (15.3 %)	74/219 (33.8 %), p = 7.9e-06
batches landing within 30 s of a rollout window	10.8 %	58.4 %, p = 1.1e-12

The per-stratum skip hazard is unchanged between the two (~55–67 % inside the window, ~4–9 % outside), so this isn't a new bug — it's the same latent one firing far more often because the exposure changed. Null control: batches produced far from any rollout window are skipped 0/94 and 1/70 (p = 0.43).

So the ordering has to be: cold-status consumers get fromOffset: 'earliest' (tracked on the backbeat side, alongside the notification and GC consumers), then this lands. With that one-line fix in place the interval change is purely beneficial.

Recorded on ZENKO-5337 and ZKOP-569, both of which are now linked as blocked by the backbeat ticket. The WIP deps.yaml commit still needs replacing with a real operator release tag too.

Issue: ZENKO-5337