registry pull retries - #2443
HarshwardhanPatil07 wants to merge 3 commits into
Conversation
| // Match the default attempt count and delay used by the Justfile's build-fetch | ||
| // retry helper. A failed attempt has to rebuild the importer, so retries are | ||
| // intentionally made at the whole-pull boundary instead of independently for | ||
| // every layer. | ||
| const PULL_MAX_ATTEMPTS: u32 = 3; | ||
| const PULL_RETRY_DELAY: Duration = Duration::from_secs(30); |
There was a problem hiding this comment.
Conceptually I think we should be matching what e.g. podman does by default. What our build system happens to do is a different unrelated thing.
On that topic see podman-container-tools/container-libs#951
In the short term, we can just copy the same defaults and add a link to that issue as a TODO to allow having bootc be configurable in the same way.
1485745 to
a2a0ced
Compare
Direct image pulls can fail when a registry transiently rejects the GetBlob request. Retry that narrowly classified failure at the whole-pull boundary with a bounded attempt count, while allowing non-registry and unrelated failures to return immediately. Related: bootc-dev#2177 Assisted-by: AI Signed-off-by: HarshwardhanPatil07 <harshpat@redhat.com>
a2a0ced to
7bd3c43
Compare
| Err(error) if retries < PULL_MAX_RETRIES && is_retryable_pull_error(&error) => { | ||
| let retry_delay = pull_retry_delay(retries); | ||
| let attempt = retries + 1; | ||
| tracing::warn!( |
There was a problem hiding this comment.
We don't currently have a ton of usage of warn!. I have a bit of a worry this is going to corrupt e.g. terminal progress rendering.
There's a kind of ongoing clash there between our usage of indicatif and tracing...we may need to do something tricky like hook into tracing and detect when indicatif is active and use that instead?
One thing we should perhaps do here is pass down the pull progress struct, and use that as our abstraction instead (yes this would be a much bigger task).
A general problem here is that we have no testing of our TUI progress rendering, because it's a bit tricky to do.
Did you test this out interactively? One good way to simulate pull failures is to e.g. inject iptables rules to reject packets to quay.io etc.
There was a problem hiding this comment.
@Johan-Liebert1 helped me debugging this but we saw that progress bar was not rendering in first place with main branch
There was a problem hiding this comment.
Wait, no, I found the issue in my script. I didn't have -it in the podman command so the progress bar never showed up
There was a problem hiding this comment.
And I remember I had removed -it because the progress bar didn't render properly on my vim emulated terminal
There was a problem hiding this comment.
Okay, I tested this interactively by rejecting both IPv4 and IPv6 HTTPS traffic during a quay.io blob pull. The retry occurred and reused completed layers, but the captured progress output was not clean
There was a problem hiding this comment.
added a shared PullProgress abstraction
testing it manually it works, showed the retry message and preserved cached layers.
There was a problem hiding this comment.
Thank you for recommending the e.g. to inject iptables rules
The non-unified installation path prepared and pulled images directly, bypassing the shared retry boundary and leaving bootc install to-filesystem exposed to transient registry failures. Route that path through the retrying pull while preserving unified-storage behavior. Related: bootc-dev#2177 Assisted-by: AI Signed-off-by: HarshwardhanPatil07 <harshpat@redhat.com>
7bd3c43 to
94cb1a5
Compare
Share the pull progress renderer across attempts and print retry notices through indicatif so tracing output cannot interfere with active progress bars. Assisted-by: AI Signed-off-by: HarshwardhanPatil07 <harshpat@redhat.com>
496fb9a to
09be175
Compare
3 participants
What
Add bounded retries for registry-backed OSTree image pulls when the containers-image proxy reports a
GetBlobrequest-initiation failure.This also routes the normal, non-unified OSTree installation path through the shared retrying pull function.
Related: #2177
Why
CI has intermittently failed during commands such as
bootc install to-filesystemwhen registries like Quay return transient errors, including502 Bad Gatewayand network timeouts.The existing CI retry wrapper covers preparatory Podman operations, but it does not cover image pulls performed internally by bootc. Consequently, one temporary registry failure can fail the entire integration job.
Retries are appropriate here because registry availability is outside bootc's control and repeating an image pull is safe.
The proxy error does not expose the underlying HTTP status. Therefore, bootc cannot distinguish a transient
502from every permanentGetBlobinitiation failure. The retry count bounds the worst-case additional delay to 60 seconds.