fix(cse): prevent CSE timeout overrun with per-op budget and pre-command guard

[djsly]

Linux CSE runs can exceed the VM provisioner's ~16-minute client window when downloads retry indefinitely or a long-running operation starts near the deadline. This PR adds a global timeout guard and per-download wall-clock budget to bound execution time.

cse_helpers.sh

• _retrycmd_internal: Add pre-attempt check_cse_timeout guard (gated on CSE_STARTTIME_SECONDS being set — no-op and no noise during VHD build). Cap per-attempt effectiveTimeout to min(timeoutVal, remainingCseTime) so a single attempt cannot outlive the provisioning window.
• _retry_file_curl_internal: Add optional maxBudget (4th arg, default 0). When > 0, caps effectiveTimeout to min(timeout, maxBudget - elapsed) per attempt, bounding total download wall-clock to the budget regardless of retry count. Loop restructured so curl fires on every iteration (previously the last attempt was skipped).
• retrycmd_get_tarball: Backward-compatible signature detection — if 3rd arg is non-numeric (old callers: <retries> <wait_sleep> <tarball> <url>), defaults timeout to 60s. New callers pass explicit <timeout> as 3rd arg.
• retrycmd_curl_file: Exposes new optional max_budget_s 6th arg (default 0).

Call sites (cse_install.sh, cse_install_ubuntu.sh, cse_install_mariner.sh)

Pass 300s per-operation budget to all provisioning-time download sites (credential-provider, oras, TLS bootstrap, CNI ×2, crictl, k8s binaries, nvidia GPG key, containerd, runc, nvidia repo). Without budget: 120 retries × 60s = ~2h10m theoretical max per download; with budget: capped at 5 min total.

cse_config.sh

Reduce nvidia-smi per-try timeout from 300s → 30s at 3 call sites (nvidia-smi is a status check, not a long-running init).

Backward compatibility

• max_budget_s defaults to 0 (no cap); VHD build callers unaffected.
• check_cse_timeout guards are skipped when CSE_STARTTIME_SECONDS is unset, preserving existing VHD build behavior exactly.

[Copilot]

Pull request overview

This PR aims to prevent Linux CSE runs from exceeding the VM provisioner’s ~16-minute client window by adding (1) a pre-attempt global timeout guard and (2) a per-download wall-clock budget, then wiring those controls into provisioning-time download call sites and updating ShellSpec coverage accordingly.

Changes:

• Add a pre-command check_cse_timeout guard in _retrycmd_internal and a per-operation maxBudget in _retry_file_curl_internal (propagated via retrycmd_curl_file / retrycmd_get_tarball).
• Update provisioning-time download call sites to pass a 300s per-operation budget and adjust retrycmd_get_tarball call signature to include an explicit timeout.
• Reduce nvidia-smi per-try timeout from 300s to 30s and update/add ShellSpec expectations for the new retry helper behaviors.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
spec/parts/linux/cloud-init/artifacts/cse_retry_helpers_spec.sh	Updates parameter tables and adds tests for pre-attempt global timeout and per-operation budget exit behavior.
spec/parts/linux/cloud-init/artifacts/cse_install_spec.sh	Updates expected retrycmd_get_tarball invocation signature to include timeout + budget.
parts/linux/cloud-init/artifacts/ubuntu/cse_install_ubuntu.sh	Adds 300s per-operation budget to provisioning-time curl downloads (NVIDIA key, containerd, runc).
parts/linux/cloud-init/artifacts/mariner/cse_install_mariner.sh	Adds 300s per-operation budget to NVIDIA repo file download.
parts/linux/cloud-init/artifacts/cse_install.sh	Adds explicit timeout + 300s per-operation budgets to provisioning-time tarball/curl downloads (credential provider, oras, CNI, crictl, k8s).
parts/linux/cloud-init/artifacts/cse_helpers.sh	Implements pre-attempt global timeout guard and per-operation budget; changes retrycmd_get_tarball signature.
parts/linux/cloud-init/artifacts/cse_config.sh	Lowers nvidia-smi retry timeout from 300s to 30s at 3 call sites.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (1)

parts/linux/cloud-init/artifacts/cse_helpers.sh:347

• The per-operation maxBudget check happens before starting timeout $timeout curl ..., but the curl attempt itself can still run longer than the remaining budget (and then exceed the budget by up to timeout + sleep). If maxBudget is intended to be a hard upper bound (“total wall-clock seconds across all retries”), cap the per-attempt timeout to min(timeout, maxBudget - elapsed) and/or re-check the budget immediately after each attempt as well.

        # Check per-operation budget if set -- prevents a single download from consuming the entire CSE window
        if [ "${maxBudget}" -gt 0 ]; then
            local opElapsed
            opElapsed=$(( $(date +%s) - opStartTime ))
            if [ "$opElapsed" -ge "$maxBudget" ]; then
                echo "Operation budget of ${maxBudget}s exceeded after ${opElapsed}s, exiting early." >&2
                return 2
            fi
        fi
        # check if global cse timeout is approaching
        if ! check_cse_timeout; then
            echo "CSE timeout approaching, exiting early." >&2
            return 2
        else
            if [ "$i" -gt 1 ]; then
                sleep $waitSleep
            fi
            timeout $timeout curl -fsSLv $url -o $filePath > $CURL_OUTPUT 2>&1
            if [ "$?" -ne 0 ]; then

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.