fix(orchestrator): suppress EIO/ENXIO on sandbox cleanup after VM crash

[AdaAibaby]

Summary

Closes #3273

When a VM crashes or is OOM-killed, NBDProvider.Close() calls sync() which issues BLKFLSBUF ioctl + fsync on /dev/nbdX. Because the NBD device is already in error state after the VM dies, the kernel returns:

• EIO — device is in error state (broken NBD connection)
• ENXIO — device is already disconnected/cleared

Both are expected — there is nothing to flush because the VM is gone. This caused every abnormal VM exit to emit a spurious ERROR:

level=error msg="failed to cleanup sandbox, will remove from cache"
  error="failed to cleanup sandbox: error flushing cow device: failed to fsync path: input/output error"

Why not suppress in NBDProvider.Close() directly?

The initial approach (commit 82a2ec2) caught EIO in NBDProvider.Close() itself. This was flagged by code review (Codex) as too broad: Sandbox.Shutdown() also calls s.Close() with the explicit comment "This should properly flush rootfs to the underlying device" — a genuine EIO there (e.g. host disk full) must still propagate as an error or a corrupt template could be silently published.

Actual fix

Handle EIO/ENXIO in setupSandboxLifecycle() in sandboxes.go — the only call site that is guaranteed to run after VM exit. The Sandbox.Shutdown() path is unaffected and still treats EIO as a hard error.

packages/orchestrator/pkg/server/sandboxes.go:

// Before
cleanupErr := sbx.Close(ctx)
if cleanupErr != nil {
    sbxlogger.I(sbx).Error(ctx, "failed to cleanup sandbox, will remove from cache", zap.Error(cleanupErr))
}

// After
cleanupErr := sbx.Close(ctx)
if cleanupErr != nil {
    if errors.Is(cleanupErr, syscall.EIO) || errors.Is(cleanupErr, syscall.ENXIO) {
        // After a VM crash the NBD device is in error state. sync() on
        // /dev/nbdX returns EIO (device error) or ENXIO (device already
        // disconnected). Both are expected — no data is lost because
        // the VM already exited.
        sbxlogger.I(sbx).Warn(ctx, "failed to flush sandbox device after VM crash (ignoring)", zap.Error(cleanupErr))
    } else {
        sbxlogger.I(sbx).Error(ctx, "failed to cleanup sandbox, will remove from cache", zap.Error(cleanupErr))
    }
}

Before / After

Before (every abnormal VM exit):

level=error msg="failed to cleanup sandbox, will remove from cache"
  error="failed to cleanup sandbox: error flushing cow device: failed to fsync path: input/output error"

After:

level=warn msg="failed to flush sandbox device after VM crash (ignoring)"
  error="failed to cleanup sandbox: error flushing cow device: failed to fsync path: input/output error"

Real non-EIO/ENXIO failures still surface at ERROR.

Test plan

• Kill a Firecracker VM with kill -9; confirm no ERROR log for "failed to cleanup sandbox" and a WARN appears instead
• Normal Sandbox.Shutdown() (pause/snapshot): confirm a genuine disk error still propagates as ERROR

[gemini-code-assist]

Code Review

This pull request updates the Close method in nbd.go to handle syscall.EIO errors when flushing the copy-on-write (COW) device, treating them as warnings since they likely indicate a VM crash. The reviewer suggested extending this logic to also ignore syscall.ENXIO errors, which can occur if the NBD device is already disconnected or cleared during a crash.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 82a2ec22e1

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".