fix(orchestrator): downgrade expected VM-exit cascade errors from ERROR to Warn

[AdaAibaby]

Summary

Closes #3274
Closes #3275

After a Firecracker VM exits (normally or abnormally), two sources of spurious ERROR-level logs appear:

1. NBD dispatch (dispatch.go): In-flight reads/writes fail with context.Canceled and response writes fail with use of closed network connection / broken pipe. All were logged at ERROR, generating noise that drowns out real backend failures.

2. FC metrics flusher (fc_metrics.go): A Go select race between p.Exit.Done() and ticker.C can cause flushMetrics() to be called against a dead Firecracker socket, emitting a spurious "failed to flush fc metrics: connection reset by peer" Warn log even after the process has exited.

Changes

packages/orchestrator/pkg/sandbox/nbd/dispatch.go

• nbd backend read/write failed: downgrade to Warn when errors.Is(err, context.Canceled || context.DeadlineExceeded); keep ERROR for real backend failures
• nbd error cmd read/write (fatal-channel-full path): downgrade to Warn; by the time this path is reached, shutdown is already in progress

packages/orchestrator/pkg/sandbox/fc/fc_metrics.go

• After case <-ticker.C fires, add a non-blocking select { case <-p.Exit.Done(): return; default: } guard to handle the race where both channels are simultaneously ready

Before / After

Before (every sandbox teardown with in-flight I/O):

level=error msg="nbd backend read failed"  error="context canceled"
level=error msg="nbd error cmd read"       error="write unix @->@: use of closed network connection"
level=error msg="nbd backend write failed" error="context canceled"
level=error msg="nbd error cmd write"      error="write unix @->@: write: broken pipe"
level=warn  msg="failed to flush fc metrics" error="connection reset by peer"   ← still emitted after process exit

After:

level=warn msg="nbd backend read cancelled (sandbox exiting)"  error="context canceled"
level=warn msg="nbd error cmd read"   error="use of closed network connection"
level=warn msg="nbd backend write cancelled (sandbox exiting)" error="context canceled"
level=warn msg="nbd error cmd write"  error="broken pipe"
                                                               ← fc metrics flush suppressed after process exit

Real backend errors (non-canceled) still surface at ERROR level.

Test plan

• Kill a Firecracker VM with kill -9 while I/O is in-flight; confirm no ERROR logs for nbd backend read/write failed with context canceled
• Confirm real NBD backend errors (e.g. backing store returning actual error) still log at ERROR
• Under high sandbox-creation load, confirm "failed to flush fc metrics" Warn disappears after VM exit

[gemini-code-assist]

Code Review

This pull request introduces improvements to the orchestrator's sandbox metrics reader and NBD dispatch logging. Specifically, it adds a guard in the metrics reader to prevent a race condition during process exit, and downgrades expected errors (such as context cancellations during shutdown or response-write failures on a closing socket) from Error to Warn level in the NBD dispatch loop. I have no feedback to provide as there are no review comments to assess.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.