Skip to content

[guest] back container bundle with scratch so sandbox mounts are on disk#2835

Open
rawahars wants to merge 2 commits into
mainfrom
harshrawat/fix-bundle-backed-disk
Open

[guest] back container bundle with scratch so sandbox mounts are on disk#2835
rawahars wants to merge 2 commits into
mainfrom
harshrawat/fix-bundle-backed-disk

Conversation

@rawahars

@rawahars rawahars commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

A sandbox:// mount is served from a directory under the container bundle, so the bundle must live on the pod scratch disk for the mount to be disk-backed. The v1 shim mounts the scratch disk at the bundle path, so this holds. The v2 shim keeps the bundle on the UVM tmpfs and mounts the scratch elsewhere, so sandbox:// mounts end up on tmpfs instead of disk.

In the guest, when the scratch dir is not already nested under the bundle, bind a scratch directory onto the bundle before mounting the overlay. The choice is path-based (bundleNeedsScratchBind), so both shims share one rule. Unmount the bind in Container.Delete, and reorder releaseResources so the guest releases the disk-backed mounts before the host detaches the scratch.

Manual validation

All of the following were run manually against a live host (v1 = runhcs-lcow, v2 = runhcs-lcow-v2), verifying the sandbox:// mount filesystem via df /sandboxmount and confirming clean teardown (mounts, dirs, and scratch disk released; no EBUSY).

Scenario Runtime sandbox:// backing Teardown What it exercises
Single pod, standard scratch v1 /dev/sd* (no regression) clean Container with its own scratch mounted at its bundle; scratch is already nested under the bundle, so bundleNeedsScratchBind skips the bind — confirms the path rule leaves working v1 layouts untouched.
Single pod, standard scratch v2 /dev/sd* (fixed) clean + state cleanup The core bug: v2 keeps the bundle on UVM tmpfs with scratch mounted elsewhere, so sandbox:// previously landed on tmpfs. The new bind puts the bundle (and thus the mount) on the scratch disk; teardown also verifies bundle/scratch dirs are removed.
Shared scratch workload v1 /dev/sd* (new bind) clean Container with reuse-scratch reuses the sandbox's scratch, so its own bundle sits on tmpfs. The path rule detects the non-nested scratch and binds — the case where v1 newly needs the bind, proving the rule is layout-based rather than shim-based.
Shared scratch workload v2 /dev/sd* (fixed) clean Same reuse-scratch container on v2; confirms the fix holds when scratch is shared and the bundle is on tmpfs.
Multi-pod, single UVM (tenant-sandbox-id) v2 /dev/sd* on every tenant clean partial + full teardown Multiple pods packed into one UVM via tenant-sandbox-id; each tenant gets its own pod namespace whose bundle is bound to that tenant's scratch, so every tenant's sandbox:// is disk-backed, and removing one tenant leaves the others (and the UVM) intact.

Additional cases:

  • Multiple sandbox:// mounts on one container — all disk-backed.
  • sandbox-tmp:// mount — remains tmpfs as intended (unchanged).
  • Container with no sandbox:// mount — starts and stops normally.
  • Exec from a sandbox:// mount — succeeds (not noexec).
  • bundleNeedsScratchBind unit test — passes (Linux/WSL).
  • Controller unit tests and GOOS=linux guest build — pass.
  • Shim + guest logs (via LogMonitor / ETW) — no bind or EBUSY errors across the full run.

A `sandbox://` mount is served from a directory under the container bundle,
so the bundle must live on the pod scratch disk for the mount to be
disk-backed. The v1 shim mounts the scratch disk at the bundle path, so
this holds. The v2 shim keeps the bundle on the `UVM tmpfs` and mounts the
scratch elsewhere, so `sandbox://` mounts end up on `tmpfs` instead of disk.

In the guest, when the scratch dir is not already nested under the bundle,
bind a scratch directory onto the bundle before mounting the overlay. The
choice is path-based (`bundleNeedsScratchBind`), so both shims share one rule.
Unmount the bind in `Container.Delete`, and reorder `releaseResources` so the
guest releases the disk-backed mounts before the host detaches the scratch.

Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
Comment thread internal/guest/runtime/hcsv2/uvm.go Outdated
Signed-off-by: Harsh Rawat <harshrawat@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants