Add Storage TSG: ghost CSV mount points (ClusterStorage.000 / .001 / .00X)

[John Neemes (1008covingtonlane)]

What this adds

A new Storage TSG for ghost CSV mount points: the numbered directories
(C:\ClusterStorage.000, .001, .00X) that appear beside the real Cluster
Shared Volume root.

They are usually harmless remnants. They become an incident when a VM, a cluster
resource, or an Azure Local platform component still references one, because
solution updates and Arc Resource Bridge operations then fail with errors that
never mention a folder name.

Why now

• No existing coverage. A sweep of the published TSG corpus and the internal
  engineering wiki returned nothing for this condition.
• Not detectable centrally. Ghost roots are per-node filesystem state and are
  not carried in Azure Local observability pipelines, so this has to be an on-box
  procedure.
• It recurs in the field. The condition shows up repeatedly across OEMs, often
  with a new numbered directory created after each solution update.

Structure

• Quick triage that fans out to every node, explicitly labelled as a first look
  rather than a clearance.
• Detection in three steps: find the roots, confirm which mount points are real,
  and check whether a ghost root still redirects to live storage.
• Classification into three outcomes, each routing to one path.
• Path A clean up when nothing references the roots.
• Path B repoint a workload VM with Move-VMStorage.
• Path C engage support when the reference is platform-managed content under
  Infrastructure_1, which is outside the customer support boundary.

Root cause and prevention

Root cause is attributed to CSV path initialization failing while another process
holds an open handle, with antivirus and filter drivers as the common trigger,
cited to the Microsoft Windows Support Team article on the behavior. Prevention
points at the documented Hyper-V antivirus exclusions for C:\ClusterStorage,
which is the actual fix for recurrence.

The guide deliberately does not claim a specific Event ID for the rename,
because none could be confirmed. It notes that Events 5120 and 5142 are related
CSV I/O events but are not this record, so neither their presence nor their
absence proves anything here.

Validation

Run end to end on a lab cluster: baseline clean, inject, detect, mitigate using
the guide's own Move-VMStorage steps, revalidate, clean up, tear down, with
zero residue afterwards.

That run surfaced three defects, all now fixed in the guide:

1. Move-VMStorage rejects the -Vhds hashtable when DestinationFilePath
   holds a PSObject-wrapped string returned by Join-Path. The error claims the
   key is missing when it is present, and the wrapper is invisible to normal
   checks: .GetType() reports String and -is [string] reports True. The
   guide now uses explicit [string] casts and explains why.
2. Passing an empty -Vhds array fails the call, so the guide says to omit the
   parameter when only configuration or paging paths are affected.
3. Windows -Filter 'ClusterStorage.*' also matches the real
   C:\ClusterStorage root. Every block pairs the filter with a
   ^ClusterStorage\.\d+$ guard, and the guide warns against simplifying it,
   since feeding an unguarded filter into a delete command would target the live
   CSV root.

Safety

• All detection is read-only.
• The pre-delete gate calls the same audit function as the classification
  step, so it cannot be narrower than the check it claims to reconfirm.
• The delete block refuses to recurse into reparse points and requires a typed
  confirmation rather than relying on the operator removing -WhatIf.
• Move-VMStorage is dry-run with -WhatIf first, and the destination is
  validated against the live CSV list before anything moves.

Review notes

• Structure and safety lint: A. All 24 PowerShell blocks parse clean and all
  15 in-page anchors resolve.
• Reviewed through deep-systems, accessibility, and literal-follower lenses across
  multiple model families; the resulting fixes include the glossary, the elevated
  PowerShell prerequisite, the reparse-point attribute check replacing an
  unreliable LinkType test, and the note that per-node Get-VM cannot see
  clustered VMs owned by other nodes.

[Copilot]

Pull request overview

Adds a Storage troubleshooting guide for detecting, classifying, remediating, and preventing ghost CSV mount points.

Changes:

• Adds cluster-wide detection and safety checks.
• Documents cleanup, VM migration, support escalation, and verification paths.
• Adds the guide to the Storage index.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md	Adds the troubleshooting guide and PowerShell procedures.
TSG/Storage/README.md	Links the new guide from the Storage index.

Suppressed comments (2)

TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md:848

• What: both runnable examples always pass -Vhds $vhds, although the preceding branch explicitly allows $vhds to be empty. Why: in the configuration/checkpoint/paging-only case, the guide states that this call is rejected, so both the dry run and real move fail unless the reader manually rewrites the command. How: build a parameter hashtable, add Vhds only when $vhds.Count -gt 0, and use the same splat for both calls.

   Move-VMStorage -Name $VMName `
       -VirtualMachinePath  $Destination `
       -SnapshotFilePath    $Destination `
       -SmartPagingFilePath $Destination `
       -Vhds                $vhds `
       -WhatIf

TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md:710

• What: the safety scan checks descendants but never checks whether a target root itself is a reparse point. Why: a matching C:\ClusterStorage.<digits> junction can pass this gate and then be sent to Remove-Item -Recurse, despite the block's promise to refuse reparse points. How: test each $t.Attributes first and skip descendant traversal when the root is a reparse point.

       $unsafe = foreach ($t in $targets) {
           Get-ChildItem -LiteralPath $t.FullName -Force -Recurse -ErrorAction SilentlyContinue |
               Where-Object { $_.Attributes -band [System.IO.FileAttributes]::ReparsePoint }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Karl Wester-Ebbinghaus (Karl-WE)]

thank you for the update to the TSG documentation. Is there any signal this root issue will be fixed? (RCA) because it happens with every AzL update.