From c1eeb21ea247fb2a7129c84df1d4246b13749dac Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 09:03:52 -0400 Subject: [PATCH 01/10] Add Storage TSG: ghost CSV mount points (ClusterStorage.000 / .001 / .00X) Azure Local and Windows Server failover clusters can leave numbered directories such as C:\ClusterStorage.000 beside the real CSV root. They are usually harmless remnants, but when a VM, cluster resource, or platform component still references one, solution updates and Arc Resource Bridge operations fail with errors that never mention a folder name. Nothing in the public TSG corpus covered this, and it is not detectable from fleet telemetry because ghost roots are per-node filesystem state. The guide covers detection, classification, and three remediation paths: clean up when unreferenced, repoint a workload VM with Move-VMStorage, or engage support when the reference is platform-managed content under Infrastructure_1. Root cause is cited to the Microsoft Windows Support Team article on CSV path initialization failing under an open handle, with antivirus and filter drivers named as the common trigger. Prevention points at the documented Hyper-V antivirus exclusions for C:\ClusterStorage. Validated end to end on a lab cluster (inject, detect, mitigate, revalidate, cleanup, teardown, zero residue). That run surfaced three defects now fixed in the guide: * Move-VMStorage rejects the -Vhds hashtable when DestinationFilePath holds a PSObject-wrapped string from Join-Path, reporting a misleading "must contain 'DestinationFilePath' key" error even though the key is present. The guide uses explicit [string] casts and explains why. * Passing an empty -Vhds array makes the call fail, so the guide says to omit the parameter when only config or paging paths are affected. * Windows -Filter 'ClusterStorage.*' also matches the real C:\ClusterStorage root, so every block pairs it with a ^ClusterStorage\.\d+$ guard and the guide warns against simplifying it. Safety: all detection is read-only, the deletion gate reuses the same audit function as the classification step, refuses to recurse into reparse points, and requires a typed confirmation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8d31ddd7-9fbb-4ff0-8aa7-3f9d88060ed7 --- TSG/Storage/README.md | 1 + ...roubleshoot-Storage-GhostCsvMountPoints.md | 1031 +++++++++++++++++ 2 files changed, 1032 insertions(+) create mode 100644 TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md diff --git a/TSG/Storage/README.md b/TSG/Storage/README.md index 016b7e09..9797bfc7 100644 --- a/TSG/Storage/README.md +++ b/TSG/Storage/README.md @@ -4,3 +4,4 @@ * [How To: Add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) * [Troubleshoot: Physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md) * [Troubleshoot: Storage pool capacity threshold warning (fixed vs thin volumes)](./Troubleshoot-Storage-StoragePoolCapacityThreshold.md) +* [Troubleshoot: Ghost CSV mount points (`C:\ClusterStorage.000`, `.001`, `.00X`)](./Troubleshoot-Storage-GhostCsvMountPoints.md) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md new file mode 100644 index 00000000..ac911798 --- /dev/null +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -0,0 +1,1031 @@ +# Troubleshoot ghost CSV mount points (`C:\ClusterStorage.000`, `.001`, `.00X`) + + + + + + + + + + + + + + + + + + +
ComponentStorage (Cluster Shared Volumes)
SeverityMedium (see At a glance; rises to + High when any object still references a ghost path)
Applicable ScenariosDay 2 Operations: solution update readiness, Arc Resource + Bridge (ARB) operations, clustered Hyper-V VM management
Affected VersionsAll Azure Local releases (and Windows Server failover + clusters using Cluster Shared Volumes)
+ +> **In plain terms:** you have found one or more extra folders next to the normal +> `C:\ClusterStorage` folder, named `C:\ClusterStorage.000`, `C:\ClusterStorage.001` +> and so on. These are leftovers, usually called *ghost folders*. Most of the time +> they are harmless clutter. They become a real problem when a virtual machine, a +> cluster resource, or an Azure Local platform component is still pointed at one of +> them, because that reference can break a solution update or stop a VM from +> starting, with an error that says nothing about folder names. +> +> **The single most important rule on this page: do not delete these folders until +> you have proved nothing references them.** Deleting first is how a tidy-up turns +> into an outage. + +## At a glance + + + + + + + + + + + + + + + + + + +
Business impactUsually low. A ghost folder that nothing references is + cosmetic. High if anything still references it: solution + updates can fail mid-flight, ARB VMs can fail to start or redeploy, and + checkpoint merges can break because a virtual disk's parent is resolved through + the wrong mount point.
Who owns thisThe customer's cluster administrator. This is not a + networking or OEM issue. If the only references are under + Infrastructure_1 (ARB or platform-managed content), stop and + engage Microsoft Support rather than self-remediating.
Typical time to resolveDetection: minutes, and read-only. Repointing a workload VM: tens of + minutes, and it moves data. Cleanup after verification: minutes.
Downtime / maintenance windowAll detection in this guide is read-only and online. + Repointing VM storage moves data and should be scheduled; depending on the VM + and the amount of data it may require the VM to be offline.
+ +## Terms used on this page + +If you are new to failover clustering, read this first. The safety decisions later +on depend on these words. + +| Term | What it means here | +| --- | --- | +| **CSV** (Cluster Shared Volume) | Shared cluster storage that every node can access at the same time. Each CSV appears as a folder under `C:\ClusterStorage`. | +| **CSV root** | The folder `C:\ClusterStorage` itself. The Cluster service owns it and puts one folder inside it per CSV. | +| **Ghost root / ghost folder** | A leftover copy of the CSV root that the Cluster service renamed to a numbered name such as `C:\ClusterStorage.000`. | +| **Mount point** | A folder that is really a doorway into a whole disk volume. Opening it shows that volume's contents. Deleting it can affect real data. | +| **Junction / reparse point** | The Windows mechanism behind a mount point: a folder that redirects somewhere else. A `-Recurse` delete can follow one into live data. | +| **Open handle** | A file or folder currently held open by a running program. Windows will not let you rename or move something while a handle is open on it. | +| **SMB** | The Windows file-sharing protocol. `Get-SmbOpenFile` shows which files other machines currently have open. | +| **ARB** (Arc Resource Bridge) | An Azure Local platform component that runs as a VM on the cluster. Managed by the platform, not by you. | +| **MOC / MocArb** | The platform layer underneath ARB. Its working files live on the infrastructure volume. Platform-managed, not customer-managed. | +| **`Infrastructure_1`** | The reserved Azure Local infrastructure volume. It holds platform configuration and working data, is not for customer workloads, and the platform blocks you from placing storage on it. | + +## Before you start + +### What you need + +- **An elevated PowerShell session** (Run as Administrator) **on one of the cluster + nodes**. Every command on this page assumes that. +- **Cluster administrator rights** on the cluster. +- The **FailoverClusters** and **Hyper-V** PowerShell modules, present by default on + Azure Local nodes. +- **PowerShell remoting working between nodes**, because the cluster-wide checks use + `Invoke-Command`. + +Confirm all of that before going further: + +```powershell +# Every value below must be True before you continue. +$id = [Security.Principal.WindowsIdentity]::GetCurrent() +[pscustomobject]@{ + RunningAsAdmin = ([Security.Principal.WindowsPrincipal]$id).IsInRole( + [Security.Principal.WindowsBuiltInRole]::Administrator) + FailoverClustersAvail = [bool](Get-Module -ListAvailable FailoverClusters) + HyperVAvail = [bool](Get-Module -ListAvailable Hyper-V) + ClusterReachable = [bool](Get-Cluster -ErrorAction SilentlyContinue) +} | Format-List +``` + +### Safety rules + +> [!WARNING] +> **Never delete a `C:\ClusterStorage.00X` folder until every check in +> [Step 3](#step-3-classify-what-you-found) reports no references.** +> A ghost folder can still contain, or still redirect to, live data. Deleting it +> while something points at it can take virtual machines or Arc Resource Bridge +> offline, and the resulting failure will not obviously point back at the deletion. + +> [!IMPORTANT] +> [Step 1](#step-1-find-the-ghost-roots) and +> [Step 2](#step-2-prove-whether-anything-references-them) are **read-only**. Run the +> whole detection pass first and decide afterwards. Do not scroll ahead and run a +> cleanup block on its own: the cleanup in +> [Path A](#path-a-no-references-found-safe-to-clean-up) assumes you have completed +> Steps 1 to 3 in order. + +## Quick triage (start here) + +> [!IMPORTANT] +> This is a fast first look, **not a clearance**. It covers the per-node checks +> only. Zero references here does **not** mean the cluster is clear, because the +> cluster-wide resource check and the mount-point check have not run yet. Treat a +> clean result as "keep going", never as "safe to delete". + +Run this on **any one node**. It fans out to every running node. + +```powershell +# The canonical ghost-path pattern used throughout this guide. +# It matches a literal dot followed by one or more digits after "ClusterStorage", +# for example C:\ClusterStorage.000\... or C:\ClusterStorage.001\... +# It deliberately does NOT match the real root C:\ClusterStorage\... +$GhostPathPattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' + +$nodes = (Get-ClusterNode | Where-Object State -eq 'Up').Name + +Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock { + param($Pattern) + + $findings = New-Object System.Collections.Generic.List[string] + + $ghostRoots = Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + Select-Object -ExpandProperty FullName + + foreach ($vm in (Get-VM -ErrorAction SilentlyContinue)) { + foreach ($d in ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue)) { + if ($d.Path -match $Pattern) { $findings.Add("VMHardDisk: $($vm.Name) -> $($d.Path)") } + } + foreach ($prop in 'ConfigurationLocation', 'SnapshotFileLocation', 'SmartPagingFilePath') { + $v = $vm.$prop + if ($v -and ($v -match $Pattern)) { $findings.Add("VMConfig: $($vm.Name).$prop -> $v") } + } + } + + foreach ($f in (Get-SmbOpenFile -ErrorAction SilentlyContinue)) { + if ($f.Path -match $Pattern) { + $findings.Add("SmbOpenFile: $($f.ClientUserName)@$($f.ClientComputerName) -> $($f.Path)") + } + } + + [pscustomobject]@{ + Node = $env:COMPUTERNAME + GhostRoots = if ($ghostRoots) { $ghostRoots -join '; ' } else { '' } + ReferenceCount = $findings.Count + Findings = if ($findings.Count) { $findings -join ' || ' } else { '' } + } +} | Select-Object Node, GhostRoots, ReferenceCount, Findings | Format-List +``` + +| What you see | What it means | Where to go | +| --- | --- | --- | +| `GhostRoots = ` on every node | No ghost folders anywhere. Nothing to do. | Stop here. | +| Ghost roots exist, `ReferenceCount = 0` | Possible cleanup candidate, **not yet cleared**. | [Step 1](#step-1-find-the-ghost-roots) | +| `ReferenceCount` above 0 on any node | Something is actively pointed at a ghost path. | [Step 1](#step-1-find-the-ghost-roots) | + +> [!NOTE] +> `Get-VM` on a node lists only the VMs currently **resident on that node**. A +> clustered VM owned by another node, or one that is powered off, can still hold a +> ghost reference these per-node passes do not see. The cluster-wide check in +> [Step 2C](#2c-cluster-resource-parameters-run-once-for-the-whole-cluster) is what +> catches those, which is why the quick triage alone is never a clearance. + +## Symptoms + +**Observable behaviors:** + +- One or more folders named `C:\ClusterStorage.000`, `C:\ClusterStorage.001`, + `C:\ClusterStorage.00X` exist alongside the normal `C:\ClusterStorage` folder. +- New numbered folders appear after every solution update, or after a node restart. +- A solution update fails part-way through, and the failure references a path + containing a numbered root. +- An Arc Resource Bridge VM fails to start, or ARB redeployment fails, after an + update. +- A checkpoint (snapshot) merge fails, or a VM reports a missing or broken parent + disk. +- A clustered VM role fails to come online on one node but works on another. + +**Common error messages:** + +Errors caused by this condition usually name the *path*, not the cause. Look for a +numbered root anywhere in the message: + +``` +... C:\ClusterStorage.000\Infrastructure_1\... +``` + +## Root cause + +### The namespace + +Cluster Shared Volumes are surfaced to every node through a single common namespace +on the system drive. Microsoft documents this: + +> Disks in CSVs are accessed using a path that appears as a numbered volume under +> the `\ClusterStorage` folder on the system drive. This path is consistent across +> all nodes in the cluster. +> +> Source: [Cluster Shared Volumes overview](https://learn.microsoft.com/windows-server/failover-clustering/failover-cluster-csvs#requirements-and-considerations-for-using-csv-in-a-failover-cluster) + +So `C:\ClusterStorage` is an ordinary directory on the system drive that the Cluster +service owns and fills with one mount point per CSV. + +### Why the numbered folders appear + +When a CSV is brought online on its owner node, the Cluster service initializes the +CSV namespace under that root. If that initialization fails, because another process +is holding an open handle on the directory at that moment, the Cluster service does +not fail the operation. As a recovery action it renames the failed directory out of +the way to the next free numbered name (`ClusterStorage.000`, then `.001`, and so +on) and creates a fresh `C:\ClusterStorage`. + +The Microsoft Windows Support Team documents this recovery behavior and attributes +the underlying failure to security software and filter-driver products blocking +access to the CSV path: + +> Source: [Break chain on multi mount points](https://jpwinsup.github.io/blog/2025/01/13/Hyper-V/Break-chain-on-multi-mount-points/), +> Microsoft Windows Support Team blog, 13 January 2025. + +That is consistent with Microsoft's own guidance to exclude `C:\ClusterStorage` from +antivirus scanning on Hyper-V hosts, which is the practical prevention for this +condition. See [Prevention](#prevention). + +The result is: + +- The cluster keeps working, so nothing obviously breaks at the time. +- The **old** directory survives on disk under the numbered name. +- Anything that recorded an absolute path while the old directory was the live root + is now pointing at a stale location. + +### Why a referenced ghost path is dangerous, not just untidy + +Having more than one mount point for what should be a single CSV can break virtual +disk parent and child differencing chains during a checkpoint merge, because a +virtual disk's parent locator can resolve through the wrong numbered mount point. +That hazard is documented in the same Microsoft Windows Support Team article. + +On Azure Local the stakes are higher again, because the platform stores its own +content on the reserved infrastructure volume, conventionally `Infrastructure_1`. +That volume holds platform configuration and working data and is explicitly not +intended for customer workloads. A stale reference there can break a platform +operation such as a solution update or an ARB lifecycle action rather than a single +VM, which is why +[Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) routes +those cases to Microsoft Support instead of self-service repair. + +## Where this shows up + +Knowing where the problem is **not** visible saves as much time as knowing where it +is. + +- **PowerShell on a cluster node**: the primary surface. `Get-ChildItem C:\` shows + the numbered folders, and the audit in + [Step 2](#step-2-prove-whether-anything-references-them) shows what references + them. +- **Component and tool log files on disk**: solution update and deployment logs under + `C:\CloudDeployment\Logs` and `C:\MASLogs` contain the numbered path when an update + failure is caused by a stale reference. See + [Step 2E](#2e-optional-search-logs-and-configuration-for-stale-references). +- **Cluster logs**: `Get-ClusterLog` output records the Cluster service's handling of + the CSV root, so it is the best place to establish *when* a ghost root appeared. + See [Step 2F](#2f-optional-establish-when-the-ghost-root-appeared). +- **Windows event logs**: there is no dedicated event that announces this rename. + Failover Clustering events appear under the `Microsoft-Windows-FailoverClustering` + provider, and CSV trouble often shows Events **5120** and **5142**, but those are + I/O availability events and are **not** the rename record. Do not treat a 5120 or + 5142 as proof of a ghost root, and do not treat their absence as proof there is + none. Use the on-disk check in [Step 1](#step-1-find-the-ghost-roots), which is + definitive. +- **Windows Failover Cluster Manager**: the ghost folders do not appear here. The + console shows CSVs by friendly name and current state, not the filesystem layout of + the system drive. +- **Azure portal**: the ghost folders do not appear here. The only indirect signal is + a failed solution update on the cluster's Updates blade, which does not name the + folder. +- **Windows Admin Center on a standalone host**: the ghost folders do not appear + here. +- **Windows Admin Center in the Azure portal**: the ghost folders do not appear here. + +> [!NOTE] +> Fleet telemetry does not carry this condition either. Ghost CSV roots are +> filesystem state on each node's system drive and are not shipped to Azure Local +> observability pipelines, so detection has to be run on the cluster using this +> guide rather than looked up centrally. + +## Step 1: find the ghost roots + +### 1A. List the numbered roots + +```powershell +Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + Select-Object FullName, CreationTime, LastWriteTime | + Sort-Object CreationTime +``` + +> [!WARNING] +> The `Where-Object` guard is not optional. Windows `-Filter 'ClusterStorage.*'` +> also matches the **real** CSV root `C:\ClusterStorage`, because the filter treats +> the part after the dot as an optional extension. Every block on this page pairs the +> filter with `^ClusterStorage\.\d+$` for that reason. If you simplify the filter and +> then feed the result into a delete command, you will target the live CSV root. + +`CreationTime` usually lines up with the update or restart that created the ghost +root. + +### 1B. Confirm which mount points are the real ones + +This is what tells ghost apart from real. Active CSV mount points must be under +`C:\ClusterStorage`, never under a numbered root. + +```powershell +Get-ClusterSharedVolume | ForEach-Object { + [pscustomobject]@{ + CSVName = $_.Name + Path = $_.SharedVolumeInfo.FriendlyVolumeName + State = $_.State + } +} | Format-Table -AutoSize +``` + +Expected output shows paths such as `C:\ClusterStorage\UserStorage_1` or +`C:\ClusterStorage\Infrastructure_1`. **Write down a destination volume from this +list**, because [Path B](#path-b-a-workload-vm-references-a-ghost-path) requires a +real path taken from here. + +> [!WARNING] +> If any **active** CSV reports a `FriendlyVolumeName` under a numbered root such as +> `C:\ClusterStorage.000\...`, that numbered root is **not** a ghost, it is the live +> namespace. Do not treat it as cleanup. Stop and engage Microsoft Support. + +### 1C. Check whether the ghost root still redirects to live data + +A ghost root can contain leftover mount points that still resolve to a real volume. +That is the difference between deleting an empty shell and detaching live storage. + +Check the **`ReparsePoint` file attribute**, which is the authoritative signal. + +> [!IMPORTANT] +> Do not rely on the `LinkType` or `Target` properties for this decision. They do not +> reliably populate for volume mount points, so a folder that really does redirect to +> a live volume can show a blank `LinkType` and be mistaken for ordinary leftover +> files. The attribute check below does not have that failure mode. + +```powershell +Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + ForEach-Object { + $root = $_.FullName + $children = Get-ChildItem -LiteralPath $root -Force -ErrorAction SilentlyContinue + if (-not $children) { + [pscustomobject]@{ GhostRoot = $root; Child = ''; IsReparsePoint = $false; Detail = '' } + } + else { + foreach ($c in $children) { + $isReparse = [bool]($c.Attributes -band [System.IO.FileAttributes]::ReparsePoint) + $detail = '' + if ($isReparse) { + $detail = (fsutil reparsepoint query "$($c.FullName)" 2>&1 | Out-String).Trim() + if ($detail.Length -gt 200) { $detail = $detail.Substring(0, 200) } + } + [pscustomobject]@{ + GhostRoot = $root + Child = $c.Name + IsReparsePoint = $isReparse + Detail = $detail + } + } + } + } | Format-Table -AutoSize +``` + +| Result | Meaning | +| --- | --- | +| `Child = ` | Empty shell. Lowest risk. | +| `IsReparsePoint = False` with files present | Ordinary leftover files. Inventory them in [Step 2D](#2d-inventory-what-is-actually-inside). | +| `IsReparsePoint = True` on any child | The ghost root **still redirects to a volume**. Treat it as live storage. Do not delete. Go to [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). | + +## Step 2: prove whether anything references them + +Each check answers "is something pointed at a ghost path?" from a different angle. +Run all of them. A single miss is what turns a cleanup into an incident. + +**Where to run each check:** + +| Check | Run it | +| --- | --- | +| 2A, 2B, 2D, 2E | On **every node**. Use the quick-triage fan-out, or repeat per node. | +| 2C, 2F | **Once**, from any one node. These are cluster-wide. | + +Define the pattern once in every session where you run these: + +```powershell +$GhostPathPattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' +``` + +> [!NOTE] +> Use this exact pattern. A simpler filter such as `ClusterStorage.` also matches the +> healthy root `C:\ClusterStorage\...`, because `.` is a regular-expression wildcard, +> which produces a false positive on every cluster. This pattern requires a literal +> dot followed by digits, so it also correctly ignores unrelated names such as +> `ClusterStorage.backup`. + +### 2A. Virtual machine disk paths + +```powershell +Get-VM | Get-VMHardDiskDrive | + Where-Object { $_.Path -match $GhostPathPattern } | + Select-Object VMName, ControllerType, ControllerNumber, ControllerLocation, Path | + Format-Table -AutoSize +``` + +### 2B. Virtual machine configuration, checkpoint, and paging paths + +A VM can have healthy disks and still be anchored to a ghost root by one of these +three properties. + +```powershell +Get-VM | Select-Object Name, ConfigurationLocation, SnapshotFileLocation, SmartPagingFilePath | + Where-Object { + $_.ConfigurationLocation -match $GhostPathPattern -or + $_.SnapshotFileLocation -match $GhostPathPattern -or + $_.SmartPagingFilePath -match $GhostPathPattern + } | Format-Table -AutoSize +``` + +### 2C. Cluster resource parameters (run once for the whole cluster) + +This is the check that catches clustered VMs owned by another node, powered-off +roles, platform-managed resources, and anything that is not a Hyper-V VM on the node +you happen to be sitting on. Run it **once** from any node. + +```powershell +Get-ClusterResource | ForEach-Object { + $r = $_ + try { + Get-ClusterParameter -InputObject $r -ErrorAction Stop | ForEach-Object { + if (($_.Value -is [string]) -and ($_.Value -match $GhostPathPattern)) { + [pscustomobject]@{ Resource = $r.Name; Parameter = $_.Name; Value = $_.Value } + } + } + } + catch { + # Some resource types do not expose parameters; ignore and continue. + } +} | Sort-Object Resource, Parameter | Format-Table -AutoSize +``` + +### 2D. Inventory what is actually inside + +If the ghost roots are empty, that alone is strong evidence nothing is using them. + +```powershell +Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + ForEach-Object { + $root = $_.FullName + Write-Host "`n==== $root ====" -ForegroundColor Cyan + $items = Get-ChildItem -LiteralPath $root -Recurse -Force -ErrorAction SilentlyContinue + if (-not $items) { + Write-Host ' (empty)' + } + else { + $items | Sort-Object LastWriteTime -Descending | + Select-Object -First 25 Mode, LastWriteTime, Length, FullName | + Format-Table -AutoSize | Out-String | Write-Host + Write-Host (" total items: {0}" -f @($items).Count) + } + } +``` + +> [!IMPORTANT] +> If the inventory shows content under a path resembling +> `...\Infrastructure_1\Shares\SU1_Infrastructure_1\...`, including MocArb or +> Orchestration working directories, this is Azure Local platform content, not +> customer data. Go to +> [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). + +### 2E. Optional: search logs and configuration for stale references + +Useful when an update or ARB operation already failed and you want to confirm the +ghost path is implicated. + +```powershell +$SearchRoots = @('C:\CloudDeployment\Logs', 'C:\MASLogs', 'C:\Windows\Cluster') + +foreach ($root in $SearchRoots) { + if (Test-Path -LiteralPath $root) { + Get-ChildItem -LiteralPath $root -Recurse -File -ErrorAction SilentlyContinue | + Where-Object { $_.Length -lt 200MB } | + Select-String -Pattern 'ClusterStorage\.\d+' -List -ErrorAction SilentlyContinue | + Select-Object -First 50 @{N='File';E={$_.Path}}, @{N='Match';E={$_.Matches[0].Value}} + } +} +``` + +> [!NOTE] +> This can take several minutes on a node with large log directories. It is read-only +> and safe to interrupt. + +### 2F. Optional: establish when the ghost root appeared + +```powershell +# Generates cluster logs covering the last 3 days into the current folder. +Get-ClusterLog -Destination . -TimeSpan 4320 + +Select-String -Path .\*.log -Pattern 'ClusterStorage\.\d+' | + Select-Object -First 40 Filename, LineNumber, Line +``` + +Correlate the timestamps with your update and restart history. That tells you which +operation created the ghost root, which is what you need to stop it recurring. + +> [!NOTE] +> Search for the numbered path itself, as above. Do not search for a specific +> sentence of cluster-log text: the exact wording is not a documented, stable string, +> so a phrase search can return nothing on a cluster that genuinely has the problem. + +## Step 3: classify what you found + +Combine the results and place the cluster in exactly one category. + +| Classification | Criteria (all must hold) | Action | +| --- | --- | --- | +| **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only stale files with no references | [Path A](#path-a-no-references-found-safe-to-clean-up) | +| **Unsafe, active references found** | Any of 2A, 2B, 2C returns a row, or an SMB open file exists under a ghost path, and the referencing object is a **customer workload VM** | [Path B](#path-b-a-workload-vm-references-a-ghost-path) | +| **Unsafe, platform references found** | Any reference is under `Infrastructure_1`, or names ARB / MOC / MocArb / Orchestration content, or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | + +> [!WARNING] +> If you are unsure which category applies, treat it as **Path C** and engage +> Microsoft Support. The cost of asking is minutes. The cost of deleting a referenced +> ghost root is an outage. + +## Path A: no references found, safe to clean up + +### Prerequisites + +- Steps 1, 2 and 3 completed **in order**, on **every** node, with no findings. +- Step 1C shows `IsReparsePoint = False` for every child of every ghost root. +- If the ghost roots contain any files at all, you have either copied them somewhere + safe or confirmed with the data owner that they are disposable. An inventory export + is a record, not a backup. + +### Steps + +1. **Paste the shared audit function.** + Every gate below calls this one function, so the final safety check is exactly as + strict as the Step 3 classification rather than a narrower subset of it. Paste it + once into your elevated session. + + ```powershell + function Get-GhostCsvAudit { + [CmdletBinding()] + param() + + $pattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' + $blockers = New-Object System.Collections.Generic.List[string] + + # (1) An active CSV mounted under a numbered root means this is not a ghost. + foreach ($csv in (Get-ClusterSharedVolume -ErrorAction SilentlyContinue)) { + if ($csv.SharedVolumeInfo.FriendlyVolumeName -match $pattern) { + $blockers.Add("ActiveCsvUnderNumberedRoot: $($csv.Name) -> $($csv.SharedVolumeInfo.FriendlyVolumeName)") + } + } + + # (2) Cluster-wide resource parameters: catches VMs owned by other nodes. + foreach ($r in (Get-ClusterResource -ErrorAction SilentlyContinue)) { + try { + foreach ($p in (Get-ClusterParameter -InputObject $r -ErrorAction Stop)) { + if (($p.Value -is [string]) -and ($p.Value -match $pattern)) { + $blockers.Add("ClusterResource: $($r.Name).$($p.Name) -> $($p.Value)") + } + } + } catch { } + } + + # (3) Per-node checks across every running node. + $nodes = (Get-ClusterNode -ErrorAction SilentlyContinue | Where-Object State -eq 'Up').Name + if (-not $nodes) { $nodes = $env:COMPUTERNAME } + + $perNode = Invoke-Command -ComputerName $nodes -ArgumentList $pattern -ScriptBlock { + param($Pattern) + $hits = New-Object System.Collections.Generic.List[string] + + foreach ($vm in (Get-VM -ErrorAction SilentlyContinue)) { + foreach ($d in ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue)) { + if ($d.Path -match $Pattern) { $hits.Add("VMHardDisk: $($vm.Name) -> $($d.Path)") } + } + foreach ($prop in 'ConfigurationLocation','SnapshotFileLocation','SmartPagingFilePath') { + $v = $vm.$prop + if ($v -and ($v -match $Pattern)) { $hits.Add("VMConfig: $($vm.Name).$prop -> $v") } + } + } + + foreach ($f in (Get-SmbOpenFile -ErrorAction SilentlyContinue)) { + if ($f.Path -match $Pattern) { $hits.Add("SmbOpenFile: $($f.Path)") } + } + + # A reparse point inside a ghost root means it still redirects to a volume. + foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { + foreach ($c in (Get-ChildItem -LiteralPath $g.FullName -Force -ErrorAction SilentlyContinue)) { + if ($c.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { + $hits.Add("ReparsePoint: $($c.FullName)") + } + } + } + + [pscustomobject]@{ Node = $env:COMPUTERNAME; Hits = @($hits) } + } + + foreach ($n in $perNode) { + foreach ($h in $n.Hits) { $blockers.Add("[$($n.Node)] $h") } + } + + [pscustomobject]@{ + SafeToDelete = ($blockers.Count -eq 0) + BlockerCount = $blockers.Count + Blockers = @($blockers) + } + } + ``` + +2. **Re-confirm immediately before acting.** + State can change between investigation and cleanup, especially if an update ran in + between. + + ```powershell + $audit = Get-GhostCsvAudit + $audit | Format-List + if (-not $audit.SafeToDelete) { $audit.Blockers } + ``` + + **`SafeToDelete` must be `True`.** If it is not, stop and return to + [Step 3](#step-3-classify-what-you-found). + +3. **Record what is there before removing it.** + + ```powershell + $stamp = Get-Date -Format 'yyyyMMdd-HHmmss' + $backup = "C:\Temp\GhostCsvBackup-$stamp" + New-Item -ItemType Directory -Path $backup -Force | Out-Null + + Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + ForEach-Object { + Get-ChildItem -LiteralPath $_.FullName -Recurse -Force -ErrorAction SilentlyContinue | + Select-Object FullName, Length, LastWriteTime | + Export-Csv -Path (Join-Path $backup "$($_.Name)-inventory.csv") -NoTypeInformation + } + Write-Host "Inventory written to $backup" + ``` + + > [!IMPORTANT] + > This writes an **inventory only**. It does not copy file contents, so it does + > not make the deletion reversible. If the ghost roots contain any file you are + > not certain is disposable, copy it somewhere safe before continuing, or treat + > the situation as + > [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). + +4. **Remove the ghost roots, one node at a time.** + This block re-runs the full audit, refuses to touch anything that is a reparse + point, and requires you to type a confirmation. Run it on a single node, confirm + the cluster is healthy, then move to the next. + + ```powershell + $audit = Get-GhostCsvAudit + if (-not $audit.SafeToDelete) { + Write-Warning "Refusing to delete. Blockers:" + $audit.Blockers + } + else { + $targets = Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } + + # Never recurse into a reparse point: that can delete the target volume's data. + $unsafe = foreach ($t in $targets) { + Get-ChildItem -LiteralPath $t.FullName -Force -Recurse -ErrorAction SilentlyContinue | + Where-Object { $_.Attributes -band [System.IO.FileAttributes]::ReparsePoint } + } + + if ($unsafe) { + Write-Warning "Reparse points found inside ghost roots. Refusing to delete:" + $unsafe | Select-Object FullName + } + elseif (-not $targets) { + Write-Host "No ghost roots on $env:COMPUTERNAME." + } + else { + Write-Host "About to permanently delete on $($env:COMPUTERNAME):" -ForegroundColor Yellow + $targets | ForEach-Object { Write-Host " $($_.FullName)" -ForegroundColor Yellow } + $answer = Read-Host "Type DELETE to confirm" + if ($answer -ceq 'DELETE') { + foreach ($t in $targets) { + Remove-Item -LiteralPath $t.FullName -Recurse -Force + Write-Host "Removed $($t.FullName)" + } + } + else { + Write-Host "Cancelled. Nothing was deleted." + } + } + } + ``` + +5. **Verify.** Go to [Verify the fix](#verify-the-fix). + +## Path B: a workload VM references a ghost path + +The fix is to move the VM's storage back onto the canonical CSV path, then confirm +nothing points at the ghost root any more. The supported primitive is +[`Move-VMStorage`](https://learn.microsoft.com/powershell/module/hyper-v/move-vmstorage), +which moves disks, configuration, checkpoints, and the smart paging file. + +### Prerequisites + +- A destination path taken from your + [Step 1B](#1b-confirm-which-mount-points-are-the-real-ones) output. Do not invent + one. +- A maintenance window, since this moves data. +- The VM is a **customer workload** VM. If it is an ARB or platform VM, or its files + are under `Infrastructure_1`, go to + [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). + +### Steps + +1. **Set your variables and validate the destination.** + Replace both placeholder values. The validation refuses to continue unless the + destination is a real, active CSV mount point on this cluster, which prevents + moving VM storage onto the non-clustered system drive by accident. + + ```powershell + # ---- replace both of these ---- + $VMName = 'REPLACE-WITH-VM-NAME' + $CsvRootPath = 'REPLACE-WITH-CSV-PATH-FROM-STEP-1B' # for example C:\ClusterStorage\UserStorage_1 + # ------------------------------- + + $GhostPathPattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' + + if ($VMName -like 'REPLACE-*' -or $CsvRootPath -like 'REPLACE-*') { + throw "Set the VM name and CSV path variables before running this block." + } + if (-not (Get-VM -Name $VMName -ErrorAction SilentlyContinue)) { + throw "VM '$VMName' not found on $env:COMPUTERNAME." + } + + # The destination must be an ACTIVE CSV mount point from Step 1B. + $validCsv = Get-ClusterSharedVolume | ForEach-Object { $_.SharedVolumeInfo.FriendlyVolumeName } + if ($CsvRootPath -notin $validCsv) { + throw "'$CsvRootPath' is not an active CSV mount point. Valid values: $($validCsv -join ', ')" + } + if ($CsvRootPath -match $GhostPathPattern) { + throw "'$CsvRootPath' is itself a ghost path. Pick a canonical CSV from Step 1B." + } + + $Destination = Join-Path $CsvRootPath $VMName + Write-Host "Destination validated: $Destination" -ForegroundColor Green + ``` + +2. **Record the current state of the affected VM.** + Keep this output. It is your rollback reference and your evidence of what changed. + + ```powershell + Get-VM -Name $VMName | + Select-Object Name, State, ConfigurationLocation, SnapshotFileLocation, SmartPagingFilePath | + Format-List + Get-VM -Name $VMName | Get-VMHardDiskDrive | + Select-Object VMName, ControllerType, ControllerNumber, ControllerLocation, Path | + Format-Table -AutoSize + ``` + +3. **Build the disk mapping and review it.** + + ```powershell + New-Item -ItemType Directory -Path $Destination -Force | Out-Null + + # The [string] casts are REQUIRED, and are the most common reason this step + # fails. Join-Path returns a PSObject-wrapped string, and Move-VMStorage + # rejects the wrapper with the misleading error + # "Hash tables in the Vhds parameter must contain 'DestinationFilePath' key" + # even though the key IS present. The wrapper is invisible to normal checks: + # .GetType() reports String and -is [string] reports True. + $vhds = @( + Get-VM -Name $VMName | Get-VMHardDiskDrive | + Where-Object { $_.Path -match $GhostPathPattern } | + ForEach-Object { + @{ + SourceFilePath = [string]$_.Path + DestinationFilePath = [string](Join-Path $Destination (Split-Path $_.Path -Leaf)) + } + } + ) + + if (-not $vhds) { Write-Host "No ghost-rooted disks on '$VMName'." } + $vhds | ForEach-Object { "{0} -> {1}" -f $_.SourceFilePath, $_.DestinationFilePath } + ``` + + **Stop and read that mapping.** Every destination must be under the path you + validated in step 1. + +4. **Dry-run the move, then perform it.** + `Move-VMStorage` supports `-WhatIf`. Always run the dry run and read its output + before the real move. + + > [!NOTE] + > If `$vhds` came back empty, because only the configuration, checkpoint, or + > paging paths were on a ghost root, **omit the `-Vhds` parameter entirely**. + > Passing an empty array makes `Move-VMStorage` reject the call. + + ```powershell + # Dry run: reports what would happen and changes nothing. + Move-VMStorage -Name $VMName ` + -VirtualMachinePath $Destination ` + -SnapshotFilePath $Destination ` + -SmartPagingFilePath $Destination ` + -Vhds $vhds ` + -WhatIf + ``` + + When the dry run looks correct, run the same command without `-WhatIf`: + + ```powershell + Move-VMStorage -Name $VMName ` + -VirtualMachinePath $Destination ` + -SnapshotFilePath $Destination ` + -SmartPagingFilePath $Destination ` + -Vhds $vhds + ``` + + > [!NOTE] + > Supply only the parameters you actually need to change. If the VM's checkpoint + > location was already correct in Step 2B, omit `-SnapshotFilePath` rather than + > moving it unnecessarily. + +5. **Confirm this VM is clean.** + + ```powershell + $vm = Get-VM -Name $VMName + $bad = @() + $bad += ($vm | Get-VMHardDiskDrive | Where-Object { $_.Path -match $GhostPathPattern }).Path + $bad += @($vm.ConfigurationLocation, $vm.SnapshotFileLocation, $vm.SmartPagingFilePath) | + Where-Object { $_ -and ($_ -match $GhostPathPattern) } + + if ($bad) { Write-Warning "Still referencing a ghost path:"; $bad } + else { Write-Host "$VMName no longer references any ghost path." -ForegroundColor Green } + ``` + +6. **Repeat for every VM found in Step 2**, then re-run + [Step 2](#step-2-prove-whether-anything-references-them) in full. + +7. **Only once every reference is cleared**, follow + [Path A](#path-a-no-references-found-safe-to-clean-up) to remove the now-unused + ghost roots. + +## Path C: references under `Infrastructure_1` or ARB, engage support + +Stop and open a support case if **any** of the following is true: + +- A reference points under `...\Infrastructure_1\...`. +- The content resembles ARB, MOC, MocArb, or Orchestration platform data. +- [Step 1C](#1c-check-whether-the-ghost-root-still-redirects-to-live-data) shows + `IsReparsePoint = True` for any child of a ghost root. +- An **active** CSV reports a `FriendlyVolumeName` under a numbered root. + +These are platform-managed paths. `Infrastructure_1` is reserved for Azure Local +system configuration, the platform deliberately blocks customer storage placement on +it, and relocating its content by hand is outside the customer support boundary. A +wrong move can leave Arc Resource Bridge or the solution update pipeline in a state +that needs a rebuild to recover. + +**Collect this before contacting support**, so the case starts with evidence: + +```powershell +$out = "C:\Temp\GhostCsvEvidence-$(Get-Date -Format 'yyyyMMdd-HHmmss')" +New-Item -ItemType Directory -Path $out -Force | Out-Null +$GhostPathPattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' + +Get-ClusterSharedVolume | ForEach-Object { + [pscustomobject]@{ CSVName = $_.Name; Path = $_.SharedVolumeInfo.FriendlyVolumeName; State = $_.State } +} | Export-Csv "$out\csv-mountpoints.csv" -NoTypeInformation + +Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | + Select-Object FullName, CreationTime, LastWriteTime | + Export-Csv "$out\ghost-roots.csv" -NoTypeInformation + +Get-ClusterResource | ForEach-Object { + $r = $_ + try { + Get-ClusterParameter -InputObject $r -ErrorAction Stop | ForEach-Object { + if (($_.Value -is [string]) -and ($_.Value -match $GhostPathPattern)) { + [pscustomobject]@{ Resource = $r.Name; Parameter = $_.Name; Value = $_.Value } + } + } + } catch { } +} | Export-Csv "$out\cluster-resource-refs.csv" -NoTypeInformation + +Write-Host "Evidence collected in $out" +``` + +Also attach the output of +[Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md). + +## Verify the fix + +Run all four. The condition is resolved only when all four are clean. + +```powershell +$GhostPathPattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' +$nodes = (Get-ClusterNode | Where-Object State -eq 'Up').Name + +# 1) No ghost roots, and no VM or SMB reference, on any node. +Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock { + param($Pattern) + $refs = @() + foreach ($vm in (Get-VM -ErrorAction SilentlyContinue)) { + $refs += ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue | + Where-Object { $_.Path -match $Pattern }).Path + $refs += @($vm.ConfigurationLocation, $vm.SnapshotFileLocation, $vm.SmartPagingFilePath) | + Where-Object { $_ -and ($_ -match $Pattern) } + } + $refs += (Get-SmbOpenFile -ErrorAction SilentlyContinue | + Where-Object { $_.Path -match $Pattern }).Path + [pscustomobject]@{ + Node = $env:COMPUTERNAME + GhostRoots = (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' }).Count + References = @($refs).Count + } +} | Select-Object Node, GhostRoots, References | Format-Table -AutoSize + +# 2) Every CSV is Online and mounted under the canonical root. +Get-ClusterSharedVolume | ForEach-Object { + [pscustomobject]@{ + CSVName = $_.Name + Path = $_.SharedVolumeInfo.FriendlyVolumeName + State = $_.State + Canonical = ($_.SharedVolumeInfo.FriendlyVolumeName -notmatch $GhostPathPattern) + } +} | Format-Table -AutoSize + +# 3) No cluster resource references a ghost path. +Get-ClusterResource | ForEach-Object { + $r = $_ + try { + Get-ClusterParameter -InputObject $r -ErrorAction Stop | ForEach-Object { + if (($_.Value -is [string]) -and ($_.Value -match $GhostPathPattern)) { + [pscustomobject]@{ Resource = $r.Name; Parameter = $_.Name; Value = $_.Value } + } + } + } catch { } +} | Format-Table -AutoSize + +# 4) Cluster roles and resources are healthy. +Get-ClusterGroup | Format-Table Name, OwnerNode, State -AutoSize +Get-ClusterResource | Where-Object State -ne 'Online' | Format-Table Name, State, OwnerGroup -AutoSize +``` + +Expected: `GhostRoots = 0` and `References = 0` on every node, every CSV `Online` +with `Canonical = True`, no rows from the third command, and no unexpected offline +resources from the fourth. + +If you reached this page because a solution update failed, re-run the update +readiness checks before retrying the update. + +## Prevention + +- **Exclude the CSV path from antivirus and endpoint scanning on every node.** This + is the single most effective prevention, because a scanner holding a handle on + `C:\ClusterStorage` is the common trigger for the rename. Microsoft publishes the + required exclusions, including `%SystemDrive%\ClusterStorage`, in + [Recommended antivirus exclusions for Hyper-V hosts](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts). + Use the exact syntax in that document, and apply the same thinking to backup and + monitoring agents, not only antivirus. +- **Check before and after every solution update.** Running the + [Quick triage](#quick-triage-start-here) as part of update readiness catches a + stale reference while it is still cheap to fix. +- **Do not create your own folders next to `C:\ClusterStorage`.** A folder matching + the numbered pattern is treated as a ghost root by this guide and by tooling. +- **Investigate recurrence rather than repeatedly cleaning up.** If new numbered + roots keep appearing after each update, something is holding an open handle under + the CSV root when the Cluster service re-establishes it. Use + [Step 2F](#2f-optional-establish-when-the-ghost-root-appeared) to identify the + timing, correlate it with backup, antivirus, monitoring, or third-party storage + agents running at that moment, and exclude the CSV path from them. +- **Keep VM storage on the canonical path.** When creating or importing a VM, target + `C:\ClusterStorage\\...` explicitly rather than accepting a recorded + absolute path from another system. + +## Related Issues + +- [Troubleshooting Storage With Support Diagnostics Tool](./Troubleshooting-Storage-With-Support-Diagnostics-Tool.md) +- [Troubleshoot: Storage pool capacity threshold warning (fixed vs thin volumes)](./Troubleshoot-Storage-StoragePoolCapacityThreshold.md) +- [Cluster Shared Volumes overview](https://learn.microsoft.com/windows-server/failover-clustering/failover-cluster-csvs) +- [Manage Cluster Shared Volumes](https://learn.microsoft.com/windows-server/failover-clustering/failover-cluster-manage-cluster-shared-volumes) +- [Recommended antivirus exclusions for Hyper-V hosts](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts) +- [`Move-VMStorage`](https://learn.microsoft.com/powershell/module/hyper-v/move-vmstorage) +- [Events 5120 and 5142 and unable to access the ClusterStorage folder](https://learn.microsoft.com/troubleshoot/windows-server/backup-and-storage/event-5120-5142-access-clusterstorage-folder) + +--- From f03e2918468ecede64cc327e58d840dfb0b4a9fb Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 09:16:48 -0400 Subject: [PATCH 02/10] Address review: state explicitly that no OEM or firmware action is required The OEM field-engineer review lens was the only reviewer scoring below 5. It asked for an explicit statement in At a glance so a vendor engineer can hand the case back without reading the whole guide. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8d31ddd7-9fbb-4ff0-8aa7-3f9d88060ed7 --- TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index ac911798..fd2889d0 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -48,7 +48,8 @@ Who owns this The customer's cluster administrator. This is not a - networking or OEM issue. If the only references are under + networking issue, and no OEM, BIOS, or firmware action is ever + required. If the only references are under Infrastructure_1 (ARB or platform-managed content), stop and engage Microsoft Support rather than self-remediating. From 47fbc0031bb245907c3ee6102e78d4b3300a7956 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:00:26 -0400 Subject: [PATCH 03/10] Storage ghost-CSV TSG: address copilot-reviewer findings Three fixes for the review bot's findings: 1. Get-GhostCsvAudit fail-CLOSED (was fail-open). SafeToDelete previously depended only on an empty findings list while every query used -ErrorAction SilentlyContinue and cluster-node discovery silently fell back to the local machine, so an unreachable node or failed cmdlet could report SafeToDelete = True without cluster-wide clearance. Now every discovery/query/remoting error is recorded as a blocker, cluster-node discovery failure is a blocker (no local-machine substitution), and every expected running node must return a result or it is a blocker. 2. Root-cause rationale now anchors on Microsoft Learn (Recommended antivirus exclusions for Hyper-V hosts; Events 5120/5142 unable to access the ClusterStorage folder) instead of a non-Microsoft blog. The community blog is retained as an explicitly non-normative secondary reference. 3. Move-VMStorage disk mapping gives each disk a unique per-disk subdirectory, so two disks that share a leaf filename no longer collide on an identical DestinationFilePath (which made Move-VMStorage fail). All 24 PowerShell blocks parse clean (PSParser AST). --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 156 ++++++++++++------ 1 file changed, 110 insertions(+), 46 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index fd2889d0..f973a7fb 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -242,17 +242,21 @@ not fail the operation. As a recovery action it renames the failed directory out the way to the next free numbered name (`ClusterStorage.000`, then `.001`, and so on) and creates a fresh `C:\ClusterStorage`. -The Microsoft Windows Support Team documents this recovery behavior and attributes -the underlying failure to security software and filter-driver products blocking -access to the CSV path: - -> Source: [Break chain on multi mount points](https://jpwinsup.github.io/blog/2025/01/13/Hyper-V/Break-chain-on-multi-mount-points/), +This failure mode is what Microsoft's own guidance addresses. Microsoft documents that +security, backup, and filter-driver products holding a handle on the CSV path cause access +failures under `C:\ClusterStorage`, and publishes the exclusions that prevent them (see +[Recommended antivirus exclusions for Hyper-V hosts](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts) +and [Events 5120 and 5142 and unable to access the ClusterStorage folder](https://learn.microsoft.com/troubleshoot/windows-server/backup-and-storage/event-5120-5142-access-clusterstorage-folder)). +Excluding `C:\ClusterStorage` from antivirus scanning on every node is the practical +prevention for this condition. See [Prevention](#prevention). + +A Microsoft Windows Support Team blog post describes the specific recovery behavior (the +rename to the next numbered name) in more detail. It is a helpful secondary write-up, not a +normative source: + +> Additional reading (non-normative): [Break chain on multi mount points](https://jpwinsup.github.io/blog/2025/01/13/Hyper-V/Break-chain-on-multi-mount-points/), > Microsoft Windows Support Team blog, 13 January 2025. -That is consistent with Microsoft's own guidance to exclude `C:\ClusterStorage` from -antivirus scanning on Hyper-V hosts, which is the practical prevention for this -condition. See [Prevention](#prevention). - The result is: - The cluster keeps working, so nothing obviously breaks at the time. @@ -588,61 +592,113 @@ Combine the results and place the cluster in exactly one category. $pattern = '[\\/]ClusterStorage\.\d+([\\/]|$)' $blockers = New-Object System.Collections.Generic.List[string] + # This gate is FAIL-CLOSED. Any discovery, query, or remoting error is recorded as a + # blocker, so an incomplete audit can never report SafeToDelete = True. Deletion is + # allowed only when every expected check ran to completion AND found nothing. + # (1) An active CSV mounted under a numbered root means this is not a ghost. - foreach ($csv in (Get-ClusterSharedVolume -ErrorAction SilentlyContinue)) { - if ($csv.SharedVolumeInfo.FriendlyVolumeName -match $pattern) { - $blockers.Add("ActiveCsvUnderNumberedRoot: $($csv.Name) -> $($csv.SharedVolumeInfo.FriendlyVolumeName)") + try { + foreach ($csv in (Get-ClusterSharedVolume -ErrorAction Stop)) { + if ($csv.SharedVolumeInfo.FriendlyVolumeName -match $pattern) { + $blockers.Add("ActiveCsvUnderNumberedRoot: $($csv.Name) -> $($csv.SharedVolumeInfo.FriendlyVolumeName)") + } } + } catch { + $blockers.Add("QueryError: Get-ClusterSharedVolume failed -> $($_.Exception.Message)") } # (2) Cluster-wide resource parameters: catches VMs owned by other nodes. - foreach ($r in (Get-ClusterResource -ErrorAction SilentlyContinue)) { - try { - foreach ($p in (Get-ClusterParameter -InputObject $r -ErrorAction Stop)) { - if (($p.Value -is [string]) -and ($p.Value -match $pattern)) { - $blockers.Add("ClusterResource: $($r.Name).$($p.Name) -> $($p.Value)") + try { + foreach ($r in (Get-ClusterResource -ErrorAction Stop)) { + try { + foreach ($p in (Get-ClusterParameter -InputObject $r -ErrorAction Stop)) { + if (($p.Value -is [string]) -and ($p.Value -match $pattern)) { + $blockers.Add("ClusterResource: $($r.Name).$($p.Name) -> $($p.Value)") + } } + } catch { + $blockers.Add("QueryError: Get-ClusterParameter on '$($r.Name)' failed -> $($_.Exception.Message)") } - } catch { } + } + } catch { + $blockers.Add("QueryError: Get-ClusterResource failed -> $($_.Exception.Message)") } - # (3) Per-node checks across every running node. - $nodes = (Get-ClusterNode -ErrorAction SilentlyContinue | Where-Object State -eq 'Up').Name - if (-not $nodes) { $nodes = $env:COMPUTERNAME } + # (3) Per-node checks across every running node. Cluster-node discovery MUST succeed: + # without the node list we cannot prove cluster-wide clearance, so a failure here is a + # blocker, NOT a silent fall back to the local machine. + $nodes = $null + try { + $nodes = @(Get-ClusterNode -ErrorAction Stop | Where-Object State -eq 'Up' | + Select-Object -ExpandProperty Name) + } catch { + $blockers.Add("DiscoveryError: Get-ClusterNode failed; cannot confirm cluster-wide clearance -> $($_.Exception.Message)") + } + if (($null -ne $nodes) -and ($nodes.Count -eq 0)) { + $blockers.Add("DiscoveryError: no cluster node reported 'Up'; cannot confirm cluster-wide clearance") + } - $perNode = Invoke-Command -ComputerName $nodes -ArgumentList $pattern -ScriptBlock { - param($Pattern) - $hits = New-Object System.Collections.Generic.List[string] + if ($nodes -and $nodes.Count -gt 0) { + # -ErrorAction SilentlyContinue keeps one unreachable node from aborting the whole + # sweep; the reported-vs-expected check below turns any missing node into a blocker. + $perNode = Invoke-Command -ComputerName $nodes -ArgumentList $pattern ` + -ErrorAction SilentlyContinue -ScriptBlock { + param($Pattern) + $hits = New-Object System.Collections.Generic.List[string] + $errors = New-Object System.Collections.Generic.List[string] + + try { + foreach ($vm in (Get-VM -ErrorAction Stop)) { + foreach ($d in ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue)) { + if ($d.Path -match $Pattern) { $hits.Add("VMHardDisk: $($vm.Name) -> $($d.Path)") } + } + foreach ($prop in 'ConfigurationLocation','SnapshotFileLocation','SmartPagingFilePath') { + $v = $vm.$prop + if ($v -and ($v -match $Pattern)) { $hits.Add("VMConfig: $($vm.Name).$prop -> $v") } + } + } + } catch { + $errors.Add("Get-VM enumeration failed -> $($_.Exception.Message)") + } - foreach ($vm in (Get-VM -ErrorAction SilentlyContinue)) { - foreach ($d in ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue)) { - if ($d.Path -match $Pattern) { $hits.Add("VMHardDisk: $($vm.Name) -> $($d.Path)") } + try { + foreach ($f in (Get-SmbOpenFile -ErrorAction Stop)) { + if ($f.Path -match $Pattern) { $hits.Add("SmbOpenFile: $($f.Path)") } + } + } catch { + $errors.Add("Get-SmbOpenFile failed -> $($_.Exception.Message)") } - foreach ($prop in 'ConfigurationLocation','SnapshotFileLocation','SmartPagingFilePath') { - $v = $vm.$prop - if ($v -and ($v -match $Pattern)) { $hits.Add("VMConfig: $($vm.Name).$prop -> $v") } + + # A reparse point inside a ghost root means it still redirects to a volume. + try { + foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Stop | + Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { + foreach ($c in (Get-ChildItem -LiteralPath $g.FullName -Force -ErrorAction SilentlyContinue)) { + if ($c.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { + $hits.Add("ReparsePoint: $($c.FullName)") + } + } + } + } catch { + $errors.Add("Ghost-root enumeration failed -> $($_.Exception.Message)") } + + [pscustomobject]@{ Node = $env:COMPUTERNAME; Hits = @($hits); Errors = @($errors) } } - foreach ($f in (Get-SmbOpenFile -ErrorAction SilentlyContinue)) { - if ($f.Path -match $Pattern) { $hits.Add("SmbOpenFile: $($f.Path)") } + foreach ($n in $perNode) { + foreach ($h in $n.Hits) { $blockers.Add("[$($n.Node)] $h") } + foreach ($e in $n.Errors) { $blockers.Add("[$($n.Node)] QueryError: $e") } } - # A reparse point inside a ghost root means it still redirects to a volume. - foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | - Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { - foreach ($c in (Get-ChildItem -LiteralPath $g.FullName -Force -ErrorAction SilentlyContinue)) { - if ($c.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { - $hits.Add("ReparsePoint: $($c.FullName)") - } + # Every expected node MUST return a result. A node that never reported (unreachable, + # WinRM down, remoting refused) is a blocker: we cannot clear what we could not inspect. + $reported = @($perNode | ForEach-Object { $_.PSComputerName }) + foreach ($expected in $nodes) { + if ($reported -notcontains $expected) { + $blockers.Add("[$expected] RemotingError: node did not return an audit result (unreachable or WinRM unavailable)") } } - - [pscustomobject]@{ Node = $env:COMPUTERNAME; Hits = @($hits) } - } - - foreach ($n in $perNode) { - foreach ($h in $n.Hits) { $blockers.Add("[$($n.Node)] $h") } } [pscustomobject]@{ @@ -812,13 +868,21 @@ which moves disks, configuration, checkpoints, and the smart paging file. # "Hash tables in the Vhds parameter must contain 'DestinationFilePath' key" # even though the key IS present. The wrapper is invisible to normal checks: # .GetType() reports String and -is [string] reports True. + # Give each disk its OWN numbered subdirectory. Two disks can share a leaf filename + # (attached from different source folders); mapping both to $Destination\ would + # produce identical DestinationFilePath values and make Move-VMStorage fail. A unique + # per-disk subdirectory keeps the mapping one-to-one while preserving each filename. + $diskIndex = 0 $vhds = @( Get-VM -Name $VMName | Get-VMHardDiskDrive | Where-Object { $_.Path -match $GhostPathPattern } | ForEach-Object { + $diskIndex++ + $diskDir = Join-Path $Destination ('Disk{0:D2}' -f $diskIndex) + New-Item -ItemType Directory -Path $diskDir -Force | Out-Null @{ SourceFilePath = [string]$_.Path - DestinationFilePath = [string](Join-Path $Destination (Split-Path $_.Path -Leaf)) + DestinationFilePath = [string](Join-Path $diskDir (Split-Path $_.Path -Leaf)) } } ) From 45c1074b58cb6eccb3f90ea8443325a0db4ec03c Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 17:13:28 -0400 Subject: [PATCH 04/10] Storage ghost-CSV TSG: address the two optional review notes 1. Get-ClusterParameter error handling made consistent across the three detection/evidence/verify blocks (2C, the Path C evidence collector, and the "Verify the fix" resource check). They previously swallowed the error silently. They now record each resource whose parameters could not be read and surface a summary (a warning for the console checks, and a companion cluster-resource-uninspected.txt for the support-evidence bundle), so a genuine query failure no longer silently reduces coverage. These remain non-fatal by design (unlike Get-GhostCsvAudit, which is the destructive gate and blocks on any error): a resource type that legitimately exposes no parameters is benign and expected. 2. Separated the per-disk-subdirectory rationale from the [string]-cast note in the Move-VMStorage mapping block so the two comments read distinctly. All 24 PowerShell blocks parse clean (PSParser AST). --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 38 +++++++++++++++++-- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index f973a7fb..2f3a7725 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -467,6 +467,7 @@ roles, platform-managed resources, and anything that is not a Hyper-V VM on the you happen to be sitting on. Run it **once** from any node. ```powershell +$paramErrors = New-Object System.Collections.Generic.List[string] Get-ClusterResource | ForEach-Object { $r = $_ try { @@ -477,9 +478,17 @@ Get-ClusterResource | ForEach-Object { } } catch { - # Some resource types do not expose parameters; ignore and continue. + # Most resource types that throw here simply do not expose parameters, which is + # benign. Record the resource so a genuine query failure stays visible instead of + # silently reducing coverage. + $paramErrors.Add($r.Name) } } | Sort-Object Resource, Parameter | Format-Table -AutoSize + +if ($paramErrors.Count) { + Write-Warning ("{0} cluster resource(s) did not return parameters and were not inspected: {1}" -f ` + $paramErrors.Count, ($paramErrors -join ', ')) +} ``` ### 2D. Inventory what is actually inside @@ -868,6 +877,7 @@ which moves disks, configuration, checkpoints, and the smart paging file. # "Hash tables in the Vhds parameter must contain 'DestinationFilePath' key" # even though the key IS present. The wrapper is invisible to normal checks: # .GetType() reports String and -is [string] reports True. + # Give each disk its OWN numbered subdirectory. Two disks can share a leaf filename # (attached from different source folders); mapping both to $Destination\ would # produce identical DestinationFilePath values and make Move-VMStorage fail. A unique @@ -980,6 +990,7 @@ Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Sil Select-Object FullName, CreationTime, LastWriteTime | Export-Csv "$out\ghost-roots.csv" -NoTypeInformation +$paramErrors = New-Object System.Collections.Generic.List[string] Get-ClusterResource | ForEach-Object { $r = $_ try { @@ -988,9 +999,19 @@ Get-ClusterResource | ForEach-Object { [pscustomobject]@{ Resource = $r.Name; Parameter = $_.Name; Value = $_.Value } } } - } catch { } + } + catch { + # Benign for resource types that expose no parameters; record the rest so the + # support case shows which resources could not be inspected. + $paramErrors.Add($r.Name) + } } | Export-Csv "$out\cluster-resource-refs.csv" -NoTypeInformation +if ($paramErrors.Count) { + $paramErrors | Set-Content "$out\cluster-resource-uninspected.txt" + Write-Host ("Note: {0} cluster resource(s) returned no parameters; listed in cluster-resource-uninspected.txt" -f $paramErrors.Count) +} + Write-Host "Evidence collected in $out" ``` @@ -1036,6 +1057,7 @@ Get-ClusterSharedVolume | ForEach-Object { } | Format-Table -AutoSize # 3) No cluster resource references a ghost path. +$paramErrors = New-Object System.Collections.Generic.List[string] Get-ClusterResource | ForEach-Object { $r = $_ try { @@ -1044,9 +1066,19 @@ Get-ClusterResource | ForEach-Object { [pscustomobject]@{ Resource = $r.Name; Parameter = $_.Name; Value = $_.Value } } } - } catch { } + } + catch { + # Benign for resource types that expose no parameters; record the rest so a genuine + # query failure does not let the verification pass on reduced coverage. + $paramErrors.Add($r.Name) + } } | Format-Table -AutoSize +if ($paramErrors.Count) { + Write-Warning ("{0} cluster resource(s) did not return parameters and were not inspected: {1}" -f ` + $paramErrors.Count, ($paramErrors -join ', ')) +} + # 4) Cluster roles and resources are healthy. Get-ClusterGroup | Format-Table Name, OwnerNode, State -AutoSize Get-ClusterResource | Where-Object State -ne 'Online' | Format-Table Name, State, OwnerGroup -AutoSize From ade7a79e0bc2c4bba9d99d4a3e0cf373d68aba67 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 08:41:58 -0400 Subject: [PATCH 05/10] Ghost CSV TSG: close the parent-chain gap and the Infrastructure_1 destination hole Applied from a tsg-forge grading run (live L3 loop plus a negative safety test on a lab cluster, and a multi-model persona panel across two frontier model families). Data-loss path (deep-systems lens): - Step 2A read only the attached disk path, so a checkpoint or differencing PARENT sitting on a ghost root was invisible to the audit and would then be deleted by Path A, breaking the chain. Step 2A now walks the full parent chain with Get-VHD, and the Path A audit walks it too, so an unreadable link is a blocker rather than a silent pass. This is the hazard the guide's own Root cause section describes, so the guide previously contradicted itself. Safety (literal-follower lens): - Path B validated the destination only as "an active CSV", but Get-ClusterSharedVolume also returns the reserved Infrastructure_1 volume, which the guide's own Path C says is off limits. The destination check now rejects the infrastructure volume first, with its own specific error, and no longer offers it in the list of valid values. - The -Vhds note told the reader to hand-edit the command when the array was empty. The move now uses a splatted parameter set that includes -Vhds only when there is something to move, removing the manual edit. Live validation on a lab cluster confirmed the detection, the fail-closed audit, and the Path A cleanup, and confirmed that a reparse point inside a ghost root correctly blocks the audit and refuses the delete while the data behind the junction survives. --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 107 ++++++++++++++---- 1 file changed, 86 insertions(+), 21 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index 2f3a7725..9c102c90 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -446,6 +446,39 @@ Get-VM | Get-VMHardDiskDrive | Format-Table -AutoSize ``` +> [!IMPORTANT] +> The command above reads only the **attached** disk path. If a VM has checkpoints or +> differencing disks, the attached disk can sit on a healthy CSV while one of its +> **parent** disks is still on a ghost root. That is the exact hazard described in +> [Root cause](#why-a-referenced-ghost-path-is-dangerous-not-just-untidy), and it is +> invisible to the command above. Deleting a ghost root that still holds a parent disk +> breaks the chain and the child disk becomes unusable. Walk the parent chain too. + +```powershell +# Walk every attached disk's FULL parent chain (checkpoints / differencing disks). +Get-VM | Get-VMHardDiskDrive | ForEach-Object { + $vmName = $_.VMName + $path = $_.Path + $depth = 0 + while ($path -and $depth -lt 50) { + if ($path -match $GhostPathPattern) { + [pscustomobject]@{ VMName = $vmName; Depth = $depth; Reference = $path } + } + $vhd = Get-VHD -Path $path -ErrorAction SilentlyContinue + if (-not $vhd) { + [pscustomobject]@{ VMName = $vmName; Depth = $depth; Reference = "UNREADABLE: $path" } + break + } + $path = $vhd.ParentPath + $depth++ + } +} | Format-Table -AutoSize +``` + +Any row returned is a reference and blocks cleanup. A row beginning `UNREADABLE:` means +the chain could not be followed, so coverage is incomplete: treat it as a reference +until you can read that disk. + ### 2B. Virtual machine configuration, checkpoint, and paging paths A VM can have healthy disks and still be anchored to a ghost root by one of these @@ -660,6 +693,25 @@ Combine the results and place the cluster in exactly one category. foreach ($vm in (Get-VM -ErrorAction Stop)) { foreach ($d in ($vm | Get-VMHardDiskDrive -ErrorAction SilentlyContinue)) { if ($d.Path -match $Pattern) { $hits.Add("VMHardDisk: $($vm.Name) -> $($d.Path)") } + + # Walk the parent chain. A checkpoint / differencing PARENT can still + # live on a ghost root while the attached child sits on a healthy CSV, + # and deleting that parent breaks the chain. An unreadable link is + # recorded as an error (a blocker), never skipped. + $p = $d.Path + $depth = 0 + while ($p -and $depth -lt 50) { + $vhd = Get-VHD -Path $p -ErrorAction SilentlyContinue + if (-not $vhd) { + if ($depth -gt 0) { + $errors.Add("Get-VHD could not read '$p' in the parent chain of $($vm.Name); coverage incomplete") + } + break + } + $p = $vhd.ParentPath + if ($p -and ($p -match $Pattern)) { $hits.Add("VMDiskParent: $($vm.Name) -> $p") } + $depth++ + } } foreach ($prop in 'ConfigurationLocation','SnapshotFileLocation','SmartPagingFilePath') { $v = $vm.$prop @@ -841,10 +893,24 @@ which moves disks, configuration, checkpoints, and the smart paging file. throw "VM '$VMName' not found on $env:COMPUTERNAME." } - # The destination must be an ACTIVE CSV mount point from Step 1B. - $validCsv = Get-ClusterSharedVolume | ForEach-Object { $_.SharedVolumeInfo.FriendlyVolumeName } + # The destination must be an ACTIVE CSV mount point from Step 1B, and it must be a + # CUSTOMER workload volume. Get-ClusterSharedVolume also returns the reserved + # infrastructure volume, so membership of that list is NOT on its own a safe test. + $InfraPathPattern = '[\\/]Infrastructure(_\d+)?([\\/]|$)' + + $allCsv = @(Get-ClusterSharedVolume | ForEach-Object { $_.SharedVolumeInfo.FriendlyVolumeName }) + $validCsv = @($allCsv | Where-Object { $_ -notmatch $InfraPathPattern }) + + # Checked FIRST so the reserved volume produces its own specific error rather than + # a generic "not in the list" message. + if ($CsvRootPath -match $InfraPathPattern) { + throw "'$CsvRootPath' is the reserved Azure Local infrastructure volume. Customer workload storage must never be placed there. Pick a customer volume from Step 1B, or go to Path C." + } + if (-not $validCsv) { + throw "No customer CSV mount point is available on this cluster. Do not use the infrastructure volume. Go to Path C." + } if ($CsvRootPath -notin $validCsv) { - throw "'$CsvRootPath' is not an active CSV mount point. Valid values: $($validCsv -join ', ')" + throw "'$CsvRootPath' is not an active customer CSV mount point. Valid values: $($validCsv -join ', ')" } if ($CsvRootPath -match $GhostPathPattern) { throw "'$CsvRootPath' is itself a ghost path. Pick a canonical CSV from Step 1B." @@ -908,35 +974,34 @@ which moves disks, configuration, checkpoints, and the smart paging file. `Move-VMStorage` supports `-WhatIf`. Always run the dry run and read its output before the real move. - > [!NOTE] - > If `$vhds` came back empty, because only the configuration, checkpoint, or - > paging paths were on a ghost root, **omit the `-Vhds` parameter entirely**. - > Passing an empty array makes `Move-VMStorage` reject the call. + The parameter set is built once, below. `-Vhds` is included **only** when there is + at least one ghost-rooted disk to move, because `Move-VMStorage` rejects an empty + array. Building it this way means you never have to hand-edit the command, which is + the step most likely to go wrong under time pressure. ```powershell + $moveParams = @{ + Name = $VMName + VirtualMachinePath = $Destination + SnapshotFilePath = $Destination + SmartPagingFilePath = $Destination + } + if ($vhds.Count -gt 0) { $moveParams['Vhds'] = $vhds } + # Dry run: reports what would happen and changes nothing. - Move-VMStorage -Name $VMName ` - -VirtualMachinePath $Destination ` - -SnapshotFilePath $Destination ` - -SmartPagingFilePath $Destination ` - -Vhds $vhds ` - -WhatIf + Move-VMStorage @moveParams -WhatIf ``` - When the dry run looks correct, run the same command without `-WhatIf`: + When the dry run looks correct, run the same parameter set without `-WhatIf`: ```powershell - Move-VMStorage -Name $VMName ` - -VirtualMachinePath $Destination ` - -SnapshotFilePath $Destination ` - -SmartPagingFilePath $Destination ` - -Vhds $vhds + Move-VMStorage @moveParams ``` > [!NOTE] > Supply only the parameters you actually need to change. If the VM's checkpoint - > location was already correct in Step 2B, omit `-SnapshotFilePath` rather than - > moving it unnecessarily. + > location was already correct in Step 2B, remove `SnapshotFilePath` from + > `$moveParams` rather than moving it unnecessarily. 5. **Confirm this VM is clean.** From 10dfda95122e6d756a4380aa171ed1dde8acc803 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 10:04:24 -0400 Subject: [PATCH 06/10] Refine platform-content routing after real-cluster testing Ran the detection read-only against a live 3-node Azure Local cluster. Every node had seven numbered roots (.000 through .006) accumulated between 2025-11 and 2026-07, one per update, with zero references from any VM disk, VM configuration, cluster resource parameter, or SMB open file, and no active CSV under a numbered root. Each ghost root held exactly one child: a stale Infrastructure_1 breadcrumb of 9 items / 2 files / 528 bytes, at Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\ AgentLifecycleManagement\FCARotation\SuccessFiles which matches the path recorded in the resolved IcM for this phenomenon. The guide as written would have routed all seven to Path C and told the operator to open a support case, purely because the content sits under Infrastructure_1. That is over-escalation on what is the most common shape in the field, and it is exactly the alarm fatigue that leads people to ignore the check. The companion CSSTools analyzer already had the better rule: platform content only matters when something still references it. Aligned the guide with that rule: * Step 2D now distinguishes referenced platform content, and ARB/MOC working data such as a .vhdx under MocArb\WorkingDirectory\ or an ImageStore folder, from small stale orchestration breadcrumbs with no references. * The Step 3 classification and Path C entry conditions were narrowed to match, with an explicit note that an Infrastructure_1 folder inside a ghost root is not by itself a support case. * Symptoms now describes the accumulate-one-per-update shape, since that is what an operator is most likely to actually see. The safety properties are unchanged: any reference, any reparse point, any ARB or MOC working data, or an active CSV under a numbered root still routes to Path C. Lint remains A, 16 anchors resolve, no prose violations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8d31ddd7-9fbb-4ff0-8aa7-3f9d88060ed7 --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 40 +++++++++++++++---- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index 9c102c90..c83f554e 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -200,6 +200,10 @@ Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock - One or more folders named `C:\ClusterStorage.000`, `C:\ClusterStorage.001`, `C:\ClusterStorage.00X` exist alongside the normal `C:\ClusterStorage` folder. - New numbered folders appear after every solution update, or after a node restart. +- **Several numbered folders have accumulated over many months**, one per update, + each holding only a small `Infrastructure_1` breadcrumb of a few hundred bytes. + This is the most common shape in the field, and on its own it is benign; the + checks below are what tell you whether it is still benign on your cluster. - A solution update fails part-way through, and the failure references a path containing a numbered root. - An Arc Resource Bridge VM fails to start, or ARB redeployment fails, after an @@ -548,11 +552,24 @@ Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Sil ``` > [!IMPORTANT] -> If the inventory shows content under a path resembling -> `...\Infrastructure_1\Shares\SU1_Infrastructure_1\...`, including MocArb or -> Orchestration working directories, this is Azure Local platform content, not -> customer data. Go to -> [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). +> Look at **what** the content is, not just where it sits. +> +> - Content that is **still referenced** by anything in Step 2, or that contains +> ARB or MOC working data such as a `.vhdx` under `MocArb\WorkingDirectory\` or +> an `ImageStore` folder, is platform data in use. Go to +> [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). +> - Small, stale breadcrumb content left behind by past orchestration, for example +> a few files under +> `Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\AgentLifecycleManagement\FCARotation\SuccessFiles`, +> with **no** references from Step 2, is an ordinary leftover. It stays on the +> normal path and is handled by +> [Path A](#path-a-no-references-found-safe-to-clean-up). +> +> The presence of an `Infrastructure_1` folder inside a ghost root is expected and +> is **not** on its own a reason to open a support case. A cluster that has taken +> several updates commonly accumulates one such folder per numbered root, each only +> a few hundred bytes. Treating every one of those as a support case creates noise +> and trains people to ignore the check. ### 2E. Optional: search logs and configuration for stale references @@ -602,7 +619,7 @@ Combine the results and place the cluster in exactly one category. | --- | --- | --- | | **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only stale files with no references | [Path A](#path-a-no-references-found-safe-to-clean-up) | | **Unsafe, active references found** | Any of 2A, 2B, 2C returns a row, or an SMB open file exists under a ghost path, and the referencing object is a **customer workload VM** | [Path B](#path-b-a-workload-vm-references-a-ghost-path) | -| **Unsafe, platform references found** | Any reference is under `Infrastructure_1`, or names ARB / MOC / MocArb / Orchestration content, or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | +| **Unsafe, platform references found** | Any reference from 2A, 2B or 2C points under `Infrastructure_1`, or the ghost root holds ARB / MOC working data (a `.vhdx` under `MocArb\WorkingDirectory\`, an `ImageStore` folder), or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | > [!WARNING] > If you are unsure which category applies, treat it as **Path C** and engage @@ -1027,12 +1044,19 @@ which moves disks, configuration, checkpoints, and the smart paging file. Stop and open a support case if **any** of the following is true: -- A reference points under `...\Infrastructure_1\...`. -- The content resembles ARB, MOC, MocArb, or Orchestration platform data. +- A reference from Step 2 points under `...\Infrastructure_1\...`. +- A ghost root holds ARB or MOC working data, for example a `.vhdx` under + `MocArb\WorkingDirectory\` or an `ImageStore` folder. - [Step 1C](#1c-check-whether-the-ghost-root-still-redirects-to-live-data) shows `IsReparsePoint = True` for any child of a ghost root. - An **active** CSV reports a `FriendlyVolumeName` under a numbered root. +> [!NOTE] +> An `Infrastructure_1` folder inside a ghost root is **not** by itself one of +> these conditions. Clusters routinely accumulate a small, stale `Infrastructure_1` +> breadcrumb per numbered root as updates run. What matters is whether anything +> still references it, or whether it holds real ARB or MOC working data. + These are platform-managed paths. `Infrastructure_1` is reserved for Azure Local system configuration, the platform deliberately blocks customer storage placement on it, and relocating its content by hand is outside the customer support boundary. A From 5037f288316f0884dcff44f8eb4f025641c792a5 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:21:14 -0400 Subject: [PATCH 07/10] Ghost CSV TSG: recursive reparse scan, fail-closed gate, non-following delete Third-model review round (xAI added to the panel) found three real defects, including two in the hardening from the previous round. All are now fixed and re-validated on a live lab cluster. - Step 1C tested only the immediate children of a ghost root for a reparse point. A volume mount nested inside an ordinary-looking leftover folder therefore classified as "ordinary files", routed to Path A, and would have been deleted as live storage. The scan is now recursive, with the reason stated. - The pre-delete safety scan used -ErrorAction SilentlyContinue, so an unreadable subtree could hide the very reparse point the gate exists to find. It now uses -ErrorAction Stop and treats an enumeration failure as unsafe, and it re-checks each root immediately before removing it. - The delete used Remove-Item -Recurse, which on Windows PowerShell 5.1 does not guarantee it will not follow a junction. It now uses [System.IO.Directory]::Delete, which removes a reparse point as a link. - The parent-chain walk added last round failed open at depth 0: an unreadable attached disk recorded no blocker, so SafeToDelete could still be true with no parent-chain coverage. It now fails closed at every depth, and also blocks when a chain exceeds the walk limit. - The DELETE confirmation is now stated as case-sensitive. Re-validated live with a junction placed one level down inside a ghost root: the cleanup correctly refused, the data behind the junction survived, and after the junction was removed as a link the root was removed cleanly with the live CSV root intact. --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 57 +++++++++++++++---- 1 file changed, 47 insertions(+), 10 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index c83f554e..69f3e744 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -382,12 +382,18 @@ Check the **`ReparsePoint` file attribute**, which is the authoritative signal. > a live volume can show a blank `LinkType` and be mistaken for ordinary leftover > files. The attribute check below does not have that failure mode. +> [!IMPORTANT] +> The scan below is **recursive** on purpose. A volume mount point does not have to sit at +> the top of the ghost root: it can be nested inside an ordinary-looking leftover folder. +> A top-level-only check would classify that as "ordinary leftover files" and send you to +> Path A, and the cleanup would then be deleting live storage. + ```powershell Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } | ForEach-Object { $root = $_.FullName - $children = Get-ChildItem -LiteralPath $root -Force -ErrorAction SilentlyContinue + $children = Get-ChildItem -LiteralPath $root -Force -Recurse -ErrorAction SilentlyContinue if (-not $children) { [pscustomobject]@{ GhostRoot = $root; Child = ''; IsReparsePoint = $false; Detail = '' } } @@ -720,15 +726,19 @@ Combine the results and place the cluster in exactly one category. while ($p -and $depth -lt 50) { $vhd = Get-VHD -Path $p -ErrorAction SilentlyContinue if (-not $vhd) { - if ($depth -gt 0) { - $errors.Add("Get-VHD could not read '$p' in the parent chain of $($vm.Name); coverage incomplete") - } + # FAIL CLOSED at every depth, including the attached leaf. + # An unreadable disk means we cannot prove its parent chain is + # clean, so it is a blocker, not a pass. + $errors.Add("Get-VHD could not read '$p' (chain of $($vm.Name), depth $depth); parent-chain coverage incomplete") break } $p = $vhd.ParentPath if ($p -and ($p -match $Pattern)) { $hits.Add("VMDiskParent: $($vm.Name) -> $p") } $depth++ } + if ($depth -ge 50) { + $errors.Add("Parent chain of $($vm.Name) exceeded 50 links; not fully walked") + } } foreach ($prop in 'ConfigurationLocation','SnapshotFileLocation','SmartPagingFilePath') { $v = $vm.$prop @@ -840,12 +850,27 @@ Combine the results and place the cluster in exactly one category. Where-Object { $_.Name -match '^ClusterStorage\.\d+$' } # Never recurse into a reparse point: that can delete the target volume's data. - $unsafe = foreach ($t in $targets) { - Get-ChildItem -LiteralPath $t.FullName -Force -Recurse -ErrorAction SilentlyContinue | - Where-Object { $_.Attributes -band [System.IO.FileAttributes]::ReparsePoint } + # This scan uses -ErrorAction Stop on purpose. Suppressing enumeration errors here + # would let an unreadable subtree hide the very reparse point this gate exists to + # find, so an enumeration failure is treated as unsafe rather than as "nothing found". + $unsafe = @() + $scanError = $null + foreach ($t in $targets) { + try { + $unsafe += Get-ChildItem -LiteralPath $t.FullName -Force -Recurse -ErrorAction Stop | + Where-Object { $_.Attributes -band [System.IO.FileAttributes]::ReparsePoint } + } + catch { + $scanError = "Could not fully enumerate $($t.FullName): $($_.Exception.Message)" + break + } } - if ($unsafe) { + if ($scanError) { + Write-Warning "Refusing to delete. The safety scan could not complete:" + Write-Warning $scanError + } + elseif ($unsafe) { Write-Warning "Reparse points found inside ghost roots. Refusing to delete:" $unsafe | Select-Object FullName } @@ -855,10 +880,22 @@ Combine the results and place the cluster in exactly one category. else { Write-Host "About to permanently delete on $($env:COMPUTERNAME):" -ForegroundColor Yellow $targets | ForEach-Object { Write-Host " $($_.FullName)" -ForegroundColor Yellow } - $answer = Read-Host "Type DELETE to confirm" + Write-Host "Type DELETE (uppercase) to confirm. Anything else cancels." -ForegroundColor Yellow + $answer = Read-Host "Confirm" if ($answer -ceq 'DELETE') { foreach ($t in $targets) { - Remove-Item -LiteralPath $t.FullName -Recurse -Force + # Re-check this specific root immediately before removing it, so a reparse + # point created between the scan above and this moment cannot be followed. + $lateCheck = @(Get-ChildItem -LiteralPath $t.FullName -Force -Recurse -ErrorAction Stop | + Where-Object { $_.Attributes -band [System.IO.FileAttributes]::ReparsePoint }) + if ($lateCheck.Count -gt 0) { + Write-Warning "Skipping $($t.FullName): a reparse point appeared since the scan." + continue + } + # [System.IO.Directory]::Delete removes a reparse point as a LINK rather than + # following it into the target, which Remove-Item -Recurse does not guarantee + # on Windows PowerShell 5.1. + [System.IO.Directory]::Delete($t.FullName, $true) Write-Host "Removed $($t.FullName)" } } From 66353ade451bcf3df4d925f0a985c54ef14cf39f Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 17:13:33 -0400 Subject: [PATCH 08/10] Ghost CSV TSG: resolve Alex's review (audit content gate, Path B parent walk, metadata, risk labels, citation, verify count) - [HIGH] Get-GhostCsvAudit now inspects content, so it no longer reports SafeToDelete=True for a root Step 3 sends to Path C. Added a PlatformWorkingData blocker (a .vhdx under MocArb\WorkingDirectory\, or an ImageStore folder); a bare Infrastructure_1 breadcrumb is deliberately not a blocker. The "exactly as strict as Step 3" claim is now accurate and enumerates what the gate covers. - [MED] Path B step 5 now walks the full parent chain (like Step 2A) instead of checking attached paths only, so a VM with a ghost-rooted checkpoint/differencing parent no longer gets a false green while Path A refuses. Step 3 detects such parents up front and routes them to Path C (Move-VMStorage cannot relocate an unattached parent). Step 1 now explains that Get-VM/Move-VMStorage are node-local and a clustered VM must be handled from its owning node. - [MED] "Verify the fix" References count filters nulls (@($refs | Where { $_ })), so a genuinely clean cluster reports 0 instead of a phantom non-zero from (empty).Path appends. - [MED] Dropped the Events 5120/5142 citation from the antivirus/handle sentence in Contributing factors: that article is about preferred-NIC SMB routing and does not support the handle claim. Kept the accurate 5120/5142 framing later and the Related Issues link. - [MED] Added the hidden azure-local-supportability/tsg-metadata/v1 marker (validates against the schema on main). - [MED] Added [HIGH RISK] to the delete step and [MEDIUM RISK] to Move-VMStorage. - [LOW] Path B destination validation now excludes Offline CSVs (State -eq 'Online'). - [LOW] Renamed the "Root cause" heading to "Contributing factors and evidence" (template + house convention). Validated: 25/25 PowerShell blocks parse clean, metadata validates, 32/32 in-page links resolve, no spaced double-hyphens in prose. --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 112 +++++++++++++++--- 1 file changed, 93 insertions(+), 19 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index 69f3e744..81973fab 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -1,3 +1,23 @@ + + # Troubleshoot ghost CSV mount points (`C:\ClusterStorage.000`, `.001`, `.00X`) @@ -221,7 +241,7 @@ numbered root anywhere in the message: ... C:\ClusterStorage.000\Infrastructure_1\... ``` -## Root cause +## Contributing factors and evidence ### The namespace @@ -249,8 +269,7 @@ on) and creates a fresh `C:\ClusterStorage`. This failure mode is what Microsoft's own guidance addresses. Microsoft documents that security, backup, and filter-driver products holding a handle on the CSV path cause access failures under `C:\ClusterStorage`, and publishes the exclusions that prevent them (see -[Recommended antivirus exclusions for Hyper-V hosts](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts) -and [Events 5120 and 5142 and unable to access the ClusterStorage folder](https://learn.microsoft.com/troubleshoot/windows-server/backup-and-storage/event-5120-5142-access-clusterstorage-folder)). +[Recommended antivirus exclusions for Hyper-V hosts](https://learn.microsoft.com/troubleshoot/windows-server/virtualization/antivirus-exclusions-for-hyper-v-hosts)). Excluding `C:\ClusterStorage` from antivirus scanning on every node is the practical prevention for this condition. See [Prevention](#prevention). @@ -460,7 +479,7 @@ Get-VM | Get-VMHardDiskDrive | > The command above reads only the **attached** disk path. If a VM has checkpoints or > differencing disks, the attached disk can sit on a healthy CSV while one of its > **parent** disks is still on a ghost root. That is the exact hazard described in -> [Root cause](#why-a-referenced-ghost-path-is-dangerous-not-just-untidy), and it is +> [Contributing factors and evidence](#why-a-referenced-ghost-path-is-dangerous-not-just-untidy), and it is > invisible to the command above. Deleting a ghost root that still holds a parent disk > breaks the chain and the child disk becomes unusable. Walk the parent chain too. @@ -645,8 +664,9 @@ Combine the results and place the cluster in exactly one category. ### Steps 1. **Paste the shared audit function.** - Every gate below calls this one function, so the final safety check is exactly as - strict as the Step 3 classification rather than a narrower subset of it. Paste it + Every gate below calls this one function, so the final safety check is as strict as + the Step 3 classification (references, reparse points, active CSV, platform working + data, and any query or remoting failure), not a narrower subset of it. Paste it once into your elevated session. ```powershell @@ -758,6 +778,9 @@ Combine the results and place the cluster in exactly one category. } # A reparse point inside a ghost root means it still redirects to a volume. + # Platform working data in use (a .vhdx under MocArb\WorkingDirectory\, or an + # ImageStore folder) is a Path C blocker regardless of references; a bare or + # stale Infrastructure_1 breadcrumb is NOT working data and stays on Path A. try { foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Stop | Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { @@ -766,6 +789,12 @@ Combine the results and place the cluster in exactly one category. $hits.Add("ReparsePoint: $($c.FullName)") } } + foreach ($item in (Get-ChildItem -LiteralPath $g.FullName -Recurse -Force -ErrorAction SilentlyContinue)) { + if (($item.FullName -match '(?i)MocArb[\\/]WorkingDirectory[\\/].+\.vhdx$') -or + ($item.PSIsContainer -and $item.Name -eq 'ImageStore')) { + $hits.Add("PlatformWorkingData: $($item.FullName)") + } + } } } catch { $errors.Add("Ghost-root enumeration failed -> $($_.Exception.Message)") @@ -834,8 +863,9 @@ Combine the results and place the cluster in exactly one category. > the situation as > [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). -4. **Remove the ghost roots, one node at a time.** - This block re-runs the full audit, refuses to touch anything that is a reparse +4. **Remove the ghost roots, one node at a time. [HIGH RISK]** + This permanently deletes the numbered roots. The block re-runs the full audit, + refuses to touch anything that is a reparse point, and requires you to type a confirmation. Run it on a single node, confirm the cluster is healthy, then move to the next. @@ -944,7 +974,7 @@ which moves disks, configuration, checkpoints, and the smart paging file. throw "Set the VM name and CSV path variables before running this block." } if (-not (Get-VM -Name $VMName -ErrorAction SilentlyContinue)) { - throw "VM '$VMName' not found on $env:COMPUTERNAME." + throw "VM '$VMName' not found on $env:COMPUTERNAME. Get-VM and Move-VMStorage are node-local, so for a clustered VM you must run Path B from the node that currently owns it. Find the owner with 'Get-ClusterGroup | Where-Object GroupType -eq ''VirtualMachine''' (its OwnerNode column), move to that node, and re-run." } # The destination must be an ACTIVE CSV mount point from Step 1B, and it must be a @@ -952,7 +982,10 @@ which moves disks, configuration, checkpoints, and the smart paging file. # infrastructure volume, so membership of that list is NOT on its own a safe test. $InfraPathPattern = '[\\/]Infrastructure(_\d+)?([\\/]|$)' - $allCsv = @(Get-ClusterSharedVolume | ForEach-Object { $_.SharedVolumeInfo.FriendlyVolumeName }) + # Only ONLINE CSVs are valid destinations. An Offline CSV still appears in this + # list but cannot host VM storage, so it is excluded here as well. + $allCsv = @(Get-ClusterSharedVolume | Where-Object { $_.State -eq 'Online' } | + ForEach-Object { $_.SharedVolumeInfo.FriendlyVolumeName }) $validCsv = @($allCsv | Where-Object { $_ -notmatch $InfraPathPattern }) # Checked FIRST so the reserved volume produces its own specific error rather than @@ -1019,13 +1052,33 @@ which moves disks, configuration, checkpoints, and the smart paging file. if (-not $vhds) { Write-Host "No ghost-rooted disks on '$VMName'." } $vhds | ForEach-Object { "{0} -> {1}" -f $_.SourceFilePath, $_.DestinationFilePath } + + # Move-VMStorage -Vhds relocates ATTACHED disks only. A ghost-rooted checkpoint or + # differencing PARENT is not an attached disk, so it will NOT move here; Step 5 and + # Path A will keep reporting it (correctly). Detect that case now so it is not a surprise. + $ghostParents = @( + Get-VM -Name $VMName | Get-VMHardDiskDrive | ForEach-Object { + $path = $_.Path; $depth = 0 + while ($path -and $depth -lt 50) { + $vhd = Get-VHD -Path $path -ErrorAction SilentlyContinue + if (-not $vhd) { break } + $path = $vhd.ParentPath; $depth++ + if ($path -and ($path -match $GhostPathPattern)) { $path } + } + } + ) + if ($ghostParents.Count -gt 0) { + Write-Warning "PARENT disk(s) on a ghost root that Move-VMStorage will NOT relocate:" + $ghostParents | Sort-Object -Unique | ForEach-Object { Write-Warning " $_" } + Write-Warning "Relocating an unattached differencing or checkpoint parent is not a hand procedure. Complete the attached-disk move if there is one, but this VM needs Path C (engage support) to clear the parent before Path A can remove the root." + } ``` **Stop and read that mapping.** Every destination must be under the path you validated in step 1. -4. **Dry-run the move, then perform it.** - `Move-VMStorage` supports `-WhatIf`. Always run the dry run and read its output +4. **Dry-run the move, then perform it. [MEDIUM RISK]** + `Move-VMStorage` relocates live VM storage. It supports `-WhatIf`. Always run the dry run and read its output before the real move. The parameter set is built once, below. `-Vhds` is included **only** when there is @@ -1061,13 +1114,34 @@ which moves disks, configuration, checkpoints, and the smart paging file. ```powershell $vm = Get-VM -Name $VMName - $bad = @() - $bad += ($vm | Get-VMHardDiskDrive | Where-Object { $_.Path -match $GhostPathPattern }).Path - $bad += @($vm.ConfigurationLocation, $vm.SnapshotFileLocation, $vm.SmartPagingFilePath) | - Where-Object { $_ -and ($_ -match $GhostPathPattern) } + $bad = New-Object System.Collections.Generic.List[string] + + # Walk every attached disk's FULL parent chain, exactly as Step 2A does. Checking only + # attached paths would report success while a checkpoint or differencing PARENT is still + # on a ghost root, which Path A would then correctly refuse to delete, leaving a green + # verdict and a blocked cleanup. An UNREADABLE link is treated as a reference. + foreach ($d in ($vm | Get-VMHardDiskDrive)) { + $path = $d.Path + $depth = 0 + while ($path -and $depth -lt 50) { + if ($path -match $GhostPathPattern) { $bad.Add("disk chain (depth $depth): $path") } + $vhd = Get-VHD -Path $path -ErrorAction SilentlyContinue + if (-not $vhd) { $bad.Add("UNREADABLE (chain not fully walked): $path"); break } + $path = $vhd.ParentPath + $depth++ + } + } + foreach ($p in @($vm.ConfigurationLocation, $vm.SnapshotFileLocation, $vm.SmartPagingFilePath)) { + if ($p -and ($p -match $GhostPathPattern)) { $bad.Add("config/checkpoint/paging: $p") } + } - if ($bad) { Write-Warning "Still referencing a ghost path:"; $bad } - else { Write-Host "$VMName no longer references any ghost path." -ForegroundColor Green } + if ($bad.Count -gt 0) { + Write-Warning "$VMName still references a ghost path (parent chains included):" + $bad + } + else { + Write-Host "$VMName no longer references any ghost path, including parent chains." -ForegroundColor Green + } ``` 6. **Repeat for every VM found in Step 2**, then re-run @@ -1168,7 +1242,7 @@ Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock Node = $env:COMPUTERNAME GhostRoots = (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction SilentlyContinue | Where-Object { $_.Name -match '^ClusterStorage\.\d+$' }).Count - References = @($refs).Count + References = @($refs | Where-Object { $_ }).Count } } | Select-Object Node, GhostRoots, References | Format-Table -AutoSize From 9f96fe14bed15c3c3d220a2996cbf3816bc78e68 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 17:24:45 -0400 Subject: [PATCH 09/10] Ghost CSV TSG: route Infrastructure_1 breadcrumb to Path C, matching the CSSTools remediation Resolves the cross-PR contract Alex flagged. Per decision, an Infrastructure_1 folder is now treated as reserved platform content on its own, even when unreferenced and holding only a stale breadcrumb, so it is routed to support (Path C) rather than hand-deleted (Path A). This matches the companion CSSTools remediation, which refuses the same content. - Get-GhostCsvAudit now adds a PlatformContent blocker for an Infrastructure_N folder (alongside the MocArb\WorkingDirectory .vhdx and ImageStore checks), so a paste-and-run operator is protected: a root containing Infrastructure_1 is not SafeToDelete. - Step 2D, Step 3 classification (both rows), and Path C prose updated: an Infrastructure_1 folder is a Path C condition, not an ordinary Path A leftover. - "Where this shows up" no longer implies the common breadcrumb shape is self-clearable. Validated: 25/25 PowerShell blocks parse clean, 32/32 in-page links resolve, no spaced double-hyphens in prose. --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 52 ++++++++++--------- 1 file changed, 28 insertions(+), 24 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index 81973fab..41b3056e 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -222,8 +222,9 @@ Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock - New numbered folders appear after every solution update, or after a node restart. - **Several numbered folders have accumulated over many months**, one per update, each holding only a small `Infrastructure_1` breadcrumb of a few hundred bytes. - This is the most common shape in the field, and on its own it is benign; the - checks below are what tell you whether it is still benign on your cluster. + This is the most common shape in the field, and on its own it is not an active + fault, but an `Infrastructure_1` folder is reserved platform storage; the checks + below tell you whether the root is clear and where to route it. - A solution update fails part-way through, and the failure references a path containing a numbered root. - An Arc Resource Bridge VM fails to start, or ARB redeployment fails, after an @@ -583,18 +584,17 @@ Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Sil > ARB or MOC working data such as a `.vhdx` under `MocArb\WorkingDirectory\` or > an `ImageStore` folder, is platform data in use. Go to > [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). -> - Small, stale breadcrumb content left behind by past orchestration, for example -> a few files under -> `Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\AgentLifecycleManagement\FCARotation\SuccessFiles`, -> with **no** references from Step 2, is an ordinary leftover. It stays on the -> normal path and is handled by -> [Path A](#path-a-no-references-found-safe-to-clean-up). +> - An `Infrastructure_1` folder is **reserved Azure Local system storage**, even +> when it holds only a small, stale breadcrumb left behind by past orchestration +> (for example a few files under +> `Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\AgentLifecycleManagement\FCARotation\SuccessFiles`). +> Do not remove it by hand. Go to +> [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). > -> The presence of an `Infrastructure_1` folder inside a ghost root is expected and -> is **not** on its own a reason to open a support case. A cluster that has taken -> several updates commonly accumulates one such folder per numbered root, each only -> a few hundred bytes. Treating every one of those as a support case creates noise -> and trains people to ignore the check. +> This matches the companion CSSTools remediation, which also refuses to delete a +> ghost root that still contains an `Infrastructure_1` folder and routes it to +> support. The Path A audit enforces the same rule, so a paste-and-run operator is +> protected even without reading this note. ### 2E. Optional: search logs and configuration for stale references @@ -642,9 +642,9 @@ Combine the results and place the cluster in exactly one category. | Classification | Criteria (all must hold) | Action | | --- | --- | --- | -| **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only stale files with no references | [Path A](#path-a-no-references-found-safe-to-clean-up) | +| **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only ordinary leftover files with no references and **no** `Infrastructure_N` folder or ARB/MOC platform content | [Path A](#path-a-no-references-found-safe-to-clean-up) | | **Unsafe, active references found** | Any of 2A, 2B, 2C returns a row, or an SMB open file exists under a ghost path, and the referencing object is a **customer workload VM** | [Path B](#path-b-a-workload-vm-references-a-ghost-path) | -| **Unsafe, platform references found** | Any reference from 2A, 2B or 2C points under `Infrastructure_1`, or the ghost root holds ARB / MOC working data (a `.vhdx` under `MocArb\WorkingDirectory\`, an `ImageStore` folder), or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | +| **Unsafe, platform references found** | Any reference from 2A, 2B or 2C points under `Infrastructure_1`, or the ghost root holds ARB / MOC working data (a `.vhdx` under `MocArb\WorkingDirectory\`, an `ImageStore` folder) or an `Infrastructure_N` folder, or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | > [!WARNING] > If you are unsure which category applies, treat it as **Path C** and engage @@ -778,9 +778,10 @@ Combine the results and place the cluster in exactly one category. } # A reparse point inside a ghost root means it still redirects to a volume. - # Platform working data in use (a .vhdx under MocArb\WorkingDirectory\, or an - # ImageStore folder) is a Path C blocker regardless of references; a bare or - # stale Infrastructure_1 breadcrumb is NOT working data and stays on Path A. + # Platform content is a Path C blocker regardless of references: an + # Infrastructure_N folder (reserved system storage), a .vhdx under + # MocArb\WorkingDirectory\, or an ImageStore folder. This matches the + # companion CSSTools remediation, which refuses the same content. try { foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Stop | Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { @@ -790,9 +791,10 @@ Combine the results and place the cluster in exactly one category. } } foreach ($item in (Get-ChildItem -LiteralPath $g.FullName -Recurse -Force -ErrorAction SilentlyContinue)) { - if (($item.FullName -match '(?i)MocArb[\\/]WorkingDirectory[\\/].+\.vhdx$') -or + if (($item.PSIsContainer -and $item.Name -match '^Infrastructure_\d+$') -or + ($item.FullName -match '(?i)MocArb[\\/]WorkingDirectory[\\/].+\.vhdx$') -or ($item.PSIsContainer -and $item.Name -eq 'ImageStore')) { - $hits.Add("PlatformWorkingData: $($item.FullName)") + $hits.Add("PlatformContent: $($item.FullName)") } } } @@ -1156,6 +1158,8 @@ which moves disks, configuration, checkpoints, and the smart paging file. Stop and open a support case if **any** of the following is true: - A reference from Step 2 points under `...\Infrastructure_1\...`. +- A ghost root **contains an `Infrastructure_1` folder** (reserved system storage), + even if nothing references it. - A ghost root holds ARB or MOC working data, for example a `.vhdx` under `MocArb\WorkingDirectory\` or an `ImageStore` folder. - [Step 1C](#1c-check-whether-the-ghost-root-still-redirects-to-live-data) shows @@ -1163,10 +1167,10 @@ Stop and open a support case if **any** of the following is true: - An **active** CSV reports a `FriendlyVolumeName` under a numbered root. > [!NOTE] -> An `Infrastructure_1` folder inside a ghost root is **not** by itself one of -> these conditions. Clusters routinely accumulate a small, stale `Infrastructure_1` -> breadcrumb per numbered root as updates run. What matters is whether anything -> still references it, or whether it holds real ARB or MOC working data. +> An `Infrastructure_1` folder is a Path C condition on its own, even when nothing +> references it and it holds only a small, stale breadcrumb. It is reserved Azure +> Local system storage, so it is routed to support rather than removed by hand. The +> companion CSSTools remediation applies the same rule. These are platform-managed paths. `Infrastructure_1` is reserved for Azure Local system configuration, the platform deliberately blocks customer storage placement on From c9f6f44b804bd9125038a969cecc711ebf0f6544 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 20:05:39 -0400 Subject: [PATCH 10/10] Ghost CSV TSG: revert Infrastructure_1 breadcrumb to Path A; widen VHD family; recursive fail-closed audit Restores the agreed cross-PR contract with the companion CSSTools remediation: a lone, unreferenced Infrastructure_1 breadcrumb (a few hundred bytes, nothing referencing the root) is ordinary cleanup and stays on Path A. Any reference or mount to the infrastructure directory, ARB/MOC working data, a reparse point, or an active CSV routes to support (Path C). This reverts option-B (9f96fe1), which had routed the lone breadcrumb to Path C and inverted the contract Alex flagged. Also mirrors the CSSTools data-loss fix and Alex's LOW finding: - Widen the audit platform-content check to the full VHD family (MocArb/ImageStore/WorkingDirectory folders, or any .vhd/.vhdx/.avhd/.avhdx/.vhds/ .vhdpmem), so an orphaned checkpoint, VHD-Set, or pmem disk in a ghost root is a Path C blocker rather than deleted. This is the identical pattern the companion remediation uses. - Make the reparse/content scan a single recursive, fail-closed pass (-Recurse plus -ErrorAction Stop): a reparse point at ANY depth blocks, and an unreadable root is never reported SafeToDelete. Fixes Alex's LOW (the reparse loop was not recursive while its sibling content loop was). - Align Step 2D, the classification table, Path C conditions, and the "most common shape" prose to the VHD family and the restored Path A routing; remove the false "companion refuses the same content" cross-references in both directions. Validated: 25/25 PowerShell blocks parse, 32/32 in-page links resolve, no spaced double-hyphens in prose. Addresses Alex review comment on PR #344 and reconciles the cross-PR contract. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: ed5bb8da-dbf5-4f8f-a012-ac482bf5e583 --- ...roubleshoot-Storage-GhostCsvMountPoints.md | 76 +++++++++---------- 1 file changed, 38 insertions(+), 38 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md index 41b3056e..27c99947 100644 --- a/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md +++ b/TSG/Storage/Troubleshoot-Storage-GhostCsvMountPoints.md @@ -222,9 +222,8 @@ Invoke-Command -ComputerName $nodes -ArgumentList $GhostPathPattern -ScriptBlock - New numbered folders appear after every solution update, or after a node restart. - **Several numbered folders have accumulated over many months**, one per update, each holding only a small `Infrastructure_1` breadcrumb of a few hundred bytes. - This is the most common shape in the field, and on its own it is not an active - fault, but an `Infrastructure_1` folder is reserved platform storage; the checks - below tell you whether the root is clear and where to route it. + This is the most common shape in the field, and on its own it is benign; the + checks below are what tell you whether it is still benign on your cluster. - A solution update fails part-way through, and the failure references a path containing a numbered root. - An Arc Resource Bridge VM fails to start, or ARB redeployment fails, after an @@ -581,20 +580,21 @@ Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Sil > Look at **what** the content is, not just where it sits. > > - Content that is **still referenced** by anything in Step 2, or that contains -> ARB or MOC working data such as a `.vhdx` under `MocArb\WorkingDirectory\` or -> an `ImageStore` folder, is platform data in use. Go to -> [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). -> - An `Infrastructure_1` folder is **reserved Azure Local system storage**, even -> when it holds only a small, stale breadcrumb left behind by past orchestration -> (for example a few files under -> `Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\AgentLifecycleManagement\FCARotation\SuccessFiles`). -> Do not remove it by hand. Go to +> ARB or MOC working data (any virtual hard disk, or a `MocArb`, `ImageStore`, or +> `WorkingDirectory` folder), is platform data in use. Go to > [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support). +> - Small, stale breadcrumb content left behind by past orchestration, for example +> a few files under +> `Infrastructure_1\Shares\SU1_Infrastructure_1\Orchestration\AgentLifecycleManagement\FCARotation\SuccessFiles`, +> with **no** references from Step 2, is an ordinary leftover. It stays on the +> normal path and is handled by +> [Path A](#path-a-no-references-found-safe-to-clean-up). > -> This matches the companion CSSTools remediation, which also refuses to delete a -> ghost root that still contains an `Infrastructure_1` folder and routes it to -> support. The Path A audit enforces the same rule, so a paste-and-run operator is -> protected even without reading this note. +> The presence of an `Infrastructure_1` folder inside a ghost root is expected and +> is **not** on its own a reason to open a support case. A cluster that has taken +> several updates commonly accumulates one such folder per numbered root, each only +> a few hundred bytes. Treating every one of those as a support case creates noise +> and trains people to ignore the check. ### 2E. Optional: search logs and configuration for stale references @@ -642,9 +642,9 @@ Combine the results and place the cluster in exactly one category. | Classification | Criteria (all must hold) | Action | | --- | --- | --- | -| **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only ordinary leftover files with no references and **no** `Infrastructure_N` folder or ARB/MOC platform content | [Path A](#path-a-no-references-found-safe-to-clean-up) | +| **Safe to clean up** | Ghost roots exist; 2A, 2B, 2C and the SMB check return nothing on **every** node; 1C shows `IsReparsePoint = False` everywhere; 2D shows the roots are empty or contain only stale files with no references | [Path A](#path-a-no-references-found-safe-to-clean-up) | | **Unsafe, active references found** | Any of 2A, 2B, 2C returns a row, or an SMB open file exists under a ghost path, and the referencing object is a **customer workload VM** | [Path B](#path-b-a-workload-vm-references-a-ghost-path) | -| **Unsafe, platform references found** | Any reference from 2A, 2B or 2C points under `Infrastructure_1`, or the ghost root holds ARB / MOC working data (a `.vhdx` under `MocArb\WorkingDirectory\`, an `ImageStore` folder) or an `Infrastructure_N` folder, or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | +| **Unsafe, platform references found** | Any reference from 2A, 2B or 2C points under `Infrastructure_1`, or the ghost root holds ARB / MOC working data (any virtual hard disk, or a `MocArb`, `ImageStore`, or `WorkingDirectory` folder), or 1C shows `IsReparsePoint = True`, or an active CSV is mounted under a numbered root | [Path C](#path-c-references-under-infrastructure_1-or-arb-engage-support) | > [!WARNING] > If you are unsure which category applies, treat it as **Path C** and engage @@ -777,23 +777,24 @@ Combine the results and place the cluster in exactly one category. $errors.Add("Get-SmbOpenFile failed -> $($_.Exception.Message)") } - # A reparse point inside a ghost root means it still redirects to a volume. - # Platform content is a Path C blocker regardless of references: an - # Infrastructure_N folder (reserved system storage), a .vhdx under - # MocArb\WorkingDirectory\, or an ImageStore folder. This matches the - # companion CSSTools remediation, which refuses the same content. + # One recursive, fail-closed pass over each ghost root. A reparse point at + # ANY depth still redirects to a volume, so the scan must recurse rather than + # check only immediate children. A read that cannot complete throws and is + # recorded as a blocker below, so an unreadable root is never reported safe. try { foreach ($g in (Get-ChildItem -Path 'C:\' -Directory -Filter 'ClusterStorage.*' -ErrorAction Stop | Where-Object { $_.Name -match '^ClusterStorage\.\d+$' })) { - foreach ($c in (Get-ChildItem -LiteralPath $g.FullName -Force -ErrorAction SilentlyContinue)) { - if ($c.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { - $hits.Add("ReparsePoint: $($c.FullName)") + foreach ($item in (Get-ChildItem -LiteralPath $g.FullName -Recurse -Force -ErrorAction Stop)) { + if ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) { + $hits.Add("ReparsePoint: $($item.FullName)") } - } - foreach ($item in (Get-ChildItem -LiteralPath $g.FullName -Recurse -Force -ErrorAction SilentlyContinue)) { - if (($item.PSIsContainer -and $item.Name -match '^Infrastructure_\d+$') -or - ($item.FullName -match '(?i)MocArb[\\/]WorkingDirectory[\\/].+\.vhdx$') -or - ($item.PSIsContainer -and $item.Name -eq 'ImageStore')) { + # Platform content is a Path C blocker regardless of references: a + # MocArb, ImageStore, or WorkingDirectory folder, or ANY virtual hard + # disk (base .vhd/.vhdx, checkpoint .avhd/.avhdx, VHD-Set .vhds, or + # .vhdpmem). This is the same pattern the companion CSSTools remediation + # uses. A bare Infrastructure_1 breadcrumb is NOT platform content and + # stays on Path A. + if ($item.Name -match '^(MocArb|ImageStore|WorkingDirectory)$|\.a?vhd(x|s|pmem)?$') { $hits.Add("PlatformContent: $($item.FullName)") } } @@ -1158,19 +1159,18 @@ which moves disks, configuration, checkpoints, and the smart paging file. Stop and open a support case if **any** of the following is true: - A reference from Step 2 points under `...\Infrastructure_1\...`. -- A ghost root **contains an `Infrastructure_1` folder** (reserved system storage), - even if nothing references it. -- A ghost root holds ARB or MOC working data, for example a `.vhdx` under - `MocArb\WorkingDirectory\` or an `ImageStore` folder. +- A ghost root holds ARB or MOC working data: any virtual hard disk (`.vhd`, + `.vhdx`, `.avhd`, `.avhdx`, `.vhds`, `.vhdpmem`), or a `MocArb`, `ImageStore`, or + `WorkingDirectory` folder. - [Step 1C](#1c-check-whether-the-ghost-root-still-redirects-to-live-data) shows `IsReparsePoint = True` for any child of a ghost root. - An **active** CSV reports a `FriendlyVolumeName` under a numbered root. > [!NOTE] -> An `Infrastructure_1` folder is a Path C condition on its own, even when nothing -> references it and it holds only a small, stale breadcrumb. It is reserved Azure -> Local system storage, so it is routed to support rather than removed by hand. The -> companion CSSTools remediation applies the same rule. +> An `Infrastructure_1` folder inside a ghost root is **not** by itself one of +> these conditions. Clusters routinely accumulate a small, stale `Infrastructure_1` +> breadcrumb per numbered root as updates run. What matters is whether anything +> still references it, or whether it holds real ARB or MOC working data. These are platform-managed paths. `Infrastructure_1` is reserved for Azure Local system configuration, the platform deliberately blocks customer storage placement on