From 7041884f6ff798f1f8cc93dad65312a90334364e Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:30:41 -0400 Subject: [PATCH 1/9] Add Storage TSG: repair jobs restart on a failing high-latency disk New troubleshooting guide for S2D storage repair/regeneration jobs that reset and never complete when a single disk is failing with high latency and IO errors. Covers detection surfaces (PowerShell, event logs, cluster log, Azure portal, FCM/WAC), the free-space and reserve check before retiring, the retire-and-rebuild resolution, pre-retire gotchas, and prevention. Adds the guide to the Storage index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd --- TSG/Storage/README.md | 1 + ...-Storage-RepairJobsRestartOnFailingDisk.md | 296 ++++++++++++++++++ 2 files changed, 297 insertions(+) create mode 100644 TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md diff --git a/TSG/Storage/README.md b/TSG/Storage/README.md index 016b7e09..cd6171ef 100644 --- a/TSG/Storage/README.md +++ b/TSG/Storage/README.md @@ -4,3 +4,4 @@ * [How To: Add physical disks to an existing Azure Local cluster](./HowTo-Storage-AddPhysicalDisksToS2DPool.md) * [Troubleshoot: Physical disks not claimed after insertion (`CanPool=False`)](./Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md) * [Troubleshoot: Storage pool capacity threshold warning (fixed vs thin volumes)](./Troubleshoot-Storage-StoragePoolCapacityThreshold.md) +* [Troubleshoot: Storage repair jobs restart and never complete (failing high-latency disk)](./Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md new file mode 100644 index 00000000..d576d611 --- /dev/null +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -0,0 +1,296 @@ +# Troubleshoot storage repair jobs that restart and never complete (failing high-latency disk) + + + + + + + + + + + + + + + + + + +
ComponentStorage
SeverityHigh
Applicable ScenariosDay 2 Operations: Storage health / Disk failure / Volume stuck at No Redundancy
Affected VersionsAll Azure Local releases (Storage Spaces Direct)
+ +## Overview + +On a Storage Spaces Direct (S2D) cluster, a virtual disk (CSV volume) can become stuck in a state where its `Repair` and `Regeneration` storage jobs start, make little or no progress, then restart from the beginning and never converge to zero. The affected volume commonly reports `OperationalStatus = No Redundancy` and `HealthStatus = Unhealthy`, which is a data-at-risk condition. + +The usual contributing factor is a single physical disk that is failing with extremely high latency and a high I/O error rate, but which the health service has not yet marked as failed (it still reports `HealthStatus = Healthy` with `OperationalStatus = "OK, Abnormal Latency"`). Because S2D still considers the drive usable, every repair attempt that must read from or write to that drive times out, and the job requeues. The volume cannot restore redundancy because the healthy copies cannot be rebuilt through the dying drive. + +The same failing drive can also hang the storage completion path under heavy write I/O and trip a `DPC_WATCHDOG_VIOLATION` (bugcheck `0x133`) on the node that hosts the drive, so a node crash and the stuck repair are often two symptoms of one underlying disk fault. + +An important trap: this scenario is frequently misread as a capacity problem, because the pool is often also over its capacity warning threshold. Freeing space does not resolve the stuck repair, and on thin volumes deleting data does not quickly return capacity to the pool. The failing disk is the actual blocker. + +## Symptoms + +- `Get-StorageJob` shows one or more `-Repair` and `-Regeneration` jobs that reset: `PercentComplete` returns to `0`, `BytesTotal` shrinks or changes between samples, and the job's elapsed time resets. Job count never drops to zero. +- `Get-VirtualDisk` shows a volume at `OperationalStatus = {No Redundancy, InService}` and `HealthStatus = Unhealthy`. +- `Get-PhysicalDisk` shows one drive with `OperationalStatus = "OK, Abnormal Latency"` while `HealthStatus` is still `Healthy`. +- `Get-HealthFault` reports a combination of: + - `Microsoft.Health.FaultType.PhysicalDisk.HighLatency.Outlier.AverageIO` (average latency thousands to millions of times the peer drives). + - `Microsoft.Health.FaultType.PhysicalDisk.HighErrorCount.Outlier.AverageIO` (I/O error count far above peer drives). + - `Microsoft.Health.FaultType.VirtualDisks.NoRedundancy` (Critical) and `Microsoft.Health.FaultType.VirtualDisks.LastCopy` (Warning). + - Often also `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded` and `Microsoft.Health.FaultType.Server.Storage.Degraded`. +- Windows event log on the node that hosts the failing drive: + - Source `disk`, Event ID **153**: "The IO operation at logical block address ... was retried." (often hundreds per hour). + - Source `Microsoft-Windows-StorageSpaces-Driver`, Event IDs **203** ("failed an IO operation. Return Code: STATUS_DEVICE_NOT_CONNECTED"), **205** ("Windows lost communication with physical disk"), **207** ("Physical disk ... arrived", indicating the drive is flapping), **209** ("failed a Read IO operation. Return Code: The I/O device reported an I/O error"). + - Event ID **312** ("Virtual disk ... has failed a write operation to all its copies") and **302** ("pool disks hosting space meta-data ... failed a space meta-data update"), which accompany the No Redundancy state. + - Event IDs **304** ("virtual disk ... is in a degraded state") and, after resolution, **305** ("Virtual disk ... is now healthy"). +- Optional and correlated: a node bugcheck `0x00000133 DPC_WATCHDOG_VIOLATION` under heavy write I/O, with the faulting stack in the storage completion path (`storport` / `CLASSPNP` / the S2D cluster block filter), on the same node that hosts the failing drive. In the System log this appears as BugCheck (Event ID 1001) and Kernel-Power 41. + +## Where the failing disk shows up (detection by surface) + +The same failing drive is visible through several tools. Use whichever the admin has in front of them. + +### PowerShell (most authoritative) + +```powershell +# 1. The drive itself: Healthy but "Abnormal Latency". +Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, Usage, HealthStatus, OperationalStatus, PhysicalLocation + +# 2. Health faults with reasons (latency and error outliers, plus the volume state). +Get-HealthFault | Select-Object FaultType, PerceivedSeverity, Reason, FaultingObjectDescription + +# 3. Reliability counters: the hard evidence. On a real failing drive you see read/write +# error totals in the thousands to millions and max latency in seconds, not milliseconds. +Get-PhysicalDisk -SerialNumber | Get-StorageReliabilityCounter | + Select-Object ReadErrorsTotal, ReadErrorsUncorrected, WriteErrorsTotal, ReadLatencyMax, WriteLatencyMax, PowerOnHours +``` + +Example from a real case: `ReadErrorsTotal = 1,252,510`, `ReadLatencyMax = 11,708` ms (healthy peers are under ~20 ms), `PowerOnHours = 32,273` (about 3.7 years). In that case `ReadErrorsUncorrected = 0`, which is why the rebuild recovered all data. A drive with non-zero *uncorrected* errors that also holds the last copy is the data-loss case to worry about. + +### Windows event log (on the node hosting the drive) + +- `disk` **153** "The IO operation ... was retried" (leading indicator, often hundreds per hour). +- `Microsoft-Windows-StorageSpaces-Driver` **203** (failed IO / STATUS_DEVICE_NOT_CONNECTED), **205** (lost communication), **207** (drive "arrived" repeatedly, meaning it is flapping), **209** (failed Read IO / I/O device error). +- `Microsoft-Windows-StorageSpaces-Driver` **312** (write failed to all copies), **302** (space metadata update failed), **304** and **305** (virtual disk degraded, then healthy). + +```powershell +Get-WinEvent -FilterHashtable @{ LogName='System'; ProviderName='disk'; Id=153; StartTime=(Get-Date).AddHours(-24) } | + Group-Object Id | Select-Object Name, Count +Get-WinEvent -LogName 'Microsoft-Windows-StorageSpaces-Driver/Operational' -MaxEvents 200 | + Group-Object Id | Sort-Object Count -Descending +``` + +### Cluster log + +The cluster log captures the health service and storage subsystem entries. Generate it and search the per-node logs for the drive's object ID and I/O status codes: + +```powershell +Get-ClusterLog -Destination C:\Temp -TimeSpan 120 +Select-String -Path C:\Temp\*_cluster.log -Pattern 'STATUS_IO_TIMEOUT','STATUS_DEVICE_NOT_CONNECTED','Abnormal Latency','NoRedundancy' +``` + +### Azure portal + +For an Arc-connected Azure Local cluster, the physical-disk and virtual-disk health faults surface as **alerts** on the cluster resource (the Monitoring / Alerts area and the resource Health blade), and in Azure Monitor if it is configured. The `PhysicalDisk.HighLatency` / `HighErrorCount` and `VirtualDisks.NoRedundancy` faults appear there with the same fault text. The portal is good for noticing the condition; retirement itself is done from PowerShell or Windows Admin Center. + +### Failover Cluster Manager and Windows Admin Center + +- **Failover Cluster Manager** shows the cluster and CSV state. A degraded volume appears under Storage with the CSV in an online-degraded or warning state. FCM does not surface per physical-disk latency well. +- **Windows Admin Center** (the recommended GUI for S2D) shows per-drive health under the cluster Drives view, including the Warning status and latency, and offers Retire and Locate actions directly. + +## What and Why + +Storage Spaces Direct keeps three copies of three-way mirror data spread across three fault domains (nodes). When a drive begins to fail slowly, its SMART and health state can still read as `Healthy`, so S2D keeps scheduling I/O to it. Repair and regeneration jobs that touch slabs on that drive issue reads and writes that never complete within the storage timeout, the job is aborted and requeued, and you observe the restart loop. If the failing drive holds the only currently readable copy of a region, that region shows as `No Redundancy` and appears in `Get-PhysicalDisk -NoRedundancy` for the affected volume. + +Under heavy write I/O, the same non-completing drive can leave a storage completion routine holding the processor dispatch level too long, which trips the DPC watchdog and bugchecks the node. That is why a node crash and a stuck repair frequently share one cause. + +Why capacity is a red herring here: + +- The pool capacity warning (`PoolCapacityThresholdExceeded`) is a `Minor` fault and is usually incidental. A repair that only needs to resync a few gigabytes is not blocked by a pool that is 85 percent full when there is still free reserve capacity. +- On thin-provisioned volumes, deleting files frees space inside the volume but does not immediately return the freed slabs to the pool. `Optimize-Volume -ReTrim` issues the unmap but pool reclaim can still lag. Do not expect deleting data to create rebuild headroom quickly. + +## Resolution + +The fix is to mark the failing drive `Retired` so S2D stops using it and rebuilds its data onto the remaining healthy drives, then physically replace it after the rebuild completes. + +### Prerequisites + +- Run all commands in an elevated PowerShell session on a cluster node. +- Confirm all cluster nodes are up: `Get-ClusterNode`. Do not start disk maintenance while a node is down, because that reduces the fault domains available for the rebuild. +- Understand the reserve-capacity model before you retire anything (see Step 3). You do **not** need a spare drive to recover. + +### Before you retire: pre-checks and gotchas + +Run through this list before Step 4. Most stuck or unsafe retires trace back to skipping one of these. + +- **All nodes up.** Confirm `Get-ClusterNode` shows every node `Up`. Retiring a drive while a node is down removes a fault domain and can block the rebuild or drop below resiliency. +- **Only one fault domain affected.** Confirm no other drive is already `Retired`, failed, or `Lost Communication` on a *different* node: `Get-PhysicalDisk | Where-Object { $_.HealthStatus -ne 'Healthy' -or $_.Usage -eq 'Retired' }`. Retiring a second drive in a second fault domain while a three-way mirror is already degraded can cause data loss. +- **Enough free reserve (Step 3).** Pool free must exceed the drive's used capacity, with reserve left over. Do not rely on deleting data to create it at the last minute; thin reclaim is slow. +- **Check for last-copy data.** Run `Get-VirtualDisk -FriendlyName | Get-PhysicalDisk -NoRedundancy`. If the failing drive is returned, some regions have no other copy; retiring is still correct but is a data-at-risk operation (see the Step 4 caveat). Also check `ReadErrorsUncorrected` on the drive: non-zero uncorrected read errors on a last-copy drive is the worst case. +- **Not during an update or CAU run.** Do not retire a drive while a solution update, Cluster-Aware Updating run, or node maintenance is in progress. Wait for a quiet window so the rebuild is not competing with reboots and storage maintenance. +- **Expect node instability if the same drive is crashing the node.** If the failing drive is also tripping `DPC_WATCHDOG_VIOLATION` bugchecks, the hosting node may reboot during triage. Retiring the drive is what stops that, but plan for the node to bounce until it is retired. +- **Do not `Remove-PhysicalDisk` before the rebuild finishes.** Removing (as opposed to retiring) pulls the drive from the pool; doing it before evacuation completes can lose data still on the drive. Retire first (Step 4), let the jobs drain (Step 5), then remove (Step 6). +- **Have a replacement on order.** You do not need the spare to recover, but order a supported-model drive so you can physically replace the retired one and restore the full drive count. + +### Steps + +#### Step 1: Confirm the restart loop and the affected volume + +```powershell +# Sample twice, ~60 seconds apart. A stuck repair shows PercentComplete resetting +# toward 0 and BytesTotal changing between samples, and the count never reaches 0. +Get-StorageJob | Select-Object Name, JobState, PercentComplete, BytesProcessed, BytesTotal + +# The affected volume reports No Redundancy / Unhealthy. +Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus, OperationalDetails +``` + +#### Step 2: Identify the failing physical disk + +```powershell +# Authoritative reasons. Look for HighLatency / HighErrorCount on a physical disk, +# plus VirtualDisks.NoRedundancy / LastCopy on the volume. +Get-HealthFault | Select-Object FaultType, PerceivedSeverity, Reason, FaultingObjectDescription + +# The failing drive: Healthy but "Abnormal Latency". +Get-PhysicalDisk | Where-Object { ($_.OperationalStatus -join ',') -match 'Abnormal Latency' } | + Select-Object FriendlyName, SerialNumber, UniqueId, PhysicalLocation, HealthStatus, OperationalStatus + +# Corroborate with the reliability counters (read/write error totals, latency). +Get-PhysicalDisk -SerialNumber | Get-StorageReliabilityCounter | + Select-Object DeviceId, ReadErrorsTotal, WriteErrorsTotal, ReadLatencyMax, WriteLatencyMax + +# Confirm whether this drive holds the LAST copy of any region of the volume. +# If it returns this drive, retiring it is a last-copy operation (see Step 4 caveat). +Get-VirtualDisk -FriendlyName | Get-PhysicalDisk -NoRedundancy | + Select-Object FriendlyName, SerialNumber, OperationalStatus +``` + +Record the drive's `UniqueId`; you will use it in Step 4 and Step 6. + +#### Step 3: Confirm there is enough reserve capacity to rebuild (no spare required) + +S2D does not use dedicated hot spares. It rebuilds a retired or failed drive's data into free **reserve capacity** distributed across the remaining drives and nodes. You do not need a replacement drive in hand to recover redundancy. What you need is enough free capacity in the surviving fault domains, with the general guidance being to keep roughly one capacity drive's worth of space free per server (up to four). + +```powershell +# 1. How much data must be rebuilt elsewhere equals the USED (allocated) capacity of the +# failing drive. A retire evacuates the drive's whole allocated content, not just a +# "dirty" delta. Read this BEFORE you retire; afterward it drops toward zero. +Get-PhysicalDisk -UniqueId | + Select-Object FriendlyName, + @{n='UsedGB';e={[math]::Round($_.AllocatedSize/1e9,1)}}, + @{n='SizeGB';e={[math]::Round($_.Size/1e9,1)}} + +# 2. Pool free space and fill level. +Get-StoragePool -FriendlyName | + Select-Object FriendlyName, HealthStatus, + @{n='UsedPct';e={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}}, + @{n='FreeTB';e={[math]::Round(($_.Size-$_.AllocatedSize)/1e12,2)}} + +# 3. Per-drive fill. No single healthy capacity drive should be at or near 100 percent, +# and the surviving nodes must have room to take the rebuilt copies. +Get-PhysicalDisk | Where-Object Usage -eq 'Auto-Select' | + Select-Object FriendlyName, + @{n='PctAlloc';e={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}} | + Sort-Object PctAlloc -Descending | Select-Object -First 5 +``` + +The check: **pool free space must exceed the failing drive's used capacity**, and you should still have reserve left afterward (about one capacity drive per node). Because three-way mirror places copies across three nodes, the free space also has to be distributed so the surviving nodes can each hold their share. If one node's drives are all near 100 percent, the rebuild for slabs that need that node stalls even when the pool total looks fine. + +Worked example from a real case: the failing 2.4 TB drive held about 2.0 TB of data (its used capacity), and the pool had about 10.8 TB free (roughly 86 percent full). Free space (10.8 TB) comfortably exceeded both the ~2.0 TB that had to be rebuilt and the ~9.6 TB reserve target (one 2.4 TB drive per node across four nodes), so the retire was safe and the rebuild completed. + +If the pool has essentially no free reserve (for example above roughly 95 percent with no per-node headroom), retiring the drive can leave S2D with nowhere to rebuild and the volume stays degraded. In that case add capacity or reduce data first, then retire. Remember that on thin volumes, deleting data does not free pool space quickly, so plan the reserve ahead of time rather than deleting at the last minute. + +#### Step 4: Retire the failing drive [MEDIUM RISK] + +```powershell +# Marks the drive do-not-use and starts the evacuation/rebuild onto healthy drives. +Set-PhysicalDisk -UniqueId -Usage Retired + +# Verify. +Get-PhysicalDisk -UniqueId | Select-Object FriendlyName, Usage, OperationalStatus +``` + +Caveat when the drive is a last-copy holder (Step 2 returned it under `-NoRedundancy`): retiring forces S2D to read those regions off the dying drive to rebuild them. Any region the drive can no longer read cannot be rebuilt and that data is lost. Retiring is still the correct action, because it triggers the evacuation while the drive is at least partly alive; leaving the drive in service guarantees the volume stays at No Redundancy. Retire sooner rather than later to maximize what can be salvaged. + +#### Step 5: Monitor the rebuild + +Retiring the drive starts an evacuation. You will see a pair of jobs, `-Repair` and `-Regeneration`, for **every** virtual disk that had data on the retired drive (typically all of the UserStorage volumes plus the small `Infrastructure_1` and `ClusterPerformanceHistory` volumes). `Repair` restores resiliency and `Regeneration` rebuilds the missing copies. They run in parallel, should climb monotonically toward 100 percent, then disappear as the job count drops to zero. This is the opposite of the reset loop in Step 1: real, increasing `BytesProcessed` against a stable `BytesTotal`. + +```powershell +# Repair/Regeneration jobs should now show a real, monotonically climbing scope +# (not the reset loop), then drain to zero. +Get-StorageJob | Select-Object Name, JobState, PercentComplete, BytesProcessed, BytesTotal + +# The volume returns to Healthy / OK and the NoRedundancy fault clears. +Get-VirtualDisk -FriendlyName | Select-Object HealthStatus, OperationalStatus +Get-HealthFault | Select-Object FaultType, PerceivedSeverity +``` + +If no repair jobs start within a few minutes of retiring the drive, trigger one explicitly: + +```powershell +Repair-VirtualDisk -FriendlyName +``` + +During evacuation the pool used percentage rises briefly as replacement copies are written, then settles as the retired drive's slabs are released. This is expected. + +The rebuild is complete when all of the following are true: `Get-StorageJob` returns nothing, every volume is `HealthStatus = Healthy` / `OperationalStatus = OK`, the `VirtualDisks.NoRedundancy` and `LastCopy` faults have cleared, and the retired drive's used capacity (`AllocatedSize`) has dropped to near zero because its data now lives elsewhere. Only then proceed to Step 6. + +#### Step 6: Physically replace and remove the drive [MEDIUM RISK] + +Only after the rebuild jobs reach zero and the volumes are Healthy: + +```powershell +# Turn on the location indicator (if supported) to find the drive in the chassis. +Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIndication + +# Remove the retired drive from the pool, then physically swap it. +Remove-PhysicalDisk -UniqueId -StoragePoolFriendlyName +``` + +A replacement drive of a supported model is claimed automatically (or add it manually per the Add-Physical-Disks TSG). No manual repair trigger is normally required; S2D rebalances onto the new drive. + +## Prevention + +- Keep reserve capacity free at all times, roughly one capacity drive per server (up to four). This is what lets you retire or lose a drive and rebuild in place without a spare, and without hitting the capacity wall mid-rebuild. +- Treat the `PhysicalDisk.HighLatency.Outlier.AverageIO` and `PhysicalDisk.HighErrorCount.Outlier.AverageIO` health faults as early warnings. A drive that is Healthy but showing "Abnormal Latency" is a retire candidate before it stalls a repair or bugchecks a node. +- Watch for repeated `disk` Event ID 153 ("was retried") and Storage Spaces Event IDs 203/205/209 against a single drive. A rising count on one drive is the leading indicator. +- Do not rely on deleting data to create rebuild headroom in a hurry. On thin volumes the freed space is not returned to the pool immediately even after `Optimize-Volume -ReTrim`. Plan reserve capacity in advance instead. + +## Data to Collect Before Opening a Support Case + +```powershell +# Cluster + storage state snapshot +Get-ClusterNode | Select-Object Name, State +Get-StoragePool -FriendlyName | Select-Object FriendlyName, HealthStatus, Size, AllocatedSize +Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus, ProvisioningType +Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, MediaType, Usage, HealthStatus, OperationalStatus +Get-StorageJob | Select-Object Name, JobState, PercentComplete, BytesProcessed, BytesTotal +Get-HealthFault | Select-Object FaultType, PerceivedSeverity, Reason, FaultingObjectDescription + +# Reliability counters for the suspect drive +Get-PhysicalDisk -SerialNumber | Get-StorageReliabilityCounter + +# Event logs from the node hosting the drive +Get-WinEvent -FilterHashtable @{ LogName='System'; Id=153; StartTime=(Get-Date).AddHours(-24) } | + Select-Object TimeCreated, Id, Message -First 50 +Get-WinEvent -LogName 'Microsoft-Windows-StorageSpaces-Driver/Operational' -MaxEvents 200 + +# Last 60 minutes of cluster log to C:\Temp +Get-ClusterLog -Destination C:\Temp -TimeSpan 60 +``` + +## Related Issues + +- Troubleshoot the storage pool capacity threshold warning (fixed vs thin volumes): `TSG/Storage/Troubleshoot-Storage-StoragePoolCapacityThreshold.md`. Use this when the primary problem is capacity rather than a failing drive. +- Troubleshoot physical disks not claimed after insertion (`CanPool=False`): `TSG/Storage/Troubleshoot-Storage-PhysicalDiskCanPoolFalse.md`. Use this when claiming the replacement drive in Step 6. +- Troubleshooting storage with the Support Diagnostics Tool: `TSG/Storage/Troubleshooting-Storage-With-Support-Diagnostics-Tool.md`. + +## References + +- Replace drives in Storage Spaces Direct (retire with `Set-PhysicalDisk -Usage Retired`, then `Remove-PhysicalDisk`): https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/replace-drives +- Plan volumes and reserve capacity for repairs: https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/plan-volumes#reserve-capacity +- Storage Spaces and Storage Spaces Direct health and operational states: https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/storage-spaces-states +- Fault tolerance and storage efficiency in Storage Spaces Direct: https://learn.microsoft.com/en-us/windows-server/storage/storage-spaces/fault-tolerance +- `Remove-PhysicalDisk` PowerShell reference: https://learn.microsoft.com/en-us/powershell/module/storage/remove-physicaldisk From 9d63f9cdff11e593d1d1024a17a2f78101c58198 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 10 Jul 2026 19:43:43 -0400 Subject: [PATCH 2/9] Address Copilot review feedback and add usability sections Resolves the four Copilot PR-review findings: (1) the pre-retire one-fault-domain check now also filters OperationalStatus (Abnormal Latency / Lost Communication) and surfaces the node, so a second stale-Healthy failing disk is not missed; (2) the Step 3 free-space query excludes the retiring disk, shows per-node free capacity, and no longer truncates to five rows; (3) the worked example now distinguishes sufficient evacuation headroom from full reserve (not restored until the replacement drive is added); (4) Step 6 uses Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ...) because -UniqueId is not a valid parameter. Also adds usability sections from a persona review: a TL;DR fastest-path, an Impact and ownership summary, a Glossary, a Scope note distinguishing RDMA/network issues, an explicit workload-impact note, a When to escalate section, and OEM/hardware framing for the physical swap. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd --- ...-Storage-RepairJobsRestartOnFailingDisk.md | 53 +++++++++++++++---- 1 file changed, 42 insertions(+), 11 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index d576d611..2030b19e 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -19,6 +19,12 @@ +**TL;DR (fastest path):** Confirm the stuck repair (`Get-StorageJob` jobs resetting, a volume at `No Redundancy`), find the failing drive (`Get-HealthFault` shows a `PhysicalDisk.HighLatency` / `HighErrorCount` outlier, and `Get-PhysicalDisk` shows it `Healthy` but `"Abnormal Latency"`), confirm the pool has free reserve, then `Set-PhysicalDisk -UniqueId -Usage Retired` and watch the rebuild drain to zero. Full detail follows; unfamiliar terms are defined in the Glossary. + +**Impact and ownership:** A volume is running without redundancy, so this is data-at-risk, but no VM or cluster downtime is needed to fix it: the retire and rebuild run online. Owner: customer or partner IT runs the retire; the OEM or hardware vendor physically replaces the drive. Duration: the rebuild usually takes from tens of minutes to a few hours, depending on how much data was on the drive. + +**Scope:** This guide is for a failing physical disk. If instead you see network or RDMA faults (for example `StorageSubsystem.RDMA.Alert`) with healthy disks, that is a storage-network problem, not this one; use the networking storage TSGs. + ## Overview On a Storage Spaces Direct (S2D) cluster, a virtual disk (CSV volume) can become stuck in a state where its `Repair` and `Regeneration` storage jobs start, make little or no progress, then restart from the beginning and never converge to zero. The affected volume commonly reports `OperationalStatus = No Redundancy` and `HealthStatus = Unhealthy`, which is a data-at-risk condition. @@ -107,7 +113,18 @@ Under heavy write I/O, the same non-completing drive can leave a storage complet Why capacity is a red herring here: - The pool capacity warning (`PoolCapacityThresholdExceeded`) is a `Minor` fault and is usually incidental. A repair that only needs to resync a few gigabytes is not blocked by a pool that is 85 percent full when there is still free reserve capacity. -- On thin-provisioned volumes, deleting files frees space inside the volume but does not immediately return the freed slabs to the pool. `Optimize-Volume -ReTrim` issues the unmap but pool reclaim can still lag. Do not expect deleting data to create rebuild headroom quickly. +- On thin-provisioned volumes, deleting files frees space inside the volume but does not necessarily return the freed slabs to the pool. `Optimize-Volume -ReTrim` issues the unmap, but the pool may not reclaim the capacity for a long time, if at all, within a maintenance window. Do not expect deleting data to create rebuild headroom quickly. + +## Glossary + +- **Three-way mirror**: the volume keeps three copies of every block on three different servers, so it survives two failures. +- **Fault domain**: a failure boundary, here a server (node). Copies of a block are placed in different fault domains. +- **Slab**: the unit of capacity S2D allocates from the pool (a chunk of a virtual disk). +- **Reserve capacity**: pool space left free on purpose so S2D can rebuild a failed or retired drive in place. S2D uses this instead of a dedicated hot spare. +- **Thin provisioning**: the volume consumes pool capacity only as data is written. Deleting data does not promptly return that capacity to the pool. +- **No Redundancy**: a volume state where some regions have no available copy. The volume is online but at risk of data loss if another failure occurs. +- **Repair and Regeneration**: the two S2D jobs that restore a volume's copies after a drive is lost or retired. +- **Retire and Remove**: `Retire` marks a drive do-not-use and evacuates its data (the drive stays in the pool). `Remove` pulls the evacuated drive out of the pool for physical replacement. ## Resolution @@ -124,7 +141,7 @@ The fix is to mark the failing drive `Retired` so S2D stops using it and rebuild Run through this list before Step 4. Most stuck or unsafe retires trace back to skipping one of these. - **All nodes up.** Confirm `Get-ClusterNode` shows every node `Up`. Retiring a drive while a node is down removes a fault domain and can block the rebuild or drop below resiliency. -- **Only one fault domain affected.** Confirm no other drive is already `Retired`, failed, or `Lost Communication` on a *different* node: `Get-PhysicalDisk | Where-Object { $_.HealthStatus -ne 'Healthy' -or $_.Usage -eq 'Retired' }`. Retiring a second drive in a second fault domain while a three-way mirror is already degraded can cause data loss. +- **Only one fault domain affected.** A second failing drive can read as `Healthy` while its `OperationalStatus` is `Abnormal Latency` or `Lost Communication`, which is exactly the condition this guide describes, so filter on operational status too, not just `HealthStatus` and `Usage`, and note each suspect drive's node: `Get-PhysicalDisk | Where-Object { $_.Usage -eq 'Retired' -or $_.HealthStatus -ne 'Healthy' -or (($_.OperationalStatus -join ',') -ne 'OK') } | Select-Object FriendlyName, SerialNumber, HealthStatus, @{n='Op';e={$_.OperationalStatus -join ','}}, PhysicalLocation` (identify a suspect drive's node with `Get-PhysicalDisk -SerialNumber | Get-StorageNode -PhysicallyConnected`). Retiring a second drive in a second fault domain while a three-way mirror is already degraded can cause data loss. - **Enough free reserve (Step 3).** Pool free must exceed the drive's used capacity, with reserve left over. Do not rely on deleting data to create it at the last minute; thin reclaim is slow. - **Check for last-copy data.** Run `Get-VirtualDisk -FriendlyName | Get-PhysicalDisk -NoRedundancy`. If the failing drive is returned, some regions have no other copy; retiring is still correct but is a data-at-risk operation (see the Step 4 caveat). Also check `ReadErrorsUncorrected` on the drive: non-zero uncorrected read errors on a last-copy drive is the worst case. - **Not during an update or CAU run.** Do not retire a drive while a solution update, Cluster-Aware Updating run, or node maintenance is in progress. Wait for a quiet window so the rebuild is not competing with reboots and storage maintenance. @@ -187,17 +204,22 @@ Get-StoragePool -FriendlyName | @{n='UsedPct';e={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}}, @{n='FreeTB';e={[math]::Round(($_.Size-$_.AllocatedSize)/1e12,2)}} -# 3. Per-drive fill. No single healthy capacity drive should be at or near 100 percent, -# and the surviving nodes must have room to take the rebuilt copies. -Get-PhysicalDisk | Where-Object Usage -eq 'Auto-Select' | - Select-Object FriendlyName, +# 3. Per-surviving-drive free capacity WITH its node. Exclude the drive being retired and do +# NOT truncate: each surviving node must have room for the rebuilt copies, so one near-full +# drive on a single node can stall the rebuild even when the pool total looks adequate. +# (Pool free can also include unusable free extents on the drive being retired, so count +# only surviving disks.) +Get-PhysicalDisk | Where-Object { $_.Usage -eq 'Auto-Select' -and $_.SerialNumber -ne '' } | + Select-Object @{n='Node';e={($_ | Get-StorageNode -PhysicallyConnected | Select-Object -First 1).Name}}, + FriendlyName, SerialNumber, + @{n='FreeGB';e={[math]::Round(($_.Size-$_.AllocatedSize)/1e9,1)}}, @{n='PctAlloc';e={[math]::Round(100*$_.AllocatedSize/$_.Size,1)}} | - Sort-Object PctAlloc -Descending | Select-Object -First 5 + Sort-Object Node, PctAlloc ``` The check: **pool free space must exceed the failing drive's used capacity**, and you should still have reserve left afterward (about one capacity drive per node). Because three-way mirror places copies across three nodes, the free space also has to be distributed so the surviving nodes can each hold their share. If one node's drives are all near 100 percent, the rebuild for slabs that need that node stalls even when the pool total looks fine. -Worked example from a real case: the failing 2.4 TB drive held about 2.0 TB of data (its used capacity), and the pool had about 10.8 TB free (roughly 86 percent full). Free space (10.8 TB) comfortably exceeded both the ~2.0 TB that had to be rebuilt and the ~9.6 TB reserve target (one 2.4 TB drive per node across four nodes), so the retire was safe and the rebuild completed. +Worked example from a real case: the failing 2.4 TB drive held about 2.0 TB of data, and the pool reported about 10.8 TB free (roughly 86 percent full). That 10.8 TB still included the ~2.4 TB sitting on the drive being retired, so the surviving disks held only about 8.4 TB free. The ~2.0 TB that had to be relocated fit easily into that 8.4 TB, so there was enough evacuation headroom and the rebuild completed and restored redundancy. Note, however, that 8.4 TB is below the ~9.6 TB full-reserve target (one 2.4 TB drive per node across four nodes): the pool ran with less than the recommended repair reserve until the failed drive was physically replaced, so add the replacement promptly to restore full reserve. If the pool has essentially no free reserve (for example above roughly 95 percent with no per-node headroom), retiring the drive can leave S2D with nowhere to rebuild and the volume stays degraded. In that case add capacity or reduce data first, then retire. Remember that on thin volumes, deleting data does not free pool space quickly, so plan the reserve ahead of time rather than deleting at the last minute. @@ -217,6 +239,8 @@ Caveat when the drive is a last-copy holder (Step 2 returned it under `-NoRedund Retiring the drive starts an evacuation. You will see a pair of jobs, `-Repair` and `-Regeneration`, for **every** virtual disk that had data on the retired drive (typically all of the UserStorage volumes plus the small `Infrastructure_1` and `ClusterPerformanceHistory` volumes). `Repair` restores resiliency and `Regeneration` rebuilds the missing copies. They run in parallel, should climb monotonically toward 100 percent, then disappear as the job count drops to zero. This is the opposite of the reset loop in Step 1: real, increasing `BytesProcessed` against a stable `BytesTotal`. +Workload impact: the rebuild runs online, so VMs and CSV volumes stay available; expect elevated storage latency until it finishes. + ```powershell # Repair/Regeneration jobs should now show a real, monotonically climbing scope # (not the reset loop), then drain to zero. @@ -245,11 +269,18 @@ Only after the rebuild jobs reach zero and the volumes are Healthy: # Turn on the location indicator (if supported) to find the drive in the chassis. Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIndication -# Remove the retired drive from the pool, then physically swap it. -Remove-PhysicalDisk -UniqueId -StoragePoolFriendlyName +# Remove the retired drive from the pool, then physically swap it. Remove-PhysicalDisk has +# no -UniqueId parameter, so resolve the disk object and pass it via -PhysicalDisks. +Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ) -StoragePoolFriendlyName ``` -A replacement drive of a supported model is claimed automatically (or add it manually per the Add-Physical-Disks TSG). No manual repair trigger is normally required; S2D rebalances onto the new drive. +The physical drive replacement is a hardware task: engage the OEM or your hardware vendor and follow their drive-replacement procedure for the chassis (the location indicator above lights the drive bay on Dell, HPE, and Lenovo servers). Expected end state: a replacement drive of a supported model is claimed automatically, S2D rebalances onto it, and every drive and volume returns to Healthy. No manual repair trigger is normally required. To add the replacement manually, see the Add-Physical-Disks TSG. + +## When to escalate + +- **To the OEM or your hardware vendor**: for the physical drive replacement, and for any drive reporting non-zero *uncorrected* read/write errors or repeated surprise-removal, which indicates hardware failure. +- **To Microsoft Support**: if the rebuild does not progress even though reserve capacity is available and no drive is failing, or if a volume stays at `No Redundancy` after the retire and rebuild complete. +- Collect the data in the section below before escalating. ## Prevention From cba32507109fe8ecf1060f39d7d783ac4c0af333 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 10 Jul 2026 21:11:53 -0400 Subject: [PATCH 3/9] Storage TSG: use Overview (house style) and correct the resiliency model Replace the "TL;DR" callout with an "## Overview" section that leads with plain-language what, why, and impact and keeps the fast at-a-glance path. Correct the resiliency description: name the Azure Local default (three-way mirror on three or more nodes, two-way mirror on two-node) and cover the other Storage Spaces Direct schemes (parity, dual parity, nested resiliency, and mirror-accelerated parity), plus Simple with no redundancy. Explain that a slow, high-latency but SMART-healthy disk is not auto-failed under any scheme, and that retire then rebuild restores redundancy on any resilient volume while a Simple volume must be restored from backup. Confirm ResiliencySettingName before acting. Update the Glossary and a Symptoms bullet to match. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd --- ...-Storage-RepairJobsRestartOnFailingDisk.md | 39 ++++++++++++++----- 1 file changed, 29 insertions(+), 10 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index 2030b19e..9ef88eaa 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -19,26 +19,26 @@ -**TL;DR (fastest path):** Confirm the stuck repair (`Get-StorageJob` jobs resetting, a volume at `No Redundancy`), find the failing drive (`Get-HealthFault` shows a `PhysicalDisk.HighLatency` / `HighErrorCount` outlier, and `Get-PhysicalDisk` shows it `Healthy` but `"Abnormal Latency"`), confirm the pool has free reserve, then `Set-PhysicalDisk -UniqueId -Usage Retired` and watch the rebuild drain to zero. Full detail follows; unfamiliar terms are defined in the Glossary. - -**Impact and ownership:** A volume is running without redundancy, so this is data-at-risk, but no VM or cluster downtime is needed to fix it: the retire and rebuild run online. Owner: customer or partner IT runs the retire; the OEM or hardware vendor physically replaces the drive. Duration: the rebuild usually takes from tens of minutes to a few hours, depending on how much data was on the drive. - -**Scope:** This guide is for a failing physical disk. If instead you see network or RDMA faults (for example `StorageSubsystem.RDMA.Alert`) with healthy disks, that is a storage-network problem, not this one; use the networking storage TSGs. - ## Overview -On a Storage Spaces Direct (S2D) cluster, a virtual disk (CSV volume) can become stuck in a state where its `Repair` and `Regeneration` storage jobs start, make little or no progress, then restart from the beginning and never converge to zero. The affected volume commonly reports `OperationalStatus = No Redundancy` and `HealthStatus = Unhealthy`, which is a data-at-risk condition. +On a Storage Spaces Direct (S2D) cluster, a data volume's `Repair` and `Regeneration` storage jobs can start, make little or no progress, then restart from the beginning and never reach zero. The affected volume commonly reports `OperationalStatus = No Redundancy` (or `Incomplete`) and `HealthStatus = Unhealthy`, which is a data-at-risk condition. The usual contributing factor is a single physical disk that is failing with extremely high latency and a high I/O error rate, but which the health service has not yet marked as failed (it still reports `HealthStatus = Healthy` with `OperationalStatus = "OK, Abnormal Latency"`). Because S2D still considers the drive usable, every repair attempt that must read from or write to that drive times out, and the job requeues. The volume cannot restore redundancy because the healthy copies cannot be rebuilt through the dying drive. The same failing drive can also hang the storage completion path under heavy write I/O and trip a `DPC_WATCHDOG_VIOLATION` (bugcheck `0x133`) on the node that hosts the drive, so a node crash and the stuck repair are often two symptoms of one underlying disk fault. +**How to resolve it (at a glance).** Confirm the stuck repair, identify the failing drive (a `Get-PhysicalDisk` disk that is still `Healthy` but reports `"Abnormal Latency"` or `Lost Communication`, and on physical hardware a `Get-HealthFault` `PhysicalDisk.HighLatency` / `HighErrorCount` outlier), confirm the pool has free reserve, then mark the drive do-not-use with `Set-PhysicalDisk -Usage Retired` and watch the rebuild drain to zero. The detailed steps, safety checks, and a glossary of terms follow below. + +**Impact and ownership.** A volume is running without full redundancy, so this is data-at-risk, but no VM or cluster downtime is needed to fix it: the retire and rebuild run online. Owner: customer or partner IT runs the retire; the OEM or hardware vendor physically replaces the drive. Duration: the rebuild usually takes from tens of minutes to a few hours, depending on how much data was on the drive and the volume's resiliency type. + +**Scope.** This guide is for a failing physical disk on an S2D pool. If instead you see network or RDMA faults (for example `StorageSubsystem.RDMA.Alert`) with healthy disks, that is a storage-network problem, not this one; use the networking storage TSGs. + An important trap: this scenario is frequently misread as a capacity problem, because the pool is often also over its capacity warning threshold. Freeing space does not resolve the stuck repair, and on thin volumes deleting data does not quickly return capacity to the pool. The failing disk is the actual blocker. ## Symptoms - `Get-StorageJob` shows one or more `-Repair` and `-Regeneration` jobs that reset: `PercentComplete` returns to `0`, `BytesTotal` shrinks or changes between samples, and the job's elapsed time resets. Job count never drops to zero. -- `Get-VirtualDisk` shows a volume at `OperationalStatus = {No Redundancy, InService}` and `HealthStatus = Unhealthy`. +- `Get-VirtualDisk` shows a volume at `OperationalStatus = {No Redundancy, InService}` (or `{Incomplete, InService}` on a two-copy volume) and `HealthStatus = Unhealthy`. - `Get-PhysicalDisk` shows one drive with `OperationalStatus = "OK, Abnormal Latency"` while `HealthStatus` is still `Healthy`. - `Get-HealthFault` reports a combination of: - `Microsoft.Health.FaultType.PhysicalDisk.HighLatency.Outlier.AverageIO` (average latency thousands to millions of times the peer drives). @@ -106,7 +106,26 @@ For an Arc-connected Azure Local cluster, the physical-disk and virtual-disk hea ## What and Why -Storage Spaces Direct keeps three copies of three-way mirror data spread across three fault domains (nodes). When a drive begins to fail slowly, its SMART and health state can still read as `Healthy`, so S2D keeps scheduling I/O to it. Repair and regeneration jobs that touch slabs on that drive issue reads and writes that never complete within the storage timeout, the job is aborted and requeued, and you observe the restart loop. If the failing drive holds the only currently readable copy of a region, that region shows as `No Redundancy` and appears in `Get-PhysicalDisk -NoRedundancy` for the affected volume. +### Resiliency types (and why the drive was not automatically failed) + +S2D stores each volume with a resiliency type that keeps redundant data across fault domains (usually nodes). **On Azure Local the default is three-way mirror (three copies) for clusters of three or more nodes, and two-way mirror (two copies) for two-node clusters.** That default is not the only option, and the volume you are repairing may use a different scheme: + +- **Two-way / three-way mirror** — two or three full copies across nodes (the defaults above); the volume shows `ResiliencySettingName = Mirror` with `NumberOfDataCopies` 2 or 3. +- **Nested resiliency** (two-node clusters) — nested two-way mirror, or nested mirror-accelerated parity, which survives two concurrent hardware failures on a two-node cluster. +- **Parity / dual parity** (erasure coding, four or more nodes) — space-efficient; reconstructs data from parity rather than a full copy (`ResiliencySettingName = Parity`). +- **Mirror-accelerated parity** — one volume that combines a mirror tier and a parity tier. +- **Simple (no resiliency)** — not recommended and never an Azure Local default; a single disk loss means the data on that volume's affected regions is lost. + +Confirm the actual scheme before you act: `Get-VirtualDisk -FriendlyName | Select-Object FriendlyName, ResiliencySettingName, NumberOfDataCopies, PhysicalDiskRedundancy`. + +**Why the drive was not automatically failed (this is scheme-independent).** S2D auto-retires a drive only when its own health/SMART state marks it failed. A drive that is merely slow or erroring, or that dropped its connection but still enumerates, can stay `HealthStatus = Healthy` (`OperationalStatus = "OK, Abnormal Latency"`, or `Lost Communication`), so S2D keeps scheduling I/O to it. Repair and regeneration jobs that must touch that drive's slabs time out and requeue, and you observe the restart loop, regardless of the resiliency scheme. + +**How the fix applies per scheme.** + +- **Any resilient scheme (mirror, parity, nested, mirror-accelerated parity):** retiring the failing drive is the correct action. S2D reconstructs the affected slabs from the surviving copies (mirror) or from parity (parity) onto free reserve capacity, and the volume returns to full redundancy. The amount of reserve and the reconstruction cost differ by scheme (a mirror rebuilds a copy; parity recomputes across the remaining columns), but the retire -> rebuild flow is the same. +- **Simple / no-resiliency volume:** there is no second copy or parity to rebuild from, so retiring the drive cannot restore the lost regions. The only path is to remove the failed drive, recreate the volume, and restore its data from backup. Check `ResiliencySettingName` first so you do not wait on a rebuild that cannot happen. + +If the failing drive holds the last currently-available copy (or a required parity element) of a region, that region shows as `No Redundancy` (or `Incomplete` on a two-copy volume), and the drive appears in `Get-PhysicalDisk -NoRedundancy` for the affected volume. Under heavy write I/O, the same non-completing drive can leave a storage completion routine holding the processor dispatch level too long, which trips the DPC watchdog and bugchecks the node. That is why a node crash and a stuck repair frequently share one cause. @@ -117,7 +136,7 @@ Why capacity is a red herring here: ## Glossary -- **Three-way mirror**: the volume keeps three copies of every block on three different servers, so it survives two failures. +- **Resiliency type**: how a volume keeps redundant data (mirror, parity, nested, or mirror-accelerated parity). The Azure Local default is three-way mirror on clusters of three or more nodes (three copies, survives two failures) and two-way mirror on two-node clusters; see What and Why for the full list. - **Fault domain**: a failure boundary, here a server (node). Copies of a block are placed in different fault domains. - **Slab**: the unit of capacity S2D allocates from the pool (a chunk of a virtual disk). - **Reserve capacity**: pool space left free on purpose so S2D can rebuild a failed or retired drive in place. S2D uses this instead of a dedicated hot spare. From 5803347fd799bea100255d0c412dba56250b57b5 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Fri, 10 Jul 2026 21:29:11 -0400 Subject: [PATCH 4/9] Storage TSG: link the exact Add-Physical-Disks path in Step 6 Reviewer consistency nit: Step 6 named the sibling by description while Related Issues uses exact paths. Point Step 6 at TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md the same way. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 4e9c61e6-56c5-46a0-9ddb-8698443047bd --- .../Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index 9ef88eaa..9edc9ea8 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -293,7 +293,7 @@ Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIndication Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ) -StoragePoolFriendlyName ``` -The physical drive replacement is a hardware task: engage the OEM or your hardware vendor and follow their drive-replacement procedure for the chassis (the location indicator above lights the drive bay on Dell, HPE, and Lenovo servers). Expected end state: a replacement drive of a supported model is claimed automatically, S2D rebalances onto it, and every drive and volume returns to Healthy. No manual repair trigger is normally required. To add the replacement manually, see the Add-Physical-Disks TSG. +The physical drive replacement is a hardware task: engage the OEM or your hardware vendor and follow their drive-replacement procedure for the chassis (the location indicator above lights the drive bay on Dell, HPE, and Lenovo servers). Expected end state: a replacement drive of a supported model is claimed automatically, S2D rebalances onto it, and every drive and volume returns to Healthy. No manual repair trigger is normally required. To add the replacement manually, see the guide for adding physical disks to the S2D pool: `TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md`. ## When to escalate From 190929a8030383a87edd1e55b3bc20429a241bbf Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 13 Jul 2026 15:01:24 -0400 Subject: [PATCH 5/9] Storage repair-jobs TSG: complete multi-surface coverage + prose polish Sanding pass from the tsg-forge grader: - Multi-surface discoverability 6/8 -> 8/8: add Windows Admin Center in the Azure portal (the same cluster Drives view for an Arc-connected cluster) and a Storage diagnostic logs (on disk) note (Get-SDDCDiagnosticInfo / Support Diagnostics Tool collect the storage subsystem's own on-disk logs). The section now shows the failing drive across all eight admin surfaces. - Prose: replace em-dashes in the resiliency-type bullets with colons (house style). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d7f2348-e3be-4fe4-8598-6bf5dcb4050f --- ...ot-Storage-RepairJobsRestartOnFailingDisk.md | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index 9edc9ea8..9bf6f1b0 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -102,7 +102,12 @@ For an Arc-connected Azure Local cluster, the physical-disk and virtual-disk hea ### Failover Cluster Manager and Windows Admin Center - **Failover Cluster Manager** shows the cluster and CSV state. A degraded volume appears under Storage with the CSV in an online-degraded or warning state. FCM does not surface per physical-disk latency well. -- **Windows Admin Center** (the recommended GUI for S2D) shows per-drive health under the cluster Drives view, including the Warning status and latency, and offers Retire and Locate actions directly. +- **Windows Admin Center (standalone host)** (the recommended GUI for S2D) shows per-drive health under the cluster Drives view, including the Warning status and latency, and offers Retire and Locate actions directly. +- **Windows Admin Center in the Azure portal**: for an Arc-connected cluster, opening Windows Admin Center in the Azure portal shows the same cluster Drives view, so the failing drive's Warning status and latency are visible there too without a separate WAC gateway. + +### Storage diagnostic logs (on disk) + +The storage subsystem also writes its own logs that a support case relies on. `Get-SDDCDiagnosticInfo` (the SDDC diagnostic tool) and the Support Diagnostics Tool collect the cluster's storage on-disk logs (the health service log, the Storage Spaces driver operational log, and the per-node cluster logs) into a single diagnostic log archive for the failing drive's node. Use these when you need the raw component logs for escalation; see the Support Diagnostics Tool guide referenced under Related Issues. ## What and Why @@ -110,11 +115,11 @@ For an Arc-connected Azure Local cluster, the physical-disk and virtual-disk hea S2D stores each volume with a resiliency type that keeps redundant data across fault domains (usually nodes). **On Azure Local the default is three-way mirror (three copies) for clusters of three or more nodes, and two-way mirror (two copies) for two-node clusters.** That default is not the only option, and the volume you are repairing may use a different scheme: -- **Two-way / three-way mirror** — two or three full copies across nodes (the defaults above); the volume shows `ResiliencySettingName = Mirror` with `NumberOfDataCopies` 2 or 3. -- **Nested resiliency** (two-node clusters) — nested two-way mirror, or nested mirror-accelerated parity, which survives two concurrent hardware failures on a two-node cluster. -- **Parity / dual parity** (erasure coding, four or more nodes) — space-efficient; reconstructs data from parity rather than a full copy (`ResiliencySettingName = Parity`). -- **Mirror-accelerated parity** — one volume that combines a mirror tier and a parity tier. -- **Simple (no resiliency)** — not recommended and never an Azure Local default; a single disk loss means the data on that volume's affected regions is lost. +- **Two-way / three-way mirror**: two or three full copies across nodes (the defaults above); the volume shows `ResiliencySettingName = Mirror` with `NumberOfDataCopies` 2 or 3. +- **Nested resiliency** (two-node clusters): nested two-way mirror, or nested mirror-accelerated parity, which survives two concurrent hardware failures on a two-node cluster. +- **Parity / dual parity** (erasure coding, four or more nodes): space-efficient; reconstructs data from parity rather than a full copy (`ResiliencySettingName = Parity`). +- **Mirror-accelerated parity**: one volume that combines a mirror tier and a parity tier. +- **Simple (no resiliency)**: not recommended and never an Azure Local default; a single disk loss means the data on that volume's affected regions is lost. Confirm the actual scheme before you act: `Get-VirtualDisk -FriendlyName | Select-Object FriendlyName, ResiliencySettingName, NumberOfDataCopies, PhysicalDiskRedundancy`. From 052621af96f3b82a5f5e7c096f9e629754fdbca8 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 19:05:32 -0400 Subject: [PATCH 6/9] RepairJobs TSG: fix Get-VirtualDisk property and reserve-example arithmetic 1. The volume-health snippet selected OperationalDetails, which is not an MSFT_VirtualDisk property (per the class MOF it has OtherOperationalStatus- Description and DetachedReason, not OperationalDetails), so that column always rendered blank. Use OtherOperationalStatusDescription. 2. The Step 3 reserve worked example subtracted the retiring drive's full 2.4 TB size from pool-free, but only ~0.4 TB of the free space sits on that drive (2.4 TB total, ~2.0 TB used). Reworded so the numbers are consistent: surviving free before the rebuild is ~10.4 TB, and ~8.4 TB is the free that remains after relocating the ~2.0 TB. The conclusion (fits, restores redundancy, but below the 9.6 TB full-reserve target) is unchanged. --- .../Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index 9bf6f1b0..d5c8446f 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -183,7 +183,7 @@ Run through this list before Step 4. Most stuck or unsafe retires trace back to Get-StorageJob | Select-Object Name, JobState, PercentComplete, BytesProcessed, BytesTotal # The affected volume reports No Redundancy / Unhealthy. -Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus, OperationalDetails +Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus, OtherOperationalStatusDescription ``` #### Step 2: Identify the failing physical disk @@ -243,7 +243,7 @@ Get-PhysicalDisk | Where-Object { $_.Usage -eq 'Auto-Select' -and $_.SerialNumbe The check: **pool free space must exceed the failing drive's used capacity**, and you should still have reserve left afterward (about one capacity drive per node). Because three-way mirror places copies across three nodes, the free space also has to be distributed so the surviving nodes can each hold their share. If one node's drives are all near 100 percent, the rebuild for slabs that need that node stalls even when the pool total looks fine. -Worked example from a real case: the failing 2.4 TB drive held about 2.0 TB of data, and the pool reported about 10.8 TB free (roughly 86 percent full). That 10.8 TB still included the ~2.4 TB sitting on the drive being retired, so the surviving disks held only about 8.4 TB free. The ~2.0 TB that had to be relocated fit easily into that 8.4 TB, so there was enough evacuation headroom and the rebuild completed and restored redundancy. Note, however, that 8.4 TB is below the ~9.6 TB full-reserve target (one 2.4 TB drive per node across four nodes): the pool ran with less than the recommended repair reserve until the failed drive was physically replaced, so add the replacement promptly to restore full reserve. +Worked example from a real case: the failing 2.4 TB drive held about 2.0 TB of data, and the pool reported about 10.8 TB free (roughly 86 percent full). About 0.4 TB of that free space sat on the drive being retired (a 2.4 TB drive with ~2.0 TB used), so the surviving disks had roughly 10.4 TB free to absorb the ~2.0 TB being relocated. That fit easily, and after the rebuild the surviving disks held about 8.4 TB free (the ~10.4 TB less the ~2.0 TB rebuilt), so redundancy was restored with headroom to spare. Note, however, that 8.4 TB is below the ~9.6 TB full-reserve target (one 2.4 TB drive per node across four nodes): the pool ran with less than the recommended repair reserve until the failed drive was physically replaced, so add the replacement promptly to restore full reserve. If the pool has essentially no free reserve (for example above roughly 95 percent with no per-node headroom), retiring the drive can leave S2D with nowhere to rebuild and the volume stays degraded. In that case add capacity or reduce data first, then retire. Remember that on thin volumes, deleting data does not free pool space quickly, so plan the reserve ahead of time rather than deleting at the last minute. From 47a578d146e5f1299bf469ce83f56e653549aa94 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 08:01:48 -0400 Subject: [PATCH 7/9] Fix Enable-PhysicalDiskIdentification cmdlet name + broaden Step 2 disk filter to Lost Communication - Step 6 LED-locate used Enable-PhysicalDiskIndication, which does not exist (Learn 404); the real Storage-module cmdlet is Enable-PhysicalDiskIdentification (Learn 200, WindowsServer2025-ps). As written the command throws 'not recognized as a cmdlet' on a node. - Step 2 identify one-liner matched only 'Abnormal Latency', missing the 'Lost Communication' operational state this guide itself names; broadened the -match to cover both. --- .../Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index d5c8446f..fdc8e238 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -193,8 +193,8 @@ Get-VirtualDisk | Select-Object FriendlyName, HealthStatus, OperationalStatus, O # plus VirtualDisks.NoRedundancy / LastCopy on the volume. Get-HealthFault | Select-Object FaultType, PerceivedSeverity, Reason, FaultingObjectDescription -# The failing drive: Healthy but "Abnormal Latency". -Get-PhysicalDisk | Where-Object { ($_.OperationalStatus -join ',') -match 'Abnormal Latency' } | +# The failing drive: Healthy but "Abnormal Latency" or "Lost Communication". +Get-PhysicalDisk | Where-Object { ($_.OperationalStatus -join ',') -match 'Abnormal Latency|Lost Communication' } | Select-Object FriendlyName, SerialNumber, UniqueId, PhysicalLocation, HealthStatus, OperationalStatus # Corroborate with the reliability counters (read/write error totals, latency). @@ -291,7 +291,7 @@ Only after the rebuild jobs reach zero and the volumes are Healthy: ```powershell # Turn on the location indicator (if supported) to find the drive in the chassis. -Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIndication +Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIdentification # Remove the retired drive from the pool, then physically swap it. Remove-PhysicalDisk has # no -UniqueId parameter, so resolve the disk object and pass it via -PhysicalDisks. From 5a7de50c25299ec759088090dd81333c4c9ac993 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 08:41:58 -0400 Subject: [PATCH 8/9] Storage repair-jobs TSG: safety gates at Step 4, glossary, and evidence caveats Applied from a tsg-forge grading run (live L4 loop re-proven on solution train 12.2610, plus a multi-model persona panel across two frontier model families). Safety (both safety lenses, both models): - Step 4 now opens with a four-point WARNING gate repeating the preconditions at the point of the destructive command, and the last-copy caveat now precedes the command rather than following it. - Added a mandatory read-back of the UniqueId before the retire, with an explicit escalation path when it does not match. - Documented that a retire is reversible. Correctness (found by the live run and by review): - Set-PhysicalDisk -Usage takes the enum AutoSelect, not the hyphenated Auto-Select that Get-PhysicalDisk displays. The hyphenated form fails outright. - The HighLatency fault multiple said "thousands to millions"; the guide's own worked example is about 585 times. - Reliability counters can return nulls or zeros behind some HBAs and NVMe, so empty counters are not evidence of a healthy drive. - Lost-communication and flapping events can be a cable, expander, or backplane fault, so the shared storage path is now ruled out before condemning a drive. - The 0x133 attribution now requires dump-stack and bugcheck-parameter confirmation instead of being asserted. Accessibility: - Added an acronym table (S2D, CSV, CAU, SDDC, SMART, HBA, DPC, RDMA) and an explicit note that angle-bracket tokens are placeholders. --- ...-Storage-RepairJobsRestartOnFailingDisk.md | 109 +++++++++++++++++- 1 file changed, 104 insertions(+), 5 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index fdc8e238..6a8150e7 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -41,7 +41,7 @@ An important trap: this scenario is frequently misread as a capacity problem, be - `Get-VirtualDisk` shows a volume at `OperationalStatus = {No Redundancy, InService}` (or `{Incomplete, InService}` on a two-copy volume) and `HealthStatus = Unhealthy`. - `Get-PhysicalDisk` shows one drive with `OperationalStatus = "OK, Abnormal Latency"` while `HealthStatus` is still `Healthy`. - `Get-HealthFault` reports a combination of: - - `Microsoft.Health.FaultType.PhysicalDisk.HighLatency.Outlier.AverageIO` (average latency thousands to millions of times the peer drives). + - `Microsoft.Health.FaultType.PhysicalDisk.HighLatency.Outlier.AverageIO` (average latency orders of magnitude above the peer drives; in the worked case below, roughly 585 times). - `Microsoft.Health.FaultType.PhysicalDisk.HighErrorCount.Outlier.AverageIO` (I/O error count far above peer drives). - `Microsoft.Health.FaultType.VirtualDisks.NoRedundancy` (Critical) and `Microsoft.Health.FaultType.VirtualDisks.LastCopy` (Warning). - Often also `Microsoft.Health.FaultType.StoragePool.PoolCapacityThresholdExceeded` and `Microsoft.Health.FaultType.Server.Storage.Degraded`. @@ -73,10 +73,38 @@ Get-PhysicalDisk -SerialNumber | Get-StorageReliabilityCounter | Example from a real case: `ReadErrorsTotal = 1,252,510`, `ReadLatencyMax = 11,708` ms (healthy peers are under ~20 ms), `PowerOnHours = 32,273` (about 3.7 years). In that case `ReadErrorsUncorrected = 0`, which is why the rebuild recovered all data. A drive with non-zero *uncorrected* errors that also holds the last copy is the data-loss case to worry about. +> [!NOTE] +> **Reliability counters are not always populated.** `Get-StorageReliabilityCounter` depends +> on the drive and the controller passing the counters through. Behind some HBAs and RAID +> controllers, and on some NVMe devices, it returns nulls or all zeros even for a drive that +> is genuinely dying. **Empty counters are not evidence that the drive is healthy.** When they +> come back empty, fall back to the other signals on this page, which do not depend on the +> counters: the `OperationalStatus` of `"OK, Abnormal Latency"` or `Lost Communication`, the +> `Get-HealthFault` latency and error outliers on physical hardware, the +> `Microsoft-Windows-StorageSpaces-Driver` events 203, 205, 207 and 209, and the stuck-repair +> job pattern itself. Also collect the vendor's own drive diagnostics. + ### Windows event log (on the node hosting the drive) - `disk` **153** "The IO operation ... was retried" (leading indicator, often hundreds per hour). - `Microsoft-Windows-StorageSpaces-Driver` **203** (failed IO / STATUS_DEVICE_NOT_CONNECTED), **205** (lost communication), **207** (drive "arrived" repeatedly, meaning it is flapping), **209** (failed Read IO / I/O device error). + +> [!IMPORTANT] +> **Rule out the shared storage path before condemning a drive.** Lost-communication and +> flapping events (203, 205, 207) say the connection to the device failed. They do not by +> themselves say the *drive* is at fault: a failing cable, SAS expander, backplane, or drive +> slot produces the same events. Two quick discriminators: +> +> - **Is it one drive or several?** If multiple drives behind the same controller, backplane, +> or enclosure are reporting together, suspect the shared path, not the drives. Group the +> suspects by `PhysicalLocation` and by node. +> - **Do the drive's own counters agree?** High `ReadErrorsUncorrected` and multi-second +> `ReadLatencyMax` on **only** this drive point at the drive. Clean counters with repeated +> 203/205/207 point at the path. +> +> If the evidence points at the path, engage the hardware vendor for the cable, backplane, or +> controller rather than replacing a healthy drive. Retiring is still safe in the meantime, +> because it only evacuates data, but a drive swap will not fix a path fault. - `Microsoft-Windows-StorageSpaces-Driver` **312** (write failed to all copies), **302** (space metadata update failed), **304** and **305** (virtual disk degraded, then healthy). ```powershell @@ -132,7 +160,7 @@ Confirm the actual scheme before you act: `Get-VirtualDisk -FriendlyName | If the failing drive holds the last currently-available copy (or a required parity element) of a region, that region shows as `No Redundancy` (or `Incomplete` on a two-copy volume), and the drive appears in `Get-PhysicalDisk -NoRedundancy` for the affected volume. -Under heavy write I/O, the same non-completing drive can leave a storage completion routine holding the processor dispatch level too long, which trips the DPC watchdog and bugchecks the node. That is why a node crash and a stuck repair frequently share one cause. +Under heavy write I/O, the same non-completing drive can leave a storage completion routine holding the processor dispatch level too long, which can trip the DPC watchdog and bugcheck the node with `DPC_WATCHDOG_VIOLATION` (`0x133`). That is a plausible shared cause when a node crash and a stuck repair happen together, but do not assume it. **Confirm the attribution before you report it**: open the dump and check that the stack actually implicates the storage completion path (`storport.sys`, `stornvme.sys`, the miniport, or `spaceport.sys`), and note the first bugcheck parameter, where `0` is a single DPC that ran too long and `1` is the system spending too long at DISPATCH_LEVEL overall. A `0x133` with an unrelated stack is a different problem that happens to coincide with this one. Why capacity is a red herring here: @@ -141,6 +169,21 @@ Why capacity is a red herring here: ## Glossary +**Reading the commands on this page.** Anything in angle brackets, such as ``, ``, ``, or ``, is a **placeholder you must replace** with the value you recorded in an earlier step. Do not paste the angle brackets. + +**Acronyms used on this page.** + +| Acronym | Meaning | +| --- | --- | +| **S2D** | Storage Spaces Direct, the software-defined storage layer that pools the drives across nodes. | +| **CSV** | Cluster Shared Volume, a volume every node can access at the same time. It appears as a folder under `C:\ClusterStorage`. | +| **CAU** | Cluster-Aware Updating, the automation that patches cluster nodes one at a time. Check for an active run with `Get-CauRun` before making storage changes. | +| **SDDC** | Software-Defined Data Center, the name of the cluster Health Service that reports health faults. | +| **SMART** | Self-Monitoring, Analysis and Reporting Technology, the drive's own health reporting. | +| **HBA** | Host Bus Adapter, the controller card that connects the server to its drives. | +| **DPC** | Deferred Procedure Call, a Windows kernel mechanism. A driver that holds a processor at DPC level too long can trigger a `DPC_WATCHDOG_VIOLATION` (`0x133`) bugcheck. | +| **RDMA** | Remote Direct Memory Access, the low-latency network transport S2D uses between nodes. | + - **Resiliency type**: how a volume keeps redundant data (mirror, parity, nested, or mirror-accelerated parity). The Azure Local default is three-way mirror on clusters of three or more nodes (three copies, survives two failures) and two-way mirror on two-node clusters; see What and Why for the full list. - **Fault domain**: a failure boundary, here a server (node). Copies of a block are placed in different fault domains. - **Slab**: the unit of capacity S2D allocates from the pool (a chunk of a virtual disk). @@ -249,15 +292,71 @@ If the pool has essentially no free reserve (for example above roughly 95 percen #### Step 4: Retire the failing drive [MEDIUM RISK] +> [!WARNING] +> **Do not run the retire command until all four of these are true.** They are +> repeated here, rather than left in +> [Before you retire](#before-you-retire-pre-checks-and-gotchas) alone, because this +> is the first step that changes storage state and the usual cause of harm is running +> it against the wrong drive or at the wrong time. +> +> 1. **Exactly one** suspect drive, in **one** fault domain (one node). Re-run the +> suspect sweep in [Before you retire](#before-you-retire-pre-checks-and-gotchas). +> If two or more drives are suspect, stop and engage Microsoft Support. +> 2. No solution update, Cluster-Aware Updating (CAU) run, or node maintenance is in +> progress. +> 3. The pool has enough free reserve on the **surviving** drives (Step 3). +> 4. You have read the last-copy caveat immediately below. + +Caveat when the drive is a last-copy holder (Step 2 returned it under +`-NoRedundancy`): retiring forces S2D to read those regions off the dying drive to +rebuild them. Any region the drive can no longer read cannot be rebuilt and that data +is lost. Retiring is still the correct action, because it triggers the evacuation +while the drive is at least partly alive; leaving the drive in service guarantees the +volume stays at No Redundancy. Retire sooner rather than later to maximize what can be +salvaged. If `-NoRedundancy` returned this drive, confirm the backup position with the +workload owner before you continue, and record that you did. + +**First, prove the `UniqueId` you are about to use really is the failing drive.** +Retiring a healthy drive on an already-degraded pool is how this step causes data +loss, and a copied or mistyped `UniqueId` is how that happens. + +```powershell +$DiskUniqueId = '' # the value you recorded in Step 2 + +# Read the drive back and confirm it is the one you diagnosed, BEFORE changing anything. +Get-PhysicalDisk -UniqueId $DiskUniqueId | + Select-Object FriendlyName, SerialNumber, HealthStatus, + @{n='Op';e={$_.OperationalStatus -join ','}}, Usage, PhysicalLocation +``` + +Confirm the output shows the **same serial number and the same node** you identified in +Step 2, and that `Op` reads `Abnormal Latency` or `Lost Communication`. If any of that +does not match, **stop and do not retire**. Re-run Step 2 to re-identify the drive. If it +still does not match after a careful re-run, treat it as an unresolved identification +problem rather than guessing: leave the pool as it is, collect the data in +[Data to Collect Before Opening a Support Case](#data-to-collect-before-opening-a-support-case), +and open a case with Microsoft Support. A degraded pool is recoverable; a healthy drive +retired by mistake alongside an already-failing one may not be. + ```powershell # Marks the drive do-not-use and starts the evacuation/rebuild onto healthy drives. -Set-PhysicalDisk -UniqueId -Usage Retired +Set-PhysicalDisk -UniqueId $DiskUniqueId -Usage Retired # Verify. -Get-PhysicalDisk -UniqueId | Select-Object FriendlyName, Usage, OperationalStatus +Get-PhysicalDisk -UniqueId $DiskUniqueId | Select-Object FriendlyName, Usage, OperationalStatus ``` -Caveat when the drive is a last-copy holder (Step 2 returned it under `-NoRedundancy`): retiring forces S2D to read those regions off the dying drive to rebuild them. Any region the drive can no longer read cannot be rebuilt and that data is lost. Retiring is still the correct action, because it triggers the evacuation while the drive is at least partly alive; leaving the drive in service guarantees the volume stays at No Redundancy. Retire sooner rather than later to maximize what can be salvaged. +> [!NOTE] +> **A retire is reversible.** `Set-PhysicalDisk -UniqueId -Usage AutoSelect` +> returns the drive to normal use and the pool rebalances. If you realize you retired +> the wrong drive, set it straight back. Do **not** escalate to `Remove-PhysicalDisk`, +> which is not reversible. +> +> Note the spelling: `Get-PhysicalDisk` **displays** the usage as `Auto-Select` with a +> hyphen, but `Set-PhysicalDisk -Usage` takes the enum value `AutoSelect` with no +> hyphen. Passing the hyphenated form fails with "Unable to match the identifier name +> Auto-Select to a valid enumerator name". The valid values are `AutoSelect`, +> `ManualSelect`, `HotSpare`, `Retired`, and `Journal`. #### Step 5: Monitor the rebuild From 0f0ff30b7178862f8c2e095c030c4f3de1e8c871 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:21:12 -0400 Subject: [PATCH 9/9] Storage repair-jobs TSG: third-model review round (correctness and Step 6 gate) A third frontier model family (xAI) was added to the review panel after the previous third slot was found to be returning empty. It caught defects the first two models approved. Correctness: - "max latency in seconds, not milliseconds" contradicted the worked example in the same paragraph, which cites ReadLatencyMax = 11,708 ms. - The guide framed the failing drive as "still Healthy". That holds for the high-latency variant, but the lost-communication variant reports HealthStatus Warning, which a live lab run on this build confirmed. Filtering on HealthStatus alone misses it, so the symptom list now documents both variants and directs filtering on OperationalStatus. - "Get-StorageJob returns nothing" is not a valid rebuild-complete predicate, because it also returns Optimize, Rebalance, and Trim jobs and can block Step 6 indefinitely. Replaced with an explicit Repair/Regeneration predicate. Safety: - Step 4's read-back told the reader to confirm the node but printed no node. It now reads Get-StorageNode -PhysicallyConnected, and also surfaces MediaType. - Step 6 runs Remove-PhysicalDisk, which is irreversible and often executed in a new session hours later, but had no identity re-check. It now re-reads the drive, hard-stops unless it is still Retired and evacuated, separates lighting the bay from removing the disk, and requires matching the chassis serial sticker before anything is unplugged. - The reversibility note was unbounded. It is now scoped to undoing a wrong retire, with an explicit warning against un-retiring a genuinely failing drive mid-rebuild. --- ...-Storage-RepairJobsRestartOnFailingDisk.md | 77 +++++++++++++++---- 1 file changed, 64 insertions(+), 13 deletions(-) diff --git a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md index 6a8150e7..b79f7954 100644 --- a/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md +++ b/TSG/Storage/Troubleshoot-Storage-RepairJobsRestartOnFailingDisk.md @@ -39,7 +39,8 @@ An important trap: this scenario is frequently misread as a capacity problem, be - `Get-StorageJob` shows one or more `-Repair` and `-Regeneration` jobs that reset: `PercentComplete` returns to `0`, `BytesTotal` shrinks or changes between samples, and the job's elapsed time resets. Job count never drops to zero. - `Get-VirtualDisk` shows a volume at `OperationalStatus = {No Redundancy, InService}` (or `{Incomplete, InService}` on a two-copy volume) and `HealthStatus = Unhealthy`. -- `Get-PhysicalDisk` shows one drive with `OperationalStatus = "OK, Abnormal Latency"` while `HealthStatus` is still `Healthy`. +- `Get-PhysicalDisk` shows one drive with `OperationalStatus = "OK, Abnormal Latency"` while `HealthStatus` is still `Healthy`. This is the high-latency variant, and the still-`Healthy` status is exactly why it is easy to miss. +- In the related **lost-communication** variant the drive reports `OperationalStatus = "Lost Communication"` and `HealthStatus` becomes **`Warning`**, not `Healthy`. Both variants produce the same stuck-repair symptom, so filter on `OperationalStatus`, never on `HealthStatus -eq 'Healthy'` alone, or you will miss this one. - `Get-HealthFault` reports a combination of: - `Microsoft.Health.FaultType.PhysicalDisk.HighLatency.Outlier.AverageIO` (average latency orders of magnitude above the peer drives; in the worked case below, roughly 585 times). - `Microsoft.Health.FaultType.PhysicalDisk.HighErrorCount.Outlier.AverageIO` (I/O error count far above peer drives). @@ -66,7 +67,8 @@ Get-PhysicalDisk | Select-Object FriendlyName, SerialNumber, Usage, HealthStatus Get-HealthFault | Select-Object FaultType, PerceivedSeverity, Reason, FaultingObjectDescription # 3. Reliability counters: the hard evidence. On a real failing drive you see read/write -# error totals in the thousands to millions and max latency in seconds, not milliseconds. +# error totals in the thousands to millions, and a max latency of thousands of ms +# (that is, whole seconds of wall-clock) where healthy peers are in the tens of ms. Get-PhysicalDisk -SerialNumber | Get-StorageReliabilityCounter | Select-Object ReadErrorsTotal, ReadErrorsUncorrected, WriteErrorsTotal, ReadLatencyMax, WriteLatencyMax, PowerOnHours ``` @@ -324,9 +326,13 @@ loss, and a copied or mistyped `UniqueId` is how that happens. $DiskUniqueId = '' # the value you recorded in Step 2 # Read the drive back and confirm it is the one you diagnosed, BEFORE changing anything. -Get-PhysicalDisk -UniqueId $DiskUniqueId | - Select-Object FriendlyName, SerialNumber, HealthStatus, - @{n='Op';e={$_.OperationalStatus -join ','}}, Usage, PhysicalLocation +$disk = Get-PhysicalDisk -UniqueId $DiskUniqueId +$disk | Select-Object FriendlyName, SerialNumber, MediaType, Usage, HealthStatus, + @{n='Op';e={$_.OperationalStatus -join ','}}, PhysicalLocation + +# The node that physically hosts the drive. The prose below asks you to confirm the node, +# so read it here rather than assuming the node you happen to be signed in to. +$disk | Get-StorageNode -PhysicallyConnected | Select-Object Name ``` Confirm the output shows the **same serial number and the same node** you identified in @@ -347,10 +353,17 @@ Get-PhysicalDisk -UniqueId $DiskUniqueId | Select-Object FriendlyName, Usage, Op ``` > [!NOTE] -> **A retire is reversible.** `Set-PhysicalDisk -UniqueId -Usage AutoSelect` -> returns the drive to normal use and the pool rebalances. If you realize you retired -> the wrong drive, set it straight back. Do **not** escalate to `Remove-PhysicalDisk`, -> which is not reversible. +> **A retire is reversible, with one important limit.** +> `Set-PhysicalDisk -UniqueId -Usage AutoSelect` returns the drive to normal +> use and the pool rebalances. Use it **only** when you have just retired the WRONG drive and +> want to undo that mistake. +> +> Do **not** use it to "cancel" a rebuild that is under way on the genuinely failing drive. +> Putting a dying drive back into service mid-evacuation restarts the original stuck-repair +> condition and can leave volumes degraded for longer. If a rebuild is slow, let it finish or +> engage support; do not un-retire the bad drive. +> +> Do **not** escalate to `Remove-PhysicalDisk`, which is not reversible. > > Note the spelling: `Get-PhysicalDisk` **displays** the usage as `Auto-Select` with a > hyphen, but `Set-PhysicalDisk -Usage` takes the enum value `AutoSelect` with no @@ -382,19 +395,57 @@ Repair-VirtualDisk -FriendlyName During evacuation the pool used percentage rises briefly as replacement copies are written, then settles as the retired drive's slabs are released. This is expected. -The rebuild is complete when all of the following are true: `Get-StorageJob` returns nothing, every volume is `HealthStatus = Healthy` / `OperationalStatus = OK`, the `VirtualDisks.NoRedundancy` and `LastCopy` faults have cleared, and the retired drive's used capacity (`AllocatedSize`) has dropped to near zero because its data now lives elsewhere. Only then proceed to Step 6. +The rebuild is complete when all of the following are true: **no `Repair` or `Regeneration` job remains**, every volume is `HealthStatus = Healthy` / `OperationalStatus = OK`, the `VirtualDisks.NoRedundancy` and `LastCopy` faults have cleared, and the retired drive's used capacity (`AllocatedSize`) has dropped to near zero because its data now lives elsewhere. Only then proceed to Step 6. + +> [!NOTE] +> Do not use a bare "`Get-StorageJob` returns nothing" as the completion test. `Get-StorageJob` +> also returns pool `Optimize`, `Rebalance`, and `Trim` jobs, which are unrelated to this repair +> and can run for a long time, so waiting for a completely empty list can block Step 6 +> indefinitely. Test for the repair jobs specifically: + +```powershell +# The rebuild-complete predicate: no Repair/Regeneration job left for any volume. +$rebuild = @(Get-StorageJob | Where-Object { $_.Name -match 'Repair|Regeneration' -and $_.JobState -notin @('Completed','Failed') }) +if ($rebuild.Count -eq 0) { "Rebuild complete." } else { $rebuild | Select-Object Name, JobState, PercentComplete } +``` #### Step 6: Physically replace and remove the drive [MEDIUM RISK] -Only after the rebuild jobs reach zero and the volumes are Healthy: +Only after the rebuild jobs reach zero and the volumes are Healthy. + +> [!WARNING] +> `Remove-PhysicalDisk` is **not reversible**, so it needs the same identity check as Step 4, +> and it is often run in a new session hours later. Re-read the drive before you remove it, +> and confirm the state below, rather than trusting an identifier pasted from earlier notes. + +```powershell +$DiskUniqueId = '' # re-paste from Step 2 and re-verify below + +# Re-read and confirm: this must still be the RETIRED drive, evacuated (AllocatedSize near 0). +$disk = Get-PhysicalDisk -UniqueId $DiskUniqueId +$disk | Select-Object FriendlyName, SerialNumber, Usage, HealthStatus, + @{n='Op';e={$_.OperationalStatus -join ','}}, + @{n='AllocatedGB';e={[math]::Round($_.AllocatedSize/1GB,2)}}, PhysicalLocation +$disk | Get-StorageNode -PhysicallyConnected | Select-Object Name + +if ($disk.Usage -ne 'Retired') { throw "Refusing: drive Usage is '$($disk.Usage)', expected 'Retired'. Do not remove it." } +if ($disk.AllocatedSize -gt 1GB) { throw "Refusing: drive still holds $([math]::Round($disk.AllocatedSize/1GB,2)) GB. Let the rebuild finish first." } +``` + +Confirm the `SerialNumber` and node above match the drive you diagnosed, then light the bay: ```powershell # Turn on the location indicator (if supported) to find the drive in the chassis. -Get-PhysicalDisk -UniqueId | Enable-PhysicalDiskIdentification +$disk | Enable-PhysicalDiskIdentification +``` +**Now go to the chassis and confirm the lit bay's serial sticker matches `SerialNumber` above before anything is unplugged.** Only then remove it from the pool: + +```powershell # Remove the retired drive from the pool, then physically swap it. Remove-PhysicalDisk has # no -UniqueId parameter, so resolve the disk object and pass it via -PhysicalDisks. -Remove-PhysicalDisk -PhysicalDisks (Get-PhysicalDisk -UniqueId ) -StoragePoolFriendlyName +# Discover with: Get-StoragePool | Where-Object IsPrimordial -eq $false | Select-Object FriendlyName +Remove-PhysicalDisk -PhysicalDisks $disk -StoragePoolFriendlyName ``` The physical drive replacement is a hardware task: engage the OEM or your hardware vendor and follow their drive-replacement procedure for the chassis (the location indicator above lights the drive bay on Dell, HPE, and Lenovo servers). Expected end state: a replacement drive of a supported model is claimed automatically, S2D rebalances onto it, and every drive and volume returns to Healthy. No manual repair trigger is normally required. To add the replacement manually, see the guide for adding physical disks to the S2D pool: `TSG/Storage/HowTo-Storage-AddPhysicalDisksToS2DPool.md`.