From 7f82e2de0e1ae01ae0901c323740daa9870b0289 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 13 Jul 2026 10:37:16 -0400 Subject: [PATCH 1/6] Add Environment Validator TSG: Azure_Kubernetes_Service_Cluster_connect New public remediation TSG for the Connectivity validator's 'Cluster connect' target (ADO 38357506). The check probes outbound HTTPS reachability from each node to the Azure Arc cluster-connect Service Bus relay azgnrelay--l1.servicebus.windows.net:443 (used to reach AKS enabled by Azure Arc without inbound ports). Documents the four real failure sub-modes harvested from fleet telemetry (EnvironmentValidatorResult), branched by the 'Layer 3 (tnc)' discriminator: DNS resolution (tnc False), firewall/outbound-443 blocked (tnc False), proxy/app-layer (tnc True), and TLS inspection (tnc True). Includes the standalone Invoke-AzStackHciConnectivityValidation repro, the .AzStackHci component log/FailedUrls.txt pointer, the multi-surface 'where this does NOT appear' note, per-sub-mode remediation with risk labels, and a verify step. Lint grade A; admin discoverability surfaces 8/8 (gate met). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d7f2348-e3be-4fe4-8598-6bf5dcb4050f --- ...zure_Kubernetes_Service_Cluster_connect.md | 254 ++++++++++++++++++ 1 file changed, 254 insertions(+) create mode 100644 TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md new file mode 100644 index 00000000..5fd138f3 --- /dev/null +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -0,0 +1,254 @@ +# Azure_Kubernetes_Service_Cluster_connect + + + + + + + + + + + + + + + + + + +
NameAzure_Kubernetes_Service_Cluster_connect
Validator / testInvoke-AzStackHciConnectivityValidation (connectivity target "Cluster connect")
ComponentConnectivity (Environment Validator / Environment Checker)
SeverityWarning in the Azure public cloud (non-blocking, but the Arc cluster-connect feature will not work until it is fixed); Critical in Azure Government (Fairfax). See the detection note.
+ +> **At a glance** +> - **What it is:** an Environment Validator (Environment Checker) connectivity check that confirms each Azure Local node can reach the **Azure Arc "cluster connect" endpoint** over outbound HTTPS. On each node it makes an outbound request to the Azure Relay (Service Bus relay) endpoint `azgnrelay--l1.servicebus.windows.net` on TCP 443 and passes when the endpoint answers. +> - **Why it matters:** cluster connect is the secure reverse tunnel that lets you reach Arc-enabled Kubernetes (AKS enabled by Azure Arc) on this cluster **without opening any inbound port**. If a node cannot reach the relay, `az connectedk8s proxy`, the Azure portal Kubernetes view, and cluster-connect-based `kubectl` access to the workload cluster stop working. +> - **Owner:** a **network / firewall / proxy / DNS** action. The fix is to allow the node's outbound connection to the relay endpoint; it is not an Azure Local configuration change. +> - **Read the Detail:** this check reports the outbound test result in a `Detail` string. The line `Test Analysis - Layer 3 (tnc): True/False` tells you whether the low-level TCP connection worked, which selects the right fix below. + +## Overview + +Azure Arc **cluster connect** provides a secure way to connect to Arc-enabled Kubernetes clusters (including AKS enabled by Azure Arc on Azure Local) from anywhere, without requiring any inbound port on the cluster's firewall. It works by having the node maintain an **outbound** connection to an Azure Relay (Service Bus relay) endpoint, and routing management traffic back down that connection. This check verifies that outbound path. + +- **What the check does:** on each node it issues an outbound HTTPS request to the cluster-connect relay endpoint for the cluster's region, `azgnrelay--l1.servicebus.windows.net` (for example `azgnrelay-eastus-l1.servicebus.windows.net`), on TCP port **443**. Reaching the endpoint is a success **even if the endpoint replies `403`** - the check is proving *reachability*, not authentication, so any HTTP response from the real endpoint (commonly `200` or `403`) passes. +- **Severity:** the check is defined at **Warning** severity in the Azure public cloud, so a failure does **not** block a solution update, but the Arc cluster-connect feature is broken until it is fixed. In Azure Government (Fairfax) the same check is **Critical**. +- **When it runs:** the Connectivity validator runs during **Deployment**, **Update**, **Scale-out (Add Node)**, and **Upgrade** readiness, and can also be run standalone at any time (see step 1). +- **The failure is always the same class of problem:** the node's outbound connection to the relay endpoint did not complete. The `Detail` string tells you *where* it broke (DNS, TCP/firewall, proxy, or TLS inspection); step 2 maps each signature to its fix. + +> **Most common in the field (start here).** The usual cause is a **firewall, proxy, or TLS-inspection appliance** that does not allow the node's outbound connection to `*.servicebus.windows.net` (the Azure Relay endpoints). Read the `Detail` string first (step 1), match the signature in step 2, and apply the matching fix in step 5. + +## Requirements + +- Outbound connectivity from **every** node to the Azure Local required endpoints, including the Azure Relay / Service Bus relay endpoints used by Arc cluster connect (`*.servicebus.windows.net`, specifically `azgnrelay--l1.servicebus.windows.net`) on TCP 443. +- If the cluster uses a **proxy**, the proxy must be configured on the nodes and must allow those endpoints. +- If the network uses **TLS inspection / deep packet inspection**, the Azure Relay endpoints must be **excluded** from interception (the relay uses a long-lived connection that inspection appliances frequently break). +- A cluster node (or a workstation with the `AzStackHci.EnvironmentChecker` module) to run `Invoke-AzStackHciConnectivityValidation` for authoritative confirmation. + +## Troubleshooting Steps + +### 1. Confirm the failure and see where it appears + +**Run the connectivity validator directly (authoritative).** On a node (or a workstation with the Environment Checker module installed), run: + +```powershell +Invoke-AzStackHciConnectivityValidation +``` + +Look for the **Azure Kubernetes Service -> Cluster connect** target in the output. A failing target is shown as **Needs Attention / Critical** with the relay URL and a help link. The run also writes its own log and report (see below), and the failing URL to `FailedUrls.txt`. + +**Read the newest health-check result on a node** and flag this check: + +```powershell +$base = 'C:\ClusterStorage\Infrastructure_1\Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' +if (-not (Test-Path $base)) { + $base = Get-ChildItem 'C:\ClusterStorage' -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'Shares\SU1_Infrastructure_1\Updates\HealthCheck\System' } | + Where-Object { Test-Path $_ } | Select-Object -First 1 +} +$latest = $null +if ($base) { + $latest = Get-ChildItem $base -Filter 'HealthCheckResult.*.json' -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 +} +if (-not $latest) { + Write-Warning "No HealthCheck result on this node; use Invoke-AzStackHciConnectivityValidation above, or read the AzStackHciEnvironmentChecker event log (Event ID 17205)." +} +else { + Get-Content $latest.FullName -Raw | ConvertFrom-Json | + Where-Object { $_.Name -eq 'Azure_Kubernetes_Service_Cluster_connect' -and $_.Status -ne 0 -and $_.Status -ne 'SUCCESS' } | + ForEach-Object { + [pscustomobject]@{ + Status = $_.Status + Source = $_.AdditionalData.Source + Target = $_.TargetResourceID + Detail = $_.AdditionalData.Detail + } + } +} +``` + +The same result is on the Windows event log as **Event ID 17205** in `AzStackHciEnvironmentChecker`, and in the Azure portal on the cluster's **Updates** tab when a pre-update health check fails. When the cluster share is unreadable, read the same result straight from the event log: + +```powershell +Get-WinEvent -LogName AzStackHciEnvironmentChecker -FilterXPath "*[System[(EventID=17205)]]" | + ForEach-Object { try { $_.Message | ConvertFrom-Json } catch { } } | + Where-Object { $_.Name -like '*Azure_Kubernetes_Service_Cluster_connect*' } | + Sort-Object { $_.Timestamp } -Descending | + Select-Object -First 10 Status, @{n='Target';e={$_.TargetResourceID}}, @{n='Detail';e={$_.AdditionalData.Detail}} +``` + +**The Environment Checker also writes its own log and report on the node that ran the check.** By default these are under the running account's `%USERPROFILE%\.AzStackHci\` folder: the text log `AzStackHciEnvironmentChecker.log`, the machine-readable `AzStackHciEnvironmentReport.json`, and, for connectivity failures, `FailedUrls.txt` (the list of endpoints that failed). These files are local to the node that executed the check; search the log for `FAILURE` or the endpoint host to find this failure and its debug detail: + +```powershell +Get-ChildItem C:\Users\*\.AzStackHci\AzStackHciEnvironmentChecker.log -ErrorAction SilentlyContinue | + Sort-Object LastWriteTime -Descending | Select-Object -First 1 | + Select-String -Pattern 'azgnrelay', 'FAILURE' | Select-Object -Last 20 +``` + +If the customer noticed this because a pending update will not start, confirm whether it is blocking (this check is a Warning in the public cloud, so it usually is not the blocker there): + +```powershell +Get-SolutionUpdate | Select-Object DisplayName, Version, State, HealthCheckResult, HealthCheckDate | Format-Table -AutoSize +``` + +**Where this does NOT appear.** This is an outbound-connectivity signal from the Environment Checker, not a Windows failover-cluster state, so do not spend time looking for it in the cluster tooling: + +- **Cluster logs:** not evident in `Get-ClusterLog` (a connectivity-validator result is not a failover-cluster event and is never written there). +- **Windows Failover Cluster Manager:** not evident in Failover Cluster Manager (endpoint reachability is not a clustered role, resource, or node property). +- **Windows Admin Center (standalone host):** not evident for this check (a standalone Windows Admin Center may show a general Azure-connection warning, but does not distinctly flag the `Azure_Kubernetes_Service_Cluster_connect` result; confirm with `Invoke-AzStackHciConnectivityValidation` above or the Azure portal). +- **Windows Admin Center in the Azure portal:** not evident for this check (use the cluster's **Updates** blade or run the validator on the node instead). + +### 2. What it looks like: example failure signatures + +A healthy node reports `Test Analysis - Overall Result: True` with an HTTP `StatusCode` of `200` (or `403` - both mean the endpoint was reached). A node that needs attention reports `Overall Result: False` with one of the following `Detail` signatures. The line `Test Analysis - Layer 3 (tnc): True/False` is the key discriminator: `tnc` is the result of `Test-NetConnection` to the endpoint on 443, so `True` means the TCP connection worked and the failure is higher up the stack (proxy or TLS inspection), while `False` means even the TCP/DNS layer failed (firewall or DNS). + +**Sub-mode 1: DNS resolution failure** (`tnc: False`). + +``` +Test Analysis - Overall Result: False +Test Analysis - Exception Message: The remote name could not be resolved: 'azgnrelay-eastus-l1.servicebus.windows.net' +Test Analysis - Layer 3 (tnc): False +``` + +The node cannot resolve the relay hostname. This is a DNS problem (step 5, "DNS resolution"). + +**Sub-mode 2: TCP blocked by the firewall** (`tnc: False`). + +``` +Test Analysis - Overall Result: False +Test Analysis - Exception Message: Unable to connect to the remote server +Test Analysis - Layer 3 (tnc): False +``` + +or + +``` +Test Analysis - Exception Message: The operation has timed out. +Test Analysis - Layer 3 (tnc): False +``` + +DNS resolved, but the TCP connection to the endpoint on 443 never completed. A firewall or route is blocking the outbound connection (step 5, "Firewall / outbound 443 blocked"). + +**Sub-mode 3: proxy or application-layer block** (`tnc: True`). + +``` +Test Analysis - Overall Result: False +Test Analysis - Exception Message: Unable to connect to the remote server +Test Analysis - Layer 3 (tnc): True +``` + +The raw TCP connection worked (`tnc: True`), but the HTTPS request still failed. A proxy that the node is not configured to use, or that is refusing the request, is the usual cause (step 5, "Proxy"). + +**Sub-mode 4: TLS inspection breaking the connection** (`tnc: True`). + +``` +Test Analysis - Overall Result: False +Test Analysis - Exception Message: The underlying connection was closed: An unexpected error occurred on a send. +Test Analysis - Layer 3 (tnc): True +``` + +The TCP connection worked, then the TLS session was torn down mid-handshake. A TLS-inspection / deep-packet-inspection appliance is intercepting the connection to the Azure Relay endpoint, which the long-lived relay connection does not tolerate (step 5, "TLS inspection"). + +> A transient network blip can produce a one-off failure. If the `Detail` shows `Overall Result: True` on a later run, it has recovered. If it persists, match the signature above and apply the fix. + +### 3. Identify the affected nodes + +The check runs per node, so confirm which nodes fail and re-test the endpoint directly from each: + +```powershell +Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { + $uri = 'azgnrelay-eastus-l1.servicebus.windows.net' # replace with the cluster's region + $tnc = Test-NetConnection -ComputerName $uri -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue + [pscustomobject]@{ TcpTo443 = $tnc } +} | Sort-Object PSComputerName | Select-Object PSComputerName, TcpTo443 +``` + +Nodes returning `True` reach the endpoint at the TCP layer (any remaining failure is proxy or TLS inspection); nodes returning `False` are blocked at the firewall or DNS layer. Substitute the cluster's real region for `` (read it from the `TargetResourceID` / `Detail` of the failing result in step 1). + +### 4. Consequences if you do not fix this + +While this check fails, **Azure Arc cluster connect does not work** for this cluster: you cannot reach Arc-enabled Kubernetes (AKS enabled by Azure Arc) through the secure reverse tunnel. That breaks `az connectedk8s proxy`, the Azure portal's Kubernetes resource view, and cluster-connect-based `kubectl` access to the workload cluster. In the Azure public cloud the check is a **Warning**, so it does not block a solution update, but the feature stays broken until outbound connectivity is restored. The cluster's local VMs and workloads keep running. In Azure Government (Fairfax) the check is **Critical** and blocks the update until it passes. + +### 5. Remediation + +Match the `Detail` signature from step 2 to the sub-mode and apply the matching fix. All of these are network-side changes; none of them drain nodes or disrupt running VMs. The canonical endpoint list is in [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and, for AKS on Azure Local, [AKS network requirements - firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). + +**Sub-mode: DNS resolution** (`The remote name could not be resolved`, `tnc: False`). + +1. On an affected node, confirm the relay hostname resolves: `Resolve-DnsName azgnrelay--l1.servicebus.windows.net`. +2. If it does not resolve, fix the node's DNS: confirm `Get-DnsClientServerAddress` points at DNS servers that can resolve public Azure names (or the customer's forwarders that do). See [Troubleshooting Connectivity Test DNS](./Troubleshooting-Connectivity-Test-Dns.md) and [Troubleshooting DNS External DNS Resolution](./Troubleshooting-DNS-External-DNS-Resolution.md). +3. Re-test with step 1. + +Risk: [LOW RISK]. Correcting DNS resolution does not disrupt running workloads. + +**Sub-mode: firewall / outbound 443 blocked** (`Unable to connect to the remote server` or `timed out`, `tnc: False`). + +1. Allow outbound **TCP 443** from every node to the Azure Relay endpoints used by cluster connect: `*.servicebus.windows.net` (specifically `azgnrelay--l1.servicebus.windows.net`). Add these to the firewall allow list per [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and the [AKS on Azure Local firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). +2. Confirm no route or NSG drops the outbound connection to the relay. +3. Re-test with `Test-NetConnection azgnrelay--l1.servicebus.windows.net -Port 443` from the node. + +Risk: [LOW RISK]. Allowing the documented outbound endpoint does not disrupt running workloads. + +**Sub-mode: proxy** (`Unable to connect to the remote server`, `tnc: True`). + +1. If the cluster uses a proxy, confirm the proxy is configured on the nodes and allows `*.servicebus.windows.net`. See the proxy guidance in [Troubleshooting External Connectivity Failures in Environment Checker](./Troubleshooting-External-Connectivity-Failures-in-Environment-Checker.md). +2. Confirm the node's proxy configuration (WinHTTP / environment) matches the cluster's documented proxy, and that the relay endpoints are not on a bypass list that routes them incorrectly. +3. Re-test with step 1. + +Risk: [LOW RISK]. A proxy allow-list change does not disrupt running workloads. + +**Sub-mode: TLS inspection** (`The underlying connection was closed: An unexpected error occurred on a send`, `tnc: True`). + +1. Exclude the Azure Relay endpoints (`*.servicebus.windows.net`) from any TLS-inspection / deep-packet-inspection / SSL-interception appliance on the outbound path. The cluster-connect relay uses a long-lived connection that inspection appliances break. +2. Confirm with the network team that the relay endpoints are on the inspection bypass list. +3. Re-test with step 1. + +Risk: [LOW RISK]. Excluding the endpoint from interception does not disrupt running workloads. + +### 6. Verification: prove the failure cleared + +Re-run the connectivity validator on the affected nodes and confirm the target is healthy: + +```powershell +Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { + Test-NetConnection azgnrelay-eastus-l1.servicebus.windows.net -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue +} | Sort-Object PSComputerName +``` + +Every node should return `True` (substitute the cluster's region). Then re-run the pre-update health check so the validator re-evaluates and refreshes the cluster-wide readiness record: + +```powershell +Invoke-SolutionUpdatePrecheck -SystemHealth +Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate +``` + +Confirm `HealthState` is `Success` with a current `HealthCheckDate`. + +> **Note:** the Azure portal readiness view and the cluster-wide health-check result refresh only when a full health check or `Invoke-SolutionUpdatePrecheck` runs, not on a targeted per-node re-test, so confirm the fix with `Invoke-AzStackHciConnectivityValidation` or `Test-NetConnection` on the nodes rather than waiting on the portal. + +## Glossary + +- **Azure Arc cluster connect:** a feature that provides secure connectivity to Arc-enabled Kubernetes clusters from anywhere without opening any inbound port, by maintaining an outbound connection from the cluster to an Azure Relay endpoint. This check verifies that outbound path. +- **Azure Relay / Service Bus relay (`*.servicebus.windows.net`):** the Azure service that hosts the cluster-connect reverse tunnel. The per-region endpoint is `azgnrelay--l1.servicebus.windows.net`. +- **`Invoke-AzStackHciConnectivityValidation`:** the Environment Checker connectivity validator. It probes each required endpoint and reports the result, including this "Cluster connect" target. Run it standalone on a node or workstation to reproduce and verify. +- **`Test Analysis - Layer 3 (tnc)`:** the `Test-NetConnection` (TCP 443) result recorded in the `Detail`. `True` = TCP reached the endpoint (a remaining failure is proxy or TLS inspection); `False` = TCP or DNS failed (firewall or DNS). +- **AKS enabled by Azure Arc:** Azure Kubernetes Service running on Azure Local, managed through Azure Arc. Cluster connect is one of the paths used to reach it. From a31246da9c0d82664981eaae2775dd70f720f0a1 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 13 Jul 2026 12:54:29 -0400 Subject: [PATCH 2/6] AKS cluster-connect TSG: note pre-update readiness is the authoritative surface Add an honest note in step 1: this connectivity target is validated as part of the pre-update / deployment readiness run and may not appear in every ad-hoc standalone Invoke-AzStackHciConnectivityValidation run on newer builds (the target set is versioned), so the authoritative confirmation for this check is the pre-update health-check result (HealthCheckResult JSON / Event ID 17205 / portal Updates tab). Use the standalone run to test raw reachability to the relay endpoint. Discovered during tier-1 live validation on a current-build lab cluster. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d7f2348-e3be-4fe4-8598-6bf5dcb4050f --- ...roubleshooting-Azure_Kubernetes_Service_Cluster_connect.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md index 5fd138f3..efa66cb6 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -47,7 +47,7 @@ Azure Arc **cluster connect** provides a secure way to connect to Arc-enabled Ku ### 1. Confirm the failure and see where it appears -**Run the connectivity validator directly (authoritative).** On a node (or a workstation with the Environment Checker module installed), run: +**Run the connectivity validator directly.** On a node (or a workstation with the Environment Checker module installed), run: ```powershell Invoke-AzStackHciConnectivityValidation @@ -55,6 +55,8 @@ Invoke-AzStackHciConnectivityValidation Look for the **Azure Kubernetes Service -> Cluster connect** target in the output. A failing target is shown as **Needs Attention / Critical** with the relay URL and a help link. The run also writes its own log and report (see below), and the failing URL to `FailedUrls.txt`. +> **Note:** this target is validated as part of the cluster's **pre-update / deployment readiness** run, and on newer builds it may not appear in every ad-hoc standalone `Invoke-AzStackHciConnectivityValidation` run (the connectivity target set is versioned). So the **authoritative** confirmation for this specific check is the pre-update health-check result below (the `HealthCheckResult.*.json`, Event ID 17205, or the portal Updates tab), which is populated by the readiness run that actually evaluates it. Use the standalone run to test raw reachability to the relay endpoint (`Test-NetConnection azgnrelay--l1.servicebus.windows.net -Port 443`), and the health-check result to confirm the check's own pass/fail. + **Read the newest health-check result on a node** and flag this check: ```powershell From 56169c4e781d13cd0d91f8f22dabdca459053875 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Mon, 13 Jul 2026 13:23:25 -0400 Subject: [PATCH 3/6] AKS cluster-connect TSG: address PR review (bot) findings - Discovery filter: use the specific HealthCheckResult.EnvironmentChecker.*.json pattern (the folder also holds HealthCheckResult.CheckCloudHealth.*.json, so the broad filter could pick a newer unrelated file and wrongly report no match). - Name match: use -like '*Azure_Kubernetes_Service_Cluster_connect*' instead of an exact -eq (health-check results can carry a prefix/suffix, so the exact compare could hide the failure); read the readable state from AdditionalData.Status. - Verification: Test-NetConnection cannot confirm the proxy / TLS-inspection sub-modes (those already have tnc:True while failing), so lead verification with the validator / Invoke-SolutionUpdatePrecheck (authoritative for every sub-mode) and demote Test-NetConnection to a DNS/TCP-only quick check with an explicit caveat. - Discoverability: add the guide to TSG/EnvironmentValidator/README.md index. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 1d7f2348-e3be-4fe4-8598-6bf5dcb4050f --- TSG/EnvironmentValidator/README.md | 1 + ...zure_Kubernetes_Service_Cluster_connect.md | 31 +++++++++++-------- 2 files changed, 19 insertions(+), 13 deletions(-) diff --git a/TSG/EnvironmentValidator/README.md b/TSG/EnvironmentValidator/README.md index c3d9c389..1f99829b 100644 --- a/TSG/EnvironmentValidator/README.md +++ b/TSG/EnvironmentValidator/README.md @@ -24,6 +24,7 @@ This folder contains the TSG's related to Environment Validators. * [Troubleshooting Secure Boot Status (AzStackHci_Security_SecureBootStatus)](./Troubleshooting-AzStackHci_Security_SecureBootStatus.md) * [Troubleshooting ASR Rule Group Policy Conflict (Security AsrRuleGPConflict)](./Troubleshooting-AzStackHci_Security_AsrRuleGPConflict.md) * [Troubleshooting ARB Resource Lock (ARBStack ARBIsLocked)](./Troubleshooting-AzStackHci_ARBStack_ARBIsLocked.md) +* [Troubleshooting Azure Kubernetes Service Cluster Connect (Azure_Kubernetes_Service_Cluster_connect)](./Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md) * [Known Issue: High Disk Space Usage in TEMP](Known-Issue-High-Disk-Space-usage-in-TEMP.md) * [Known Issue: WinRM cannot process the configuration request](Known-Issue-WinRM-cannot-process-the-configuration-request.md) * [Known Issue: This module requires Az.Accounts version 5.3.0](Known-Issue-This-module-requires-Az-Accounts-version-5-3-0.md) diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md index efa66cb6..35c4353d 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -55,7 +55,7 @@ Invoke-AzStackHciConnectivityValidation Look for the **Azure Kubernetes Service -> Cluster connect** target in the output. A failing target is shown as **Needs Attention / Critical** with the relay URL and a help link. The run also writes its own log and report (see below), and the failing URL to `FailedUrls.txt`. -> **Note:** this target is validated as part of the cluster's **pre-update / deployment readiness** run, and on newer builds it may not appear in every ad-hoc standalone `Invoke-AzStackHciConnectivityValidation` run (the connectivity target set is versioned). So the **authoritative** confirmation for this specific check is the pre-update health-check result below (the `HealthCheckResult.*.json`, Event ID 17205, or the portal Updates tab), which is populated by the readiness run that actually evaluates it. Use the standalone run to test raw reachability to the relay endpoint (`Test-NetConnection azgnrelay--l1.servicebus.windows.net -Port 443`), and the health-check result to confirm the check's own pass/fail. +> **Note:** this target is validated as part of the cluster's **pre-update / deployment readiness** run, and on newer builds it may not appear in every ad-hoc standalone `Invoke-AzStackHciConnectivityValidation` run (the connectivity target set is versioned). So the **authoritative** confirmation for this specific check is the pre-update health-check result below (the `HealthCheckResult.EnvironmentChecker.*.json`, Event ID 17205, or the portal Updates tab), which is populated by the readiness run that actually evaluates it. Use the standalone run to test raw reachability to the relay endpoint (`Test-NetConnection azgnrelay--l1.servicebus.windows.net -Port 443`), and the health-check result to confirm the check's own pass/fail. **Read the newest health-check result on a node** and flag this check: @@ -68,7 +68,10 @@ if (-not (Test-Path $base)) { } $latest = $null if ($base) { - $latest = Get-ChildItem $base -Filter 'HealthCheckResult.*.json' -ErrorAction SilentlyContinue | + # Use the Environment Checker result file specifically. The folder also holds + # HealthCheckResult.CheckCloudHealth.*.json (other checks), so a broad filter could + # pick a newer unrelated file and wrongly report no match for this validator. + $latest = Get-ChildItem $base -Filter 'HealthCheckResult.EnvironmentChecker.*.json' -ErrorAction SilentlyContinue | Sort-Object LastWriteTime -Descending | Select-Object -First 1 } if (-not $latest) { @@ -76,10 +79,10 @@ if (-not $latest) { } else { Get-Content $latest.FullName -Raw | ConvertFrom-Json | - Where-Object { $_.Name -eq 'Azure_Kubernetes_Service_Cluster_connect' -and $_.Status -ne 0 -and $_.Status -ne 'SUCCESS' } | + Where-Object { $_.Name -like '*Azure_Kubernetes_Service_Cluster_connect*' -and $_.Status -ne 0 -and $_.Status -ne 'SUCCESS' } | ForEach-Object { [pscustomobject]@{ - Status = $_.Status + Status = if ($_.AdditionalData.Status) { $_.AdditionalData.Status } else { $_.Status } Source = $_.AdditionalData.Source Target = $_.TargetResourceID Detail = $_.AdditionalData.Detail @@ -228,24 +231,26 @@ Risk: [LOW RISK]. Excluding the endpoint from interception does not disrupt runn ### 6. Verification: prove the failure cleared -Re-run the connectivity validator on the affected nodes and confirm the target is healthy: +**Re-run the actual check (authoritative for every sub-mode).** The validator makes the real HTTPS request to the endpoint, so it is the only reliable confirmation for the proxy and TLS-inspection sub-modes (where the TCP layer already succeeds while HTTPS still fails). Re-run the pre-update health check so the validator re-evaluates and refreshes the cluster-wide readiness record: ```powershell -Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { - Test-NetConnection azgnrelay-eastus-l1.servicebus.windows.net -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue -} | Sort-Object PSComputerName +Invoke-SolutionUpdatePrecheck -SystemHealth +Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate ``` -Every node should return `True` (substitute the cluster's region). Then re-run the pre-update health check so the validator re-evaluates and refreshes the cluster-wide readiness record: +Confirm `HealthState` is `Success` with a current `HealthCheckDate`. You can also re-run `Invoke-AzStackHciConnectivityValidation` on an affected node and confirm the **Cluster connect** target now reports `Overall Result: True`. + +**Quick lower-layer check (DNS and firewall/TCP sub-modes only).** For sub-modes 1 and 2, `Test-NetConnection` confirms the DNS/TCP layer is now open: ```powershell -Invoke-SolutionUpdatePrecheck -SystemHealth -Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate +Invoke-Command -ComputerName (Get-ClusterNode).Name -ScriptBlock { + Test-NetConnection azgnrelay-eastus-l1.servicebus.windows.net -Port 443 -InformationLevel Quiet -WarningAction SilentlyContinue +} | Sort-Object PSComputerName ``` -Confirm `HealthState` is `Success` with a current `HealthCheckDate`. +Every node should return `True` (substitute the cluster's region). Note that `True` only proves the TCP connection succeeds; for the **proxy** and **TLS-inspection** sub-modes it does **not** prove the fix (those already have `tnc: True` while failing), so confirm those with the validator / precheck above. -> **Note:** the Azure portal readiness view and the cluster-wide health-check result refresh only when a full health check or `Invoke-SolutionUpdatePrecheck` runs, not on a targeted per-node re-test, so confirm the fix with `Invoke-AzStackHciConnectivityValidation` or `Test-NetConnection` on the nodes rather than waiting on the portal. +> **Note:** the Azure portal readiness view and the cluster-wide health-check result refresh only when a full health check or `Invoke-SolutionUpdatePrecheck` runs, not on a targeted per-node re-test, so confirm the fix with `Invoke-AzStackHciConnectivityValidation` or the precheck rather than waiting on the portal. ## Glossary From 34a94210eec3d9bb5757ca0af38a4308cb5dd693 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Tue, 18 Aug 2026 19:06:21 -0400 Subject: [PATCH 4/6] AKS Cluster Connect TSG: remove dead 'See the detection note.' reference The severity cell ended with 'See the detection note.', which points to no section in the guide. Remove the dangling reference; the cell already states the per-cloud severity and effect. --- .../Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md index 35c4353d..130df201 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -15,7 +15,7 @@ Severity - Warning in the Azure public cloud (non-blocking, but the Arc cluster-connect feature will not work until it is fixed); Critical in Azure Government (Fairfax). See the detection note. + Warning in the Azure public cloud (non-blocking, but the Arc cluster-connect feature will not work until it is fixed); Critical in Azure Government (Fairfax). From 9928515f9c65e5d8dbfd6531b8c12dbb36ba8dc2 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 08:41:58 -0400 Subject: [PATCH 5/6] AKS cluster-connect TSG: correct the risk labels and gate the network changes Applied from a tsg-forge grading run with a multi-model persona panel across two frontier model families. Both models independently flagged the remediation section as the weakest part of the guide. - Every sub-mode was labelled [LOW RISK]. Repointing a cluster node's DNS is not low risk: a node's DNS also resolves Active Directory, the cluster name, and its management endpoints. That sub-mode is now [MEDIUM RISK] when node settings are changed, and [LOW RISK] only for the recommended path of fixing the customer's DNS forwarders. - Added an explicit warning never to point a node at a public resolver such as 8.8.8.8, which was the specific failure mode the accessibility lens predicted. - Added a current-state capture before any DNS or proxy change so every change is reversible, including concrete commands for the proxy case. - Added an ownership and approval gate: DNS, firewall, proxy, and TLS-inspection policy are customer network assets outside the Azure Local support boundary. - Clarified that the firewall to change is normally the perimeter device, not the Windows firewall on the node. - Expanded NSG and TLS inspection on first use, and directed proxy mismatches to the platform owner rather than an ad hoc node edit. - Added an instruction to apply one sub-mode fix at a time. --- ...zure_Kubernetes_Service_Cluster_connect.md | 59 +++++++++++++++---- 1 file changed, 47 insertions(+), 12 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md index 130df201..24dbc1cd 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -195,39 +195,74 @@ While this check fails, **Azure Arc cluster connect does not work** for this clu ### 5. Remediation -Match the `Detail` signature from step 2 to the sub-mode and apply the matching fix. All of these are network-side changes; none of them drain nodes or disrupt running VMs. The canonical endpoint list is in [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and, for AKS on Azure Local, [AKS network requirements - firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). +Match the `Detail` signature from step 2 to the sub-mode and apply **only** the matching fix. The canonical endpoint list is in [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and, for AKS on Azure Local, [AKS network requirements - firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). + +> [!IMPORTANT] +> **Apply one sub-mode fix at a time, and stop when the check passes.** Every fix below +> changes network configuration that other systems depend on. If you are unsure which +> sub-mode applies, re-read step 2 rather than applying several fixes at once. +> +> **These changes are owned by the customer's network or platform team**, not by this +> guide. DNS servers, firewall rules, proxy configuration, and TLS-inspection policy are +> all outside the Azure Local support boundary. Confirm the change with the owning team +> and follow the customer's change process before you make it. Record the current value +> first so every change can be reversed. **Sub-mode: DNS resolution** (`The remote name could not be resolved`, `tnc: False`). +> [!WARNING] +> A cluster node's DNS configuration is also how it finds Active Directory, the cluster +> name, and its own management endpoints. Pointing a node at a public resolver such as +> `8.8.8.8` to make this one lookup succeed will break domain and cluster name +> resolution. **Never set a node's DNS to a public resolver.** The correct fix is to make +> the customer's existing DNS servers resolve the public name, normally by fixing their +> forwarders. + 1. On an affected node, confirm the relay hostname resolves: `Resolve-DnsName azgnrelay--l1.servicebus.windows.net`. -2. If it does not resolve, fix the node's DNS: confirm `Get-DnsClientServerAddress` points at DNS servers that can resolve public Azure names (or the customer's forwarders that do). See [Troubleshooting Connectivity Test DNS](./Troubleshooting-Connectivity-Test-Dns.md) and [Troubleshooting DNS External DNS Resolution](./Troubleshooting-DNS-External-DNS-Resolution.md). -3. Re-test with step 1. +2. Record the current setting before changing anything, so the change is reversible: + + ```powershell + Get-DnsClientServerAddress -AddressFamily IPv4 | + Select-Object InterfaceAlias, InterfaceIndex, ServerAddresses + ``` -Risk: [LOW RISK]. Correcting DNS resolution does not disrupt running workloads. +3. If it does not resolve, the fix belongs on the DNS servers the node already uses: confirm `Get-DnsClientServerAddress` points at the customer's DNS servers, and have the DNS owner make those servers resolve public Azure names (normally by correcting their forwarders). Do not repoint the node. See [Troubleshooting Connectivity Test DNS](./Troubleshooting-Connectivity-Test-Dns.md) and [Troubleshooting DNS External DNS Resolution](./Troubleshooting-DNS-External-DNS-Resolution.md). +4. Re-test with step 1. + +Risk: [MEDIUM RISK] if a node's DNS servers are changed, because a node's DNS configuration also serves Active Directory and cluster name resolution, and a wrong value can break domain join, cluster communication, and management. [LOW RISK] when the fix is made on the DNS servers themselves (correcting forwarders), which is the recommended path and does not disrupt running workloads. **Sub-mode: firewall / outbound 443 blocked** (`Unable to connect to the remote server` or `timed out`, `tnc: False`). -1. Allow outbound **TCP 443** from every node to the Azure Relay endpoints used by cluster connect: `*.servicebus.windows.net` (specifically `azgnrelay--l1.servicebus.windows.net`). Add these to the firewall allow list per [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and the [AKS on Azure Local firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). -2. Confirm no route or NSG drops the outbound connection to the relay. +1. Allow outbound **TCP 443** from every node to the Azure Relay endpoints used by cluster connect: `*.servicebus.windows.net` (specifically `azgnrelay--l1.servicebus.windows.net`). Add these to the firewall allow list per [Azure Arc-enabled Kubernetes network requirements](https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/network-requirements) and the [AKS on Azure Local firewall URL exceptions](https://learn.microsoft.com/en-us/azure/aks/hybrid/aks-hci-network-system-requirements#firewall-url-exceptions). This is normally the **perimeter or edge firewall**, not the Windows firewall on the node; Azure Local nodes do not block this outbound traffic by default. Confirm with the network owner which device enforces the block before changing anything. +2. Confirm no route or Network Security Group (NSG) drops the outbound connection to the relay. Route and NSG changes are owned by the network team. 3. Re-test with `Test-NetConnection azgnrelay--l1.servicebus.windows.net -Port 443` from the node. -Risk: [LOW RISK]. Allowing the documented outbound endpoint does not disrupt running workloads. +Risk: [LOW RISK] to the cluster. Allowing the documented outbound endpoint does not disrupt running workloads, but it is a change to customer network policy and needs the network owner's approval. **Sub-mode: proxy** (`Unable to connect to the remote server`, `tnc: True`). 1. If the cluster uses a proxy, confirm the proxy is configured on the nodes and allows `*.servicebus.windows.net`. See the proxy guidance in [Troubleshooting External Connectivity Failures in Environment Checker](./Troubleshooting-External-Connectivity-Failures-in-Environment-Checker.md). -2. Confirm the node's proxy configuration (WinHTTP / environment) matches the cluster's documented proxy, and that the relay endpoints are not on a bypass list that routes them incorrectly. +2. Confirm the node's proxy configuration matches the cluster's documented proxy, and that the relay endpoints are not on a bypass list that routes them incorrectly. Read the current values before changing them: + + ```powershell + netsh winhttp show proxy + [Environment]::GetEnvironmentVariable('HTTPS_PROXY','Machine') + [Environment]::GetEnvironmentVariable('HTTP_PROXY','Machine') + [Environment]::GetEnvironmentVariable('NO_PROXY','Machine') + ``` + + Report a mismatch to the platform owner rather than editing it ad hoc: the proxy configuration is applied and maintained by the Azure Local deployment, and an out-of-band edit can be overwritten or can break other platform traffic. 3. Re-test with step 1. -Risk: [LOW RISK]. A proxy allow-list change does not disrupt running workloads. +Risk: [MEDIUM RISK] if node proxy settings are edited directly, because the proxy carries all node outbound traffic including Arc and update traffic. [LOW RISK] for a proxy allow-list change made on the proxy itself. **Sub-mode: TLS inspection** (`The underlying connection was closed: An unexpected error occurred on a send`, `tnc: True`). -1. Exclude the Azure Relay endpoints (`*.servicebus.windows.net`) from any TLS-inspection / deep-packet-inspection / SSL-interception appliance on the outbound path. The cluster-connect relay uses a long-lived connection that inspection appliances break. -2. Confirm with the network team that the relay endpoints are on the inspection bypass list. +1. Exclude the Azure Relay endpoints (`*.servicebus.windows.net`) from any TLS-inspection, deep-packet-inspection, or SSL-interception appliance on the outbound path. These are middleboxes that terminate and re-sign TLS so they can inspect the traffic; the cluster-connect relay uses a long-lived connection that they break. +2. Confirm with the network team that the relay endpoints are on the inspection bypass list. This is a security-policy change and needs their approval and their change record. 3. Re-test with step 1. -Risk: [LOW RISK]. Excluding the endpoint from interception does not disrupt running workloads. +Risk: [LOW RISK] to the cluster. Excluding the endpoint from interception does not disrupt running workloads, but it changes the customer's security enforcement and must be agreed with the security or network owner. ### 6. Verification: prove the failure cleared From 9a3ccbdd1d947b2137b3df4cc5f625a946aa36e5 Mon Sep 17 00:00:00 2001 From: 1008covingtonlane <42551186+1008covingtonlane@users.noreply.github.com> Date: Wed, 19 Aug 2026 14:21:13 -0400 Subject: [PATCH 6/6] AKS cluster-connect TSG: fix the Azure Government endpoint and the tnc label Third-model review round (xAI added to the panel) found a wrong-cloud endpoint that would have sent every Azure Government reader to a host that does not exist for them. - The guide marks this check Critical in Azure Government but listed only the commercial suffix servicebus.windows.net. Azure Government uses servicebus.usgovcloudapi.net, so testing or allow-listing the documented host on a Fairfax cluster cannot fix the check, and there it also blocks the solution update. Added an explicit callout and directed readers to take the exact host from the check's own TargetResourceID. - The DNS warning banned replacing a node's DNS with a public resolver but not adding one alongside the customer's servers, which still breaks Active Directory and cluster name resolution and is harder to diagnose. Both are now covered. - "Layer 3 (tnc)" is the validator's own label, but Test-NetConnection -Port 443 is a TCP handshake, which is layer 4. More importantly it is not proxy-aware, so on a proxy-only network tnc reads False even though the proxy path is the supported route. Documented both, so tnc False plus a configured proxy is read as the proxy sub-mode rather than automatically as a firewall problem. - HealthState is the aggregate pre-update result, so an unrelated failing check keeps it non-Success and a Success does not prove this target passed. The per-target validator output is now named as the authoritative confirmation. --- ...zure_Kubernetes_Service_Cluster_connect.md | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md index 24dbc1cd..6b2bb85f 100644 --- a/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md +++ b/TSG/EnvironmentValidator/Troubleshooting-Azure_Kubernetes_Service_Cluster_connect.md @@ -31,6 +31,16 @@ Azure Arc **cluster connect** provides a secure way to connect to Arc-enabled Ku - **What the check does:** on each node it issues an outbound HTTPS request to the cluster-connect relay endpoint for the cluster's region, `azgnrelay--l1.servicebus.windows.net` (for example `azgnrelay-eastus-l1.servicebus.windows.net`), on TCP port **443**. Reaching the endpoint is a success **even if the endpoint replies `403`** - the check is proving *reachability*, not authentication, so any HTTP response from the real endpoint (commonly `200` or `403`) passes. - **Severity:** the check is defined at **Warning** severity in the Azure public cloud, so a failure does **not** block a solution update, but the Arc cluster-connect feature is broken until it is fixed. In Azure Government (Fairfax) the same check is **Critical**. + +> [!IMPORTANT] +> **Azure Government uses a different endpoint suffix.** Every relay FQDN on this page is the +> Azure **public cloud** form `azgnrelay--l1.servicebus.windows.net`. In Azure +> Government the Service Bus / Relay suffix is **`servicebus.usgovcloudapi.net`**, so the host +> to test and allow is `azgnrelay--l1.servicebus.usgovcloudapi.net` (for example +> the `usgovvirginia` or `usgovarizona` region). Testing or allow-listing the commercial name +> on a Fairfax cluster will not fix the check, and because the check is **Critical** there it +> also blocks the solution update. Confirm the exact host for your cloud and region from the +> `TargetResourceID` in the check's own `Detail` output rather than from an example. - **When it runs:** the Connectivity validator runs during **Deployment**, **Update**, **Scale-out (Add Node)**, and **Upgrade** readiness, and can also be run standalone at any time (see step 1). - **The failure is always the same class of problem:** the node's outbound connection to the relay endpoint did not complete. The `Detail` string tells you *where* it broke (DNS, TCP/firewall, proxy, or TLS inspection); step 2 maps each signature to its fix. @@ -214,7 +224,10 @@ Match the `Detail` signature from step 2 to the sub-mode and apply **only** the > A cluster node's DNS configuration is also how it finds Active Directory, the cluster > name, and its own management endpoints. Pointing a node at a public resolver such as > `8.8.8.8` to make this one lookup succeed will break domain and cluster name -> resolution. **Never set a node's DNS to a public resolver.** The correct fix is to make +> resolution. **Never point a node at a public resolver, whether by replacing its DNS servers +> or by adding one to the list.** An added public resolver still gets used for lookups and +> still breaks Active Directory and cluster name resolution intermittently, which is harder to +> diagnose than an outright break. The correct fix is to make > the customer's existing DNS servers resolve the public name, normally by fixing their > forwarders. @@ -273,7 +286,15 @@ Invoke-SolutionUpdatePrecheck -SystemHealth Get-SolutionUpdateEnvironment | Format-List HealthState, HealthCheckDate ``` -Confirm `HealthState` is `Success` with a current `HealthCheckDate`. You can also re-run `Invoke-AzStackHciConnectivityValidation` on an affected node and confirm the **Cluster connect** target now reports `Overall Result: True`. +Confirm `HealthState` is `Success` with a current `HealthCheckDate`. + +> [!NOTE] +> `HealthState` is the **aggregate** result of the whole pre-update health check, not this one +> target. An unrelated failing check keeps it non-`Success` even after cluster connect is +> fixed, and a `Success` does not on its own prove that this target passed. The authoritative +> per-target confirmation is the validator output itself, so always also re-run +> `Invoke-AzStackHciConnectivityValidation` on an affected node and confirm the **Cluster +> connect** target reports `Overall Result: True`. **Quick lower-layer check (DNS and firewall/TCP sub-modes only).** For sub-modes 1 and 2, `Test-NetConnection` confirms the DNS/TCP layer is now open: @@ -293,4 +314,12 @@ Every node should return `True` (substitute the cluster's region). Note that `Tr - **Azure Relay / Service Bus relay (`*.servicebus.windows.net`):** the Azure service that hosts the cluster-connect reverse tunnel. The per-region endpoint is `azgnrelay--l1.servicebus.windows.net`. - **`Invoke-AzStackHciConnectivityValidation`:** the Environment Checker connectivity validator. It probes each required endpoint and reports the result, including this "Cluster connect" target. Run it standalone on a node or workstation to reproduce and verify. - **`Test Analysis - Layer 3 (tnc)`:** the `Test-NetConnection` (TCP 443) result recorded in the `Detail`. `True` = TCP reached the endpoint (a remaining failure is proxy or TLS inspection); `False` = TCP or DNS failed (firewall or DNS). + > **Note on the label.** The validator prints "Layer 3", but `Test-NetConnection -Port 443` + > completes a **TCP** handshake, which is layer 4. Read the field as "did name resolution + > plus a direct TCP connection succeed", not as an IP-layer test. It is also **not** + > proxy-aware: `Test-NetConnection` connects directly, while the validator's HTTPS request + > honors the node's proxy. On a network where direct 443 is blocked and all traffic must go + > through a proxy, `tnc` can therefore read `False` even though the proxy path is the only + > supported route, so treat `tnc: False` plus a configured proxy as the proxy sub-mode, not + > automatically as a firewall problem. - **AKS enabled by Azure Arc:** Azure Kubernetes Service running on Azure Local, managed through Azure Arc. Cluster connect is one of the paths used to reach it.