diff --git a/docs/plugins.md b/docs/plugins.md index bc6f38645..a494219b9 100644 --- a/docs/plugins.md +++ b/docs/plugins.md @@ -23,19 +23,20 @@ repo instead - see [Advanced](#advanced-hidden-flags) - but it bypasses the cluster and is not the intended flow.) The access model: - Authentication: the **Bearer token** from your kubeconfig (client - certificates do not work). -- Authorization: the ClusterRole - `d8:registry-packages-proxy:packages-download`, bound by the cluster - administrator. Authorization is cached for about 5 minutes, so after the - binding is created, retry with a fresh token. -- Endpoint: discovered automatically through your kubeconfig's API server; - override with `--rpp-endpoint` / `D8_RPP_ENDPOINT`, pass a private CA with + certificates work only on clusters whose proxy trusts the cluster CA). +- Authorization: the ClusterRole `d8:registry-packages-proxy:cli-download`, + bound by the cluster administrator. Authorization is cached for about 5 + minutes, so after the binding is created, retry with a fresh token. +- Endpoint: the cluster is asked where the proxy is, and the endpoints it + offers are tried in order - master addresses first, then the public host. + Override with `--rpp-endpoint` / `D8_RPP_ENDPOINT`, pass a private CA with `--rpp-ca-file`. The access model is shared with d8 self-update (see [self-update.md - How access works](self-update.md#how-access-works) for the -OIDC-kubeconfig and endpoint-discovery details), but the ClusterRole differs: -plugins need `packages-download`, CLI self-update needs `cli-download`. +OIDC-kubeconfig and endpoint-discovery details), including the ClusterRole: +plugins live under `deckhouse-cli/plugins/`, so they are served from +`/v1/images/` and covered by the same `cli-download` role. ## Commands @@ -134,7 +135,7 @@ The persistent flags above are shared by every `d8 plugins` subcommand; the |---|---|---| | `image or tag not found` (404) | that plugin - or that specific version - is not published in this cluster's registry | check with `d8 plugins versions `; publishing is the plugin CI's job | | `... unauthorized (401)` | no accepted Bearer token (a client-certificate kubeconfig is not enough) | use an OIDC-token kubeconfig (Kubeconfig Generator or `d8 login`) | -| `... forbidden (403)` | your identity may not download plugins | ask an admin to bind the ClusterRole `d8:registry-packages-proxy:packages-download`; authorization is cached ~5 min, so retry with a fresh token | +| `... forbidden (403)` | your identity may not download plugins | ask an admin to bind the ClusterRole `d8:registry-packages-proxy:cli-download`; authorization is cached ~5 min, so retry with a fresh token | | `... requirements not satisfied` | mandatory **plugin** dependencies are missing or version-incompatible | run `d8 plugins contract `; on `install` deps auto-install, but at plugin *run* time install them manually as the hint says (`d8 plugins install `) | | `... requires Kubernetes/Deckhouse/module ...` | a **cluster-side** requirement is unmet (a different message from the row above) | upgrade the cluster/module, or pass `--skip-cluster-checks` to bypass verification | | `... upstream error (5xx)` | the proxy could not reach the backing registry | retry shortly, or check the `registry-packages-proxy` pods in `d8-cloud-instance-manager` | diff --git a/docs/self-update.md b/docs/self-update.md index 31a9d3991..e6dc64c71 100644 --- a/docs/self-update.md +++ b/docs/self-update.md @@ -23,7 +23,7 @@ d8 cli update │ Bearer token from your kubeconfig ▼ -registry-packages-proxy. (found automatically via Ingress) +registry-packages-proxy (address taken from the cluster) │ TokenReview + SubjectAccessReview (kube-rbac-proxy) ▼ cluster registry (credentials live only inside the cluster) @@ -137,7 +137,7 @@ $ d8 cli use v0.13.0 # repeated: "deckhouse-cli is already at v0.13.0 | `... unauthorized` (401) | no token in kubeconfig, or a client-certificate identity | use an OIDC kubeconfig from the Kubeconfig Generator | | `... forbidden` (403) | the `cli-download` role is not bound to you | ask the administrator for the ClusterRoleBinding | | 403 right after the role was bound | the proxy caches authorization for ~5 min per token | retry with a fresh token or wait 5 minutes | -| `x509: certificate signed by unknown authority` | the proxy endpoint uses a CA your system does not trust | pass `--rpp-ca-file ` | -| `x509: ... doesn't contain any IP SANs` | you are connecting to a pod IP instead of the Ingress host | set `--rpp-endpoint https://registry-packages-proxy.` | +| `x509: certificate signed by unknown authority` | the endpoint you named with `--rpp-endpoint` uses a CA your system does not trust | pass `--rpp-ca-file `; discovered endpoints already use the CA the cluster publishes | +| `no endpoint answered` | every endpoint the cluster offered was rejected, and the message names each one with its reason | master addresses need network access to port 4219, the public host needs a valid certificate; or name an endpoint yourself with `--rpp-endpoint` | | `deckhouse-cli is already up to date` | you run the latest version | use `--version X` to install an exact (older) one | | `d8 cli use X` downloads although X was installed before | the local store was cleaned, or X was installed on another machine/user | it will download once and stay installed | diff --git a/internal/plugins/README.md b/internal/plugins/README.md index 4a791065e..474515dec 100644 --- a/internal/plugins/README.md +++ b/internal/plugins/README.md @@ -39,9 +39,11 @@ The `pluginSource` interface (`source.go`) has two implementations, chosen in side (ADR #386: deckhouse-cli reaches the registry exclusively through the proxy, so every command needs a reachable cluster). See `internal/selfupdate/README.md` for what RPP is and how authorization works - - plugin download is gated by the `d8:registry-packages-proxy:packages-download` - ClusterRole, distinct from self-update's `cli-download`. The plugin routes are - `/v1/images/deckhouse-cli/plugins//{tags,manifests/,images/}`. + plugin download is gated by the `d8:registry-packages-proxy:cli-download` + ClusterRole, the same one self-update needs. The plugin routes are + `/v1/images/deckhouse-cli/plugins//{tags,manifests/,images/}`, + and kube-rbac-proxy authorizes the whole `/v1/images/` prefix through the + `deployments/cli-binary` subresource. - **`registryPluginSource` (`source_legacy.go`) - a temporary, hidden `--source` bypass.** It pulls straight from a registry repo with go-containerregistry, skipping the proxy and the cluster, and force-sets `--skip-cluster-checks`. It diff --git a/internal/plugins/cmd/contract.go b/internal/plugins/cmd/contract.go index 074bfa7d3..8f34ffda7 100644 --- a/internal/plugins/cmd/contract.go +++ b/internal/plugins/cmd/contract.go @@ -44,6 +44,10 @@ func newContractCommand(manager *plugins.Manager, logger *dkplog.Logger) *cobra. ctx := cmd.Context() + if err := manager.InitPluginServices(ctx); err != nil { + return err + } + latestVersion, err := manager.LatestVersion(ctx, pluginName) if err != nil { return fmt.Errorf("failed to fetch latest version: %w", err) diff --git a/internal/plugins/cmd/errdetect/diagnose.go b/internal/plugins/cmd/errdetect/diagnose.go index 731fed55f..3efe75f71 100644 --- a/internal/plugins/cmd/errdetect/diagnose.go +++ b/internal/plugins/cmd/errdetect/diagnose.go @@ -36,12 +36,16 @@ func Diagnose(err error) *diagnostic.HelpfulError { switch { case errors.Is(err, rpp.ErrUnauthorized): return help(err, "registry-packages-proxy: unauthorized (401)", - "no accepted Bearer token (a client-certificate kubeconfig is not enough)", - "use a kubeconfig with an OIDC token (Kubeconfig Generator or 'd8 login')") + "the kubeconfig credential was rejected: an invalid or expired token, or a client certificate the proxy does not trust", + "use a kubeconfig with an OIDC token", + "a client-certificate kubeconfig works only when the certificate is signed by the cluster CA") case errors.Is(err, rpp.ErrForbidden): + // Plugins live under deckhouse-cli/plugins/, so they are served from + // /v1/images/ and authorized by cli-download. The packages-download role + // covers /v1/packages/ and grants nothing here. return help(err, "registry-packages-proxy: forbidden (403)", "the identity may not download plugins", - "bind the ClusterRole 'd8:registry-packages-proxy:packages-download' to the user/group", + "bind the ClusterRole 'd8:registry-packages-proxy:cli-download' to the user/group", "authorization is cached ~5 min - after binding, retry with a fresh token") case errors.Is(err, rpp.ErrNotFound): return help(err, "registry-packages-proxy: plugin or version not found (404)", @@ -53,11 +57,11 @@ func Diagnose(err error) *diagnostic.HelpfulError { "the proxy could not reach the backing registry", "retry shortly, or check the registry-packages-proxy pods in d8-cloud-instance-manager") case errors.Is(err, rpp.ErrEndpointDiscovery): - return help(err, "registry-packages-proxy: endpoint discovery via the Kubernetes API failed", - "discovery reaches the proxy through your kubeconfig's API server, which was unreachable or presented an invalid certificate", - "this is the Kubernetes API endpoint (kubeconfig 'server:'), not the proxy - confirm it is reachable and its TLS certificate is valid for that host", - "skip discovery: pass --rpp-endpoint https://registry-packages-proxy. (or set D8_RPP_ENDPOINT)", - "on a master node, point the kubeconfig at the local API (https://127.0.0.1:6445, CA /etc/kubernetes/pki/ca.crt) with an OIDC token") + return help(err, "registry-packages-proxy: no usable endpoint found", + "either the cluster could not be asked where the proxy is, or none of the endpoints it offered answered", + "the message above names what was tried: a Kubernetes API failure means the kubeconfig 'server:' is unreachable or its certificate is invalid", + "a per-endpoint failure means the proxy was not reachable there - master addresses need access to port 4219, the public host needs a valid certificate", + "skip discovery: pass --rpp-endpoint (or set D8_RPP_ENDPOINT), adding --rpp-ca-file when its CA is not publicly trusted") default: return nil } diff --git a/internal/plugins/cmd/errdetect/diagnose_test.go b/internal/plugins/cmd/errdetect/diagnose_test.go index 655e46837..51d721062 100644 --- a/internal/plugins/cmd/errdetect/diagnose_test.go +++ b/internal/plugins/cmd/errdetect/diagnose_test.go @@ -37,10 +37,10 @@ func TestDiagnose(t *testing.T) { wantSol string }{ {"401", rpp.ErrUnauthorized, "unauthorized (401)", "OIDC"}, - {"403", rpp.ErrForbidden, "forbidden (403)", "packages-download"}, + {"403", rpp.ErrForbidden, "forbidden (403)", "cli-download"}, {"404", rpp.ErrNotFound, "plugin or version not found (404)", "deckhouse-cli/plugins"}, {"5xx", rpp.ErrUpstream, "upstream error (5xx)", "registry-packages-proxy pods"}, - {"discovery", rpp.ErrEndpointDiscovery, "endpoint discovery via the Kubernetes API failed", "--rpp-endpoint"}, + {"discovery", rpp.ErrEndpointDiscovery, "no usable endpoint found", "--rpp-endpoint"}, } for _, tc := range cases { diff --git a/internal/plugins/cmd/install.go b/internal/plugins/cmd/install.go index aa42aed19..d2f27812a 100644 --- a/internal/plugins/cmd/install.go +++ b/internal/plugins/cmd/install.go @@ -43,6 +43,10 @@ func newInstallCommand(manager *plugins.Manager) *cobra.Command { pluginName := args[0] ctx := cmd.Context() + if err := manager.InitPluginServices(ctx); err != nil { + return err + } + opts := []plugins.InstallOption{ plugins.InstallWithVersion(version), plugins.InstallWithMajorVersion(useMajor), diff --git a/internal/plugins/cmd/plugins.go b/internal/plugins/cmd/plugins.go index 26306bf45..906d24e42 100644 --- a/internal/plugins/cmd/plugins.go +++ b/internal/plugins/cmd/plugins.go @@ -53,23 +53,15 @@ func NewCommand(logger *dkplog.Logger, builtinCommands []string) *cobra.Command " " + rppflags.EnvCAFile + " PEM CA bundle for proxy TLS verification\n" + " KUBECONFIG path to the kubeconfig file", Hidden: true, - PersistentPreRunE: func(cmd *cobra.Command, _ []string) error { + PersistentPreRunE: func(_ *cobra.Command, _ []string) error { // The plugins directory was captured at registration time, BEFORE flag // parsing - re-read it here so --plugins-dir is honored (the env // path DECKHOUSE_CLI_PATH is applied earlier, at registration). manager.SetDirectory(flags.DeckhousePluginsDir) - // init plugin services for subcommands after flags are parsed. - // PersistentPreRunE is outside wrapProxyDiagnostics, so classify - // proxy/discovery failures here too. - if err := manager.InitPluginServices(cmd.Context()); err != nil { - if diag := errdetect.Diagnose(err); diag != nil { - return diag - } - - return err - } - + // The proxy client is NOT built here: only the network subcommands need + // it (they call InitPluginServices themselves), and the local ones - + // list, remove - must work without a reachable cluster. if err := manager.EnsureInstallRoot(); err != nil { logger.Warn("failed to ensure plugin root directory", slog.String("error", err.Error())) } diff --git a/internal/plugins/cmd/update.go b/internal/plugins/cmd/update.go index 1b909d19d..8e0ddbe16 100644 --- a/internal/plugins/cmd/update.go +++ b/internal/plugins/cmd/update.go @@ -38,6 +38,11 @@ func newUpdateCommand(manager *plugins.Manager) *cobra.Command { Args: cobra.ExactArgs(1), RunE: func(cmd *cobra.Command, args []string) error { pluginName := args[0] + + if err := manager.InitPluginServices(cmd.Context()); err != nil { + return err + } + fmt.Printf("Updating plugin: %s\n", pluginName) return manager.InstallPlugin(cmd.Context(), pluginName, plugins.InstallWithMajorVersion(useMajor)) @@ -58,6 +63,10 @@ func newUpdateAllCommand(manager *plugins.Manager) *cobra.Command { Short: "Update all installed plugins", Long: "Update all installed plugins to their newest cluster-compatible version within each plugin's current major.", RunE: func(cmd *cobra.Command, _ []string) error { + if err := manager.InitPluginServices(cmd.Context()); err != nil { + return err + } + fmt.Println("Updating all installed plugins...") if err := manager.UpdateAll(cmd.Context()); err != nil { diff --git a/internal/plugins/cmd/versions.go b/internal/plugins/cmd/versions.go index c11ff4e11..d859de10c 100644 --- a/internal/plugins/cmd/versions.go +++ b/internal/plugins/cmd/versions.go @@ -70,6 +70,10 @@ func newVersionsCommand(manager *plugins.Manager) *cobra.Command { return err } + if err := manager.InitPluginServices(cmd.Context()); err != nil { + return err + } + versions, err := manager.PublishedVersions(cmd.Context(), pluginName) if err != nil { return err diff --git a/internal/plugins/init.go b/internal/plugins/init.go index 2aae7aba1..3c19af0de 100644 --- a/internal/plugins/init.go +++ b/internal/plugins/init.go @@ -30,7 +30,15 @@ import ( // reaching the proxy by the user's kubeconfig identity. ctx bounds endpoint // discovery, so a Ctrl-C during command startup is honored. The proxy is the only // plugin source (ADR: deckhouse-cli reaches the registry exclusively through it). +// +// Each network entry point calls this before touching the source; local +// subcommands never do, so they work without a reachable cluster. A repeated +// call is a no-op, keeping the entry points independent of each other. func (m *Manager) InitPluginServices(ctx context.Context) error { + if m.service != nil { + return nil + } + // legacy --source bypass (temporary): pull straight from a registry, skipping // the proxy and the cluster. See internal/plugins/source_legacy.go. if d8flags.SourceRegistryRepo != "" { diff --git a/internal/rpp/client.go b/internal/rpp/client.go index caea1c7d6..b1e1a91cd 100644 --- a/internal/rpp/client.go +++ b/internal/rpp/client.go @@ -17,6 +17,7 @@ limitations under the License. package rpp import ( + "bytes" "context" "encoding/json" "fmt" @@ -26,6 +27,7 @@ import ( "net/url" "os" "strings" + "time" "unicode" "k8s.io/client-go/rest" @@ -47,6 +49,14 @@ const ( loggerName = "rpp" + // healthzPath is served without authentication, which makes it the cheapest + // way to learn whether an endpoint answers at all. + healthzPath = "/healthz" + + // probeTimeout bounds one reachability check, so walking the candidate list + // stays quick when an endpoint is unroutable. + probeTimeout = 3 * time.Second + // maxTagsResponseBytes caps the tags JSON read so a misbehaving endpoint cannot // make the client buffer an unbounded response; real tag lists are a few KiB. maxTagsResponseBytes int64 = 4 << 20 @@ -95,6 +105,16 @@ func WithInsecureSkipTLSVerify() Option { } } +// collectOptions applies the options and returns the resulting settings. +func collectOptions(opts []Option) options { + var o options + for _, opt := range opts { + opt(&o) + } + + return o +} + // validate rejects contradictory TLS options instead of silently resolving them. func (o options) validate() error { if o.insecure && (o.caFile != "" || len(o.caData) > 0) { @@ -113,17 +133,31 @@ func (o options) validate() error { type Client struct { baseURL string http *http.Client - logger *dkplog.Logger + + // probe answers "does this endpoint exist and pass TLS verification" and + // carries no identity: an endpoint is not trusted with the kubeconfig + // credential before it answers that question. + probe *http.Client + + // restConfig and tlsOpts are what the transports were built from. healTrust + // rebuilds the transports around a re-read CA; a hand-built client + // (NewWithHTTPClient) has no restConfig and never heals. + restConfig *rest.Config + tlsOpts options + + // refreshTrust re-reads the CA the cluster publishes. It is set only when the + // endpoint was discovered together with that CA; nil pins the trust for the + // client's lifetime. + refreshTrust func(ctx context.Context) ([]byte, error) + + logger *dkplog.Logger } // New builds a Client whose requests carry the kubeconfig identity from // restConfig. baseURL is the proxy endpoint root, for example // "https://10.0.0.1:4219". func New(baseURL string, restConfig *rest.Config, logger *dkplog.Logger, opts ...Option) (*Client, error) { - var o options - for _, opt := range opts { - opt(&o) - } + o := collectOptions(opts) if err := o.validate(); err != nil { return nil, err @@ -147,32 +181,104 @@ func New(baseURL string, restConfig *rest.Config, logger *dkplog.Logger, opts .. return nil, fmt.Errorf("build RPP HTTP client: %w", err) } - return newClient(baseURL, httpClient, logger), nil + probeClient, err := buildProbeClient(o) + if err != nil { + return nil, fmt.Errorf("build RPP probe client: %w", err) + } + + client := newClient(baseURL, httpClient, probeClient, logger) + client.restConfig = restConfig + client.tlsOpts = o + + return client, nil } // NewWithHTTPClient builds a Client around a pre-built HTTP client. It is used in -// tests and by callers that construct the transport themselves. +// tests and by callers that construct the transport themselves. The probe shares +// the supplied client: its trust settings live inside the opaque transport, and +// a probe that verifies differently from the real requests would be meaningless. func NewWithHTTPClient(baseURL string, httpClient *http.Client, logger *dkplog.Logger) *Client { - return newClient(baseURL, httpClient, logger) + return newClient(baseURL, httpClient, nil, logger) +} + +// refuseRedirects keeps every response on the original host: the transport stamps +// the kubeconfig credential on each hop, so following a redirect would replay it +// to whatever host the response names. A 3xx then surfaces as an unexpected-status +// error. +func refuseRedirects(*http.Request, []*http.Request) error { + return http.ErrUseLastResponse +} + +// guardedCopy returns a copy of the client with redirects refused. The caller's +// client is not mutated. +func guardedCopy(client *http.Client) *http.Client { + guarded := *client + guarded.CheckRedirect = refuseRedirects + + return &guarded } -func newClient(baseURL string, httpClient *http.Client, logger *dkplog.Logger) *Client { - // Refuse redirects: the transport stamps the kubeconfig credential on every - // hop, so a redirect would replay it to whatever host the response names. - // A 3xx then surfaces as an unexpected-status error. - // The caller's client is copied, not mutated. - guarded := *httpClient - guarded.CheckRedirect = func(*http.Request, []*http.Request) error { - return http.ErrUseLastResponse +func newClient(baseURL string, httpClient, probeClient *http.Client, logger *dkplog.Logger) *Client { + guarded := guardedCopy(httpClient) + + // No separate probe client supplied: probe with the main one. + probe := guarded + + if probeClient != nil { + probe = guardedCopy(probeClient) } return &Client{ baseURL: strings.TrimRight(baseURL, "/"), - http: &guarded, + http: guarded, + probe: probe, logger: logger.Named(loggerName), } } +// healTrust re-reads the published CA and rebuilds both transports around it. It +// reports whether a retry makes sense: the CA source answered and gave a bundle +// different from the one the failed handshake used. Transports are swapped in +// place - commands drive the client sequentially, which is what makes that safe. +func (c *Client) healTrust(ctx context.Context) bool { + if c.refreshTrust == nil || c.restConfig == nil { + return false + } + + caPEM, err := c.refreshTrust(ctx) + if err != nil { + c.logger.Debug("re-reading the published CA failed", dkplog.Err(err)) + + return false + } + + // Nothing new: a retry would fail the same handshake again. + if len(caPEM) == 0 || bytes.Equal(caPEM, c.tlsOpts.caData) { + return false + } + + o := c.tlsOpts + o.caData = caPEM + + httpClient, err := buildHTTPClient(c.restConfig, o) + if err != nil { + return false + } + + probeClient, err := buildProbeClient(o) + if err != nil { + return false + } + + c.tlsOpts = o + c.http = guardedCopy(httpClient) + c.probe = guardedCopy(probeClient) + + c.logger.Debug("the published CA changed; transports rebuilt around it") + + return true +} + // validateBaseURL ensures the explicit endpoint is a usable https URL, so a // misconfigured --rpp-endpoint fails with a clear message instead of an opaque // transport error on the first request. @@ -285,11 +391,51 @@ func (c *Client) GetManifest(ctx context.Context, image ImageRef, ref string) ([ return raw, nil } +// reachable reports whether the endpoint answers. Any HTTP status counts as an +// answer: the public host routes only /v1/images/, so its 404 still proves the +// network path and the TLS handshake. Only a transport failure rules an endpoint +// out, which is what makes walking a candidate list possible. +// +// The request goes through the credential-free probe client: /healthz needs no +// authentication, and with --rpp-insecure-skip-tls-verify any answering host +// would otherwise receive the kubeconfig identity. +func (c *Client) reachable(ctx context.Context) error { + ctx, cancel := context.WithTimeout(ctx, probeTimeout) + defer cancel() + + req, err := http.NewRequestWithContext(ctx, http.MethodGet, c.baseURL+healthzPath, nil) + if err != nil { + return fmt.Errorf("build probe request: %w", err) + } + + resp, err := c.probe.Do(req) + if err != nil && isCertVerificationError(err) && c.healTrust(ctx) { + resp, err = c.probe.Do(req.Clone(ctx)) + } + + if err != nil { + return fmt.Errorf("GET %s: %w", healthzPath, err) + } + + defer func() { _ = resp.Body.Close() }() + + return nil +} + // do executes the request and, on a non-2xx status, closes the body and maps the // status to a sentinel error. On success the response is returned with its body // still open for the caller to consume. +// +// A certificate verification failure gets one retry after healTrust: the cluster +// rotates the proxy's trust root together with its certificate, and the +// published CA is where the new root comes from. Requests here are GETs with no +// body, which is what makes the replay safe. func (c *Client) do(req *http.Request) (*http.Response, error) { resp, err := c.http.Do(req) + if err != nil && isCertVerificationError(err) && c.healTrust(req.Context()) { + resp, err = c.http.Do(req.Clone(req.Context())) + } + if err != nil { return nil, fmt.Errorf("%s %s: %w", req.Method, req.URL.Path, err) } diff --git a/internal/rpp/clusterconfig.go b/internal/rpp/clusterconfig.go new file mode 100644 index 000000000..2c3050efd --- /dev/null +++ b/internal/rpp/clusterconfig.go @@ -0,0 +1,113 @@ +/* +Copyright 2026 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package rpp + +import ( + "context" + "encoding/json" + "fmt" + "net/url" + + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" +) + +const ( + // proxyConfigMapName is published by the registry-packages-proxy module. It + // answers both questions a client has: where the proxy is, and which CA signs + // the certificate it serves. + proxyConfigMapName = "registry-packages-proxy-config" + + configKeyEndpoints = "endpoints" + configKeyCA = "ca.crt" + configKeyPublicEndpoint = "publicEndpoint" +) + +// clusterConfig is the published configuration of the proxy. +// +// A cluster running an older module version publishes nothing, and an identity +// without the cli-download role cannot read it. Both cases leave every field +// empty instead of failing: discovery still works through the Ingress. +type clusterConfig struct { + // endpoints are base URLs built from the master addresses, verifiable with caPEM. + endpoints []string + + // publicEndpoint is the base URL of the public Ingress, empty when the proxy + // is not published under a public domain. + publicEndpoint string + + // caPEM verifies the certificate served on the master addresses. + caPEM []byte +} + +// readClusterConfig returns the configuration published by the module. An absent +// ConfigMap and a denied read are reported as an empty config with no error, +// because both are normal on clusters this CLI still has to work with. A 401 is +// neither: the credentials themselves are rejected, so it surfaces as an error +// instead of silently degrading discovery. Malformed content is an error: it +// means the published contract changed. +func readClusterConfig(ctx context.Context, kube kubernetes.Interface) (clusterConfig, error) { + configMap, err := kube.CoreV1().ConfigMaps(proxyNamespace).Get(ctx, proxyConfigMapName, metav1.GetOptions{}) + if err != nil { + if apierrors.IsNotFound(err) || apierrors.IsForbidden(err) { + return clusterConfig{}, nil + } + + return clusterConfig{}, fmt.Errorf("get configmap %s/%s: %w", proxyNamespace, proxyConfigMapName, err) + } + + endpoints, err := parseConfigEndpoints(configMap.Data[configKeyEndpoints]) + if err != nil { + return clusterConfig{}, err + } + + config := clusterConfig{ + endpoints: endpoints, + publicEndpoint: configMap.Data[configKeyPublicEndpoint], + } + + if ca := configMap.Data[configKeyCA]; ca != "" { + config.caPEM = []byte(ca) + } + + return config, nil +} + +// parseConfigEndpoints turns the published "host:port" list into base URLs. +func parseConfigEndpoints(raw string) ([]string, error) { + if raw == "" { + return nil, nil + } + + var addresses []string + if err := json.Unmarshal([]byte(raw), &addresses); err != nil { + return nil, fmt.Errorf("parse %q in configmap %s/%s: %w", configKeyEndpoints, proxyNamespace, proxyConfigMapName, err) + } + + endpoints := make([]string, 0, len(addresses)) + + for _, address := range addresses { + if address == "" { + continue + } + + endpoints = append(endpoints, (&url.URL{Scheme: proxyScheme, Host: address}).String()) + } + + return endpoints, nil +} diff --git a/internal/rpp/clusterconfig_test.go b/internal/rpp/clusterconfig_test.go new file mode 100644 index 000000000..c40ca09a6 --- /dev/null +++ b/internal/rpp/clusterconfig_test.go @@ -0,0 +1,150 @@ +/* +Copyright 2026 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package rpp + +import ( + "context" + "errors" + "testing" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" + "k8s.io/client-go/kubernetes/fake" + clienttesting "k8s.io/client-go/testing" +) + +func proxyConfigMap(data map[string]string) *corev1.ConfigMap { + return &corev1.ConfigMap{ + ObjectMeta: metav1.ObjectMeta{Name: proxyConfigMapName, Namespace: proxyNamespace}, + Data: data, + } +} + +func TestReadClusterConfigReturnsPublishedValues(t *testing.T) { + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219","192.168.0.2:4219"]`, + "ca.crt": "CA-PEM", + "publicEndpoint": "https://registry-packages-proxy.example.com", + })) + + config, err := readClusterConfig(context.Background(), kube) + require.NoError(t, err) + + assert.Equal(t, []string{"https://192.168.0.1:4219", "https://192.168.0.2:4219"}, config.endpoints) + assert.Equal(t, "https://registry-packages-proxy.example.com", config.publicEndpoint) + assert.Equal(t, []byte("CA-PEM"), config.caPEM) +} + +func TestReadClusterConfigWithoutPublicEndpoint(t *testing.T) { + // A cluster with no public domain publishes the master addresses only, and + // that is the case this whole path exists for. + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219"]`, + "ca.crt": "CA-PEM", + })) + + config, err := readClusterConfig(context.Background(), kube) + require.NoError(t, err) + + assert.Equal(t, []string{"https://192.168.0.1:4219"}, config.endpoints) + assert.Empty(t, config.publicEndpoint) + assert.Equal(t, []byte("CA-PEM"), config.caPEM) +} + +func TestReadClusterConfigWithoutCertificate(t *testing.T) { + // The module publishes no CA until the certificate is issued. The addresses + // are still useful, but nothing can verify them yet. + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219"]`, + })) + + config, err := readClusterConfig(context.Background(), kube) + require.NoError(t, err) + + assert.Equal(t, []string{"https://192.168.0.1:4219"}, config.endpoints) + assert.Nil(t, config.caPEM) +} + +func TestReadClusterConfigTreatsAbsentConfigMapAsEmpty(t *testing.T) { + config, err := readClusterConfig(context.Background(), fake.NewSimpleClientset()) + require.NoError(t, err, "an older cluster publishes no config, and that is not a failure") + + assert.Empty(t, config.endpoints) + assert.Empty(t, config.publicEndpoint) + assert.Empty(t, config.caPEM) +} + +func TestReadClusterConfigTreatsDeniedReadAsEmpty(t *testing.T) { + kube := fake.NewSimpleClientset() + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden( + schema.GroupResource{Resource: "configmaps"}, proxyConfigMapName, errors.New("no access")) + }) + + config, err := readClusterConfig(context.Background(), kube) + require.NoError(t, err, "an identity without the role falls back to the Ingress instead of failing") + + assert.Empty(t, config.endpoints) +} + +func TestReadClusterConfigSurfacesUnauthorized(t *testing.T) { + kube := fake.NewSimpleClientset() + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewUnauthorized("credentials rejected") + }) + + _, err := readClusterConfig(context.Background(), kube) + require.Error(t, err, "rejected credentials fail every later cluster call, so hiding the 401 only moves the failure") + assert.Contains(t, err.Error(), "credentials rejected") +} + +func TestReadClusterConfigRejectsMalformedEndpoints(t *testing.T) { + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `192.168.0.1:4219`, + })) + + _, err := readClusterConfig(context.Background(), kube) + require.Error(t, err, "a broken contract must be reported, not silently ignored") + assert.Contains(t, err.Error(), configKeyEndpoints) +} + +func TestReadClusterConfigSurfacesAPIFailure(t *testing.T) { + kube := fake.NewSimpleClientset() + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, errors.New("tls: failed to verify certificate") + }) + + _, err := readClusterConfig(context.Background(), kube) + require.Error(t, err) + assert.Contains(t, err.Error(), "tls: failed to verify certificate") +} + +func TestReadClusterConfigSkipsEmptyAddresses(t *testing.T) { + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219",""]`, + })) + + config, err := readClusterConfig(context.Background(), kube) + require.NoError(t, err) + + assert.Equal(t, []string{"https://192.168.0.1:4219"}, config.endpoints) +} diff --git a/internal/rpp/connect.go b/internal/rpp/connect.go index 34d25b7e9..d742943db 100644 --- a/internal/rpp/connect.go +++ b/internal/rpp/connect.go @@ -18,7 +18,9 @@ package rpp import ( "context" + "fmt" "log/slog" + "strings" "k8s.io/client-go/kubernetes" "k8s.io/client-go/rest" @@ -27,10 +29,11 @@ import ( ) // NewClusterClient builds a Client for the proxy reachable from the given cluster -// connection. The endpoint is used as-is when set, otherwise discovered (Ingress -// preferred, pod IPs as fallback; see chooseDiscoveredEndpoint). +// connection. The endpoint is used as-is when set, otherwise the candidates the +// cluster offers are tried in order (see discoverCandidates). // caFile / insecure select TLS verification (mutually exclusive; New reports the -// contradiction). +// contradiction). Either one applies to every candidate and replaces the CA the +// cluster published. func NewClusterClient( ctx context.Context, kube kubernetes.Interface, @@ -39,26 +42,67 @@ func NewClusterClient( endpoint, caFile string, insecure bool, ) (*Client, error) { - if endpoint == "" { - discovered, source, err := chooseDiscoveredEndpoint(ctx, kube) + if endpoint != "" { + return New(endpoint, restConfig, logger, flagOptions(caFile, insecure)...) + } + + // Judge the flags once. Left to the loop below, a contradictory pair would be + // reported as one rejected candidate per endpoint, hiding both the real cause + // and the ErrUnsupportedConfig the caller matches on. + if err := collectOptions(flagOptions(caFile, insecure)).validate(); err != nil { + return nil, err + } + + candidates, err := discoverCandidates(ctx, kube) + if err != nil { + return nil, fmt.Errorf("%w: %w", ErrEndpointDiscovery, err) + } + + rejected := make([]string, 0, len(candidates)) + + for _, c := range candidates { + client, err := New(c.endpoint, restConfig, logger, candidateOptions(c, caFile, insecure)...) if err != nil { - return nil, err + rejected = append(rejected, fmt.Sprintf("%s %s: %s", c.source, c.endpoint, err)) + + continue } - logger.Debug("discovered registry-packages-proxy endpoint", - slog.String("endpoint", discovered), slog.String("discovered_via", source)) - - if source == "pod" { - // The pod fallback is a master-node IP: unreachable from outside the - // cluster network, and its certificate carries no IP SANs - say so - // before the connection fails with an opaque TLS/timeout error. - logger.Debug("no registry-packages-proxy Ingress found; the pod endpoint is reachable " + - "only from the cluster network and needs --rpp-insecure-skip-tls-verify (or pass --rpp-endpoint)") + // A master endpoint is verified by the CA published next to it, and that + // CA rotates together with the serving certificate. The published bundle + // is the only source of the new root, so let the client re-read it when + // verification fails. Explicit TLS flags pin the trust instead. + if c.source == sourceMaster && len(c.caPEM) > 0 && caFile == "" && !insecure { + client.refreshTrust = func(ctx context.Context) ([]byte, error) { + config, err := readClusterConfig(ctx, kube) + if err != nil { + return nil, err + } + + return config.caPEM, nil + } + } + + if err := client.reachable(ctx); err != nil { + logger.Debug("registry-packages-proxy candidate did not answer", + slog.String("endpoint", c.endpoint), slog.String("candidate_source", c.source), dkplog.Err(err)) + + rejected = append(rejected, fmt.Sprintf("%s %s: %s", c.source, c.endpoint, err)) + + continue } - endpoint = discovered + logger.Debug("discovered registry-packages-proxy endpoint", + slog.String("endpoint", c.endpoint), slog.String("discovered_via", c.source)) + + return client, nil } + return nil, fmt.Errorf("%w: no endpoint answered: %s", ErrEndpointDiscovery, strings.Join(rejected, "; ")) +} + +// flagOptions turns the TLS flags into client options. +func flagOptions(caFile string, insecure bool) []Option { var opts []Option if insecure { @@ -69,5 +113,20 @@ func NewClusterClient( opts = append(opts, WithCAFile(caFile)) } - return New(endpoint, restConfig, logger, opts...) + return opts +} + +// candidateOptions keeps the flags authoritative: a CA file or insecure given by +// the caller replaces the CA the cluster published, and the two never combine +// (New rejects that). +func candidateOptions(c candidate, caFile string, insecure bool) []Option { + if insecure || caFile != "" { + return flagOptions(caFile, insecure) + } + + if len(c.caPEM) > 0 { + return []Option{WithCAData(c.caPEM)} + } + + return nil } diff --git a/internal/rpp/connect_test.go b/internal/rpp/connect_test.go new file mode 100644 index 000000000..a40ce5f86 --- /dev/null +++ b/internal/rpp/connect_test.go @@ -0,0 +1,361 @@ +/* +Copyright 2026 Flant JSC + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package rpp + +import ( + "context" + "crypto/ecdsa" + "crypto/elliptic" + "crypto/rand" + "crypto/tls" + "crypto/x509" + "crypto/x509/pkix" + "encoding/pem" + "fmt" + "io" + "math/big" + "net" + "net/http" + "net/http/httptest" + "net/url" + "os" + "sync" + "sync/atomic" + "testing" + "time" + + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/client-go/rest" + clienttesting "k8s.io/client-go/testing" + + dkplog "github.com/deckhouse/deckhouse/pkg/log" +) + +// tlsEndpoint starts a TLS server that answers every request, and returns its +// host:port together with its certificate in PEM form. +// +// Every server gets a freshly generated certificate of its own, unlike +// httptest.NewTLSServer which reuses one built-in certificate for all servers. +// Without that, two test endpoints would be indistinguishable to a client +// verifying against a CA, and the case this file is about could not be written. +func tlsEndpoint(t *testing.T, tagsBody string) (string, []byte) { + t.Helper() + + return tlsEndpointWithHandler(t, http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, tagsBody) + })) +} + +func tlsEndpointWithHandler(t *testing.T, handler http.Handler) (string, []byte) { + t.Helper() + + certPEM, keyPEM := selfSignedLocalhostCert(t) + + pair, err := tls.X509KeyPair(certPEM, keyPEM) + require.NoError(t, err) + + srv := httptest.NewUnstartedServer(handler) + srv.TLS = &tls.Config{Certificates: []tls.Certificate{pair}, MinVersion: tls.VersionTLS12} + srv.StartTLS() + t.Cleanup(srv.Close) + + parsed, err := url.Parse(srv.URL) + require.NoError(t, err) + + return parsed.Host, certPEM +} + +func selfSignedLocalhostCert(t *testing.T) ([]byte, []byte) { + t.Helper() + + key, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader) + require.NoError(t, err) + + template := &x509.Certificate{ + SerialNumber: big.NewInt(1), + Subject: pkix.Name{CommonName: "registry-packages-proxy-test"}, + NotBefore: time.Now().Add(-time.Hour), + NotAfter: time.Now().Add(time.Hour), + KeyUsage: x509.KeyUsageDigitalSignature | x509.KeyUsageCertSign, + ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth}, + BasicConstraintsValid: true, + IsCA: true, + IPAddresses: []net.IP{net.ParseIP("127.0.0.1")}, + } + + der, err := x509.CreateCertificate(rand.Reader, template, template, &key.PublicKey, key) + require.NoError(t, err) + + keyDER, err := x509.MarshalECPrivateKey(key) + require.NoError(t, err) + + return pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), + pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}) +} + +// TestNewClusterClientSkipsCandidateWithUnverifiableCertificate is the case that +// motivated the candidate list: one endpoint answers with a certificate the +// published CA does not cover, and the CLI moves on instead of failing. +func TestNewClusterClientSkipsCandidateWithUnverifiableCertificate(t *testing.T) { + const tagsBody = `{"name":"deckhouse-cli","tags":["v0.13.1"]}` + + strangerHost, _ := tlsEndpoint(t, tagsBody) + trustedHost, trustedCA := tlsEndpoint(t, tagsBody) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q,%q]`, strangerHost, trustedHost), + "ca.crt": string(trustedCA), + })) + + client, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", false) + require.NoError(t, err) + assert.Equal(t, "https://"+trustedHost, client.baseURL) + + tags, err := client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err) + assert.Equal(t, []string{"v0.13.1"}, tags) +} + +func TestNewClusterClientProbesWithoutCredentials(t *testing.T) { + // The probe runs before an endpoint has earned any trust, so it must not + // carry the kubeconfig identity. The real requests after it must. + var mu sync.Mutex + + authByPath := map[string][]string{} + + host, ca := tlsEndpointWithHandler(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + mu.Lock() + authByPath[r.URL.Path] = append(authByPath[r.URL.Path], r.Header.Get("Authorization")) + mu.Unlock() + + _, _ = io.WriteString(w, `{"name":"deckhouse-cli","tags":["v0.13.1"]}`) + })) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, host), + "ca.crt": string(ca), + })) + + client, err := NewClusterClient(context.Background(), kube, + &rest.Config{BearerToken: "secret-token"}, dkplog.NewNop(), "", "", false) + require.NoError(t, err) + + _, err = client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err) + + mu.Lock() + defer mu.Unlock() + + require.NotEmpty(t, authByPath[healthzPath], "discovery probes /healthz") + + for _, auth := range authByPath[healthzPath] { + assert.Empty(t, auth, "the probe must not carry the kubeconfig credential") + } + + tagsAuth := authByPath["/v1/images/deckhouse-cli/tags"] + require.NotEmpty(t, tagsAuth) + assert.Equal(t, "Bearer secret-token", tagsAuth[0], "real requests keep the identity") +} + +func TestNewClusterClientPrefersTheFirstAnsweringCandidate(t *testing.T) { + host, ca := tlsEndpoint(t, `{"name":"deckhouse-cli","tags":[]}`) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, host), + "ca.crt": string(ca), + "publicEndpoint": "https://registry-packages-proxy.example.com", + })) + + client, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", false) + require.NoError(t, err) + assert.Equal(t, "https://"+host, client.baseURL, "a verifiable master endpoint wins over the public host") +} + +func TestNewClusterClientReportsEveryRejectedCandidate(t *testing.T) { + strangerHost, _ := tlsEndpoint(t, `{}`) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, strangerHost), + "ca.crt": "not a certificate", + })) + + _, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", false) + require.Error(t, err) + assert.ErrorIs(t, err, ErrEndpointDiscovery) + assert.Contains(t, err.Error(), strangerHost, "the message names what was tried") + assert.Contains(t, err.Error(), sourceMaster) +} + +func TestNewClusterClientInsecureAcceptsAnyCandidate(t *testing.T) { + // --rpp-insecure-skip-tls-verify replaces the published CA, which is how the + // pod fallback stays usable on a cluster that publishes no certificate. + strangerHost, _ := tlsEndpoint(t, `{"name":"deckhouse-cli","tags":["v0.13.1"]}`) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, strangerHost), + })) + + client, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", true) + require.NoError(t, err) + + tags, err := client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err) + assert.Equal(t, []string{"v0.13.1"}, tags) +} + +func TestNewClusterClientHealsFromStaleCAAtDiscovery(t *testing.T) { + // A converge can rotate the serving certificate and the published CA between + // the ConfigMap read and the probe. The first read returns the stale CA, + // every later one the current bundle - the probe retry must pick it up. + host, currentCA := tlsEndpoint(t, `{"name":"deckhouse-cli","tags":["v0.13.1"]}`) + _, staleCA := tlsEndpoint(t, `{}`) + + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, host), + "ca.crt": string(currentCA), + })) + + reads := 0 + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + reads++ + if reads == 1 { + return true, proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, host), + "ca.crt": string(staleCA), + }), nil + } + + // Fall through to the tracker, which holds the current CA. + return false, nil, nil + }) + + client, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", false) + require.NoError(t, err, "the probe fails against the stale CA, re-reads the published one and retries") + assert.Equal(t, "https://"+host, client.baseURL) + + tags, err := client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err) + assert.Equal(t, []string{"v0.13.1"}, tags) +} + +func TestClientHealsFromTrustRotationBetweenRequests(t *testing.T) { + // The platform reissues the proxy certificate together with its root, so a + // client built before the rotation holds a CA the server no longer matches. + // On the next handshake it must re-read the published CA and retry. + certA, keyA := selfSignedLocalhostCert(t) + certB, keyB := selfSignedLocalhostCert(t) + + pairA, err := tls.X509KeyPair(certA, keyA) + require.NoError(t, err) + pairB, err := tls.X509KeyPair(certB, keyB) + require.NoError(t, err) + + var serving atomic.Pointer[tls.Certificate] + + serving.Store(&pairA) + + srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + _, _ = io.WriteString(w, `{"name":"deckhouse-cli","tags":["v0.13.1"]}`) + })) + // GetConfigForClient, not GetCertificate: the client connects by IP, so the + // hello carries no SNI, and without SNI a non-empty Certificates list (which + // StartTLS enforces) would win over GetCertificate. + srv.TLS = &tls.Config{ + Certificates: []tls.Certificate{pairA}, + GetConfigForClient: func(*tls.ClientHelloInfo) (*tls.Config, error) { + return &tls.Config{Certificates: []tls.Certificate{*serving.Load()}, MinVersion: tls.VersionTLS12}, nil + }, + MinVersion: tls.VersionTLS12, + } + srv.StartTLS() + t.Cleanup(srv.Close) + + parsed, err := url.Parse(srv.URL) + require.NoError(t, err) + + var mu sync.Mutex + + publishedCA := certA + + kube := fake.NewSimpleClientset() + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + mu.Lock() + defer mu.Unlock() + + return true, proxyConfigMap(map[string]string{ + "endpoints": fmt.Sprintf(`[%q]`, parsed.Host), + "ca.crt": string(publishedCA), + }), nil + }) + + client, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), "", "", false) + require.NoError(t, err) + + _, err = client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err, "sanity: the pre-rotation trust works") + + // Rotate: the server presents certB, the cluster publishes certB. + mu.Lock() + publishedCA = certB + mu.Unlock() + serving.Store(&pairB) + srv.CloseClientConnections() + + tags, err := client.ListTags(context.Background(), CLIImage()) + require.NoError(t, err, "the client re-reads the published CA and retries once") + assert.Equal(t, []string{"v0.13.1"}, tags) +} + +func TestNewClusterClientRejectsContradictoryFlags(t *testing.T) { + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["10.0.0.1:4219","10.0.0.2:4219"]`, + "ca.crt": "CA-PEM", + })) + + _, err := NewClusterClient(context.Background(), kube, &rest.Config{}, dkplog.NewNop(), + "", writeTempCA(t, []byte("CA-PEM")), true) + require.Error(t, err) + + assert.ErrorIs(t, err, ErrUnsupportedConfig, "the caller matches on this") + assert.NotErrorIs(t, err, ErrEndpointDiscovery, "the endpoints are fine, the flags are not") + assert.NotContains(t, err.Error(), "10.0.0.2", "the flags are judged once, not per candidate") +} + +func TestNewClusterClientUsesExplicitEndpointWithoutDiscovery(t *testing.T) { + host, ca := tlsEndpoint(t, `{"name":"deckhouse-cli","tags":[]}`) + + caFile := writeTempCA(t, ca) + + // An empty cluster offers no candidates, so reaching the endpoint proves + // discovery was skipped entirely. + client, err := NewClusterClient(context.Background(), fake.NewSimpleClientset(), &rest.Config{}, + dkplog.NewNop(), "https://"+host, caFile, false) + require.NoError(t, err) + assert.Equal(t, "https://"+host, client.baseURL) +} + +func writeTempCA(t *testing.T, caPEM []byte) string { + t.Helper() + + path := t.TempDir() + "/ca.pem" + require.NoError(t, os.WriteFile(path, caPEM, 0o600)) + + return path +} diff --git a/internal/rpp/endpoint.go b/internal/rpp/endpoint.go index f9c6aab8d..08b726548 100644 --- a/internal/rpp/endpoint.go +++ b/internal/rpp/endpoint.go @@ -48,45 +48,104 @@ const ( proxyScheme = "https" ) +// Candidate sources, as reported in logs and in the failure message. +const ( + sourceMaster = "master" + sourcePublic = "public" + sourcePod = "pod" +) + // errIngressUnusable marks the Ingress lookup as "the API answered, but the -// Ingress is absent or has no host" - the only case where the in-cluster pod -// fallback is worth trying. A transport/TLS/auth failure reaching the API is NOT -// this: it would fail pod listing identically, so it is surfaced as-is. +// Ingress is absent or has no host". The public candidate is then simply absent. var errIngressUnusable = errors.New("registry-packages-proxy ingress unusable") -// chooseDiscoveredEndpoint resolves the proxy endpoint when none was given -// explicitly. It PREFERS the public Ingress (a valid TLS certificate, reachable -// from a workstation) and falls back to in-cluster pod IPs (which need -// --rpp-insecure-skip-tls-verify and cluster-network reachability). The second -// return value names the source ("ingress" / "pod") for logging. +// candidate is one way to reach the proxy: an endpoint plus the trust anchor that +// verifies it. An empty caPEM means "verify against the system roots". +type candidate struct { + endpoint string + source string + caPEM []byte +} + +// discoverCandidates returns the ways to reach the proxy, best first: // -// Only an unusable Ingress (see errIngressUnusable) triggers the pod fallback. -// Any other error is an API-leg failure, surfaced as ErrEndpointDiscovery. -func chooseDiscoveredEndpoint(ctx context.Context, kube kubernetes.Interface) (string, string, error) { - endpoint, err := discoverIngressEndpoint(ctx, kube) - if err == nil { - return endpoint, "ingress", nil +// 1. master addresses published by the module, verifiable with the published CA: +// they need no public domain, no DNS record and no external certificate authority +// 2. the public endpoint, verifiable with the system roots: the only way in for a +// client with no network path to the master nodes +// 3. pod IPs, kept for clusters that publish no configuration yet; their +// certificate carries no verifiable CA, so this one needs insecure TLS +// +// A denied read is not a failure: it drops the candidates it would have produced +// and leaves the rest, so a narrowly permitted identity still gets a usable path. +func discoverCandidates(ctx context.Context, kube kubernetes.Interface) ([]candidate, error) { + config, err := readClusterConfig(ctx, kube) + if err != nil { + return nil, err } - if !errors.Is(err, errIngressUnusable) { - return "", "", fmt.Errorf("%w: %w", ErrEndpointDiscovery, err) + candidates := make([]candidate, 0, len(config.endpoints)+2) + + for _, endpoint := range config.endpoints { + candidates = append(candidates, candidate{endpoint: endpoint, source: sourceMaster, caPEM: config.caPEM}) + } + + public, err := publicEndpoint(ctx, kube, config) + if err != nil { + return nil, err } - endpoint, err = discoverEndpoint(ctx, kube) + if public != "" { + candidates = append(candidates, candidate{endpoint: public, source: sourcePublic}) + } + + pods, err := discoverPodEndpoints(ctx, kube) if err != nil { - return "", "", fmt.Errorf("%w: %w", ErrEndpointDiscovery, err) + return nil, err + } + + for _, endpoint := range pods { + candidates = append(candidates, candidate{endpoint: endpoint, source: sourcePod}) } - return endpoint, "pod", nil + if len(candidates) == 0 { + return nil, errors.New("no registry-packages-proxy endpoint found in the cluster") + } + + return candidates, nil +} + +// publicEndpoint prefers the endpoint the module published and reads the Ingress +// only when there is none, so a cluster with the published config needs no +// permission on Ingresses at all. +func publicEndpoint(ctx context.Context, kube kubernetes.Interface, config clusterConfig) (string, error) { + if config.publicEndpoint != "" { + return config.publicEndpoint, nil + } + + endpoint, err := discoverIngressEndpoint(ctx, kube) + switch { + case err == nil: + return endpoint, nil + case errors.Is(err, errIngressUnusable), isReadDenied(err): + return "", nil + default: + return "", err + } } // discoverIngressEndpoint returns the public proxy endpoint (https://) taken -// from the registry-packages-proxy Ingress. This path has a valid TLS certificate -// and is reachable from outside the cluster - the right default for a workstation. +// from the registry-packages-proxy Ingress. // -// An absent Ingress or one with no host yields errIngressUnusable, signalling the -// caller to try the in-cluster pod fallback. Any other error is returned raw so -// the caller can surface the API-leg failure instead of falling back. +// The scheme is always https and is deliberately not taken from the Ingress TLS +// block. Every request carries the kubeconfig bearer token, so a plain http +// endpoint would put it on the wire in the clear. The TLS block also says nothing +// about a TLS terminator standing in front of the cluster, where https is the +// right choice even though the Ingress itself serves http. An endpoint that cannot +// complete a handshake is simply skipped, see discoverCandidates. +// +// An absent Ingress or one with no host yields errIngressUnusable. Any other error +// is returned raw so the caller can surface the API-leg failure. func discoverIngressEndpoint(ctx context.Context, kube kubernetes.Interface) (string, error) { ingress, err := kube.NetworkingV1().Ingresses(proxyNamespace).Get(ctx, proxyIngressName, metav1.GetOptions{}) if err != nil { @@ -106,20 +165,23 @@ func discoverIngressEndpoint(ctx context.Context, kube kubernetes.Interface) (st return "", fmt.Errorf("%w: ingress %q has no host", errIngressUnusable, proxyIngressName) } -// discoverEndpoint returns a proxy endpoint base URL from the first serving -// registry-packages-proxy pod (running, ready, with an IP), joined to the proxy -// port. Terminating and not-yet-ready pods are skipped. No failover: one serving -// pod is enough. +// discoverPodEndpoints returns an endpoint per serving proxy pod. Terminating and +// not-yet-ready pods are skipped. // -// This is a master-node pod IP, reachable from inside the cluster network. A -// workstation outside the cluster usually cannot reach it and should pass an -// explicit endpoint (for example the public Ingress) instead. -func discoverEndpoint(ctx context.Context, kube kubernetes.Interface) (string, error) { +// These are master-node pod IPs, reachable from inside the cluster network. A +// workstation outside the cluster usually cannot reach them. +func discoverPodEndpoints(ctx context.Context, kube kubernetes.Interface) ([]string, error) { pods, err := kube.CoreV1().Pods(proxyNamespace).List(ctx, metav1.ListOptions{LabelSelector: proxyPodSelector}) if err != nil { - return "", fmt.Errorf("list registry-packages-proxy pods: %w", err) + if isReadDenied(err) { + return nil, nil + } + + return nil, fmt.Errorf("list registry-packages-proxy pods: %w", err) } + endpoints := make([]string, 0, len(pods.Items)) + for i := range pods.Items { pod := &pods.Items[i] if !podIsServing(pod) { @@ -127,11 +189,10 @@ func discoverEndpoint(ctx context.Context, kube kubernetes.Interface) (string, e } base := url.URL{Scheme: proxyScheme, Host: net.JoinHostPort(pod.Status.PodIP, strconv.Itoa(proxyPort))} - - return base.String(), nil + endpoints = append(endpoints, base.String()) } - return "", fmt.Errorf("no ready registry-packages-proxy pods found in namespace %q", proxyNamespace) + return endpoints, nil } // podIsServing reports whether the pod is a usable proxy endpoint: running, not @@ -149,3 +210,11 @@ func podIsServing(pod *corev1.Pod) bool { return false } + +// isReadDenied reports whether the API refused this read to a valid identity. +// Such a read yields no candidates, while the rest of discovery carries on. +// A 401 is not a denied read: rejected credentials fail every cluster call, +// so it surfaces as an error instead of thinning the candidate list. +func isReadDenied(err error) bool { + return apierrors.IsForbidden(err) +} diff --git a/internal/rpp/endpoint_test.go b/internal/rpp/endpoint_test.go index 5b02829bf..51ae48286 100644 --- a/internal/rpp/endpoint_test.go +++ b/internal/rpp/endpoint_test.go @@ -25,8 +25,10 @@ import ( "github.com/stretchr/testify/require" corev1 "k8s.io/api/core/v1" networkingv1 "k8s.io/api/networking/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/apimachinery/pkg/runtime" + "k8s.io/apimachinery/pkg/runtime/schema" "k8s.io/client-go/kubernetes/fake" clienttesting "k8s.io/client-go/testing" ) @@ -69,7 +71,7 @@ func conditionStatus(ready bool) corev1.ConditionStatus { return corev1.ConditionFalse } -func TestDiscoverEndpoint(t *testing.T) { +func TestDiscoverPodEndpointsKeepsOnlyServingPods(t *testing.T) { kube := fake.NewSimpleClientset( proxyPod("not-ready", "10.0.0.2", corev1.PodRunning, false, false), proxyPod("terminating", "10.0.0.3", corev1.PodRunning, true, true), @@ -78,18 +80,30 @@ func TestDiscoverEndpoint(t *testing.T) { proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false), ) - endpoint, err := discoverEndpoint(context.Background(), kube) + endpoints, err := discoverPodEndpoints(context.Background(), kube) require.NoError(t, err) - assert.Equal(t, "https://10.0.0.1:4219", endpoint) + assert.Equal(t, []string{"https://10.0.0.1:4219"}, endpoints) } -func TestDiscoverEndpointNoneServing(t *testing.T) { +func TestDiscoverPodEndpointsNoneServing(t *testing.T) { kube := fake.NewSimpleClientset( proxyPod("not-ready", "10.0.0.2", corev1.PodRunning, false, false), ) - _, err := discoverEndpoint(context.Background(), kube) - require.Error(t, err) + endpoints, err := discoverPodEndpoints(context.Background(), kube) + require.NoError(t, err, "no serving pod is not a failure, it just yields no candidate") + assert.Empty(t, endpoints) +} + +func TestDiscoverPodEndpointsToleratesDeniedList(t *testing.T) { + kube := fake.NewSimpleClientset() + kube.PrependReactor("list", "pods", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden(schema.GroupResource{Resource: "pods"}, "", errors.New("no access")) + }) + + endpoints, err := discoverPodEndpoints(context.Background(), kube) + require.NoError(t, err, "an identity without pod list rights still uses the other candidates") + assert.Empty(t, endpoints) } func TestDiscoverIngressEndpoint(t *testing.T) { @@ -103,7 +117,7 @@ func TestDiscoverIngressEndpoint(t *testing.T) { func TestDiscoverIngressEndpointAbsent(t *testing.T) { _, err := discoverIngressEndpoint(context.Background(), fake.NewSimpleClientset()) require.Error(t, err) - assert.ErrorIs(t, err, errIngressUnusable, "an absent Ingress signals the pod fallback") + assert.ErrorIs(t, err, errIngressUnusable, "an absent Ingress just yields no public candidate") } func TestDiscoverIngressEndpointNoHost(t *testing.T) { @@ -111,47 +125,109 @@ func TestDiscoverIngressEndpointNoHost(t *testing.T) { _, err := discoverIngressEndpoint(context.Background(), kube) require.Error(t, err) - assert.ErrorIs(t, err, errIngressUnusable, "an Ingress with no host signals the pod fallback") + assert.ErrorIs(t, err, errIngressUnusable, "an Ingress with no host just yields no public candidate") } -func TestChooseDiscoveredEndpointPrefersIngress(t *testing.T) { +func TestDiscoverCandidatesPrefersPublishedMasters(t *testing.T) { + // The published masters come first: they need no public domain and no external + // certificate authority. The public host and the pods follow as fallbacks. kube := fake.NewSimpleClientset( - proxyIngress("registry-packages-proxy.example.com"), + proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219"]`, + "ca.crt": "CA-PEM", + "publicEndpoint": "https://registry-packages-proxy.example.com", + }), proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false), ) - endpoint, source, err := chooseDiscoveredEndpoint(context.Background(), kube) + candidates, err := discoverCandidates(context.Background(), kube) require.NoError(t, err) - assert.Equal(t, "https://registry-packages-proxy.example.com", endpoint) - assert.Equal(t, "ingress", source) + require.Len(t, candidates, 3) + + assert.Equal(t, "https://192.168.0.1:4219", candidates[0].endpoint) + assert.Equal(t, sourceMaster, candidates[0].source) + assert.Equal(t, []byte("CA-PEM"), candidates[0].caPEM) + + assert.Equal(t, "https://registry-packages-proxy.example.com", candidates[1].endpoint) + assert.Equal(t, sourcePublic, candidates[1].source) + assert.Empty(t, candidates[1].caPEM, "the public host is verified with the system roots") + + assert.Equal(t, "https://10.0.0.1:4219", candidates[2].endpoint) + assert.Equal(t, sourcePod, candidates[2].source) } -func TestChooseDiscoveredEndpointFallsBackToPods(t *testing.T) { - // No Ingress -> fall back to in-cluster pod IPs. +func TestDiscoverCandidatesReadsIngressWhenNothingPublished(t *testing.T) { + // A cluster running an older module version: no published config, so the + // public endpoint comes from the Ingress object as before. kube := fake.NewSimpleClientset( + proxyIngress("registry-packages-proxy.example.com"), proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false), ) - endpoint, source, err := chooseDiscoveredEndpoint(context.Background(), kube) + candidates, err := discoverCandidates(context.Background(), kube) require.NoError(t, err) - assert.Equal(t, "https://10.0.0.1:4219", endpoint) - assert.Equal(t, "pod", source) + require.Len(t, candidates, 2) + assert.Equal(t, sourcePublic, candidates[0].source) + assert.Equal(t, "https://registry-packages-proxy.example.com", candidates[0].endpoint) + assert.Equal(t, sourcePod, candidates[1].source) } -func TestChooseDiscoveredEndpointSurfacesAPIFailure(t *testing.T) { - // A transport/TLS failure reaching the API (not an absent Ingress) is surfaced - // as ErrEndpointDiscovery, not masked by falling back to pod listing - even - // when a serving pod exists. - kube := fake.NewSimpleClientset( - proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false), - ) +func TestDiscoverCandidatesSkipsIngressWhenPublished(t *testing.T) { + // With the published public endpoint the Ingress is never read, so the CLI + // needs no permission on Ingresses. + kube := fake.NewSimpleClientset(proxyConfigMap(map[string]string{ + "endpoints": `["192.168.0.1:4219"]`, + "publicEndpoint": "https://published.example.com", + })) + kube.PrependReactor("get", "ingresses", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, errors.New("ingresses must not be read") + }) + + candidates, err := discoverCandidates(context.Background(), kube) + require.NoError(t, err) + require.Len(t, candidates, 2) + assert.Equal(t, "https://published.example.com", candidates[1].endpoint) +} + +func TestDiscoverCandidatesToleratesDeniedIngressRead(t *testing.T) { + kube := fake.NewSimpleClientset(proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false)) kube.PrependReactor("get", "ingresses", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewForbidden(schema.GroupResource{Resource: "ingresses"}, "", errors.New("no access")) + }) + + candidates, err := discoverCandidates(context.Background(), kube) + require.NoError(t, err) + require.Len(t, candidates, 1) + assert.Equal(t, sourcePod, candidates[0].source) +} + +func TestDiscoverCandidatesSurfacesUnauthorized(t *testing.T) { + // Rejected credentials fail every candidate source the same way, so a 401 + // stops discovery instead of thinning the candidate list. + kube := fake.NewSimpleClientset(proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false)) + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { + return true, nil, apierrors.NewUnauthorized("credentials rejected") + }) + + _, err := discoverCandidates(context.Background(), kube) + require.Error(t, err) + assert.Contains(t, err.Error(), "credentials rejected") +} + +func TestDiscoverCandidatesSurfacesAPIFailure(t *testing.T) { + // A transport or TLS failure reaching the API is surfaced instead of being + // masked: every candidate source would fail the same way. + kube := fake.NewSimpleClientset(proxyPod("serving", "10.0.0.1", corev1.PodRunning, true, false)) + kube.PrependReactor("get", "configmaps", func(clienttesting.Action) (bool, runtime.Object, error) { return true, nil, errors.New("tls: failed to verify certificate") }) - _, _, err := chooseDiscoveredEndpoint(context.Background(), kube) + _, err := discoverCandidates(context.Background(), kube) require.Error(t, err) - assert.ErrorIs(t, err, ErrEndpointDiscovery) - assert.NotErrorIs(t, err, errIngressUnusable) assert.Contains(t, err.Error(), "tls: failed to verify certificate") } + +func TestDiscoverCandidatesWithoutAnyEndpoint(t *testing.T) { + _, err := discoverCandidates(context.Background(), fake.NewSimpleClientset()) + require.Error(t, err, "an empty cluster offers no way to reach the proxy") +} diff --git a/internal/rpp/transport.go b/internal/rpp/transport.go index 3fd866196..3ec5d4e7f 100644 --- a/internal/rpp/transport.go +++ b/internal/rpp/transport.go @@ -19,6 +19,7 @@ package rpp import ( "crypto/tls" "crypto/x509" + "errors" "fmt" "net/http" "time" @@ -106,6 +107,51 @@ func certPoolWith(caPEM []byte) (*x509.CertPool, error) { return pool, nil } +// isCertVerificationError reports whether err is a server-certificate +// verification failure - the one transport error a trust refresh can fix. +// Timeouts, refusals and protocol errors stay final. +func isCertVerificationError(err error) bool { + var verification *tls.CertificateVerificationError + if errors.As(err, &verification) { + return true + } + + // Platform verifiers may surface the x509 error without the tls wrapper. + var unknownAuthority x509.UnknownAuthorityError + + var hostname x509.HostnameError + + return errors.As(err, &unknownAuthority) || errors.As(err, &hostname) +} + +// buildProbeClient builds the credential-free client behind the reachability +// probe. Server trust mirrors the main client - the supplied CA on top of the +// system roots, or no verification when insecure - but no identity fields are +// attached, so a probed endpoint learns nothing about the caller. +func buildProbeClient(o options) (*http.Client, error) { + tlsConfig := &tls.Config{} + + switch { + case o.insecure: + tlsConfig.InsecureSkipVerify = true + case len(o.caData) > 0: + pool, err := certPoolWith(o.caData) + if err != nil { + return nil, err + } + + tlsConfig.RootCAs = pool + } + + return &http.Client{ + Transport: &http.Transport{ + TLSClientConfig: tlsConfig, + TLSHandshakeTimeout: tlsHandshakeTimeout, + ResponseHeaderTimeout: responseHeaderTimeout, + }, + }, nil +} + // withTunedTransport returns a transport wrapper that, innermost to outermost: // - clones the base *http.Transport // - bounds connection setup (TLS handshake, response-header wait) diff --git a/internal/selfupdate/cmd/errdetect/diagnose.go b/internal/selfupdate/cmd/errdetect/diagnose.go index 45f819cb2..fa4a2513a 100644 --- a/internal/selfupdate/cmd/errdetect/diagnose.go +++ b/internal/selfupdate/cmd/errdetect/diagnose.go @@ -36,8 +36,9 @@ func Diagnose(err error) *diagnostic.HelpfulError { switch { case errors.Is(err, rpp.ErrUnauthorized): return help(err, "registry-packages-proxy: unauthorized (401)", - "no accepted Bearer token (a client-certificate kubeconfig is not enough)", - "use a kubeconfig with an OIDC token (Kubeconfig Generator or 'd8 login')") + "the kubeconfig credential was rejected: an invalid or expired token, or a client certificate the proxy does not trust", + "use a kubeconfig with an OIDC token", + "a client-certificate kubeconfig works only when the certificate is signed by the cluster CA") case errors.Is(err, rpp.ErrForbidden): return help(err, "registry-packages-proxy: forbidden (403)", "the identity may not download the CLI", @@ -52,11 +53,11 @@ func Diagnose(err error) *diagnostic.HelpfulError { "the proxy could not reach the backing registry", "retry shortly, or check the registry-packages-proxy pods in d8-cloud-instance-manager") case errors.Is(err, rpp.ErrEndpointDiscovery): - return help(err, "registry-packages-proxy: endpoint discovery via the Kubernetes API failed", - "discovery reaches the proxy through your kubeconfig's API server; that server was unreachable or presented an invalid certificate", - "this is the Kubernetes API endpoint (kubeconfig 'server:'), not the proxy; confirm it is reachable with a valid TLS certificate for that host", - "skip discovery: pass --rpp-endpoint https://registry-packages-proxy. (or set D8_RPP_ENDPOINT)", - "on a master node, point the kubeconfig at the local API (https://127.0.0.1:6445, CA /etc/kubernetes/pki/ca.crt) with an OIDC token") + return help(err, "registry-packages-proxy: no usable endpoint found", + "either the cluster could not be asked where the proxy is, or none of the endpoints it offered answered", + "the message above names what was tried: a Kubernetes API failure means the kubeconfig 'server:' is unreachable or its certificate is invalid", + "a per-endpoint failure means the proxy was not reachable there - master addresses need access to port 4219, the public host needs a valid certificate", + "skip discovery: pass --rpp-endpoint (or set D8_RPP_ENDPOINT), adding --rpp-ca-file when its CA is not publicly trusted") default: return nil } diff --git a/internal/selfupdate/cmd/errdetect/diagnose_test.go b/internal/selfupdate/cmd/errdetect/diagnose_test.go index 80a315d7e..b20ad35bc 100644 --- a/internal/selfupdate/cmd/errdetect/diagnose_test.go +++ b/internal/selfupdate/cmd/errdetect/diagnose_test.go @@ -40,7 +40,7 @@ func TestDiagnose(t *testing.T) { {"403", rpp.ErrForbidden, "forbidden (403)", "cli-download"}, {"404", rpp.ErrNotFound, "version not found (404)", "d8 cli versions"}, {"5xx", rpp.ErrUpstream, "upstream error (5xx)", "registry-packages-proxy pods"}, - {"discovery", rpp.ErrEndpointDiscovery, "endpoint discovery via the Kubernetes API failed", "--rpp-endpoint"}, + {"discovery", rpp.ErrEndpointDiscovery, "no usable endpoint found", "--rpp-endpoint"}, } for _, tc := range cases {