Add replicas and sessionStorage to MCPRemoteProxy for HA

[aron-muon]

Summary

When MCPRemoteProxy runs with multiple replicas behind a load balancer that doesn't preserve client-IP affinity (e.g. AWS ALB across multiple AZs), every non-initialize request fails with Session not found. The transparent proxy validates Mcp-Session-Id against pod-local in-memory state on every hop — transparent_proxy.go even calls this out:

// Guard: reject non-initialize requests with unknown session IDs.
// When multiple proxyrunner replicas share a Redis session store,
// a valid session will always be found.

On top of that, MCPRemoteProxy had no way to run multiple replicas in the first place: the operator hardcoded replicas: 1 on Deployment creation, so operators had to pin counts externally (e.g. a min=max HPA workaround that exploits the reconciler preserving the live count).

This PR completes the HA story by mirroring the two pieces MCPServer (#4368) and VirtualMCPServer (#4367) already have: a spec.replicas field and Redis-backed shared session storage.

What changed:

• Add MCPRemoteProxySpec.Replicas — same *int32 semantics as VirtualMCPServer.spec.replicas: nil leaves Deployment.Spec.Replicas unset (apiserver defaults to 1; an HPA or other external controller owns scaling and is never fought by the reconciler), non-nil is operator-authoritative — set on create, drift-checked in deploymentNeedsUpdate, synced on update.
• Add MCPRemoteProxySpec.SessionStorage — same SessionStorageConfig shape used by MCPServer and VirtualMCPServer. populateScalingConfigForRemoteProxy writes the non-sensitive Redis parameters into runner.ScalingConfig.SessionRedis (before PopulateMiddlewareConfigs, since rate limiting reads SessionRedis); buildRedisPasswordEnvVarForRemoteProxy injects THV_SESSION_REDIS_PASSWORD via SecretKeyRef so the password never lands in the RunConfig ConfigMap.
• Surface the shared SessionStorageWarning condition when replicas > 1 without Redis session storage — status-condition parity with the MCPServer and VirtualMCPServer validators.
• Cross-reference sessionAffinity and sessionStorage in the field docs — Redis-backed storage delivers HA only with sessionAffinity: None, and None without Redis is the exact failure mode this PR fixes.

The change is intentionally a near-verbatim mirror of the MCPServer / VirtualMCPServer implementations to keep review easy — it's the same pattern reviewers have already accepted twice.

Type of change

• Bug fix
• New feature

Test plan

• Unit tests — full suite via task test. New coverage: TestPopulateScalingConfigForRemoteProxy (4 cases incl. password-never-serialized), TestBuildRedisPasswordEnvVarForRemoteProxy (4 cases), TestDeploymentForMCPRemoteProxyReplicas (nil/0/3), TestMCPRemoteProxyDeploymentNeedsUpdateReplicas (drift matrix), TestMCPRemoteProxyEnsureDeploymentReplicaSync (fake-cluster preserve-vs-sync), TestValidateSessionStorageForReplicasRemoteProxy (condition matrix)
• Build verification — task build and task build-operator
• CRDs/docs regenerated — task operator-generate, task operator-manifests, task crdref-gen (controller-gen v0.17.3)

API Compatibility

• This PR does not break the v1beta1 API. Both added fields (spec.replicas, spec.sessionStorage) are optional; when omitted, behavior is identical to today (the only delta: a freshly created Deployment carries no explicit replicas: 1 and relies on the apiserver default of 1).

Changes

File	Lines	Change
cmd/thv-operator/api/v1beta1/mcpremoteproxy_types.go	+39	Add Replicas *int32 and SessionStorage *SessionStorageConfig with cross-referenced docs
cmd/thv-operator/api/v1beta1/zz_generated.deepcopy.go	+10	Generated DeepCopy for both fields (controller-gen v0.17.3)
cmd/thv-operator/controllers/mcpremoteproxy_controller.go	+53	Replica drift check in deploymentNeedsUpdate; non-nil sync / nil preserve on update; validateSessionStorageForReplicas advisory condition
cmd/thv-operator/controllers/mcpremoteproxy_deployment.go	+36	Deployment.Spec.Replicas from spec (was hardcoded 1); Redis password env var via SecretKeyRef
cmd/thv-operator/controllers/mcpremoteproxy_runconfig.go	+29	populateScalingConfigForRemoteProxy, called before PopulateMiddlewareConfigs
cmd/thv-operator/controllers/mcpremoteproxy_replicas_test.go	+255	Replica create/drift/sync + condition tests
cmd/thv-operator/controllers/mcpremoteproxy_deployment_test.go	+66	Redis password env var tests; nil-replicas assertion
cmd/thv-operator/controllers/mcpremoteproxy_runconfig_test.go	+86	TestPopulateScalingConfigForRemoteProxy
deploy/charts/operator-crds/{files,templates}/...mcpremoteproxies.yaml	+162 ea	Generated replicas + sessionStorage schema
docs/operator/crd-api.md	+4/−1	Generated CRD API reference

Does this introduce a user-facing change?

Yes. Two new optional fields on MCPRemoteProxy:

• spec.replicas — desired proxy replica count. Leave unset to keep scaling external (HPA-compatible: the operator never overwrites the live count); set it to make the operator authoritative.
• spec.sessionStorage — Redis-backed shared session state across replicas, same shape as MCPServer.spec.sessionStorage. For HA set replicas >= 2, sessionStorage.provider: redis, and sessionAffinity: None; a SessionStorageWarning status condition flags replicas > 1 without Redis.

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 69.73%. Comparing base (7d1ebf5) to head (20cfd13).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5237      +/-   ##
==========================================
+ Coverage   69.71%   69.73%   +0.02%     
==========================================
  Files         645      645              
  Lines       65598    65651      +53     
==========================================
+ Hits        45729    45782      +53     
- Misses      16522    16523       +1     
+ Partials     3347     3346       -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Wires Redis-backed session storage into MCPRemoteProxy so multi-replica proxy deployments behind load balancers without client-IP affinity can share session state, closing the parity gap with MCPServer (#4368) and VirtualMCPServer (#4367).

Changes:

• Adds MCPRemoteProxySpec.SessionStorage (mirrors the existing SessionStorageConfig schema) and regenerates DeepCopy/CRD/docs.
• Adds populateScalingConfigForRemoteProxy to copy address/db/keyPrefix into runner.ScalingConfig.SessionRedis (password intentionally excluded).
• Adds buildRedisPasswordEnvVarForRemoteProxy to inject THV_SESSION_REDIS_PASSWORD via SecretKeyRef on the proxy Deployment, with corresponding unit tests.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
cmd/thv-operator/api/v1beta1/mcpremoteproxy_types.go	New optional SessionStorage field.
cmd/thv-operator/api/v1beta1/zz_generated.deepcopy.go	Generated DeepCopy for the new field.
cmd/thv-operator/controllers/mcpremoteproxy_runconfig.go	Populates ScalingConfig.SessionRedis from spec.
cmd/thv-operator/controllers/mcpremoteproxy_runconfig_test.go	Unit tests for the new populator.
cmd/thv-operator/controllers/mcpremoteproxy_deployment.go	Injects Redis password env var via SecretKeyRef.
cmd/thv-operator/controllers/mcpremoteproxy_deployment_test.go	Unit tests covering env-var injection cases.
deploy/charts/operator-crds/files/crds/toolhive.stacklok.dev_mcpremoteproxies.yaml	Regenerated CRD with sessionStorage subschema.
deploy/charts/operator-crds/templates/toolhive.stacklok.dev_mcpremoteproxies.yaml	Template counterpart of the regenerated CRD.
docs/operator/crd-api.md	Doc additions for the new field.

Files not reviewed (1)

• cmd/thv-operator/api/v1beta1/zz_generated.deepcopy.go: Language not supported

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aron-muon]

Fixed in 4c065e7.

[yrobla]

Functional Review — Wire SessionStorage into MCPRemoteProxy

Context

The fix addresses a real architectural gap. When MCPRemoteProxy runs with >1 replica behind a load balancer that lacks client-IP affinity (AWS ALB across AZs, GCP NEGs, etc.), every non-initialize request fails with Session not found because the transparent proxy validates Mcp-Session-Id against pod-local in-memory state. MCPServer and VirtualMCPServer already had the Redis-backed escape hatch; MCPRemoteProxy was the only component missing it.

What changed and does it make sense?

The PR adds three wiring pieces:

1. MCPRemoteProxySpec.SessionStorage — same optional field shape used by MCPServer/VirtualMCPServer. Additive and backward-compatible. ✅
2. populateScalingConfigForRemoteProxy — writes non-sensitive Redis params into runner.ScalingConfig.SessionRedis in the RunConfig ConfigMap. ✅
3. buildRedisPasswordEnvVarForRemoteProxy — injects THV_SESSION_REDIS_PASSWORD as a SecretKeyRef pod env var, so the password never lands in the ConfigMap. ✅

The security split (connection params in ConfigMap, password as pod secret) matches the established pattern and is correct.

Bug: populateScalingConfigForRemoteProxy is called after PopulateMiddlewareConfigs

In mcpserver_runconfig.go the analogous call order is:

// Must run before PopulateMiddlewareConfigs because rate limiting reads SessionRedis.
populateScalingConfig(runConfig, m)
if err := runner.PopulateMiddlewareConfigs(runConfig); err != nil { ... }

PopulateMiddlewareConfigs → addRateLimitMiddleware reads config.ScalingConfig.SessionRedis and errors when nil and rate limiting is configured. The PR reverses this in mcpremoteproxy_runconfig.go. MCPRemoteProxy has no rateLimiting field today so this is benign now, but it diverges from the documented invariant and will break the moment rate limiting is added to MCPRemoteProxy. See inline comment for suggested fix.

Design note: sessionAffinity / sessionStorage interaction needs documentation

MCPRemoteProxySpec.SessionAffinity defaults to ClientIP. The two fields now coexist without guidance:

sessionAffinity	sessionStorage	Result
ClientIP (default)	nil / memory	✅ works (pod-sticky routing)
ClientIP	redis	Redis wired but Service stickiness makes it a no-op
None	redis	✅ intended HA configuration
None	nil / memory	❌ Session not found on any cross-pod request

The last row is the exact failure mode this PR fixes — but users can still hit it if they set sessionAffinity: None without configuring Redis. The SessionStorage field comment should mention setting sessionAffinity: None for the HA case, or a kubebuilder XValidation could enforce it at admission time (similar to how MCPServer validates that rate limiting requires Redis).

Alternative approaches

• Consistent hashing at the ingress (hash Mcp-Session-Id to a specific pod): avoids Redis but requires custom LB config outside ToolHive's control.
• Session migration between pods: more complex with no clear benefit over Redis here.

Redis is the right call — same decision already made for MCPServer and VirtualMCPServer, and the operational overhead is low (one Redis shared across all three CRD types).

Summary

Finding	Severity	Action
populateScalingConfigForRemoteProxy called after PopulateMiddlewareConfigs	Medium — latent bug	Reorder above PopulateMiddlewareConfigs
No guidance on sessionAffinity + sessionStorage interaction	Low — docs/UX	Add cross-reference in field comments
Implementation correctness, security split, API compatibility	✅	No action needed

[yrobla]

Ordering bug (latent): populateScalingConfigForRemoteProxy must be called before PopulateMiddlewareConfigs, not after.

PopulateMiddlewareConfigs → addRateLimitMiddleware reads config.ScalingConfig.SessionRedis and returns an error when it is nil and a rate-limit config is present. The equivalent call in mcpserver_runconfig.go explicitly documents this:

// Must run before PopulateMiddlewareConfigs because rate limiting reads SessionRedis.
populateScalingConfig(runConfig, m)
if err := runner.PopulateMiddlewareConfigs(runConfig); err != nil { ... }

MCPRemoteProxy has no rateLimiting field today so this is harmless right now, but the moment rate limiting is added (a natural follow-on given MCPServer already has it) this ordering will cause a confusing runtime error. Please swap the two blocks:

Suggested change

	populateScalingConfigForRemoteProxy(runConfig, proxy)
	// Populate ScalingConfig.SessionRedis from spec.sessionStorage so the
	// proxy runner has the address/db/keyPrefix needed to construct a
	// shared Redis-backed session store. The Redis password is intentionally
	// excluded here — it is injected as the THV_SESSION_REDIS_PASSWORD env
	// var by buildRedisPasswordEnvVar in mcpremoteproxy_deployment.go.
	// Must run before PopulateMiddlewareConfigs because rate limiting reads SessionRedis.
	populateScalingConfigForRemoteProxy(runConfig, proxy)
	// Populate middleware configs from the configuration fields
	// This ensures that middleware_configs is properly set for serialization
	if err := runner.PopulateMiddlewareConfigs(runConfig); err != nil {
	return nil, fmt.Errorf("failed to populate middleware configs: %w", err)
	}
	return runConfig, nil

[aron-muon]

Fixed in 4c065e7 — applied the suggestion as-is; populateScalingConfigForRemoteProxy now runs before PopulateMiddlewareConfigs, mirroring the MCPServer ordering.

[yrobla]

Consider documenting the sessionAffinity + sessionStorage interaction.

These two fields now coexist without any cross-reference, which creates a footgun. When sessionStorage.provider == redis the correct HA setup also requires sessionAffinity: None — otherwise the k8s Service's ClientIP stickiness makes the Redis store redundant. Conversely, sessionAffinity: None without Redis is the broken configuration this PR was written to fix.

At minimum, the SessionStorage field comment should say: "When using the Redis provider, also set sessionAffinity: None so the Service routes requests round-robin and all replicas rely on the shared session store rather than pod-local state."

A stronger option is a kubebuilder XValidation that requires sessionAffinity == "None" when sessionStorage.provider == "redis", catching the misconfiguration at admission time.

[aron-muon]

Documented in 9364db0 — both field comments now cross-reference each other (regenerated into the CRD schema and crd-api.md). Went with docs-only rather than the CEL rule: ClientIP + redis is still a valid combination (the shared store preserves sessions when a pod dies and affinity re-pins the client), so rejecting it at admission felt too strict.

[aron-muon]

Pushed d6264234fe: added spec.replicas alongside spec.sessionStorage, mirroring the VirtualMCPServer semantics — nil leaves scaling to an HPA/external controller (the reconciler never overwrites the live count), non-nil is operator-authoritative with drift reconciliation, and the shared SessionStorageWarning condition flags replicas > 1 without Redis. Previously the operator hardcoded replicas: 1, so multi-replica deployments required pinning the count with an external min=max HPA workaround. Also merged latest main. PR title/description updated to reflect the fuller HA scope.