Distributed cluster queries with interserver secret fail for JWT-authenticated users

[BorisTyshkevich]

Summary

A query against a Hybrid table fails for an OAuth/JWT-authenticated user, although the same user can query each source independently. The failure is an internal native-TCP remote hop, not an HTTP client or SQL Browser failure.

Downstream report: Altinity/altinity-sql-browser#363

Environment

• ClickHouse: 26.3.10.20001.altinityantalya
• Authentication: jwt_dynamic_jwks (the user and oauth_demo_role are resolved from the presented JWT)
• Hybrid base segment: cluster('{cluster}', ontime.fact_ontime)
• The configured remote_servers cluster has an interserver <secret>
• The OAuth role has SELECT on both sources and READ ON REMOTE
• The outbound host allowlist permits the cluster peer and Iceberg endpoints

Reproduction

1. Authenticate over HTTP with a valid OAuth/JWT identity that receives oauth_demo_role.
2. Run a query on the Hybrid table, for example:

SELECT
    Carrier,
    count() AS flights,
    round(100 * countIf(ArrDel15 = 0) / count(), 1) AS on_time_pct
FROM ontime.fact_ontime_hybrid
WHERE Year >= 2012 AND Carrier = 'WN'
GROUP BY Carrier
ORDER BY on_time_pct DESC;

3. The query returns partial progress, then fails with:

Code: 210. DB::Exception: Connection reset by peer ... While executing Remote

The server log for the corresponding internal connection shows:

Authentication failed: Code: 192: There is no user '<JWT subject>' in `token`
TCPHandler: Code: 516 ... authentication failed

Control tests

All queries below succeeded for a temporary static user whose only role was oauth_demo_role:

• direct ontime.fact_ontime
• cluster('antalya', ontime.fact_ontime)
• ontime.fact_ontime_hybrid

The temporary user was deleted after testing. This rules out the Hybrid query plan, source-table permissions, READ ON REMOTE, and the remote host filter.

Expected behavior

The internal Distributed/Hybrid connection should be able to execute under the JWT-authenticated initiator identity, or ClickHouse should provide a supported secure way to delegate that identity/role to the internal hop.

Actual behavior

The internal native connection carries an initial username but not the JWT. The receiving server attempts to resolve that username in the token context and fails, causing the outer query to surface only a generic network error.

Impact

This prevents OAuth/JWT users from querying Hybrid tables whose native segment uses cluster(), despite having the necessary role grants.

[BorisTyshkevich]

Additional finding: prefer_localhost_replica is a practical workaround.

On the affected Antalya deployment, the compiled ClickHouse default is 1, but the effective setting for the OAuth user was 0 because the server's default settings profile explicitly sets:

prefer_localhost_replica = 0

Control queries against the same Hybrid table and JWT identity:

Setting	Result
prefer_localhost_replica = 0	Code 210 / internal-hop failure
prefer_localhost_replica = 1	query completed successfully

The query log has both controls: the disabled run ended with code 210, and the enabled run completed with code 0.

This makes the mechanism clearer: when enabled, the cluster() segment uses the local Antalya replica and avoids the internal native-TCP authentication hop that cannot resolve the JWT-derived user. This is a workaround, not a fix for JWT identity delegation across Distributed/Hybrid hops.

[BorisTyshkevich]

Deeper code-path analysis identifies both the trigger and the underlying incompatibility.

Trigger

The affected OAuth user inherits the deployment's default settings profile, which explicitly sets:

prefer_localhost_replica = 0

(The compiled ClickHouse default is 1.) With 0, SelectStreamFactory does not select the local shard replica and instead builds a remote stream, even though the cluster() segment targets the same Antalya server.

Remote authentication path

The configured remote_servers cluster has a <secret>.

1. In Connection.cpp, a nonempty cluster_secret selects the interserver handshake: it sends the interserver-user marker and an empty password. It does not take the JWT handshake branch.
2. After verifying the interserver-secret hash, TCPHandler.cpp calls:

   session->authenticate(AlwaysAllowCredentials{client_info.initial_user}, ...)

3. TokenAccessStorage::authenticateImpl() accepts only TokenCredentials, because it needs the bearer JWT to validate/provision the dynamic user and map token roles. It cannot resolve an AlwaysAllowCredentials username alone.

That produces the observed “There is no user '' in token”, followed by the peer reset and outer Code 210.

Why localhost preference succeeds

With prefer_localhost_replica = 1, SelectStreamFactory constructs a local plan, retaining the already HTTP/JWT-authenticated query context. No native remote connection or second authentication occurs.

Therefore this is not specific to Hybrid: Hybrid exposes a generic Dynamic-JWT-user + Distributed/cluster() + interserver-secret incompatibility.

The localhost setting is useful as a workaround, but not a general resolution: it will not cover nonlocal/stale replicas, other shards, or paths where parallel-replica handling bypasses it. A durable fix needs a secure supported way for an interserver query to resolve/delegate a dynamic token identity (and its mapped roles), rather than relying only on the username.