Add metastore read replica routing

[shuheiktgw]

Description

This PR adds support for routing read-only metastore requests to a PostgreSQL read replica.

The current scope is limited to metastore requests issued by searchers. Other read-only metastore call sites may also benefit from this later, but this PR keeps the initial coverage focused on the search path.

How It Works

1. Users can configure metastore_read_replica_uri alongside the existing metastore_uri.
2. On metastore nodes, Quickwit resolves the regular metastore_uri as the primary metastore and, when configured, resolves metastore_read_replica_uri as a read-only PostgreSQL metastore client. The read replica client uses its own PostgreSQL connection pool, with the same pool settings as the regular PostgreSQL metastore configuration.
3. The metastore gRPC service is wrapped in ReadReplicaGrpcService, which contains the primary service and an optional read replica service. Incoming requests with the qw-use-read-replica: true gRPC metadata header are routed to the read replica service. If no read replica is configured, the wrapper falls back to the primary service.
4. Searchers use a dedicated read-only metastore client, MetastoreReadServiceClient, which exposes only the read-only subset of metastore RPCs. The underlying gRPC client attaches the qw-use-read-replica: true header through an interceptor, so eligible searcher metastore requests are routed to the read replica when available.
5. Searchers always attach the read-replica header on this read-only client, even when metastore_read_replica_uri is unset. This is safe because metastore nodes fall back to the primary service when no read replica service is configured.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: bf5e1023f1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".