Skip to content

Commit 951d8e8

Browse files
ericallamclaude
ericallam
and
claude
authored
feat(webapp): per-client database pool metrics that survive the driver adapter (#4541)
## What Follow-up to #4539. The driver-adapter work is inert until a client flips to the pg driver adapter, but the moment one does, our database observability degrades: the OTel metrics pipeline reads pool stats from Prisma's `$metrics`, which is owned by the Rust engine's `quaint` pool. Under the adapter, `pg.Pool` owns the pool, so those gauges read zero. The pipeline also only ever scraped a single client (the control-plane writer singleton). This PR makes database metrics driver-agnostic and per-client: - Every configured client registers a metrics source: control-plane writer/replica, run-ops writer/replica, legacy writer/replica. Previously only the control-plane writer singleton was scraped. - Each OTel instrument is observed per client with `db_client` and `db_driver` (`quaint` | `pg-adapter`) attributes. `db_client` uses our canonical datasource-role labels (`control-plane-writer`, `control-plane-replica`, `run-ops-writer`, `run-ops-replica`, `legacy-run-ops-writer`, `legacy-run-ops-replica`) — the same strings used for the `db.datasource` span attribute, so a metric and a trace point at the same pool. - Pool figures come from the authoritative source per driver: - **pg-adapter**: `pg.Pool` (`totalCount`/`idleCount`/`waitingCount`, plus cumulative opened/closed from `connect`/`remove` events). - **quaint**: the Rust engine's `$metrics` pool gauges/counters, exactly as before. - Query counters and duration histograms still come from `$metrics` for both drivers (the Rust engine executes queries in both cases). - New `db.pool.connections.waiting` gauge (pg.Pool exposes this; quaint reports 0). - Stops exporting Prisma metrics from the Prometheus `/metrics` route. Pool observability now lives entirely in the OTel pipeline, per driver, per client. ## Why So we can flip any client (including the control-plane writer, the primary desync-fix target) to the driver adapter without losing pool visibility. Existing dashboards keyed on the same metric names keep working; they gain a per-client dimension. ## Testing Unit (`apps/webapp/app/utils/databaseMetrics.server.test.ts`): the pure normalizer — quaint reads pool from `$metrics`; adapter reads pool from `pg.Pool` and keeps engine query metrics; `busy` never goes negative; graceful zeroing when `$metrics` is unavailable (adapter still reports live pool figures). Live smoke test against a prod-shaped local stack: three physically-distinct Postgres DBs (control-plane, run-ops, legacy) behind dual PgBouncers, split mode on, with a mix of adapter and quaint clients. Reading the actual emitted OTel metrics, every pool shows up as its own series: ``` db.pool.connections.total{db_client="control-plane-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="control-plane-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="run-ops-writer", db_driver="pg-adapter"} = 1 db.pool.connections.total{db_client="run-ops-replica", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-writer", db_driver="quaint"} = 1 db.pool.connections.total{db_client="legacy-run-ops-replica",db_driver="quaint"} = 1 db.client.queries.total{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing db.client.queries.duration.count{db_client="control-plane-writer",db_driver="pg-adapter"} = incrementing ``` Confirms: metrics are attributed per pool with the correct driver; adapter pools' figures come from `pg.Pool`; and query counters/duration histograms keep incrementing under the pg adapter. Also verified `/metrics` (Prometheus) now returns zero `prisma_*` series while still serving the app's own metrics. `pnpm run typecheck --filter webapp` passes. ## Notes - `/metrics` (Prometheus) no longer includes `prisma_*` series. Anything scraping that endpoint for Prisma metrics should read the equivalent `db.*` metrics from the OTel exporter instead. - **PgBouncer + `?schema=` gotcha (separate from this PR, worth flagging for rollout):** since #4539 parses `?schema=` from the DSN and passes `{ schema }` to the adapter, node-postgres sends `search_path` as a startup parameter. A transaction-mode PgBouncer rejects that with `FATAL: unsupported startup parameter: search_path`. Our prod control-plane DSNs use the default `public` schema with no `?schema=` param, so this is latent, but any client we flip to the adapter must not carry `?schema=` in its DSN (or the pooler needs `ignore_startup_parameters = search_path`). --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent bd8ce4a commit 951d8e8

6 files changed

Lines changed: 493 additions & 147 deletions

File tree

‎.server-changes/db-pool-metrics-per-client.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
area: webapp
3+
type: improvement
4+
---
5+
6+
Database connection metrics are now reported for every configured database connection instead of only the primary one, and stay accurate regardless of connection type.

‎apps/webapp/app/db.server.ts‎

Lines changed: 120 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,7 @@ import {
2424
logTransactionInfrastructureError,
2525
} from "./utils/prismaErrors";
2626
import { singleton } from "./utils/singleton";
27+
import { registerDatabaseMetricsSource } from "./utils/databaseMetrics.server";
2728
import {
2829
isSplitEnabled,
2930
assertSplitRealtimeInterlock,
@@ -247,16 +248,16 @@ export function selectRunOpsTopology(
247248
if (config.legacySharesControlPlane) {
248249
legacyRunOps = controlPlane;
249250
} else {
250-
const legacyWriter = builders.buildLegacyWriter(config.legacyUrl, "run-ops-legacy-writer");
251+
const legacyWriter = builders.buildLegacyWriter(config.legacyUrl, "legacy-run-ops-writer");
251252
const legacyReplica: PrismaReplicaClient = config.legacyReplicaUrl
252-
? builders.buildLegacyReplica(config.legacyReplicaUrl, "run-ops-legacy-reader")
253+
? builders.buildLegacyReplica(config.legacyReplicaUrl, "legacy-run-ops-replica")
253254
: legacyWriter;
254255
legacyRunOps = { writer: legacyWriter, replica: legacyReplica };
255256
}
256257

257-
const newWriter = builders.buildNewWriter(config.newUrl, "run-ops-new-writer");
258+
const newWriter = builders.buildNewWriter(config.newUrl, "run-ops-writer");
258259
const newReplica: RunOpsPrismaClient = config.newReplicaUrl
259-
? builders.buildNewReplica(config.newReplicaUrl, "run-ops-new-reader")
260+
? builders.buildNewReplica(config.newReplicaUrl, "run-ops-replica")
260261
: newWriter;
261262

262263
return {
@@ -430,19 +431,25 @@ function getClient() {
430431

431432
return buildWriterClient({
432433
url,
433-
clientType: "writer",
434+
clientType: "control-plane-writer",
434435
poolTimeout: env.DATABASE_WRITER_POOL_TIMEOUT,
435436
connectTimeout: env.DATABASE_WRITER_CONNECTION_TIMEOUT,
436437
useDriverAdapter: env.CONTROL_PLANE_DATABASE_WRITER_DRIVER_ADAPTER === "1",
437438
});
438439
}
439440

441+
type DriverAdapterPool = {
442+
adapter: PrismaPg;
443+
pool: Pool;
444+
poolCounters: { opened: () => number; closed: () => number };
445+
};
446+
440447
function buildDriverAdapterPool(
441448
connectionString: string,
442449
clientType: string,
443450
poolTimeoutSeconds: number,
444451
connectionLimit: number
445-
): PrismaPg {
452+
): DriverAdapterPool {
446453
const pool = new Pool({
447454
connectionString,
448455
max: connectionLimit,
@@ -457,14 +464,27 @@ function buildDriverAdapterPool(
457464
});
458465
});
459466

467+
let opened = 0;
468+
let closed = 0;
469+
pool.on("connect", () => {
470+
opened += 1;
471+
});
472+
pool.on("remove", () => {
473+
closed += 1;
474+
});
475+
460476
let schema: string | undefined;
461477
try {
462478
schema = new URL(connectionString).searchParams.get("schema") ?? undefined;
463479
} catch {
464480
schema = undefined;
465481
}
466482

467-
return new PrismaPg(pool, { schema, disposeExternalPool: true });
483+
return {
484+
adapter: new PrismaPg(pool, { schema, disposeExternalPool: true }),
485+
pool,
486+
poolCounters: { opened: () => opened, closed: () => closed },
487+
};
468488
}
469489

470490
// Generalized writer builder shared by the control-plane client and the run-ops
@@ -548,21 +568,34 @@ export function buildWriterClient({
548568
: []) satisfies Prisma.LogDefinition[]),
549569
] satisfies Prisma.LogDefinition[];
550570

551-
const client = useDriverAdapter
552-
? new PrismaClient({
553-
adapter: buildDriverAdapterPool(
554-
url,
555-
clientType,
556-
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
557-
env.DATABASE_CONNECTION_LIMIT
558-
),
559-
log: logConfig,
560-
})
571+
const driverPool = useDriverAdapter
572+
? buildDriverAdapterPool(
573+
url,
574+
clientType,
575+
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
576+
env.DATABASE_CONNECTION_LIMIT
577+
)
578+
: undefined;
579+
580+
const client = driverPool
581+
? new PrismaClient({ adapter: driverPool.adapter, log: logConfig })
561582
: new PrismaClient({
562583
datasources: { db: { url: databaseUrl.href } },
563584
log: logConfig,
564585
});
565586

587+
registerDatabaseMetricsSource(
588+
driverPool
589+
? {
590+
clientType,
591+
usesDriverAdapter: true,
592+
client,
593+
pool: driverPool.pool,
594+
poolCounters: driverPool.poolCounters,
595+
}
596+
: { clientType, usesDriverAdapter: false, client }
597+
);
598+
566599
// Only use structured logging if we're not already logging to stdout
567600
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
568601
client.$on("info", (log) => {
@@ -631,7 +664,7 @@ function getReplicaClient() {
631664

632665
return buildReplicaClient({
633666
url,
634-
clientType: "reader",
667+
clientType: "control-plane-replica",
635668
poolTimeout: env.DATABASE_READ_REPLICA_POOL_TIMEOUT,
636669
connectTimeout: env.DATABASE_READ_REPLICA_CONNECTION_TIMEOUT,
637670
useDriverAdapter: env.CONTROL_PLANE_DATABASE_REPLICA_DRIVER_ADAPTER === "1",
@@ -719,21 +752,34 @@ export function buildReplicaClient({
719752
: []) satisfies Prisma.LogDefinition[]),
720753
] satisfies Prisma.LogDefinition[];
721754

722-
const replicaClient = useDriverAdapter
723-
? new PrismaClient({
724-
adapter: buildDriverAdapterPool(
725-
url,
726-
clientType,
727-
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
728-
env.DATABASE_CONNECTION_LIMIT
729-
),
730-
log: logConfig,
731-
})
755+
const driverPool = useDriverAdapter
756+
? buildDriverAdapterPool(
757+
url,
758+
clientType,
759+
poolTimeout ?? env.DATABASE_POOL_TIMEOUT,
760+
env.DATABASE_CONNECTION_LIMIT
761+
)
762+
: undefined;
763+
764+
const replicaClient = driverPool
765+
? new PrismaClient({ adapter: driverPool.adapter, log: logConfig })
732766
: new PrismaClient({
733767
datasources: { db: { url: replicaUrl.href } },
734768
log: logConfig,
735769
});
736770

771+
registerDatabaseMetricsSource(
772+
driverPool
773+
? {
774+
clientType,
775+
usesDriverAdapter: true,
776+
client: replicaClient,
777+
pool: driverPool.pool,
778+
poolCounters: driverPool.poolCounters,
779+
}
780+
: { clientType, usesDriverAdapter: false, client: replicaClient }
781+
);
782+
737783
// Only use structured logging if we're not already logging to stdout
738784
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
739785
replicaClient.$on("info", (log) => {
@@ -813,14 +859,18 @@ function buildRunOpsWriterClient({
813859
}`
814860
);
815861

816-
const client = useDriverAdapter
862+
const driverPool = useDriverAdapter
863+
? buildDriverAdapterPool(
864+
url,
865+
clientType,
866+
env.RUN_OPS_DATABASE_WRITER_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
867+
env.DATABASE_CONNECTION_LIMIT
868+
)
869+
: undefined;
870+
871+
const client = driverPool
817872
? new RunOpsPrismaClient({
818-
adapter: buildDriverAdapterPool(
819-
url,
820-
clientType,
821-
env.RUN_OPS_DATABASE_WRITER_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
822-
env.DATABASE_CONNECTION_LIMIT
823-
),
873+
adapter: driverPool.adapter,
824874
log: [
825875
{ emit: "event", level: "error" },
826876
{ emit: "event", level: "info" },
@@ -844,6 +894,18 @@ function buildRunOpsWriterClient({
844894
],
845895
});
846896

897+
registerDatabaseMetricsSource(
898+
driverPool
899+
? {
900+
clientType,
901+
usesDriverAdapter: true,
902+
client,
903+
pool: driverPool.pool,
904+
poolCounters: driverPool.poolCounters,
905+
}
906+
: { clientType, usesDriverAdapter: false, client }
907+
);
908+
847909
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
848910
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
849911
client.$on("warn", (log) => logger.warn("RunOpsPrismaClient warn", { clientType, event: log }));
@@ -894,14 +956,18 @@ function buildRunOpsReplicaClient({
894956
}`
895957
);
896958

897-
const client = useDriverAdapter
959+
const driverPool = useDriverAdapter
960+
? buildDriverAdapterPool(
961+
url,
962+
clientType,
963+
env.RUN_OPS_DATABASE_READ_REPLICA_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
964+
env.RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT ?? env.DATABASE_CONNECTION_LIMIT
965+
)
966+
: undefined;
967+
968+
const client = driverPool
898969
? new RunOpsPrismaClient({
899-
adapter: buildDriverAdapterPool(
900-
url,
901-
clientType,
902-
env.RUN_OPS_DATABASE_READ_REPLICA_POOL_TIMEOUT ?? env.DATABASE_POOL_TIMEOUT,
903-
env.RUN_OPS_DATABASE_READ_REPLICA_CONNECTION_LIMIT ?? env.DATABASE_CONNECTION_LIMIT
904-
),
970+
adapter: driverPool.adapter,
905971
log: [
906972
{ emit: "event", level: "error" },
907973
{ emit: "event", level: "info" },
@@ -925,6 +991,18 @@ function buildRunOpsReplicaClient({
925991
],
926992
});
927993

994+
registerDatabaseMetricsSource(
995+
driverPool
996+
? {
997+
clientType,
998+
usesDriverAdapter: true,
999+
client,
1000+
pool: driverPool.pool,
1001+
poolCounters: driverPool.poolCounters,
1002+
}
1003+
: { clientType, usesDriverAdapter: false, client }
1004+
);
1005+
9281006
if (process.env.PRISMA_LOG_TO_STDOUT !== "1") {
9291007
client.$on("info", (log) => logger.info("RunOpsPrismaClient info", { clientType, event: log }));
9301008
client.$on("warn", (log) => logger.warn("RunOpsPrismaClient warn", { clientType, event: log }));

‎apps/webapp/app/routes/metrics.ts‎

Lines changed: 1 addition & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,4 @@
11
import type { LoaderFunctionArgs } from "@remix-run/server-runtime";
2-
import { prisma } from "~/db.server";
32
import { metricsRegister } from "~/metrics.server";
43

54
export async function loader({ request }: LoaderFunctionArgs) {
@@ -13,17 +12,7 @@ export async function loader({ request }: LoaderFunctionArgs) {
1312
}
1413
}
1514

16-
// We need to remove empty lines from the prisma metrics, grafana doesn't like them
17-
let prismaMetrics = "";
18-
try {
19-
prismaMetrics = (await prisma.$metrics.prometheus()).replace(/^\s*[\r\n]/gm, "");
20-
} catch {
21-
prismaMetrics = "";
22-
}
23-
const coreMetrics = await metricsRegister.metrics();
24-
25-
// Order matters, core metrics end with `# EOF`, prisma metrics don't
26-
const metrics = prismaMetrics + coreMetrics;
15+
const metrics = await metricsRegister.metrics();
2716

2817
return new Response(metrics, {
2918
headers: {

0 commit comments

Comments
 (0)