publish-content: no retry on transient HANA pool-timeout 500 → fatal prod rebuild

[jung-thomas]

What happened

Prod content rebuild (single slug abap-environment-create-rap-business-events) failed fatally: run

[publish-content] #672 short-circuit disengaged: cannot reach /content/source-hashes: Error: HTTP 500 [status=500]
Fatal: HTTP 500: {"error":"Pool resource could not be acquired within 1s"}

The srv recovered within minutes — a later GET /content/source-hashes returned 200. So this was transient HANA connection-pool exhaustion, not a content or DB fault: a momentary load spike (rebuild fetches source-hashes + nav metadata for ~1400 tutorials + the publish PUT, colliding with live traffic / scheduled jobs) starved the pool, and the acquire timed out.

Root cause

• No DB pool config in the project → CAP/@cap-js/hana defaults, incl. acquireTimeoutMillis: 1000 (the "within 1s"). Aggressive under load.
• scripts/publish-content.ts has no retry/backoff on a 500, so a single transient pool-timeout aborts the whole rebuild. (The source-hashes fetch already fails soft — "short-circuit disengaged" → full delta — but the final publish PUT is fatal.)

Proposed fix

1. Retry-with-backoff (e.g. 3 attempts, jittered) in publish-content.ts on 500 whose body matches Pool resource could not be acquired (and similar transient acquire/timeout errors).
2. Consider raising cds.requires.db.pool.acquireTimeoutMillis (and/or max) for the srv so brief contention waits instead of erroring.

Impact

Prod rebuilds intermittently abort under load; requires manual re-run. Recurs whenever the pool briefly saturates.

[jung-thomas]

Escalation: this is systemic on prod, not a one-off transient

Retried the prod rebuild after the srv looked recovered; it failed again — this time at the Hugo build guard, because a /build/* fetch drew a bad connection:

[fetch-tags] WARN: status 500 — writing empty payload
[build:hugo] verb-shelf guard FAILED: verb_definitions.json has no verbs — the feed baked empty.

Plain unauthenticated probes of the prod tutorials-srv (no rebuild running), 3× each, show a ~1/3 failure rate across every endpoint — so it's the shared DB pool, not any one handler:

endpoint	results
/build/verb-definitions	200, 200, 500
/build/tags	500, 200, 200
/build/catalog	500, 500, 200
/content/source-hashes	200, 200, 200

All 500s are the same Pool resource could not be acquired within 1s.

Immediate mitigation: restart the prod tutorials-srv app to clear leaked/wedged pool connections, then re-run the rebuild.

Investigate (raises scope beyond client retry):

• connection leak — a checked-out pool connection never released (candidate: raw db.run BLOB-locator paths in content-store.js / embedding-query.js on an error path);
• default pool max too small for prod concurrency (no cds.requires.db.pool config exists);
• a long-running query or hot scheduled job holding connections.

Client-side retry (original issue body) is still worth doing but will only paper over a genuinely saturated pool — the srv-side leak/sizing is the real fix.

[jung-thomas]

Fixed in #2290 (merged to DEV).

Root cause

beginSession in scripts/publish-content.ts was the only publish HTTP call in the happy path with neither withRetry nor a try/catch — appendBatch, commit, and the three render-* phases are all wrapped, begin was not. A transient HANA pool-acquire 500 (Pool resource could not be acquired within 1s) thrown by beginSession bubbled straight to the top-level main().catch → Fatal: HTTP 500, aborting the whole rebuild. Matches the trace exactly: the #672 short-circuit soft-fails first (disengaged), then the fatal 500. The delta /content/hashes fetch had the same gap (exits 1 on a transient 500 before a session opens).

Fix

1. Wrapped beginSession in withRetry (5 attempts, [2000,5000,10000,20000], ±20% jitter) — same posture as the existing commit retry. 500/502/503/504 are already classified transient in publish-retry.ts, so no body-matching was needed.
2. Wrapped the delta fetchRemoteHashes in withRetry (4 attempts, [1000,3000,6000], jitter).
3. Added cds.requires.db.pool for hana [hybrid]/[production] (acquireTimeoutMillis: 5000, max: 100, min: 0, fifo: true). There was no pool config → defaults incl. acquireTimeoutMillis: 1000 (the "within 1s"); brief contention now waits instead of erroring.

Tests

scripts/__tests__/publish-client.test.ts: beginSession under withRetry recovers from a transient pool-timeout 500, and does not retry a permanent 409. Full scripts/__tests__ suite: 937 passing.

Note: pool config takes effect on next deploy (config-only change, not yet deployed).