fix(#1748): bound the LML-client limiter queue with a deadline + circuit breaker - #1883
Open
jakebromberg wants to merge 1 commit into
Open
fix(#1748): bound the LML-client limiter queue with a deadline + circuit breaker#1883jakebromberg wants to merge 1 commit into
jakebromberg wants to merge 1 commit into
Conversation
…uit breaker The shared LML client's process-wide Semaphore(5) did an unbounded, un-timed acquire() before the HTTP span even opened, so under sustained LML slowness callers could queue for minutes with no deadline and no visible span. Add a bounded admission wait (Semaphore.acquire(maxWaitMs)) and a consecutive- failure circuit breaker (LmlCircuitBreaker) to createLmlLimiter, wired into the runtime defaultLimiter via LML_LIMITER_QUEUE_DEADLINE_MS / LML_CIRCUIT_BREAKER_THRESHOLD / LML_CIRCUIT_BREAKER_RESET_MS. A shed throws LmlSheddedError (extends LmlClientError), so every existing catch arm that already treats an LML failure as "leave the row for the recovery sweep" needs no changes. Backfill/job-level limiters keep the original unbounded shape by omitting the new opt-in config. CI overrides the breaker threshold on both surfaces so existing specs that simulate consecutive LML 500s can't trip it mid-suite.
This was referenced Jul 31, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The shared LML client's process-wide
Semaphore(5)did an unbounded, un-timedacquire()before the HTTP span even opened. Under sustained LML slowness a caller could queue for minutes with no deadline and no visible span — long pastSTRANDED_TTL_SECONDS(60s), so the row it was holding open would strand anyway. This bounds that admission wait and adds a fast-fail path.What changed
Semaphore.acquire(maxWaitMs)rejects with a newLmlSheddedError(extendsLmlClientError) once a caller has waited past the queue deadline, instead of blocking indefinitely. FIFO ordering is preserved so a timed-out waiter can't jump the queue.LmlCircuitBreaker(closed / open / half-open) fast-fails admission afterLML_CIRCUIT_BREAKER_THRESHOLDconsecutive failures and probes recovery afterLML_CIRCUIT_BREAKER_RESET_MS, so a hard-down LML sheds immediately rather than making every caller eat the full deadline.createLmlLimitergains optionalqueueDeadlineMs/breaker. Omitting them preserves the original unbounded shape, so backfill/job-level limiters are unchanged. Only the runtimedefaultLimiter(interactive path) opts in, wired viaLML_LIMITER_QUEUE_DEADLINE_MS/LML_CIRCUIT_BREAKER_THRESHOLD/LML_CIRCUIT_BREAKER_RESET_MS.LmlSheddedError, a subtype of the existingLmlClientError, so every catch arm that already treats an LML failure as "leave the row for the recovery sweep" ([C6] Retune flowsheet-metadata-backfill cron as safety-net (hourly, 15-min grace) #895 backfill cron) needs no change. No new shed-handling branches..github/workflows/test.yml+dev_env/docker-compose.ymlpinLML_CIRCUIT_BREAKER_THRESHOLDhigh so integration specs that simulate consecutive LML 500s (tests/integration/metadata.spec.js) can't trip the breaker mid-suite (mirrors the BS#955 precedent).Defaults
LML_LIMITER_QUEUE_DEADLINE_MS15000STRANDED_TTL_SECONDS=60s so a shed always beats the strand.LML_CIRCUIT_BREAKER_THRESHOLD5LML_CIRCUIT_BREAKER_RESET_MS30000Testing
tests/unit/shared/lml-client/limiter-shed.test.ts(new, 517 lines) — fake-timer coverage of the bounded semaphore (FIFO, deadline rejection), the breaker state machine, the limiter wiring, and an end-to-end path through the reallookupMetadatawith mockedfetch.tests/unit/services/lml.client.test.ts— added_resetLmlClientLimitersForTest()tobeforeEach(the breaker adds shared mutable state on the process-widedefaultLimiter).prettier --check✓,typecheck✓,eslint0 errors ✓, targeted unit suites 160/160 ✓.Open questions for review
15000 / 5 / 30000are first-pass; the only hard constraint is deadline <STRANDED_TTL_SECONDS. Comfortable with these, or tune?lml.shedbreadcrumb/counter could be a fast follow if we want the shed rate to be first-class queryable.Scope
+899 / −16 across 8 files. Part of [Epic B #876] (BS↔LML single-coordinator contract) — this is the client-side admission-control slice.
Closes #1748