Repair a stale busy entry instead of spinning on it - #173
Merged
Conversation
The spin guard from #160 made the wedge survivable: after 1000 no-progress passes the worker forces the connection closed and returns to the pool. That is a last resort wearing the wrong costume -- the client it closes is innocent, and 1000 ENQ/DEQ round trips hammer the very lock every healthy request queues behind. serve_client() can do better than survive, because it holds the one fact httppc() does not have: ownership. In worker mode a client comes off the work queue and belongs to exactly one worker -- this loop. Nothing else processes it: clients never enter httpd->httpc in worker mode, so http_process_clients() walks an empty array, and no module calls the busy API through the vector (checked across mvsmf, httprexx, httplua). So when a pass makes no progress and the client turns out to be in httpd->busy, that entry is stale by definition -- left behind by an earlier failure on this worker, or by a freed client that occupied the same storage. Clear it on the FIRST no-progress pass, say so (HTTPD907W), and keep serving: the request completes normally and the wedge never forms. If the stalled client is NOT in busy, the stall has a different cause -- http_set_busy() failing on a full region is the known one -- and clearing nothing is correct: the spin guard remains as the last resort for that. The recovery sits in serve_client() and not in httppc() deliberately. httppc()'s busy-exit is legitimate concurrency control wherever ownership is not certain: select mode (http_process_clients walking the client array) and any future vector caller. Only the owner may conclude "stale"; serve_client() is the owner. Refs #159
This was referenced Aug 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the remaining httpd half of #159: a stale
httpd->busyentry is nowrepaired on first contact instead of spinning a worker into the #160
escape.
What changes
serve_client()holds the one facthttppc()does not: ownership. Inworker mode a client comes off the work queue and belongs to exactly one
worker. Nothing else processes it -- clients never enter
httpd->httpcinworker mode, so
http_process_clients()walks an empty array, and no module inthe fleet calls the busy API through the vector (checked across mvsmf,
httprexx, httplua).
So when a pass makes no progress and the client turns out to be in
httpd->busy, that entry is stale by definition -- left behind by anearlier failure on this worker, or by a freed client that occupied the same
storage. The loop now clears it on the FIRST no-progress pass, reports
HTTPD907W, and keeps serving. The request completes normally; the wedgenever forms; the connection survives.
If the stalled client is NOT in busy, the stall has a different cause
(
http_set_busy()failing on a full region is the known one) and nothing iscleared -- the #160 spin guard remains as the last resort for that, unchanged.
The recovery sits in
serve_client()and not inhttppc()deliberately:the busy-exit is legitimate concurrency control wherever ownership is not
certain (select mode, future vector callers). Only the owner may conclude
"stale", and
serve_client()is the owner.What this does and does not claim
The wedge has not been reproduced since the original specimens, so this is
defense at the choke point, verified by review and regression, not by
reproducing the wedge. What the change guarantees: no stale busy entry can
cost more than one log line, where before it cost either a worker (pre-#160)
or an innocent connection plus 1000 ENQ/DEQ round trips (#160's escape).
The two mechanisms behind the original specimens are accounted for elsewhere:
___try()fix, onservers mid-CGI-abend-cascade -- i.e. with every CRT-anchored call on the
worker resolving through a dead module's runtime. That environment cannot
recur as-then.
CSTATE_IN, 10 s timeout never firing)now has a concrete code-level candidate:
ecb_timed_waitlist()stores afailed STIMER's rc and waits anyway, on an ECB only that timer would post.
Storage-gated, which matches "only on degraded servers, never fresh".
Filed as ecb_timed_waitlist() ignores a failed STIMER and waits on an ECB nothing will post libc370#94.
Tests
MVS suite: 264 PASS, 0 FAIL (JOB00870), host suite 63 PASS, 0 FAIL. The
recovery itself is not black-box reachable (a stale busy entry cannot be
fabricated from outside), which is exactly the property the fix restores.
Refs #159