Skip to content

serve_client: escape the spin, and instrument the worker wedge - #160

Merged
mgrossmann merged 3 commits into
mainfrom
debug-217-worker-wedge
Aug 9, 2026
Merged

serve_client: escape the spin, and instrument the worker wedge#160
mgrossmann merged 3 commits into
mainfrom
debug-217-worker-wedge

Conversation

@mgrossmann

Copy link
Copy Markdown
Contributor

serve_client() (src/httpd.c) had no escape condition:

while (httpc->state != CSTATE_CLOSE) {
    http_process_client(httpc);
    if (work->state == CTHDWORK_STATE_SHUTDOWN) break;
}

It re-calls http_process_client() with no wait and no timeout, discards the
return value, and keys progress solely on httpc->state — a field the callee
is not obliged to touch. httppc() has exactly such a path, the busy-exit at
the top (if (http_is_busy(httpc)) goto quit;). A client left in
httpd->busy therefore spins the worker forever on ENQ/DEQ, and that worker is
lost to the pool for the life of the address space.

Measured on a live degraded instance: two workers with their client frozen at
CSTATE_DONE and permanently in httpd->busy, where every problem-state
PSW sample landed in @@ENQDEQ (i.e. lock()/unlock()) and not one in
http_done, http_send, recv or fclose. Details in #159.

What this changes

A spin guard in serve_client() — counts passes that leave httpc->state
unchanged and, past a threshold, reports and forces the close instead of taking
the address space down. It also calls http_reset_busy() first: reaching the
guard means httppc() kept taking its busy-exit and never got to
http_reset_busy(), so the client is still in httpd->busy — and
http_close() only removes from httpd->httpc, never from busy. Without
that call the freed HTTPC address stays in the array and wedges the next client
allocated at it.

Two diagnostics, behind HTTPD_DEBUG_217 (include/httpd.h):

HTTPD900D the busy-exit fires — names client, state, socket, request count
HTTPD902D http_gets' poll loop runs past its deadline

HTTPD902D exists to separate the two hang shapes: "the wait returns but the
deadline never fires" from "the wait never returned at all". A worker that
wedges with no message from either is stuck inside cthread_timed_wait(),
which would make it a libc370 problem rather than an httpd one.

Removability

Deliberately split, since these have different lifetimes:

  • The spin guard and its HTTPD901E message are not behind the switch.
    They are a fix, not a diagnostic, and are meant to stay.
  • The two diagnostics are. Set HTTPD_DEBUG_217 to 0 to silence them, or
    delete the define together with the two #if HTTPD_DEBUG_217 blocks to drop
    them entirely — they are self-contained and touch nothing else.

Verified to build clean under -Wall -Werror with the switch in both positions.

Note

There is also a comment in httpd.c recording why the storage probe that
briefly lived in the socket thread is gone: it bisected with malloc() and
abended the thread S878, because malloc reaches @@GETM, which issues
GETMAIN RU — unconditional. On this platform allocating is not a way to
measure free storage. That is the root cause of #217's S80A cascade and is
filed as mvslovers/libc370#81.

Refs #159, mvslovers/mvsmf#217, mvslovers/libc370#81

serve_client() had no escape condition: it re-calls http_process_client()
with no wait and no timeout, discards the return value, and keys progress
solely on httpc->state -- a field the callee need not touch. httppc() has
exactly such a path, the busy-exit at the top, so a client left in
httpd->busy makes the worker spin on ENQ/DEQ forever and lose itself to
the pool. Count passes that change nothing, report, and force the close
rather than taking the address space down with it.

Three probes for the open investigation:

  HTTPD900D  the busy-exit fires, naming the client and its state
  HTTPD901E  the spin guard trips in serve_client()
  HTTPD902D  http_gets' poll loop runs past its deadline, which separates
             "the wait returns but the timeout never fires" from "the wait
             never returned at all"

Also records why the storage probe that used to sit in the socket thread
is gone: it bisected with malloc() and abended the thread S878, because
malloc goes to @@getm, which issues GETMAIN RU -- unconditional. On this
platform allocating is not a way to measure free storage.

Refs #159, mvslovers/mvsmf#217, mvslovers/libc370#81
The spin guard rescued the worker but left the cause behind. Reaching it
means httppc() kept taking its busy-exit and never got to
http_reset_busy(), so the client is still in httpd->busy -- and
http_close() only removes from httpd->httpc, never from busy. Without
this the freed HTTPC address stays in the array and wedges the next
client that happens to be allocated at it.

Refs #159
HTTPD900D and HTTPD902D are temporary and should be easy to switch off
or delete once #159 is understood. Guard them with a single define so
silencing them is a one-character change and removing them is two
self-contained blocks plus the define.

The spin guard in serve_client() is deliberately left unguarded: it is a
fix rather than a diagnostic, and its HTTPD901E message is the whole
point of having it.

Verified to build clean with the switch in both positions.

Refs #159
@mgrossmann
mgrossmann merged commit 76a39e5 into main Aug 9, 2026
1 check passed
@mgrossmann
mgrossmann deleted the debug-217-worker-wedge branch August 9, 2026 18:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant