Skip to content

Report the storage shortage in the env-var path instead of resetting silently - #167

Merged
mgrossmann merged 1 commit into
mainfrom
issue-162-env-oom-observability
Aug 9, 2026
Merged

Report the storage shortage in the env-var path instead of resetting silently#167
mgrossmann merged 1 commit into
mainfrom
issue-162-env-oom-observability

Conversation

@mgrossmann

@mgrossmann mgrossmann commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Fixes #162

The regression

Before libc370#82 a storage shortage inside the request pipeline abended S878;
try(serve_client, ...) caught it, logged HTTPD062E, reset busy and closed
cleanly. Now the same shortage makes http_set_env() return -1 → goto failed
CSTATE_RESET: a connection reset with no response and no log line,
indistinguishable from a malformed request. Observability got worse.

Where the message goes, and why not at the reset

Not at the failed: labels. Both of them (httpin.c:183, httppars.c:142) are
shared with socket errors, strtok returning NULL and missing env vars, so a
message there could only guess at the cause — or need errno archaeology to tell
the cases apart.

httpsenv() is the one choke point every caller passes through — httpin,
httppars, httpshen, httpsqen, httppc, and any CGI going through the
HTTPX vector — and it is the last place that still knows which variable
failed. That is strictly more useful to an operator than "the connection was
reset".

http_new_env() has two rejects that both return NULL, so it now sets errno
to distinguish them:

  • HTTPD904E No storage for environment variable <name> client(<HTTPC>)
  • HTTPD905E Environment variable <name> too large client(<HTTPC>)

(the client(...) operand is the HTTPC pointer, matching HTTPD900D's style, not the client IP)

Its signature is unchanged, so the HTTPX vector entry is untouched.

Two unchecked return values in the same path

Both named in the issue:

  • httppc.c dispatched the CGI even when SCRIPT_FILENAME could not be
    set, i.e. ran it against no file at all. It now answers 500.
  • parse_cookies() ignored set_cookie()'s rc in a loop. Under a
    shortage a request with 20 cookies would have emitted 20 of the new messages.
    It now stops at the first failure — that is what bounds the message to once
    per request, the same cadence HTTPD062E had.

parse_cookies() also returned ENOMEM (12) where every other setter in this
path returns -1. Normalized; every caller only tests truthiness.

Testing

The oversize reject needs no storage shortage to reach, so it is tested.
TSTNENV gains three assertions: a name+value over the sanity limit is rejected
with E2BIG, and one byte under the limit still succeeds (so the reject cannot
quietly swallow legitimate variables).

Run on the MVS target, and confirmed to actually discriminate — against the
unfixed httpnenv.c it fails, with the fix it passes:

TSTNENV    FAIL CC 1      FAIL CC 1      | 28 PASS, 2 FAIL     <- httpnenv.c stashed
TSTNENV    ok CC 0        ok CC 0        | 30 PASS, 0 FAIL     <- with the fix

The ENOMEM branch itself is inspection-only. It needs a real storage
shortage; there is no allocator interposition on this toolchain and building
injection into production code for it would be worse than saying so.

What was verified live instead is that every path this PR touches still behaves
on the non-OOM side — deployed, activated, restarted:

path exercise result
httpin env plain GET /.dsrv 200
httpsqen /.dm?m=10&l=16&t=CVTPTR 200
parse_cookies / set_cookie request with 5 cookies 200
CGI dispatch /zosmf/restfiles/ds?dslevel=… 200
httppc SCRIPT_FILENAME extension route *.httpdsrv 200
malformed request GARBAGE\r\n\r\n still reset, unchanged

The SCRIPT_FILENAME branch only runs for extension-based routes, of which the
production Parmlib has none — so it was exercised through a temporary alternate
member (MOD=HTTPDSRV → derived *.httpdsrv), started with S HTTPD,M=… so
HTTPPRM0 was never touched. GET /x.httpdsrv?target=HTTPD dispatched and
rendered, confirming the new guard falls through correctly on success. Member
deleted, server restarted on HTTPPRM0, production member verified unchanged.

Also make modules / make test clean under -Wall -Werror, make test-host
4 suites / 63 assertions / 0 fail.

Not included

The issue's third note — reinstating the malloc-bisecting storage probe now that
malloc() fails with NULL instead of S878 — is marked there as a separate
enhancement and is left out.

…silently

Before libc370#82 a shortage inside the request pipeline abended S878,
try(serve_client) caught it and logged HTTPD062E, and the connection
closed cleanly. Now http_set_env() just returns -1, every caller does
goto failed -> CSTATE_RESET, and the client sees a connection reset
with no response and nothing in the log -- indistinguishable from a
malformed request. Observability got worse for this path.

Report it at httpsenv(), the one choke point every caller passes
through and the last place that still knows which variable failed --
the reset labels in httpin.c and httppars.c are shared with socket
errors and malformed-request paths, so a message there could only
guess. httpnenv() now sets errno so the two rejects are told apart:
HTTPD905E for a variable over the sanity limit, HTTPD904E for an
actual shortage.

Two unchecked return values in the same path, both from the issue:

  - httppc.c dispatched the CGI even when SCRIPT_FILENAME could not be
    set, which runs it against no file at all; it now answers 500.
  - parse_cookies() ignored set_cookie()'s rc in a loop, so a request
    with 20 cookies would emit 20 messages instead of one. It now
    stops at the first failure, which is what bounds the new message
    to once per request -- the same cadence HTTPD062E had.

parse_cookies() also returned ENOMEM (12) where every other setter in
this path returns -1; normalized, callers only test truthiness.

TSTNENV covers the half that needs no storage shortage to reach: an
oversized name+value is rejected with E2BIG and one byte under the
limit still succeeds. Both assertions fail against the unfixed
httpnenv.c.

Fixes #162
@mgrossmann

Copy link
Copy Markdown
Contributor Author

Checked the one thing this design could get wrong: does the errno set in httpnenv() read back correctly in httpsenv(), given the call goes through the HTTPX vector?

Yes, and the vector is not the boundary it looks like:

  • errno is *(__errno()), and __errno() (libc370 @@errno.c) returns &crt->crterrno from __crtget().
  • __crtget() (@@crtget.c) finds the CRT by matching crttcb against the current TCB (PSATOLD out of the PSA) in the PPA's CRT array. So it is task-scoped, not module-scoped or WSA-scoped — unlike credkey(), which is the per-GRT trap this resembles.
  • Both httpnenv() and httpsenv() are HTTPD's own statically linked copies, and the vector entry httpx->http_new_env points straight back at HTTPD's httpnenv. So the write and the read happen in the same module on the same task, resolving the same crterrno — including when a CGI drives http_set_env through the vector, because the CGI's own libc370 copy is never involved in either half.

Also deliberate, not incidental: ENOMEM is the fallback and E2BIG is the specific test. If errno ever carried a stale value from an earlier call, the message degrades to "no storage" — the honest default for an allocation path — rather than falsely claiming the variable was too large.

@mgrossmann
mgrossmann merged commit faf4ba2 into main Aug 9, 2026
1 check passed
@mgrossmann
mgrossmann deleted the issue-162-env-oom-observability branch August 9, 2026 22:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allocation failure in the env-var path is a silent connection reset since libc370#81

1 participant