os_bitand/os_bitor/os_bitxor/os_x2c allocate script-controlled sizes without checking: 10 unchecked mallocs in the MVS glue

[mgrossmann]

The vendored Lua 5.4 core is clean (allocator NULL handled everywhere via LUA_ERRMEM), but the hand-written MVS glue in src/loslib.c allocates buffers sized by the script's string lengths and dereferences them immediately:

• src/loslib.c:580,581,582 — os_bitand buf1/buf2/resp, memcpy into them two lines later
• src/loslib.c:623,624,625 — os_bitor, same shape
• src/loslib.c:666,667,668 — os_bitxor, same shape
• src/loslib.c:706 — os_x2c, writes resp[j++] at :719

Since libc370#81 (fixed by libc370#82, in the sysroot as of 2026-08-09) malloc() returns NULL on shortage, so each of these is an S0C4 — in the lua370 CLI, or inside httpd when the same code runs under httplua.

Fix: allocate through Lua's own luaM_/luaL_Buffer machinery (which raises LUA_ERRMEM properly) or check and luaL_error — the sizes are attacker/script-controlled, so the failure must land in the script's error handling, not in the host process.

Sweep context: mvslovers/libc370#81 (comment)

[mgrossmann]

Implementation notes, from reading os_bitand (src/loslib.c:569-610; bitor/bitxor are the same shape, os_x2c a one-buffer variant):

char *buf1 = calloc(1, len+1);   /* input 1, copied only to apply pad */
char *buf2 = calloc(1, len+1);   /* input 2, copied only to apply pad */
char *resp = calloc(1, len+1);   /* result */
memcpy(buf1, str1, len1); ...

Preferred shape — zero raw allocations. buf1/buf2 exist only so the
tail can be padded; the loop can pad per index instead and read the Lua
strings in place (lua_tolstring already hands out stable pointers to
immutable strings):

char c1 = (i < len1) ? str1[i] : *pad;
char c2 = (i < len2) ? str2[i] : *pad;
resp[i] = c1 & c2;

and the result buffer comes from Lua itself: luaL_Buffer b; char *resp = luaL_buffinitsize(L, &b, len); ... luaL_pushresultsize(&b, len);
That turns 3 allocations per call into 1, and that one is Lua-owned: if it
cannot be obtained, the failure raises LUA_ERRMEM through the allocator —
which works end-to-end since libc370#82 (verified down to the @@setj/
@@longj assembler; a script OOM becomes an error, not an abend) — and on
any unwind the buffer is GC'd, nothing leaks. On this platform the memory
arithmetic matters too: with the 64-byte GETMAIN rounding, three len+1
callocs cost roughly triple what they look like.

If the minimal check-and-raise route is taken instead, one trap:
luaL_error() unwinds via longjmp, so any raw-malloc'd buffer still live
at the raise leaks permanently (nothing frees it on an httpd worker). The
checks must free everything already obtained before raising — three
allocations, three partial-failure orderings. The luaL_Buffer route
sidesteps that class entirely.

Also worth knowing: allocations routed through Lua's luaM_/l_alloc get
one emergency full GC and a retry before failing (lmem.c), which raw
malloc() in the glue never gets. And since libc370#82 every failed
malloc() emits an 'Out of memory' WTO + save area traceback on the
console — the sizes here are script-controlled, so keeping the failure
inside Lua's error handling is also what keeps a greedy script from
becoming an operator-console event.