perf: get unpack() out of the NYI list

[MyNameIsTrez]

While writing a transpiler to Lua I hit a 20x performance cliff caused by unpack.

unpack is listed as 2.1 stitch on tarantool's LuaJIT Not Yet Implemented page, which explains the cliff. I confirmed this with -jv.

This PR resolves the cliff by making unpack fully compiled.

The original performance cliff and my workaround

Running the MRE below against the default branch (v2.1-agentzh) 20 times produces only fast runs (~0.025s) or slow runs (~0.3s), never anything in between.

The Dockerfile below reads mre.lua from the current working directory. Every Dockerfile in this PR description is run with docker build -t luajit2-test . && docker run --rm -it luajit2-test.

Fast runs are expected. Slow runs happen when LuaJIT blacklists empty_fn() in response to unpack NYIs. Pass -jv to luajit to see these sporadic blacklisted messages.

Dockerfile that runs mre.lua

FROM alpine:latest

# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev

WORKDIR /workspace

# Clone the default branch and compile
RUN git clone https://github.com/openresty/luajit2.git && \
    cd luajit2 && \
    make -j$(nproc)

# Copy the MRE script from your host's current working directory
COPY mre.lua /workspace/mre.lua

WORKDIR /workspace/luajit2

# Run the benchmark 20 times
CMD ["sh", "-c", "for i in $(seq 20); do src/luajit ../mre.lua; done"]

mre.lua

-- empty_fn takes no args here because many game functions don't
-- take args either; this MRE is meant to mirror that.
-- Note: replacing () with (...) makes the benchmark report fast times,
-- since it avoids the NYI shown below.
local function empty_fn() end

local function run_unpack()
    pcall(empty_fn, unpack({}))
end

-- On my laptop (AMD Ryzen AI 9 HX 370), 80k iterations is the
-- nondeterministic tipping point between always fast (~70k) 
-- and always slow (~90k).
for _ = 1, 80000 do
    run_unpack()
end

local start = os.clock()

-- Because of the previous loop, this unrelated hot-loop is a coin toss
-- between being JIT compiled (fast) or stuck in the interpreter (slow).
for _ = 1, 100000000 do
    empty_fn()
end

print(os.clock() - start)

To work around this, I updated my transpiler to generate a specialized wrapper per argument count instead of forwarding arguments through unpack. Each wrapper indexes the args table directly and is cached, so the code generation cost is paid once while execution stays fully traceable by LuaJIT. This workaround will stay relevant for years, since many programs never update the LuaJIT version they embed.

Workaround mre.lua

local pcall_wrappers = {}

local function get_pcall_wrapper(arg_count)
    if pcall_wrappers[arg_count] then
        return pcall_wrappers[arg_count]
    end

    local arg_list = {}
    for i = 1, arg_count do
        arg_list[i] = "args[" .. i .. "]"
    end

    -- Generate a specialized wrapper to avoid `unpack` (which triggers a LuaJIT NYI).
    -- Example (arg_count=2): return function(fn, args) return pcall(fn, args[1], args[2]) end
    local args_str = #arg_list > 0 and (", " .. table.concat(arg_list, ", ")) or ""
    local code = string.format("return function(fn, args) return pcall(fn%s) end", args_str)

    local wrapper = loadstring(code)()
    pcall_wrappers[arg_count] = wrapper
    return wrapper
end

-- empty_fn takes no args here because many game functions don't
-- take args either; this MRE is meant to mirror that.
-- Note: replacing () with (...) makes the benchmark report fast times,
-- since it avoids the NYI shown below.
local function empty_fn() end

local function run_unpack()
    local args = {}
    local wrapper = get_pcall_wrapper(#args)
    wrapper(empty_fn, args)
end

-- On my laptop (AMD Ryzen AI 9 HX 370), 80k iterations is the
-- nondeterministic tipping point between always fast (~70k) 
-- and always slow (~90k).
for _ = 1, 80000 do
    run_unpack()
end

local start = os.clock()

-- Because of the previous loop, this unrelated hot-loop is a coin toss
-- between being JIT compiled (fast) or stuck in the interpreter (slow).
for _ = 1, 100000000 do
    empty_fn()
end

print(os.clock() - start)

Running the 32 tests I wrote for recff_unpack

Dockerfile that runs t/unpack.t its 32 recff_unpack tests

FROM alpine:latest

# Install build dependencies, perl, perl-utils (for prove), and cpanminus
RUN apk add --no-cache git make gcc musl-dev perl perl-utils perl-app-cpanminus

# Install Perl test dependencies
RUN cpanm --notest IPC::Run3 Test::Base Test::LongString Parallel::ForkManager

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Build, install to system paths, and run the test
CMD ["sh", "-c", "make -j$(nproc) && make install && prove t/unpack.t"]

It prints this:

t/unpack.t .. ok
All tests successful.
Files=1, Tests=96,  1 wallclock secs ( 0.03 usr  0.00 sys +  0.16 cusr  0.10 csys =  0.29 CPU)
Result: PASS

Confirming this fixed the original mre.lua

Dockerfile that checks out this PR's branch, to demonstate the original mre.lua now always runs fast

FROM alpine:latest

# Install build dependencies
RUN apk add --no-cache git make gcc musl-dev

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Compile LuaJIT
RUN make -j$(nproc)

# Copy the MRE script from your host's current working directory
COPY mre.lua ./

# Run the benchmark 20 times
CMD ["sh", "-c", "for i in $(seq 20); do src/luajit mre.lua; done"]

Running unimut to mutation test recff_unpack

Although I brought line and branch coverage to 100%, I couldn't be sure I was covering every edge case, or that there were no redundant sections I could cut.

To address this, I wrote unimut (universal mutator, pip install unimut) for this PR. It is called universal because it lets users register backends for other languages too:

unimut can be run like unimut --file src/lj_ffrecord.c --run 'make -j$(nproc) && prove t/unpack.t'. The Dockerfile below compiles with ASan and UBSan, which brings surviving mutants down from 11 to 9, and adds temporary // unimut on and // unimut off markers around the recff_unpack function this PR adds:

Dockerfile that mutation tests recff_unpack

FROM alpine:latest

# Install build dependencies, perl, perl-utils (for prove), cpanminus, and Python/pip
RUN apk add --no-cache git make gcc musl-dev perl perl-utils perl-app-cpanminus python3 py3-pip

# Install Perl test dependencies
RUN cpanm --notest IPC::Run3 Test::Base Test::LongString Parallel::ForkManager

# Install unimut globally
RUN pip install --break-system-packages unimut

WORKDIR /luajit2

# Clone repository and checkout PR #269 into branch 'fix-unpack-nyi'
RUN git clone https://github.com/openresty/luajit2 . && \
    git fetch origin pull/269/head:fix-unpack-nyi && \
    git checkout fix-unpack-nyi

# Inject unimut markers around the recff_unpack function
RUN perl -0777 -pi -e 's|(/\* unpack\(t, \[i, \[j\]\]\) \*/\nstatic void LJ_FASTCALL recff_unpack)|// unimut on\n$1|' src/lj_ffrecord.c && \
    perl -0777 -pi -e 's|(\nstatic void LJ_FASTCALL recff_tonumber)|\n// unimut off\n$1|' src/lj_ffrecord.c

# Run unimut with AddressSanitizer, UndefinedBehaviorSanitizer flags, and increased timeout
CMD unimut \
    --file src/lj_ffrecord.c \
    --jobs 16 \
    --timeout 120 \
    --run 'make -j$(nproc) PREFIX="$(pwd)/build" \
    TARGET_CFLAGS="-fsanitize=address,undefined -fno-sanitize=alignment,shift -fno-sanitize-recover=undefined -fno-omit-frame-pointer -ftrivial-auto-var-init=pattern -g -DLUAJIT_USE_SYSMALLOC -DLUA_USE_ASSERT -DLUA_USE_APICHECK" \
    TARGET_LDFLAGS="-fsanitize=address,undefined" \
    && make install PREFIX="$(pwd)/build" \
    && PATH="$(pwd)/build/bin:$PATH" prove -I. t/unpack.t'

It prints this concise diff

src/lj_ffrecord.c:376
- if (tref_isk(tri))

src/lj_ffrecord.c:377
- emitir(IRTGI(IR_EQ), tri, lj_ir_kint(J, i));
+ ;

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn == 0) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn < 0) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k != n; k++) {

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k <= n; k++) {

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn <= (0 + 1)) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:391
- if (maxn <= 0 || span >= (uint32_t)maxn)
+ if ((maxn <= (0 - 1)) || (span >= ((uint32_t) maxn)))

src/lj_ffrecord.c:397
- for (k = 0; k < n; k++) {
+ for (k = 0; k < (n + 1); k++) {

Survived: 9/146

The 9 surviving mutants are expected. They involve checks against internal LuaJIT implementation details that Lua-level tests can't, or shouldn't, cover.

The CI already fails on the base v2.1-agentzh branch

The Travis CI pipeline fails on the Valgrind job, but the latest commit on the base v2.1-agentzh branch fails with the exact same error. This PR doesn't introduce any new test failures.