Skip to content

Fix integer overflow in StationTessellatorImpl buffer growth#262

Open
matthewperiut wants to merge 1 commit into
ModificationStation:developfrom
matthewperiut:fix/tessellator-buffer-overflow
Open

Fix integer overflow in StationTessellatorImpl buffer growth#262
matthewperiut wants to merge 1 commit into
ModificationStation:developfrom
matthewperiut:fix/tessellator-buffer-overflow

Conversation

@matthewperiut

Copy link
Copy Markdown
Contributor

Fix integer overflow in StationTessellatorImpl buffer growth

Symptom

After a long play session, the Arsenic tessellator crashes while rendering GUI items
(e.g. items in the inventory/hotbar) with:

java.lang.IllegalArgumentException: capacity < 0: (-2147483648)
    at java.base/java.nio.Buffer.createCapacityException
    at java.base/java.nio.ByteBuffer.allocateDirect
    ... GlAllocationUtils.allocateByteBuffer
    ... StationTessellatorImpl.ensureBufferCapacity

Root cause

StationTessellatorImpl.ensureBufferCapacity grows the tessellator buffer by
doubling bufferSize (an int) every time the buffer fills, and never shrinks it
back. Vanilla Tessellator.reset() resets vertexCount/bufferPosition but does NOT
reset bufferSize, so the doubled size accumulates across the whole session.

The buffer normally never grows, because vanilla Tessellator.vertex() auto-flushes
(draw()) when it nears full. Arsenic's direct-write quad() path bypasses that
auto-flush and instead calls ensureBufferCapacity(48), which grows the buffer rather
than flushing it. Over a long session the buffer therefore doubles repeatedly:

base bufferSize = 2_097_152            (2^21)
after 8 doublings: bufferSize = 536_870_912   (2^29)
bufferSize * 4 (the byte-buffer allocation) = 2^31 -> overflows signed int
                                                   = Integer.MIN_VALUE = -2147483648

GlAllocationUtils.allocateByteBuffer(-2147483648) then calls
ByteBuffer.allocateDirect(MIN_VALUE), which throws capacity < 0. (A latent ~2 GB
Arrays.copyOf(buffer, bufferSize) allocation sits one line earlier as well.)

Fix

Minimal, contained change in ensureBufferCapacity:

  • Cap bufferSize at a sane maximum (MAX_BUFFER_SIZE = 1 << 28 ints = a 1 GiB direct
    buffer) so growth is bounded and bufferSize * 4 can never overflow a signed int.
    The base size is 2^21, so this still permits several doublings for any plausible load.
  • Once the buffer is already at the ceiling, return without reallocating instead of
    doubling into an invalid (negative) capacity.
  • Perform the byte-count arithmetic in long before the allocation, so even at the
    cap the computed capacity stays a correct positive int.

The diff touches only StationTessellatorImpl.java and preserves the existing logging
and growth behaviour up to the cap.

Verification

  • Reproduced the arithmetic: from the 2^21 base, the 8th doubling yields bufferSize
    2^29 and bufferSize * 4 == 2^31, which overflows to Integer.MIN_VALUE -- matching the
    reported crash capacity exactly. With the cap, bufferSize stops at 2^28 and the byte
    count tops out at 2^30 (a valid positive int), so the overflow can no longer occur.
  • Built the changed module with the project toolchain (JDK 21):
    ./gradlew :station-renderer-api-v0:build -> BUILD SUCCESSFUL (only the project's
    pre-existing -source 17 / deprecation warnings, none from this change).

@matthewperiut matthewperiut marked this pull request as ready for review June 14, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant