Skip to content

fix: cnum destroyed numeric literals outside int64 range (silent wrong arithmetic) - #51

Merged
pyrex41 merged 2 commits into
mainfrom
fix/cnum-int64-saturation
Jul 29, 2026
Merged

fix: cnum destroyed numeric literals outside int64 range (silent wrong arithmetic)#51
pyrex41 merged 2 commits into
mainfrom
fix/cnum-int64-saturation

Conversation

@pyrex41

@pyrex41 pyrex41 commented Jul 29, 2026

Copy link
Copy Markdown
Owner

The bug

Silent wrong arithmetic on ordinary finite literals — nothing to do with infinity.

(> 1e300 1e100)       ->  false   (should be true)
(/ 1e300 1e290)       ->  1       (should be 1e10)
(= 1e19 (* 1e18 10))  ->  false   (should be true)

Root cause

compiler.lua cnum rendered any integral float with string.format("%d", n).
That is only sound inside int64 range. The hazard was already known — there was a
math.tointeger guard — but math.tointeger is nil under LuaJIT/5.1, so the
guard only covered PUC 5.3+, the platform where %d errors loudly. On LuaJIT,
where it fails silently, control fell straight through to the saturating call:

$ luajit -e 'print(string.format("%d", 1e300))'
9223372036854775807

The literal was destroyed in the generated Lua source, at compile time, before
evaluation ever ran:

before:  return (APP(S(">"), 9223372036854775807, 9223372036854775807))
after:   return (APP(S(">"), 1.0000000000000001e+300, 1e+100))

Both operands were literally the same number by the time Lua parsed them, which
is why > answered false and / answered 1.

catom (compiler.lua:282) already used %.17g for the same job — the two
literal paths disagreed with each other.

The fix

compiler.lua:302 — range-check up front, on every host, instead of relying on a
guard that only exists on one runtime:

local TWO63 = 9223372036854775808.0   -- 2^63, exact as a double
local function cnum(n)
  if n == math.floor(n) and n > -TWO63 and n < TWO63 then
    ...  -- %d path, unchanged
  end
  return string.format("%.17g", n)
end

Anything outside int64 falls through to the %.17g form catom already used.
Every finite double round-trips through it exactly (verified over the range in the
new spec). Integral values that genuinely fit keep their exact %d rendering — no
.0, no exponent form.

Non-finite values (±inf, NaN) fail the comparisons and take the same
fall-through path they took before: behaviour unchanged, deliberately.

Tests

test/numeric_literal_spec.lua (new, registered in scripts/run-tests.lua),
committed failing first (eec9690) so the fix demonstrably causes it to pass:

before fix 71 pass, 26 fail
after fix 97 pass, 0 fail

It covers three things: (1) every emitted literal is valid Lua that parses back to
the bit-identical double, across 0, ±1, 2^31, 2^52, 2^53, the largest
double below 2^63, 2^63, 2^64, 1e19, 1e100, 1e300, 1e308, 1e-300,
0.1, 1/3, pi; (2) ordinary integer literals still render exactly; (3) the
end-to-end comparisons and arithmetic.

Full suites, before (main @ dfa7950) and after:

make test      567 pass / 0 fail across 16 specs   ->   664 pass / 0 fail across 17 specs
make certify   134 passed / 0 failed / 100%        ->   134 passed / 0 failed / 100%

Cross-port

expression shen-go shen-cl shen-rust shen-lua before shen-lua after
(> 1e300 1e100) true true true false true
(/ 1e300 1e290) 10000000000 10000000000 10000000000.0 1 10000000000
(= 1e19 (* 1e18 10)) true true true false true

Out of scope (deliberately not addressed)

The identical %d-saturation pattern also sits on two rendering paths:
runtime.lua to_str (the printer) and prims.lua numToStr (the str
primitive). Left alone, because rendering of large finite values is a known open
cross-port question — shen-go prints 1e300 as 9223372036854775807 too, while
shen-cl and shen-rust print the full positional integer. This PR only fixes the
value. Worth a separate issue.

Also verified as non-issues: the boot bytecode cache keys on compiler.lua's
source (boot.lua cache_key), so it self-invalidates; and KDATA serialization
already used %.17g/tonumber and was never affected.

Reuben Brooks and others added 2 commits July 29, 2026 15:17
compiler.lua `cnum` renders every integral float with
string.format("%d", n). On LuaJIT/5.1 there is no math.tointeger to
guard with, and %d there SATURATES at 2^63-1 rather than erroring, so a
literal outside int64 range is destroyed in the emitted Lua source at
compile time, before evaluation:

  (> 1e300 1e100)      -> false  (should be true)
  (/ 1e300 1e290)      -> 1      (should be 1e10)
  (= 1e19 (* 1e18 10)) -> false  (should be true)

Generated source for the first, today:

  return (APP(S(">"), 9223372036854775807, 9223372036854775807))

Both operands are the same number by the time Lua sees them.

This commit adds the spec only; it FAILS (71 pass, 26 fail) so the fix
that follows is demonstrably what makes it pass. Ground truth for the
end-to-end assertions was taken from shen-go, shen-cl and shen-rust,
which all answer true / 10000000000 / true.

Rendering of large finite values and non-finite handling are known open
cross-port questions and are deliberately NOT asserted here.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`cnum` took the string.format("%d", n) path for ANY integral float. That
is only sound inside int64 range. PUC 5.3+ was guarded via
math.tointeger, but math.tointeger is nil under LuaJIT/5.1 -- and there
%d does not error, it SATURATES at 2^63-1. The literal was therefore
destroyed in the emitted Lua source, at compile time:

  before:  return (APP(S(">"), 9223372036854775807, 9223372036854775807))
  after:   return (APP(S(">"), 1.0000000000000001e+300, 1e+100))

Range-check up front on every host instead. Values outside int64 (and
non-finite values, whose handling is unchanged) fall through to the
%.17g form catom already uses for the same job; every finite double
round-trips through it exactly. Integral values that genuinely fit keep
their exact %d rendering -- no ".0", no exponent form.

  (> 1e300 1e100)       false -> true
  (/ 1e300 1e290)       1     -> 10000000000
  (= 1e19 (* 1e18 10))  false -> true

matching shen-go, shen-cl and shen-rust.

test/numeric_literal_spec.lua: 71 pass / 26 fail -> 97 pass / 0 fail.

The boot bytecode cache keys on compiler.lua's source (boot.lua
cache_key), so it self-invalidates; KDATA serialization already used
%.17g and was never affected.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pyrex41
pyrex41 merged commit c83c40e into main Jul 29, 2026
2 checks passed
@pyrex41
pyrex41 deleted the fix/cnum-int64-saturation branch July 29, 2026 20:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant