Skip to content

fix: render out-of-int64 values positionally instead of saturating - #52

Merged
pyrex41 merged 2 commits into
mainfrom
fix/render-int64-saturation
Jul 29, 2026
Merged

fix: render out-of-int64 values positionally instead of saturating#52
pyrex41 merged 2 commits into
mainfrom
fix/render-int64-saturation

Conversation

@pyrex41

@pyrex41 pyrex41 commented Jul 29, 2026

Copy link
Copy Markdown
Owner

c83c40e fixed the int64-saturation class at compile time (cnum), so
shen-lua now computes large values correctly. It deliberately left the
rendering paths out of scope, and they still saturated:

(* 1000000000.0 10000000000.0)   =>  9223372036854775807
(str 1e19)                       =>  9223372036854775807
1e300                            =>  9223372036854775807
-1e19                            => -9223372036854775808

Two sites, same root cause: string.format("%d", n) on a finite integral
double. Under LuaJIT/5.1 there is no math.tointeger to guard with and %d
saturates rather than erroring, so everything at or beyond int64 range
collapsed onto one sentinel.

  • runtime.lua to_str — the printer (REPL echo, shen.tostring, specs)
  • prims.lua numToStr — the Shen str primitive

The rule, now in exactly one place

runtime.num_to_str, called by both surfaces. They are two surfaces on one
rule and have diverged in this repo before, so prims.lua no longer carries
its own copy.

input rendering
finite integral, ` x
finite integral, ` x
non-integral finite shortest_float — unchanged (issue #24)
+inf / -inf / NaN inf / -inf / nan — unchanged; overflow policy deliberately untouched

1e19 -> 10000000000000000000. 1e300 -> 301 digits.

Why shortest-round-trip and not the double's exact value (%.0f)? It
round-trips; it is what shen-rust prints for the same double (checked
byte-for-byte, below); and it is the rule this printer already applies to every
non-integral value. %.0f would match nobody. The boundary is sign-symmetric
(|x| >= 2^63, matching cnum's existing > -TWO63) so -2^63 renders as the
negation of 2^63 instead of exposing the two's-complement asymmetry of %d.

Cross-port verification

Given the same double, the new renderer is byte-for-byte identical to a
fresh shen-rust on all 20 anchored expressions tested (1e19, -1e19, 1e20,
1e21, 1e100, 1e300, -1e300, 1e308, 2^63, -2^63, 2^64,
12345678901234567890, and reader-independent products such as
(* 4611686018427387904.0 2.0)).

expression before after shen-cl shen-rust shen-go
1e19 9223372036854775807 10000000000000000000 10000000000000000000 10000000000000000000 1e+19
(str 1e19) 9223372036854775807 10000000000000000000 10000000000000000000 10000000000000000000 1e+19
(* 1000000000.0 10000000000.0) 9223372036854775807 10000000000000000000 same same 1e+19
-1e19 -9223372036854775808 -10000000000000000000 same same -1e+19
1e300 9223372036854775807 1000...0 (301 digits) 301 digits identical to ours 1.0000000000000002e+300
18446744073709551616 9223372036854775807 18446744073709552000 18446744073709551616 18446744073709552000 1.8446744073709552e+19
+inf / -inf / NaN inf / -inf / nan unchanged raises inf/-inf/NaN inf/-inf/nan
42, 0, -42, 2^53, 0.1, 1/3, 1e-300 all unchanged

Where shen-lua and shen-cl still differ on a big literal (2^64,
12345678901234567890) it is because shen-cl has bignums and keeps the
literal exact; shen-lua only has doubles and cannot. Where shen-lua and
shen-rust differ on 1e300's digits it is a reader difference, not a
renderer one — shen-lua, shen-go and shen-rust all read 1e300 as
1.0000000000000002e300 and print identical digits from it.

The .0 question (investigated, not "fixed")

shen-rust prints (/ 1e300 1e290) as 10000000000.0 but 1e19 as
10000000000000000000. This is not a magnitude rule and shen-cl does not
agree with it:

expression shen-cl shen-rust shen-lua shen-go
(/ 1e300 1e290) 10000000000 10000000000.0 10000000000 10000000000
(* 1.0 5) 5 5.0 5 5
(* 2.5 2) 5.0 5.0 5 5
(/ 10.0 2.0) 5 5.0 5 5

The .0 is a type tag, not a formatting choice. shen-cl and shen-rust both
have a distinct integer type alongside floats and suffix .0 on the float one;
they disagree with each other because their readers disagree about which
literals are integers (shen-cl's reader makes 1.0 and 1e300 exact integers;
shen-rust's does not). shen-lua and shen-go have a single number type — a
double — and render every integral value bare. Reproducing .0 would require
introducing an int/float distinction across the whole port and would turn 42
into 42.0. Out of scope, orthogonal to this defect, and reported rather than
guessed at.

Tests

test/numeric_render_spec.lua (new, registered in scripts/run-tests.lua),
committed failing first (caaedd6: 153 pass / 42 fail) then fixed
(009e51f: 195 pass / 0 fail). It asserts both rendering paths on every
case and asserts that they agree with each other, plus the collateral:
ordinary integers, negatives, zero, 2^53, the largest double below 2^63,
non-integral floats, 1e-300, and inf/-inf/nan.

make test      664 pass / 0 fail / 17 specs   ->   859 pass / 0 fail / 18 specs
make certify   134 / 0 / 100%                 ->   134 / 0 / 100%

Not changed: overflow policy, compiler.lua cnum, boot.lua's %.17g image
serialisation, and shen-go / shen-rust (owned elsewhere).

Reuben Brooks and others added 2 commits July 29, 2026 17:03
Both rendering paths -- runtime.lua to_str (the printer) and prims.lua
numToStr (the str primitive) -- render a finite integral double with
string.format("%d", n). Under LuaJIT/5.1 there is no math.tointeger to
guard with and %d SATURATES rather than erroring, so every value outside
int64 range prints as the same sentinel:

    (* 1000000000.0 10000000000.0)  =>  9223372036854775807
    (str 1e19)                      =>  9223372036854775807
    1e300                           =>  9223372036854775807

c83c40e fixed the compile-time literal path (cnum) so the VALUE is now
right; only the rendering is destroyed. shen-cl and shen-rust both print
the full positional integer.

The spec asserts both paths, asserts they agree with each other, and
pins the collateral (ordinary integers, negatives, zero, 2^53, the max
double below 2^63, non-integral floats, 1e-300, inf/-inf/nan). Every
big-value expectation was taken byte-for-byte from a fresh shen-rust.

Currently: 153 pass, 42 fail.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
runtime.lua to_str (the printer) and prims.lua numToStr (the `str`
primitive) each rendered a finite integral double with
string.format("%d", n). LuaJIT/5.1 have no math.tointeger to guard with
and there %d SATURATES instead of erroring, so every value at or beyond
int64 range printed as one sentinel:

    (* 1000000000.0 10000000000.0)  =>  9223372036854775807
    (str 1e19)                      =>  9223372036854775807
    1e300                           =>  9223372036854775807

c83c40e fixed the same class at compile time (cnum) so the VALUE was
already right; only the rendering was destroyed.

Rule now, in ONE place (runtime.num_to_str):
  * finite integral, |x| < 2^63 -- exact %d, completely unchanged;
  * finite integral, |x| >= 2^63 -- shortest round-trippable decimal,
    expanded positionally: no exponent at any magnitude. 1e19 renders as
    10000000000000000000, 1e300 as 301 digits;
  * everything else (non-integral, +-inf, NaN) -- shortest_float,
    unchanged. Overflow policy is deliberately untouched.

Shortest-round-trip rather than the double's exact value (%.0f) because
it round-trips, it is what shen-rust prints for the same double (checked
byte-for-byte on 20 expressions), and it is the rule this printer already
applies to non-integral values (issue #24). The boundary is
sign-symmetric so -2^63 is the negation of 2^63 rather than exposing the
two's-complement asymmetry of %d.

prims.lua no longer carries its own copy of the rule -- it now calls
R.num_to_str. The duplicate is how the two paths diverged before.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pyrex41
pyrex41 merged commit 9f6fb35 into main Jul 29, 2026
2 checks passed
@pyrex41
pyrex41 deleted the fix/render-int64-saturation branch July 29, 2026 22:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant