Fix precision of real32quant for 32 bit float - #86
Conversation
3734f5d to
6e3fe69
Compare
hahnjo
left a comment
There was a problem hiding this comment.
I think this change would benefit from more details in the commit message: what does ROOT do (compute in double), what does the macro request (single-precision float so the value is cast), why is that not possible in JavaScript (no single precision float type), and how does the change solve it (Math.fround "cutting off" the mantissa).
Also you need to update jsroot/types/fundamental/README.md to remove the caveat documented so far.
6e3fe69 to
3008012
Compare
|
regarding the commit message:
Not quite, the quantization is eventually written as integers with a fixed number of bits. The important point is that the in-memory field type can be single- or double-precision, depending on the requested template argument.
The mantissa of 32-bit floating-point numbers are 23 bits. I think the important aspect of |
3008012 to
d5e2232
Compare
|
@undefined-panda can you please rebase to resolve the (trivial) conflict? Proposal for the commit message: |
e20e1b4 to
4a34699
Compare
Real32Quant is stored as integers on disk and then transformed back to floating-point numbers using double-precision arithmetic. In the ROOT (C++) implementation, the value is then cast according to the requested field type, which can have either single-precision (float) or double-precision. Since JavaScript only has double-precision floats, Math.fround is used to get the nearest 32-bit single-precision number.
4a34699 to
a2dfbaa
Compare
No description provided.