MDEV-35715: UBSAN: runtime error: 1e+19 is outside the range of representable values of type 'long long' in Field_bit::store on INSERT by kjarir · Pull Request #5081 · MariaDB/server

kjarir · 2026-05-15T15:35:44Z

This patch fixes a UBSAN float-cast-overflow runtime error triggered when inserting large floating-point values (e.g. 1e+19) into BIT, ENUM, or SET columns. The root cause was a direct (longlong) cast of an out-of-range double in Field_bit::store(double), Field_enum::store(double), and Field_set::store(double).

The fix adds explicit bounds checking before any cast. Out-of-range values emit ER_WARN_DATA_OUT_OF_RANGE as a warning or error depending on thd->really_abort_on_warning(), matching the behavior of Field_bit::store(const char *). In-range values are safely cast via (ulonglong) first to avoid undefined behavior.

How I tested it:
Added test cases to mysql-test/main/func_math.test covering overflow and underflow for BIT(1), BIT(8), BIT(64), ENUM, and SET columns in both strict and non-strict sql_mode. Ran the test suite with:
bash./mysql-test-run.pl --suite=main func_math --record
All tests passed with no UBSAN errors.

CLAassistant · 2026-05-15T15:36:05Z

All committers have signed the CLA.

gemini-code-assist

Code Review

This pull request performs extensive security hardening and bug fixes, notably replacing sprintf with snprintf across the codebase and the CONNECT storage engine to prevent buffer overflows. Key enhancements include strict numeric validation for master.info, LENENC support for long passwords in COM_CHANGE_USER, and improved handling of negative zero in floating-point storage. It also resolves various crash scenarios in GIS and JSON functions, memory leaks, and replication inconsistencies. Review feedback identified a critical type mismatch in sql/wsrep_applier.cc that could lead to undefined behavior during debug logging, and a potential stack buffer overflow in sql/mysqld.cc when handling Unix socket paths.

I am having trouble creating individual review comments. Click here to see my feedback.

sql/wsrep_applier.cc (103-104)

The change from a byte buffer to a std::vectorstd::string (implied by push_back(err_str)) seems to conflict with the subsequent WSREP_DEBUG call (line 106-108 in context) which uses dst.data() with a %s format specifier. If dst is now a vector of strings, dst.data() returns a std::string*, which will cause Undefined Behavior or a crash when passed to %s. Please ensure the debug print is updated to access the string content correctly, e.g., dst.empty() ? "" : dst[0].c_str().

sql/mysqld.cc (2707)

The use of strmov to copy path into addr.sun_path is potentially dangerous if path exceeds the fixed size of the sun_path array (typically 108 bytes). While there is a length check later in network_init, this utility function should ideally be self-protecting. Consider using strmake or adding an explicit length check before the copy to prevent a stack buffer overflow.

  if (strlen(path) >= sizeof(addr.sun_path))
  {
    sql_print_error("Unix socket path is too long: %s", path);
    unireg_abort(1);
  }
  strmov(addr.sun_path, path);

grooverdan · 2026-05-16T01:24:15Z

Lets get a working local test environment so mtr matches results before submitting.

grooverdan · 2026-05-26T07:30:09Z

-  return Field_bit::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());
+  if (nr < 0)
+  {


like Field_bit::store(const char * lets do error if thd->really_abort_on_warning() and warning otherwise. The implementation of this corresponds to the sql_mode=STRICT_ALL_TABLES which can form part of the tests.

Thank you for pointing this out. I'll update the implementation to check thd->really_abort_on_warning() and return an error in strict mode, and emit a warning otherwise matching the behavior of Field_bit::store(const char *). I'll also add a corresponding test case covering both strict and non-strict sql_mode behavior

Good attempt. Looking at other store cases error >0 is returned for both warnings and errors in *::store implementations.

Taking the store(const char * condition

if (get_thd()->really_abort_on_warning()) set_warning(ER_DATA_TOO_LONG, 1); else set_warning(ER_WARN_DATA_OUT_OF_RANGE, 1);

We don't want to return 1 until we've done the actual Field_bit::store(long/char

As we are passing a value that does error, to the ::store we still need to set the return value.

At error condition:

Field_bit::store(0LL, TRUE); return 1;

Likewise for the overflow case

ulonglong val= ULLONG_MAX; Field_bit::store((longlong)val, TRUE); return 1;

With these changes in place. if you look back at the code, its duplicating works. so restructure:

ulonglong val;; if (nr < 0.0) { val= 0; goto err; } else if (nr > ULLONG_MAX) { val= (longlong) ULLONG_MAX; goto err; } val= (longlong) nr; return Field_bit::store((ulonglong)val, TRUE); err: if (get_thd()->really_abort_on_warning()) set_warning(ER_DATA_TOO_LONG, 1); else set_warning(ER_WARN_DATA_OUT_OF_RANGE, 1); Field_bit::store((ulonglong)val, TRUE); return 1; }

grooverdan · 2026-05-26T07:30:36Z

+    set_warning(Sql_condition::WARN_LEVEL_WARN, ER_WARN_DATA_OUT_OF_RANGE, 1);
+    return Field_bit::store(0LL, TRUE);
+  }
+  if (nr >= 18446744073709551616.0) /* 2^64 */


nicely commented.

ULLONG_MAX! 😄

You're right @grooverdan, since BIT fields are unsigned, ULLONG_MAX is the correct upper bound here. I'll use ULLONG_MAX for the overflow clamp.

The point was to replace the number with the ULLONG_MAX define in the code.

grooverdan · 2026-05-26T07:47:56Z

+  {
+    set_warning(Sql_condition::WARN_LEVEL_WARN, ER_WARN_DATA_OUT_OF_RANGE, 1);
+    return Field_bit::store((longlong)ULLONG_MAX, TRUE);
+  }


You're casting twice below. Just (ulonglong) cast is sufficient. I note that (longlong) along generates the UBSAN error we aimed to avoid.

Understood. I'll remove the redundant double cast and use a single (ulonglong) cast, and then pass it directly. I'll make sure this path doesn't re-introduce the UBSAN error.

look closer 👀 , you actually didn't.

gkodinov

Thank you for your contribution! This is a preliminary review.

Couple of things:

your PR explanation does not match what you did.
you only cover bit. Neither enum nor set, as mentioned in the PR. Please add coverage for these.

gkodinov · 2026-05-26T08:43:23Z

+    set_warning(Sql_condition::WARN_LEVEL_WARN, ER_WARN_DATA_OUT_OF_RANGE, 1);
+    return Field_bit::store(0LL, TRUE);
+  }
+  if (nr >= 18446744073709551616.0) /* 2^64 */


gkodinov · 2026-05-26T08:44:25Z

 int Field_bit::store(double nr)
 {
-  return Field_bit::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());


use the utility class instead!

Which class?

I'm not sure what utility class @gkodinov is referring too.

Field_bit::store(const char *` where this function call ends up handles this assertion anyway. Could omit the assertion IMHO? WDYT @gkodinov ?

grooverdan · 2026-05-26T22:31:53Z

Thank you for your contribution! This is a preliminary review.

Couple of things:
* your PR explanation does not match what you did.

It did change over time. The commit message is good, though will need to be corrected once there is a warning option under the different SQL_MODE.

* you only cover bit. Neither enum nor set, as mentioned in the PR. Please add coverage for these.

If you're out of time for enum/set let us know and we can clone/edit the JIRA ticket and leaving it for someone else.

Direct (longlong) cast of out-of-range double values in Field_bit::store(double), Field_enum::store(double), and Field_set::store(double) causes undefined behavior (float-cast-overflow) flagged by UBSAN. Fix by adding explicit bounds checking before the cast: - Negative values clamp to 0 with ER_WARN_DATA_OUT_OF_RANGE warning/error. - Values >= 2^64 clamp to ULLONG_MAX with ER_WARN_DATA_OUT_OF_RANGE warning/error. - Warning vs error is determined by thd->really_abort_on_warning() to respect sql_mode=STRICT_ALL_TABLES behavior, matching Field_bit::store(const char *). - In-range values are safely cast via (ulonglong) before passing to the longlong overload, avoiding any direct double-to-longlong cast. Tests added to mysql-test/main/func_math.test covering BIT, ENUM, and SET overflow/underflow boundaries in both strict and non-strict sql_mode.

grooverdan

Thanks for adding set/enum. Good commit message. See if you can make a title that shows its more than just a Field_bit keeping in mind the length limit. (Field_*::double as last resort)

In CI - https://buildbot.mariadb.org/#/grid?branch=refs%2Fpull%2F5081%2Fhead - the main.type_enum test added an extra warning. I think with the recommended changes I described it shouldn't happen. Test it to be sure.

Suggest moving the enum test cases to main.type_enum. Take a look at MDEV-39043 around behaviours, implementation and results enum out of bound values.

Likewise for set tests into main.type_set.

grooverdan · 2026-05-27T23:35:24Z

+    set_warning(Sql_condition::WARN_LEVEL_WARN, ER_WARN_DATA_OUT_OF_RANGE, 1);
+    return Field_bit::store(0LL, TRUE);
+  }
+  if (nr >= 18446744073709551616.0) /* 2^64 */


The point was to replace the number with the ULLONG_MAX define in the code.

grooverdan · 2026-05-27T23:42:02Z

 int Field_bit::store(double nr)
 {
-  return Field_bit::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());


I'm not sure what utility class @gkodinov is referring too.

Field_bit::store(const char *` where this function call ends up handles this assertion anyway. Could omit the assertion IMHO? WDYT @gkodinov ?

grooverdan · 2026-05-28T01:11:25Z

-  return Field_bit::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());
+  if (nr < 0)
+  {


Good attempt. Looking at other store cases error >0 is returned for both warnings and errors in *::store implementations.

Taking the store(const char * condition

if (get_thd()->really_abort_on_warning()) set_warning(ER_DATA_TOO_LONG, 1); else set_warning(ER_WARN_DATA_OUT_OF_RANGE, 1);

We don't want to return 1 until we've done the actual Field_bit::store(long/char

As we are passing a value that does error, to the ::store we still need to set the return value.

At error condition:

Field_bit::store(0LL, TRUE); return 1;

Likewise for the overflow case

ulonglong val= ULLONG_MAX; Field_bit::store((longlong)val, TRUE); return 1;

With these changes in place. if you look back at the code, its duplicating works. so restructure:

ulonglong val;; if (nr < 0.0) { val= 0; goto err; } else if (nr > ULLONG_MAX) { val= (longlong) ULLONG_MAX; goto err; } val= (longlong) nr; return Field_bit::store((ulonglong)val, TRUE); err: if (get_thd()->really_abort_on_warning()) set_warning(ER_DATA_TOO_LONG, 1); else set_warning(ER_WARN_DATA_OUT_OF_RANGE, 1); Field_bit::store((ulonglong)val, TRUE); return 1; }

grooverdan · 2026-05-28T01:12:34Z

+  {
+    set_warning(Sql_condition::WARN_LEVEL_WARN, ER_WARN_DATA_OUT_OF_RANGE, 1);
+    return Field_bit::store((longlong)ULLONG_MAX, TRUE);
+  }


look closer 👀 , you actually didn't.

grooverdan · 2026-05-28T01:15:14Z

-  return Field_enum::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());
+  THD *thd= get_thd();
+  if (nr < 0.0 || nr >= 18446744073709551616.0)


While bit fields cannot be less than 0, sets and enums can.

grooverdan · 2026-05-28T01:25:53Z

 int Field_enum::store(double nr)
 {
-  return Field_enum::store((longlong) nr, FALSE);
+  DBUG_ASSERT(marked_for_write_or_computed());


if we look at the entire Field_enum::store(longlong nr, bool unsigned_val) it applies limits that do not include < 0 and the max value is typelib->count.

Given the short Field_enum::store(longlong nr, bool unsigned_val) it seem prudent to just copy the function and use keep nr as a double (without casting on the first comparision because that would cause a UBSAN error`.

grooverdan · 2026-05-28T01:44:12Z


+int Field_set::store(double nr)
+{
+  DBUG_ASSERT(marked_for_write_or_computed());


Like Field_enum::store(longlong nr, bool unsigned_val it seems more readable to follow Field_set::store(longlong nr, bool unsigned_val) but use double nr rather than trying to wrap it with similar boundary conditions.

Note that the number is a bit mapping. So keep the conversion in mind for your GSoC MDEV wrt to set handling.

When copied and nr is a double the code block (cast removed) is:

if (nr > max_nr) { nr&= max_nr; /* this is weird, even in the longlong case */ set_warning(WARN_DATA_TRUNCATED, 1); error=1; }

With longlong case the &= was weird ( why did > max_nr retain it lower bits - plausable case with replication to a reduced set), with double its unpredicatble as to the lower bits and precision.

Lets avoid some double casting with:

ulonglong nrl; /*define a beginning of function */ ... if (nr > max_nr) { nrl= max_nr; set_warning(WARN_DATA_TRUNCATED, 1); error=1; } else nrl= (ulonglong) nr; store_type(nrl); return error; }

grooverdan · 2026-05-28T01:48:15Z

+DROP TABLE t1;
+
+--echo # ENUM tests
+CREATE TABLE t1 (c ENUM('a', 'b', 'c'));


Try some numeric values (including negative) quoted in the enum types and see if you can insert a corresponding double.

grooverdan · 2026-05-28T01:52:36Z

+--error ER_WARN_DATA_OUT_OF_RANGE
+INSERT INTO t1 VALUES (-1.0);
+SET STATEMENT sql_mode='' FOR INSERT INTO t1 VALUES (-1.0);
+SELECT c, c+0 FROM t1;


I'm a bit surprised that a blank string is the result. I'm going to ask around if intentional or need to preserve this.

c+0 is what looks like an attempt to show its empty value. LENGTH(c) as l` is more direct.

grooverdan · 2026-05-28T01:57:19Z

+DROP TABLE t1;
+
+--echo # SET tests
+CREATE TABLE t1 (c SET('a', 'b', 'c'));


logic around inserting numbers into set seems to be considered a bitmap rather than a mapping to the set members. Examine other test for current behaviour.

I'd like to see here that integer values behave like their equivillent double values, and not just for error/warning conditions.

Include some 1.0, 2.0, 3.0 values and see the results.

gkodinov

It's good progress: thanks for covering the rest of the conversions mentioned.
Also, the test is failing in buildbot: please always check that before submitting for review since this is usually the first thing I look at.

gkodinov · 2026-05-28T07:44:01Z

+CREATE TABLE t1 (c BIT(1));
+INSERT INTO t1 VALUES (0.0), (0.1), (0.9), (1.0), (1.5);
+Warnings:
+Warning	1264	Out of range value for column 'c' at row 5


It'd be nice I guess to say which of the 5 values this was. But this is for the final reviewer to decide on.

it does say "row 5"

gkodinov · 2026-05-28T07:48:36Z

 SELECT SIGN(-' 0 ');

 --echo # End of 13 tests
+


I'd move these to type_xxx.test. It's not really math functions.

gkodinov · 2026-05-28T07:49:24Z


 int Field_enum::store(double nr)
 {
-  return Field_enum::store((longlong) nr, FALSE);


Converter_double_to_longlong please

enums being character values need some of the double properties to be compared.

Alternate already suggested:

#5081 (comment)

gkodinov · 2026-05-28T07:49:48Z


+int Field_set::store(double nr)
+{
+  DBUG_ASSERT(marked_for_write_or_computed());


Converter_double_to_longlong please

Sets are using this store as a bitmask, which becomes slightly odd with a store of double with is precision differences (that can exist within 64 bits).

Alternate: #5081 (comment)

kjarir force-pushed the MDEV-35715-fix-10.11 branch 2 times, most recently from c20fff0 to a6268f5 Compare May 15, 2026 15:40

gemini-code-assist Bot reviewed May 15, 2026

View reviewed changes

grooverdan self-assigned this May 16, 2026

grooverdan marked this pull request as draft May 16, 2026 01:16

gkodinov added the External Contribution All PRs from entities outside of MariaDB Foundation, Corporation, Codership agreements. label May 18, 2026

kjarir marked this pull request as ready for review May 24, 2026 05:55

kjarir force-pushed the MDEV-35715-fix-10.11 branch from a6268f5 to 0fbd468 Compare May 24, 2026 06:03

kjarir changed the base branch from main to 10.11 May 24, 2026 06:05

kjarir force-pushed the MDEV-35715-fix-10.11 branch 7 times, most recently from b7ae0fb to 4163676 Compare May 24, 2026 07:24

grooverdan requested changes May 26, 2026

View reviewed changes

gkodinov requested changes May 26, 2026

View reviewed changes

kjarir force-pushed the MDEV-35715-fix-10.11 branch 2 times, most recently from 8567003 to 0e547fc Compare May 27, 2026 10:20

kjarir force-pushed the MDEV-35715-fix-10.11 branch from 0e547fc to 6e85cfa Compare May 27, 2026 10:28

kjarir requested review from gkodinov and grooverdan May 27, 2026 10:48

grooverdan requested changes May 28, 2026

View reviewed changes

gkodinov requested changes May 28, 2026

View reviewed changes

Uh oh!

Conversation

kjarir commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

CLAassistant commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

sql/wsrep_applier.cc (103-104)

sql/mysqld.cc (2707)

Uh oh!

grooverdan commented May 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kjarir May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gkodinov left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

grooverdan commented May 26, 2026

Uh oh!

grooverdan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kjarir commented May 15, 2026 •

edited

Loading

CLAassistant commented May 15, 2026 •

edited

Loading

kjarir May 27, 2026 •

edited

Loading

grooverdan May 29, 2026 •

edited

Loading