Add changes for compatibility with WASM components and collocated UDF servers

[kesmit13]

This PR makes several changes to allow the singlestoredb work in WASM environments. Many of these changes benefit standard installations as well such as lazy loading of numpy, pandas, polars, and pyarrow. Others move imports that are only needed in certain environments, but not within WASM.

A new collocated UDF server implementation is also included that uses a high-performance loop in the C extension to parse and call Python functions on each row. This function is used both by standard collocated servers as well as WASM-based UDF handlers.

---

Note

High Risk
High risk due to substantial changes to the C extension’s row serialization/parsing (new DECIMAL/DATE/TIME/TIMESTAMP handling, pointer-safe reads) and addition of fd-level socket/mmap I/O helpers used in a new UDF server, which could affect correctness, memory safety, and protocol compatibility.

Overview
Adds a new collocated Python UDF server (singlestoredb.functions.ext.collocated) with a CLI entry point (python-udf-server) that speaks the Rust wasm-udf-server-compatible Unix-socket + mmap protocol, supports control signals (@@health, @@functions, @@register), and can run in thread or pre-fork process mode.

Extends the _singlestoredb_accel C extension with call_function_accel (combined load/call/dump hot path) plus fd-level helpers (mmap_read, mmap_write, recv_exact with optional poll-based timeouts), and updates rowdat parsing/dumping to support DECIMAL/NEWDECIMAL, DATE, TIME, and DATETIME/TIMESTAMP (including Python decimal.Decimal, datetime/date/timedelta conversions) while replacing unsafe unaligned pointer casts with memcpy and tightening length/error checks.

Improves WASM/optional-dependency compatibility by lazily importing numpy/pandas/polars/pyarrow, moving jwt imports to call sites and deferring auth handler initialization, broadening result-shape handling in Connection._iquery (DataFrame/arrow/numpy/list), adding VECTOR() dtype construction, and updating JSON encoding and tests for newer pandas string dtypes; also includes a resources/build_wasm.sh wheel build helper.

^{Reviewed by Cursor Bugbot for commit 4c7fb60. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

This PR introduces WASM-compatibility improvements (primarily by lazy-loading heavyweight optional dependencies and moving environment-specific imports into call sites) and adds a new collocated Python UDF server implementation, including a new C-extension hot path to accelerate rowdat_1 decode → Python call → rowdat_1 encode.

Changes:

• Added a WIT interface definition and a WASM build helper script for external UDF component workflows.
• Refactored optional dependency handling (numpy/pandas/polars/pyarrow, IPython, JWT) to be more robust in constrained/WASM-like environments.
• Added a new collocated UDF server (socket + mmap protocol, thread/process modes, dynamic registration) and a C-extension accelerator entry point (call_function_accel).

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
wit/udf.wit	Defines the external UDF WIT interface and exported world.
singlestoredb/utils/_lazy_import.py	Adds cached lazy imports for heavy optional deps.
singlestoredb/utils/dtypes.py	Converts dtype maps to lazily-evaluated, cached getters.
singlestoredb/utils/results.py	Switches result formatting to lazy imports + cached type maps.
singlestoredb/utils/events.py	Broadens IPython import failure handling.
singlestoredb/converters.py	Uses lazy numpy import in vector converters.
singlestoredb/connection.py	Adjusts internal result-to-dict conversion to avoid importing pandas.
singlestoredb/mysql/connection.py	Adds WASM-friendly DEFAULT_USER detection (handles OSError).
singlestoredb/auth.py	Moves jwt import into call site.
singlestoredb/management/utils.py	Moves jwt import into call sites for WASM-friendliness.
singlestoredb/management/manager.py	Moves jwt import into is_jwt call site.
singlestoredb/functions/dtypes.py	Updates exports to use dtype-map getter functions.
singlestoredb/functions/ext/rowdat_1.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/json.py	Replaces eager dtype maps with lazy getter functions.
singlestoredb/functions/ext/collocated/*	Adds collocated server, protocol handling, registry, control signals, and WASM adapter.
singlestoredb/tests/test_connection.py	Makes pandas string dtype assertions version-tolerant.
resources/build_wasm.sh	Adds a build helper for wasm32-wasip2 wheels.
pyproject.toml	Adds python-udf-server CLI entry point.
accel.c	Adds call_function_accel C hot path and exports it from the extension module.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated 11 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[cursor]

Socket timeout permanently lost after partial read recovery

Medium Severity

_recv_exact_py calls sock.settimeout(None) when recovering from a partial-read timeout, but the caller in _handle_udf_loop never restores the 0.1s timeout afterward. Once this rare path executes, all subsequent recv_into calls block indefinitely, meaning the shutdown_event.is_set() check in the while loop is never reached. This prevents graceful shutdown of the Python-fallback worker. The C accel path avoids this by using internal poll() that doesn't mutate the socket state.

Additional Locations (1)

• singlestoredb/functions/ext/collocated/connection.py#L269-L286

 

[cursor]

NEWDECIMAL added to string_types shadows decimal handler

High Severity

Value 246 (ft.NEWDECIMAL) was added to string_types, but it's also in the new decimal_types set. Since string_types is checked before decimal_types in the if-elif chains of _load, _load_vectors, _dump, and _dump_vectors, the decimal_types handler is unreachable for NEWDECIMAL. This causes NEWDECIMAL values to be returned as plain strings instead of decimal.Decimal on load, and on dump it will call .encode('utf-8') on a Decimal object, raising an AttributeError.

Additional Locations (1)

• singlestoredb/functions/ext/rowdat_1.py#L92-L93

 

^{Reviewed by Cursor Bugbot for commit a7fb74d. Configure here.}

[cursor]

PyLong key leaked in PyDict_SetItem calls

Medium Severity

PyLong_FromUnsignedLongLong(u64) returns a new reference, but PyDict_SetItem does not steal references — it increments the refcount of both key and value. The temporary PyLong key is never Py_DECREF'd, leaking memory on every non-null DECIMAL, DATE, TIME, and DATETIME/TIMESTAMP value processed through the numpy loading path (load_rowdat_1_numpy).

Additional Locations (2)

• accel.c#L2609-L2610
• accel.c#L2646-L2673

 

^{Reviewed by Cursor Bugbot for commit a7fb74d. Configure here.}

[cursor]

C TIME encoding disagrees with Python pack algorithm

High Severity

The C TIME encoding computes total_secs from days*86400 + seconds and keeps microseconds separate, then negates them independently. The Python _pack_time instead computes total microseconds via total_seconds() * 1_000_000 first, then decomposes. For negative timedeltas with non-zero microseconds (e.g., -22:59:59.5), the C approach produces a corrupted packed value because sign_v * total_us has the opposite sign from the hhmmss component, causing the packed digits to bleed. The round-trip gives a wrong timedelta.

Additional Locations (2)

• accel.c#L4990-L4998
• accel.c#L5773-L5781

 

^{Reviewed by Cursor Bugbot for commit f3d069a. Configure here.}

[cursor]

Null DECIMAL doesn't advance data past string bytes

Low Severity

In the DECIMAL handling for both load_rowdat_1_numpy (second pass) and load_rowdat_1, when is_null is true, data advances past the 8-byte length prefix but not past i64 bytes of string data. The first sizing pass always advances by both 8 and i64. If a producer ever emits a non-zero length for a null DECIMAL value, the data pointer would desync with subsequent columns and rows. Currently safe only because null encoding always uses length zero.

Additional Locations (1)

• accel.c#L4553-L4567

 

^{Reviewed by Cursor Bugbot for commit f3d069a. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

There are 7 total unresolved issues (including 5 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 4c7fb60. Configure here.}

[cursor]

NULL datetime/date/decimal crashes Python fallback deserializer

High Severity

In _load and _load_vectors, the new datetime/date/decimal deserialization runs unconditionally before the is_null check at row.append(None if is_null else val). For NULL values, the packed wire value is 0, so _unpack_datetime(0) calls datetime(0, 0, 0) which raises ValueError (year/month/day must be >= 1), _unpack_date(0) similarly crashes, and decimal.Decimal('') raises InvalidOperation. The C accelerator correctly checks is_null before unpacking, but the Python fallback does not — which is the path used in WASM, the primary target of this PR.

Additional Locations (1)

• singlestoredb/functions/ext/rowdat_1.py#L270-L287

 

^{Reviewed by Cursor Bugbot for commit 4c7fb60. Configure here.}

[cursor]

Floating-point precision loss in _pack_time for timedeltas

Low Severity

_pack_time computes int(td.total_seconds() * 1_000_000) which involves float64 arithmetic. total_seconds() returns a float, and the multiplication can lose microsecond precision for large timedelta values. Using integer-only arithmetic like td.days * 86400_000_000 + td.seconds * 1_000_000 + td.microseconds would avoid this entirely. The C code reads days, seconds, microseconds as separate integers and avoids this issue.

 

^{Reviewed by Cursor Bugbot for commit 4c7fb60. Configure here.}