Skip to content

Fix memory safety issues#3188

Merged
texodus merged 1 commit into
masterfrom
memory-safety-fixes
Jun 23, 2026
Merged

Fix memory safety issues#3188
texodus merged 1 commit into
masterfrom
memory-safety-fixes

Conversation

@texodus

@texodus texodus commented Jun 19, 2026

Copy link
Copy Markdown
Member

I asked Claude to find memory safety bugs in Perspective and write a PR, while I played Balatro on my phone. Here's its own summary of what it found:

Fixes

  • Uneven column lengths in columnar table create/update (table.cpp
    Table::from_cols, Table::update_cols): the table was sized from a single
    column's length while every column was filled to its own length. Now the
    table is sized to the longest column and shorter columns are null-padded, so
    all writes stay in bounds.

  • NDJSON row under-allocation (table.cppTable::from_ndjson): capacity
    was estimated from the newline count but one row was written per parsed
    object. The table is now grown per row (amortized O(1) via geometric capacity
    growth), so concatenated objects without newline separators can no longer
    overrun the buffer.

  • Arrow row-count truncation (arrow_loader.cppArrowLoader::row_count):
    Arrow's 64-bit row count was silently truncated to 32 bits, under-sizing the
    destination table relative to the data written during fill. Oversized/negative
    counts are now rejected instead of truncated.

  • Arrow time32 element width (arrow_loader.cppcopy_array): time32
    values are 32-bit and map to a 4-byte column, but the loader copied 8 bytes
    per element, over-reading the source and over-writing the destination. Now
    copies 4 bytes per element.

  • first/last aggregate with a missing sort dependency (sparse_tree.cpp
    first_last_helper): the helper assumed the aggregate spec always carried
    both a value and a sort dependency and indexed the dependency list
    unconditionally. A view whose sort column falls outside the visible set could
    produce a spec without the sort dependency, causing an out-of-bounds read. Now
    guarded (covers first, last, and last − first).

  • count aggregate over a string column reads at the wrong stride
    (aggregate.cppt_aggregate::init, AGGTYPE_COUNT / DTYPE_STR): a
    string column's backing store holds 4-byte t_uindex vocabulary indices, but
    the count aggregate was instantiated as t_aggimpl_count<std::uint64_t, …>,
    so build_aggregatet_column::fill read the input column at an 8-byte
    (uint64) stride — double the real element width — and ran off the end of the
    buffer. The read is pure waste (t_aggimpl_count::reduce discards the values;
    the count is filled in later from nstrands), but it still tripped a storage
    bounds check. The raw type is now t_uindex, matching the storage width. This
    was a latent over-read for years, masked by push_back's ~20% capacity
    headroom; tighter column sizing in a later optimization removed the slack and
    exposed it. It is reachable from every pivoted (group_by/split_by) view
    with a string column, since the default per-column aggregate for a string is
    count and that aggregate is built during view construction — so the abort
    fired before the view's actual query (get_min_max, to_columns, etc.) ever
    ran. Found by updating and enabling PSP_STORAGE_VERIFY in debug builds.

  • Residency eviction data race (residency.cpp, residency.h): the
    shared pending-eviction vectors were cleared outside the manager's mutex, so
    concurrent request-thread eviction passes could double-free. All mutations now
    occur under the lock, and a dedicated mutex serializes each eviction cycle.

  • Unvalidated Arrow input (arrow_loader.cppArrowLoader::initialize):
    Arrow's IPC reader does not validate buffer contents, yet the fill paths index
    value/offset/dictionary buffers directly. The loaded (remotely supplied) table
    is now fully validated (ValidateFull()) before those buffers are trusted, so
    a malformed payload — bad offsets, out-of-range dictionary indices, inconsistent
    chunk lengths — is rejected instead of read out of bounds. This is the systemic
    defense behind the time32 and row-count fixes above. Note: validation is
    O(data) per ingested payload; Validate() plus targeted bounds checks would be
    a cheaper alternative if ingest throughput matters.

  • Out-of-bounds access in expression vector functions (computed_function.cpp
    diff3, norm3, cross_product3, dot_product3): these operate on
    3-element vectors and indexed v[0..2] unconditionally, but their exprtk
    parameter sequences ("VVV"/"VV"/"V") enforce vector type, not length.
    A user expression can declare shorter vectors (e.g. var v[2] := {1,2}; norm3(v)), causing an out-of-bounds read — and for diff3/cross_product3,
    an out-of-bounds write to the (short) output vector. Each function now
    clears its result for vectors shorter than 3 before indexing; this surfaces
    as an invalid expression (rejected at view creation) rather than an
    out-of-bounds access. Vectors of length 3 are unaffected.

@texodus texodus added the bug Concrete, reproducible bugs label Jun 19, 2026
@texodus texodus requested a review from timkpaine June 19, 2026 14:22
@texodus texodus force-pushed the memory-safety-fixes branch 3 times, most recently from aa54fef to 30d26c8 Compare June 22, 2026 03:28
hex.match(/.{2}/g)!.map((b) => parseInt(b, 16)),
);
await expect(perspective.table(bytes)).rejects.toThrow(
/row count exceeds maximum supported size/,

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is brittle and irrelevent, we don't cate about the message.

Comment thread rust/perspective-js/test/js/constructors/arrow.spec.ts Outdated
Comment thread rust/perspective-js/test/js/constructors/arrow.spec.ts Outdated
expect(paths.length).toEqual(3);
expect(paths).toContainEqual([]);
expect(paths).toContainEqual(["x"]);
expect(paths).toContainEqual(["y"]);

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These data structures are small, for readability its worth just asserting the entire structure instead of multiple playwright assertion calls in a row.

const auto validation = m_table->ValidateFull();
if (!validation.ok()) {
PSP_COMPLAIN_AND_ABORT(validation.ToString());
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked that this has a minimal (nonexistent) performance impact.

if (n < 0
|| static_cast<std::uint64_t>(n)
> std::numeric_limits<std::uint32_t>::max()) {
PSP_COMPLAIN_AND_ABORT(

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not something Perspective itself will check internally, e.g. a subsequent update call will still cause the same issue this verbose error messages "fixes".

if (v1.size() < 3 || v2.size() < 3 || out.size() < 3) {
rval.m_status = STATUS_CLEAR;
return rval;
}

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an actual good catch but we can validate this better by just checking the exact cases and erroring else?

case AGGTYPE_WEIGHTED_MEAN: {
auto pkeys = get_pkeys(nidx);

if (spec.get_dependencies().size() < 2) {

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think empty categories can be constructed with the current config but this fix satisfies the analysis.

Signed-off-by: Andrew Stein <steinlink@gmail.com>
@texodus texodus force-pushed the memory-safety-fixes branch from 30d26c8 to c4cb7a3 Compare June 22, 2026 04:48
@texodus texodus merged commit 82eaa40 into master Jun 23, 2026
44 of 46 checks passed
@texodus texodus deleted the memory-safety-fixes branch June 23, 2026 04:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Concrete, reproducible bugs

Development

Successfully merging this pull request may close these issues.

1 participant