[WIP][core] Fix column masking correctness in query-auth reads#8570
Draft
plusplusjiajia wants to merge 4 commits into
Draft
[WIP][core] Fix column masking correctness in query-auth reads#8570plusplusjiajia wants to merge 4 commits into
plusplusjiajia wants to merge 4 commits into
Conversation
plusplusjiajia
force-pushed
the
core-mask-remap-fail-closed
branch
4 times, most recently
from
July 12, 2026 04:39
6bb3c6c to
117e3e5
Compare
plusplusjiajia
force-pushed
the
core-mask-remap-fail-closed
branch
14 times, most recently
from
July 12, 2026 12:54
62a53ad to
5a49a17
Compare
plusplusjiajia
force-pushed
the
core-mask-remap-fail-closed
branch
3 times, most recently
from
July 12, 2026 16:52
62849c7 to
17823a0
Compare
plusplusjiajia
changed the title
Contributor
|
plusplusjiajia
changed the title
Member
Author
|
@JingsongLi Agreed. I'll fold the masked-column pushdown safety into this PR so masking is correct end to end in one place, and close #8582. Updating this PR shortly. |
plusplusjiajia
changed the title
JingsongLi
marked this pull request as draft
July 14, 2026 13:23
A cross-column mask (e.g. display := concat_ws('-', first, last)) threw at
read time when the query projected the masked target but not the mask's
input columns: "Column masking refers to field 'first' which is not present
in output row type".
Row-filter operands are already added to the read projection and projected
back out (apache#8447); do the same for column-mask inputs, transitively. The
widened read type is pushed before planning so file-level column pruning
keeps the files those columns live in, and every split is projected back to
the query's read type.
Stale rules -- columns absent from the latest schema after a rename or drop
-- fail closed at plan time. The read schema is fixed once the first split
reader exists, so auth-added columns no longer leak into later splits of the
same TableRead; that leak also affected the existing row-filter path.
Scoped to query-auth.enabled tables.
A column mask changes the value domain of its target, so a predicate on a masked column must evaluate on the masked value. Pushing it into raw statistics compares it against the raw value instead and prunes away the files the query should have matched -- an equality filter on a masked column returned an empty result. Scan side: the query filter is deferred to plan(), and only the conjuncts free of masked columns feed statistics and partition pruning. Masked conjuncts drop rows at read time only, so their presence also keeps limit/TopN split pruning off; TopN pruning additionally skips masked ordering columns. A mask appearing on an already-pushed filter column fails closed, since raw statistics were already consumed. Read side: the filter is not forwarded to the reader internals; conjuncts on masked columns are evaluated inside the auth read, post-mask. A masked filter column the query does not project is widened in and projected back out, reusing the mask-input machinery. Scoped to query-auth.enabled tables.
plusplusjiajia
force-pushed
the
core-mask-remap-fail-closed
branch
from
July 25, 2026 01:50
17823a0 to
fecdac2
Compare
Follow-up on the two commits above, fixing three defects found while reviewing them. A conjunct is retained for post-mask evaluation when it references any masked column, but splitAnd does not split a disjunction, so `masked_a = 'x' OR b = 'y'` is retained whole while only `a` was widened into the read schema. Reading then failed with "Filter on masked columns [a] cannot be evaluated on read schema", rejecting a valid query. Widen with every column the retained conjuncts reference, not just the masked ones. Scan planning seeded the auth widening from the query projection alone, so a filter on an unprojected masked column -- and whatever that mask reads -- did not reach the planning read type, and column-file pruning could drop the files holding those columns before the reader widens them back in. Seed the widening with the post-mask filter columns as well. listPartitionEntries() bypasses plan() and pushed the filter with an empty mask set, comparing conjuncts on masked columns against raw partition values; a later plan() on the same scan then rejected the masks it discovered as a rule change against an already-pushed filter. Resolve the auth rules in the listing path too.
…umns Two defects from reviewing the previous commit. validateReadableWithoutRename only rejected a masked column exposed under a different name. A column that was dropped and later re-added keeps its name but gets a fresh field id, so a time-travel read of a pre-drop snapshot passed the name check and the rule then resolved to the unrelated historical column -- masking with, or exposing, values it was never written for. Compare field ids and fail closed when the same name is a different column. Scan planning added the post-mask filter columns to the seed of requiredAuthFields, but that method returns what the rules read (filter operands and mask inputs), not the seed itself, so an unmasked operand of a retained conjunct was still absent from the planning read type and its column file could be pruned. The read side already added them explicitly; do the same here.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
A cross-column mask (e.g. display := concat_ws('-', first, last)) throws at read time when the query projects the masked target but not the mask's input columns:
Column masking refers to field 'first' which is not present in output row type
Row-filter operands are already added to the read projection (#8447); this PR does the same for column-mask inputs, and hardens the paths that surfaced while testing it:
(preserving nested pruning), the widened columns are projected back out after masking, and the scan pushes the widened read type before planning so file-level column pruning (e.g. data evolution) keeps the
files those columns live in.
re-validated only when the fetched rules change. Projected system fields (e.g. _ROW_ID) stay valid; unprojected ones, nested-pruned columns and blob-view columns are rejected.
only to columns readable from the query, so a column merely retained in a previously widened schema cannot activate a mask after a rules change. Previously the first createReader call mutated this.readType, so
later splits of the same TableRead leaked the auth-added columns — also on the pre-existing row-filter path.
output type.
Behavior changes are scoped to query-auth.enabled tables; tables without query auth and the write path are unaffected.
Out of scope: raw-value pushdown on masked columns (filter/TopN/limit statistics pruning, masked partition predicates) is unsafe today on master for the already-merged masking feature. This PR does not change
any pushdown behavior; the fix is #8582 (stacked on this PR), where a predicate on a masked column evaluates on the masked value.
Tests
changing between reads of one TableRead, repeated reads.