Filter pgsqlite internal objects from client sqlite_master queries#82
Closed
robertjbass wants to merge 4 commits into
Closed
Filter pgsqlite internal objects from client sqlite_master queries#82robertjbass wants to merge 4 commits into
robertjbass wants to merge 4 commits into
Conversation
pgsqlite keeps its own bookkeeping in tables named __pgsqlite_* (schema, migrations, enum metadata, string/numeric constraints, datetime cache, etc.). The catalog handlers that pgsqlite synthesizes itself already exclude these (e.g. src/catalog/pg_class.rs appends "AND name NOT LIKE '__pgsqlite_%'"), but a sqlite_master / sqlite_schema query written directly by a client is passed through verbatim, so tools that introspect the native SQLite catalog see pgsqlite's internal tables alongside the user's own. Add a branch to CatalogInterceptor::intercept_query that runs before the pg_catalog / information_schema gate (a sqlite_master query contains none of those substrings). It parses the query with the bundled sqlparser, finds every sqlite_master / sqlite_schema relation in the FROM clause (alias-aware and JOIN-safe, matching the SQLite sqlite_schema alias too), ANDs "<relation>.name NOT LIKE '__pgsqlite_%'" into the WHERE clause, and executes the rewritten query. It fails open (runs the original query unchanged) on any parse failure or when sqlite_master is not an actual FROM relation, so ordinary queries are untouched. Only the __pgsqlite_% prefix is filtered; sqlite_% internal objects are left as-is. Covered by new unit tests (rewrite: plain / aliased / joined / sqlite_schema / fail-open / literal-not-table / similarly-named-table) and an integration test (catalog_sqlite_master_filter_test) asserting the internal tables are hidden over the wire while user tables, indexes and views are still returned.
…lite_master The sqlite_master / sqlite_schema filter previously hid only the __pgsqlite_* bookkeeping tables. Extend the rewritten predicate to also exclude the four pg_catalog tables that are physically materialized as real SQLite tables (pg_attrdef, pg_constraint, pg_depend, pg_index) and any index or trigger owned by an internal table, by filtering on tbl_name as well as name. This keeps an unqualified SELECT * FROM sqlite_master clean, not just a type = 'table' scan. The four table names are a closed set defined next to the __pgsqlite_ prefix as PG_CATALOG_INTERNAL_TABLES, with src/migration/registry.rs cited as the source of truth. Matching is exact-set, not a pg_% wildcard, so a user table such as pg_indexes is still returned. The rewrite keeps its alias/JOIN awareness and fail-open-on-parse-failure behavior.
…e_master Extends the client sqlite_master/sqlite_schema filter to exclude the 24 pg_catalog and information_schema compatibility views pgsqlite materializes as real SQLite views (source of truth: migration/registry.rs) in addition to the __pgsqlite_* tables, the four materialized pg_catalog tables, and their internal indexes. A raw type='view' scan or an unqualified SELECT * now shows only the user's own schema. The exclusion is an exact-name set, never a pg_% / information_schema_% wildcard, so a user-owned view or table such as pg_my_report is still returned. Filter applies to both name and tbl_name for symmetry, keeps the same alias/JOIN awareness, and fails open on any parse failure.
The sqlite_master / sqlite_schema internal-table filter parsed the client query, injected filter predicates into the AST, and re-serialized the whole statement back to SQL. That full round-trip corrupted moderately complex predicates: a client query with a large name NOT IN (...) list plus a name NOT LIKE '\_%' ESCAPE '\' clause returned zero rows once the IN list grew past about 16 entries, breaking a plain list-tables query. Rewrite the filter to never re-serialize the user's statement. sqlparser is now used only to analyze the query (confirm a single SELECT over sqlite_master, capture the alias, and locate the WHERE condition or FROM clause by byte span); the output is produced by string-level splicing onto the original query text, so every user token (ESCAPE clauses, IN lists, JOINs to pragma_table_info, aliases) is preserved byte for byte. A final re-parse gate makes any low-confidence splice fail open and run the original query unmodified. Add unit and integration coverage for the exact failing shape (a type='table' scan with an ESCAPE clause and a 30+ entry NOT IN, ordered by name) plus a large-IN-list regression lock.
Author
|
Superseded by #83. That PR takes a safer approach: instead of rewriting the client's sqlite_master query (which this PR's splice/rewrite could corrupt for complex predicates), #83 runs the query verbatim and filters the internal objects out of the RESULT rows, so it can never corrupt a query. Please review #83 instead. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Client
sqlite_master/sqlite_schemaqueries return pgsqlite's own internal objects alongside the user's tables:__pgsqlite_*bookkeeping tables,pg_attrdef,pg_constraint,pg_depend,pg_index),pg_class,pg_type,information_schema_tables, ...),pgsqlite already filters
__pgsqlite_%from the catalog queries it synthesizes itself (e.g.src/catalog/pg_class.rs), but a client's ownsqlite_masterscan passes through unfiltered, so these internal objects leak into schema browsers, ORMs, and\dt-style tooling.Change
CatalogInterceptor::intercept_querygains a branch (above the existing catalog-substring early-return, since a rawsqlite_masterquery contains none of those substrings) that rewrites a clientsqlite_master/sqlite_schemaSELECTto exclude pgsqlite's internal objects, then executes the rewrite. It:sqlparser, walks the FROM clause (relation + JOINs) alias-aware,name/tbl_namematching__pgsqlite_%, the 4 internal pg_catalog table names, and the internal pg_catalog/information_schema view names (both sets defined as consts citingsrc/migration/registry.rsas the source of truth),The pg_catalog names are matched as an exact reserved-name set, never a
pg_%wildcard, so a user-owned object named e.g.pg_my_reportis preserved (covered by a test).Tests
Unit + wire-level integration tests (real
tokio_postgresclient against a live server): each internal object hidden fromtype='table',type='view', and unqualifiedSELECT *scans; user tables/views/indexes returned; a user-ownedpg_my_reportreturned; unparseable SQL falls through unchanged.