Skip to content

perf(sidebar): stop hierarchical schema search and refresh reading every schema - #3099

Merged
datlechin merged 5 commits into
mainfrom
fix/hierarchical-schema-query-storms
Sep 23, 2026
Merged

datlechin merged 5 commits into
mainfrom
fix/hierarchical-schema-query-storms

Conversation

@datlechin

Copy link
Copy Markdown
Member

Found while investigating #3048 (#3060). Stacked on #3095, which keys these same lists by database; base is its branch, so this diff is only the change below.

Both defects are on the engines grouped by hierarchical schema: Oracle, Snowflake, BigQuery, Spanner, Trino, Dameng and Cloudflare R2 SQL. Their sidebar lists every schema of the database and reads each schema's objects on its own, two to four queries a schema (tables and routines, plus triggers on Oracle).

Defect 1: one keystroke in the sidebar filter read every schema

SidebarTreeView.scheduleSearchLoad loaded every schema not yet loaded, 300 ms after the filter text changed, so it could judge whether each one held a match. The outline also expanded every schema it could not judge yet, so each of those expansions would have read the schema even without that call. A database with 200 schemas cost 400 to 600 queries on the first keystroke.

Fix. The hierarchical tree judges an unread schema from the all-schema listing #3060 added, the way #3060 made the Tree layout judge an unexpanded schema. hierarchicalSchemaIsVisible becomes hierarchicalSchemaSearchVerdict, which returns the same match / noMatch / unknown as schemaSearchVerdict:

  • a schema whose name matches, or whose read objects hold a match, is a match;
  • a schema read since the last catalog change answers for itself;
  • otherwise the listing answers for its tables: a matching table is a match, none is no match;
  • a schema the listing has not answered for yet stays on screen, collapsed.

Only a match expands, so only a matching schema is read. scheduleSearchLoad is gone, and SidebarViewModel asks for the browsed database's listing on a hierarchical connection too, through the same AllSchemaTablesDemand, which asks again after a catalog change or reconnect.

Procedures, functions, triggers and types. The listing holds tables and views only, and no engine here can list routines across schemas in one call. So a routine, trigger or type matches only in a schema whose objects have been read: one expanded since connecting, or one an earlier search matched and expanded. A schema read before the last catalog change still counts for these, since the match expands it and the expansion reads it again. This is the same reach #3060 gave the Tree layout, and docs/features/connection-window.mdx now says so. Keeping the old reach would mean reading every schema again, which is the defect.

Defect 2: every COMMIT read every loaded schema again

Every COMMIT reports a catalog change (.objects and .schemas), and SchemaRefreshService.performRefresh then called refreshLoadedSchemaObjects, which reloaded every schema in the loaded or loading state, one after another. After a search had loaded every schema, each COMMIT re-ran two to four queries per schema.

Fix: stale-while-revalidate, as #3060 did for its listing. A catalog change marks every loaded schema stale and keeps what each one shows (markLoadedSchemaObjectsStale), then reads at once only the schemas something is about to judge: the browsed schema, which is the browse catalog, and any schema holding a queued truncate or drop, which the change prunes against the refreshed lists when it finishes. Every other schema is read by its next reader: an expanded tree row (schemaObjectsNeedFetch replaces the outline's .idle guard) or a caller of loadSchemaObjects.

  • Freshness is CatalogFreshness, keyed by schema, with one addition: it records the revision a fetch started at, so a reader asks once per change. A failed read is not retried until the next change, since the failure publishes a change every reader observes. A read cut short by a cancel is asked for again.
  • The fetch dedup key carries the revision, so a read after a change starts its own fetch instead of joining one that began before it. The generation fence (schemaLoadGenerations) is unchanged: the latest fetch to start is the one that commits.
  • A refresh never clears the cache it refreshes: stale rows stay on screen until the new ones arrive, and a failed read keeps them.
  • A stale list cannot say an object is gone. CatalogEditAdoption.loadedBrowseCatalog judges queued operations against current lists only, and no longer counts a hierarchical engine's browsed schema as covered by the flat list, which is empty there.
  • Open Quickly lets the listing answer for a stale schema once the listing has arrived, and keeps the stale rows standing in until then.

Measured

Mock driver counting every catalog read; in tests the Oracle plugin is not loaded, so a schema read is 2 queries (tables and routines); with the plugin it is 3.

Scenario (200 schemas) Before After
First keystroke of a search, three schemas hold a match, engine lists every table in one call 400 (every schema read) 7: 1 listing + 3 matched schemas x 2
Same, engine listed schema by schema (the listing's fallback until the single-call overrides land) 400 208: 1 + 1 schema list + 200 table reads + 6
One COMMIT after a search left all 200 loaded 400 per-schema reads, one schema after another 2 (the browsed schema), plus the unchanged top-level 2

After is measured (HierarchicalSchemaSearchCostTests, SchemaRefreshCommitCostTests). Before is the code path: scheduleSearchLoad read every unloaded schema and refreshLoadedSchemaObjects reloaded every loaded one, 2 reads each here. On Oracle with its plugin loaded each is 3, so before is 600. A broad search still reads every schema it matches, since a matched schema expands to show its matches; that is the same cost #3060's Tree layout has.

Tests

New:

  • SchemaServiceStaleSchemaTests (8): a change reads nothing and keeps rows; fetchingNow reads only the named loaded schema; a reader reads a stale schema once; a failed read keeps rows and waits for the next change; a fetch that began before a change shows its rows but stays stale; a read after a change does not join the earlier fetch; a cancelled read is asked for again; only current lists can prune a queued truncate or drop.
  • SchemaRefreshCommitCostTests (1): the measured COMMIT case. It compiles against fix(sidebar): key per-schema object lists by database #3095's API; the edit that turns it red is putting back the reload of every loaded schema.
  • HierarchicalSchemaSearchTests (16): the verdict's cases, routines in unread schemas included, and databasesListedForSearch. HierarchicalSchemaSearchCostTests (2): the measured search cases. The old verdict had no listing input, so these cannot run against it; the edit that turns them red is returning .unknown for every unread schema, which is what made the outline expand and read them all.
  • CatalogFreshnessTests (+5) for the fetch-started record.

Updated: DatabaseTreeFilterTests, DatabaseTreeFilterQualifiedSearchTests (new verdict API), QuickSwitcherCrossSchemaTests (staleSchemas, +2 cases), SchemaServiceRefreshTests, SchemaServiceDatabaseSwitchTests (new refresh signature).

Verification

  • verify.sh test over 21 suites (the new ones plus SchemaServiceDatabaseSwitchTests SchemaServiceTests SchemaServiceRefreshTests SchemaServiceHierarchicalTests SchemaServiceSideObjectsTests SchemaServiceRoutinesTests SchemaRefreshServiceTests SchemaRefreshAfterWriteTests CatalogEditAdoptionTests SidebarViewModelTests DatabaseTreeAllSchemaTablesTests QuickSwitcherViewModelTests AllSchemaTablesDemandTests CatalogFreshnessTests DatabaseTreeFilterTests DatabaseTreeFilterQualifiedSearchTests QuickSwitcherCrossSchemaTests): PASS, 267 executed, 267 passed.
  • SwiftLint --strict on the 19 changed Swift files: 0 violations. verify.sh docs: PASS.
  • main at the time of writing did not compile on its own (MainSplitViewController+RecentTabs.swift lacks .versionHistory, fixed by fix(tabs): give version history tabs a symbol in the recent tabs list #3093; FileTabExternalChangeTests and OracleColumnStatementsTests break the test target, fixed by test: fix the two test files that stop the test target compiling on main #3098). I patched those three locally to run the suites; none of that is in this PR.

Reviewed with a feature-dev:code-reviewer agent (Codex is out of quota until Sep 29). It found nothing at its confidence bar. Its one note, that a settled-but-stale schema's verdict is untested, is covered by staleReadYieldsToTheListing, staleReadKeepsAProcedureMatch and staleReadWithoutAMatchIsHidden in HierarchicalSchemaSearchTests, which it did not open.

No UI automation: the defects are query counts against a live server with hundreds of schemas, which no UI test here can stand up. The decisions are pure functions and service calls, tested directly.

Base automatically changed from fix/schema-service-hierarchical-caches to main September 23, 2026 19:24
…ma-query-storms

# Conflicts:
#	CHANGELOG.md
#	TablePro/Core/Services/Query/SchemaRefreshService.swift
#	TablePro/Core/Services/Query/SchemaService.swift
#	TablePro/ViewModels/QuickSwitcherViewModel.swift
#	TableProTests/Services/SchemaServiceDatabaseSwitchTests.swift
#	TableProTests/Services/SchemaServiceRefreshTests.swift
#	TableProTests/ViewModels/QuickSwitcherCrossSchemaTests.swift
@mintlify

mintlify Bot commented Sep 23, 2026

Copy link
Copy Markdown

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated
TablePro 🟢 Ready View Preview Sep 23, 2026, 8:05 PM

💡 Tip: Enable Automations to automatically generate PRs for you.

@datlechin
datlechin merged commit 044042e into main Sep 23, 2026
4 checks passed

This branch was successfully deployed

1 active deployment
staging - docs b169c577 Deployed Sep 23, 2026 by mintlify[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant