Task Summary
Foundation step of the per-user BYO-S3 warehouse feature (part of #6870). Today the storage layer
only ever talks to one warehouse; this teaches it to handle several — without changing any
behavior yet.
- Per-warehouse REST catalog cache. The Iceberg catalog client — which resolves tables and
reads/writes their metadata against Lakekeeper — is today a single shared instance
(IcebergCatalogInstance on the JVM, iceberg_catalog_instance.py for Python workers). Turn it
into a cache keyed by warehouse name, so one worker process can hold several REST catalogs at
once, one per warehouse it touches. Only the REST (Lakekeeper) catalog varies by warehouse; the
hadoop/postgres catalogs stay warehouse-agnostic and keep sharing one. getInstance(warehouse)
falls back to the configured default when no warehouse is given.
- Warehouse-scoped storage URIs. Every stored result is identified by an internal VFS URI (a
path). Add the warehouse name as a leading /wh/<name> segment so the URI itself says which
warehouse the data lives in; the reader parses it back out (VFSURIFactory.warehouseFromURI) and
DocumentFactory opens the matching catalog. When the segment is absent, the URI is
byte-for-byte identical to today's — existing data and behavior are untouched.
- Per-execution warehouse in the context. Add a
warehouse: Option[String] field to
WorkflowContext so the selected warehouse flows from the request down to the workers.
Backward-compatible no-op: with no warehouse selected (the default), the catalog and URIs behave
exactly as they do today.
Task Type
Task Summary
Foundation step of the per-user BYO-S3 warehouse feature (part of #6870). Today the storage layer
only ever talks to one warehouse; this teaches it to handle several — without changing any
behavior yet.
reads/writes their metadata against Lakekeeper — is today a single shared instance
(
IcebergCatalogInstanceon the JVM,iceberg_catalog_instance.pyfor Python workers). Turn itinto a cache keyed by warehouse name, so one worker process can hold several REST catalogs at
once, one per warehouse it touches. Only the REST (Lakekeeper) catalog varies by warehouse; the
hadoop/postgres catalogs stay warehouse-agnostic and keep sharing one.
getInstance(warehouse)falls back to the configured default when no warehouse is given.
path). Add the warehouse name as a leading
/wh/<name>segment so the URI itself says whichwarehouse the data lives in; the reader parses it back out (
VFSURIFactory.warehouseFromURI) andDocumentFactoryopens the matching catalog. When the segment is absent, the URI isbyte-for-byte identical to today's — existing data and behavior are untouched.
warehouse: Option[String]field toWorkflowContextso the selected warehouse flows from the request down to the workers.Backward-compatible no-op: with no warehouse selected (the default), the catalog and URIs behave
exactly as they do today.
Task Type