feat(rest): add scan plan endpoint support to REST catalog client - #783
gsandeep1241 wants to merge 7 commits into
Conversation
| auto table_catalog = std::make_shared<TableScopedCatalog>( | ||
| shared_from_this(), context, identifier, table_config, table_session); | ||
|
|
||
| if (supported_endpoints_.contains(Endpoint::PlanTableScan())) { |
There was a problem hiding this comment.
This should also gate on the effective scan-planning-mode, not only endpoint support. Java defaults to client-side planning and lets the table config override the client config, so a table can otherwise be forced into REST planning even when the server says client, or silently fall back when the server says server but the endpoint is missing.
| request.case_sensitive = context_.case_sensitive; | ||
| request.min_rows_requested = context_.min_rows_requested; | ||
|
|
||
| if (context_.from_snapshot_id.has_value() && context_.to_snapshot_id.has_value()) { |
There was a problem hiding this comment.
We need to set use-snapshot-schema for snapshot/time-travel and incremental scans. Java sends it for useSnapshot and start/end snapshot scans, and the REST spec says time travel should use the snapshot schema. Without it, schema-evolved tables can be planned against the current schema.
| rest_context_.client->Post(path, json_request, /*headers=*/{}, | ||
| *PlanErrorHandler::Instance(), *rest_context_.session)); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto json, FromJsonString(response.body())); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto result, |
There was a problem hiding this comment.
Planning responses can include storage-credentials. Java switches to a scan-scoped FileIO built from those credentials, and the spec expects clients to use them for the returned tasks. Ignoring them means servers that vend temporary storage credentials can plan successfully but reads may fail.
|
|
||
| switch (result.plan_status) { | ||
| case PlanStatus::kCompleted: | ||
| return ResolveScanTasks(result.plan_tasks, result.file_scan_tasks, specs); |
There was a problem hiding this comment.
Once a plan-id is returned, the server may hold resources until all plan tasks are fetched or the plan is cancelled. If resolving paginated tasks fails partway through, this returns without cancelling the remaining plan; Java cancels from the scan-task iterable cleanup path.
There was a problem hiding this comment.
Updated here. Are you also suggesting that we change the return signature of PlanFiles to have an iterable and let the blocking happen at the caller instead of here?
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
|
Hey @gsandeep1241, do you want to revive this? |
|
@wgtmac Thanks for re-opening it. Apologies for the long delay, I'll get back on it this week - next set of changes should be out for review in the next couple of days! |
When a table is loaded from a REST catalog that advertises the PlanTableScan
endpoint, NewScan() now returns a RestTableScanBuilder whose Build() produces
a RestTableScan. PlanFiles() on that scan delegates manifest resolution to
the server via POST /plan, GET /plan/{id} (with exponential backoff),
POST /tasks/{id}, and DELETE /plan/{id} (best-effort cancel), instead of
reading manifests locally.
- Add RestTable, RestTableScanBuilder, RestTableScan and RestScanContext
- Promote DataTableScan::PlanFiles and TableScanBuilder::Build to virtual
- Convert RestCatalog::client_ and paths_ to shared_ptr so RestScanContext
can share ownership with live scans
- Cancel server-side plan when ResolveScanTasks fails partway through - Propagate use_snapshot_schema from scan context to PlanTableScanRequest: true for UseSnapshot/AsOfTime/tag refs and incremental scans, false for branch refs and default scans - Gate RestTable creation on effective scan-planning-mode config (table config overrides client config, default is client); error if server mode is requested but endpoint is not advertised - Add ScanPlanningMode enum and ScanPlanningModeFrom() parser to RestCatalogProperties - Make HttpClient methods virtual and add HttpResponse::MakeForTesting() to support unit test mocking - Add tests: use_snapshot_schema in table_scan_test, ScanPlanningModeFrom parsing in catalog_properties_test, and RestTableScan HTTP flow tests in rest_table_scan_test
Upstream changed Table and DataTableScanBuilder constructors to require full_name/table_name and MetricsReporter parameters. Updated RestTable, RestTableScanBuilder, and their callers accordingly.
fb8da85 to
d8a7d6c
Compare
| auto table_catalog = std::make_shared<TableScopedCatalog>( | ||
| shared_from_this(), context, identifier, table_config, table_session); | ||
|
|
||
| if (supported_endpoints_.contains(Endpoint::PlanTableScan())) { |
| request.case_sensitive = context_.case_sensitive; | ||
| request.min_rows_requested = context_.min_rows_requested; | ||
|
|
||
| if (context_.from_snapshot_id.has_value() && context_.to_snapshot_id.has_value()) { |
|
|
||
| switch (result.plan_status) { | ||
| case PlanStatus::kCompleted: | ||
| return ResolveScanTasks(result.plan_tasks, result.file_scan_tasks, specs); |
There was a problem hiding this comment.
Updated here. Are you also suggesting that we change the return signature of PlanFiles to have an iterable and let the blocking happen at the caller instead of here?
| rest_context_.client->Post(path, json_request, /*headers=*/{}, | ||
| *PlanErrorHandler::Instance(), *rest_context_.session)); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto json, FromJsonString(response.body())); | ||
| ICEBERG_ASSIGN_OR_RAISE(auto result, |
| return IOError("Scan planning failed: {}", | ||
| result.error ? result.error->message : "unknown error"); | ||
| case PlanStatus::kCancelled: | ||
| return IOError("Scan planning was cancelled for plan_id={}", plan_id); |
There was a problem hiding this comment.
Do we want to introduce a new error type? Or use InvalidArgument instead?
06fb014 to
2bab861
Compare
Parse storage-credentials from PlanTableScanResponse, FetchPlanningResultResponse, and FetchScanTasksResponse. When credentials are present, build a scan-scoped FileIO via MakeTableFileIO and expose it through RestTableScan::effective_io() for callers to use when reading the returned scan tasks.
2bab861 to
bcc9da4
Compare
RestTableScanBuilder::Build() calls context_.Validate() across the iceberg_rest/iceberg library boundary. Without ICEBERG_EXPORT on TableScanContext the symbol is hidden in the shared library and the linker fails on arm64.
…port MSVC does not export the implicitly-generated move constructor of a template class instantiation. RestTableScanBuilder (introduced in iceberg_rest) is exported with ICEBERG_REST_EXPORT, so its compiler- generated move constructor must call the base TableScanBuilder move constructor as an imported symbol. Explicitly defaulting it makes it part of the explicit template instantiation and therefore exported.
Thanks @wgtmac for reviving this! This is now ready for review. Please take a look when you can :) |
2 participants
When a table is loaded from a REST catalog that advertises the PlanTableScan endpoint, NewScan() now returns a RestTableScanBuilder whose Build() produces a RestTableScan. PlanFiles() on that scan delegates manifest resolution to the server via POST /plan, GET /plan/{id} (with exponential backoff), POST /tasks/{id}, and DELETE /plan/{id} (best-effort cancel), instead of reading manifests locally.