feat(datafusion) Add snapshot_id parameter to pin table reads

[NoahKusaba]

Which issue does this PR close?

• Closes: Add snapshot_id parameter for Datafusion IcebergTableProvider (Distributed reads) #2861

Related to : #2613

What changes are included in this PR?

Optional Snapshot ID parameter to IcebergTableProvider to pin table reads

Are these changes tested?

• test_pinned_snapshot_reads_historical_data: Tests that pinning and unpinning snapshot Id's for a table provider with multiple snapshots, correctly reads from the correct snapshot by asserting against expected output.

[NoahKusaba]

@CTTY do you have time to look at this?
Tried to make this change as simple as possible.
7 lines of functional code change, everything else is comments or test.

[xanderbailey]

Hey, thanks for the PR, just wondering what the benefit of this is vs using the StaticTableProvider?

If we decide this is the right path forward, I wonder if we can make it generic such that this works for incremental append scans etc also? Maybe with a ScanRange enum or something? Happy to hear thoughts here!

[NoahKusaba]

what the benefit of this is vs using the StaticTableProvider?

If we decide this is the right path forward, I wonder if we can make it generic such that this works for incremental append scans etc also? Maybe with a ScanRange enum or something? Happy to hear thoughts here!

My idea is to register tables to datafusion using IcebergTableProvider, so that the same registration also allows for inserts/deletes/merges, etc. Whereas StaticTableProvider is strictly scan only.
Also a staticTableProvider can never go back to current_state as it never refreshes, with my PR unpinning
with_snapshot_id(None)
will return back to current state.

There's also a distributed angle (my motivating use case is a Ballista integration): the flow is Client -> LogicalCodec serializes the provider's recipe -> Scheduler rebuilds the provider and does physical planning -> PhysicalCodec -> Executors. The provider needs to hold everything required to recreate the table connection on remote executors: snapshot_id here, and catalog config (catalog_type, name, storage_properties) in a follow-up PR.
This is split out from #2727, where the fuller picture is visible.

I think the ScanRange suggestion is great, and since the current snapshot_id field is private we can replace it with ScanRange in the future without breaking anything. This would also require some plumbing to IcebergTableScan to work, which I think should live in a separate PR to keep this minimal.
We can make a stub field too for the time being, and put snapshot_id in it?

[xanderbailey]

Something seems a little off here to me. Read/write skew seems like a footgun. A single registered table that reads snapshot A but writes to HEAD is a confusing object. INSERT INTO t; SELECT * FROM t won't show the row you just inserted. That's a strange contract to hand a user in my opinion.

I also think schema isn't being handled here correctly for pinned snapshots, we should resolve against the snapshot's schema rather than current, the static table provider handles this correctly.

[NoahKusaba]

Something seems a little off here to me. Read/write skew seems like a footgun. A single registered table that reads snapshot A but writes to HEAD is a confusing object. INSERT INTO t; SELECT * FROM t won't show the row you just inserted. That's a strange contract to hand a user in my opinion.

I also think schema isn't being handled here correctly for pinned snapshots, we should resolve against the snapshot's schema rather than current, the static table provider handles this correctly.

insert-read problem:
Sorry for that, I understand what you are saying on careful review. I was so tunnel visioned on the ballista integration, that I had missed actual users directly calling snapshot_id.

The way I have it wired in my Ballista integration is that snapshot_id is resolved by the scheduler when it encodes the physical plan to executors, so a read -> insert -> read would properly show the new data as the snapshot_id would be reset to head by the scheduler as queries are sequentially made.

I think it makes sense to add a write guard, if the snapshot_id is pinned (not None), to constrain confusing behavior? I added it in a new commit, let me know how you think this should be handled.

[xanderbailey]

Thanks for iterating with me! What are we gaining here over using the StaticTableProvider now?

[NoahKusaba]

Thanks for iterating with me! What are we gaining here over using the StaticTableProvider now?

Thanks for helping me with this too (still fixing the schema update with snapshot id problem), and sorry for responding late.

To answer your question, anyone not using the iceberg-ballista integration likely won't benefit from these changes, and they should keep using IcebergStaticTableProvider for time-travel reads. After this PR the two resolve schemas through the same helper and both refuse writes when pinned, so read semantics are identical.

Ballista requires you instantiate one TableProvider, through which queries are routed (explained above). This creates two requirements:

1. The TableProvider needs to carry all the information required to rebuild it on the scheduler, which de-serializes the logical plan and re-plans it (this PR + another one add the missing requisite information: snapshot-id + catalog-configuration). StaticTableProvider is built from an already-loaded Table and carries no catalog identity, so there is nothing a codec can rebuild it from.
2. If I want the same instantiated TableProvider to allow for both Scanning + Insert + etc, it needs functional .scan() and .insert_into() implementations. StaticTableProvider has the methods (the trait requires them), but its .insert_into() unconditionally errors, so writes are structurally impossible for it.

[NoahKusaba]

Schema Problem:
Following StaticTableProvider's approach (per your suggestion), pinning now re-derives the exposed schema from the pinned snapshot.

**Also now re-loads the table to reload the schema.

While addressing this, I found a second issue: with_snapshot_id cached the Arrow schema at construction time (the current schema). Pinning across a schema evolution would then push down non-existent column names from the old schema.

Fix: Schema resolution is now centralized in a shared snapshot_arrow_schema helper that looks up the snapshot’s own schema via snapshot.schema(metadata). Both with_snapshot_id and IcebergStaticTableProvider constructors use this helper, ensuring a single resolution path. with_snapshot_id is now fallible, as the snapshot ID is validated during pinning.

[NoahKusaba]

Hey @xanderbailey can you take a look at this again when you have time?
Hopefully it is in a more acceptable state.