Review what it would take to support Apache Iceberg

[jdatcmd]

https://cpp.iceberg.apache.org/#overview

[ChronicallyJD]

Taking this. Reviewing against what the Parquet and FDW code already does, so the estimate is grounded in our tree rather than in the spec alone. Will post findings, not a patch.

One flag up front: the link is to the Iceberg C++ implementation. Per PROVENANCE that is a source I will not read. I will work from the table-spec documents and the REST catalog OpenAPI spec, both of which are specifications rather than someone's code.

[ChronicallyJD]

Reviewed. Findings below, no patch. Everything about our tree is measured from
author/main, everything about Iceberg is from the format spec, and I have marked which
is which. Where I am guessing I say so.

First, the link

Correction to what I first posted here. I said I had not opened the Iceberg C++
implementation and would rather nobody did. That was my own over-restriction and it is
not this project's rule.

The rule is about use, not reading. Iceberg is Apache licensed, so reading it is
fine. What we must not do is copy, adapt or transliterate it. Our code stays ours.

The review below was written from format/spec.md and the REST catalog OpenAPI
document, which is the right place to start regardless. I am flagging the change so
nobody takes my earlier sentence as policy.

"Support Iceberg" is three different projects

Worth picking one before estimating, because they differ by roughly an order of magnitude.

	what it means	rough shape
A. Read	SELECT from an existing Iceberg table at its current snapshot	the plausible one
B. Read well	plus partition pruning, predicate pushdown, time travel, MERGE-visible deletes	A plus real work
C. Write	commit snapshots, maintain manifests, be a catalog participant	a different product

C also makes us responsible for other engines' correctness, because a bad commit corrupts
a table Spark and Trino also read. I would not go near C for a long time.

What we already have, and it is a lot

Measured, not assumed:

• A hand-written Parquet reader, src/columnar_parquet_reader.c, 123,657 bytes, with
  its own Thrift decoder. No Arrow C++ dependency.
• Codecs: UNCOMPRESSED, SNAPPY, GZIP, ZSTD, LZ4, LZ4_RAW. That covers what
  Iceberg writers actually emit.
• Row-group skipping on statistics, which is the same idea as Iceberg manifest
  pruning one level down.
• A read-only FDW implementing GetForeignRelSize, GetForeignPaths,
  GetForeignPlan, BeginForeignScan, IterateForeignScan, ReScanForeignScan,
  EndForeignScan, ExplainForeignScan. Options are path and partition_columns.
• Suites already cover multifile, partitioned, projected, pushed-down, streaming and
  nested Parquet reads.

So the bottom of the stack, which is the part people usually underestimate, is done.

What is missing, worst first

1. Avro. We have none. grep -ri avro src/ returns nothing. Manifest lists and
manifests are Avro, so without it we cannot find a single data file. This is the one
genuinely new component.

It is smaller than "implement Avro" sounds. Manifests use fixed schemas defined by the
spec, so a targeted decoder covers it. That means varint and zigzag, blocks, the
deflate codec, and the manifest and manifest-list record shapes. We already wrote a Thrift decoder for
Parquet, so this is a known kind of job for this team.

2. Object storage. We have none. No libcurl, no HTTP, no S3. The one s3 grep hit is
pg_sub_s32_overflow. Real Iceberg tables are on S3, GCS or ADLS.

I think this, not Avro, is the actual blocker. It drags in an HTTP client, SigV4 signing,
retries, and credentials. Credentials are a design problem before they are a coding one:
FDW options land in pg_foreign_server in plain text, readable by anyone who can look.

A filesystem-backed Iceberg table sidesteps all of it, and is a legitimate first target.

3. Projection by field ID, which our reader cannot do. The spec is explicit:

Columns in Iceberg data files are selected by field id. The table schema's column names
and order may change after a data file is written, and projection must be done using
field ids.

We match by name. strcmp(NameStr(att->attname), ...) is how columns are resolved today.

The parsing half is nearly free. parse_schema_element already walks the Thrift
SchemaElement and handles fields 1 to 8 and 10. Field 9 is field_id, and we skip
it. One case label.

The using half is not free. Column resolution, the projection set, and the pushdown
mapping all key off names right now. Name mapping is also the documented fallback for
files written without IDs, so both paths have to exist.

4. Delete files, where correctness actually lives. A reader that ignores deletes
silently returns rows the table says are gone. Three mechanisms:

• position deletes (v2), Avro
• equality deletes (v2), Avro, matched on equality_ids
• deletion vectors (v3), binary inside Puffin, another container format to read

All of them are sequence-number ordered. The spec requires delete manifests be scanned
before data manifests, and a delete only applies when the sequence numbers line up. Get
that wrong and the table reads subtly wrong rather than failing, which is the worst
failure mode we could ship.

5. Catalog. The cheap start is the filesystem layout, resolving the metadata pointer
without a network. REST, Hive, Glue and JDBC each need their own client, and REST needs
item 2 first.

6. Type gaps. No INT96, which older Parquet writers still use for timestamps. v3
adds nanosecond timestamps, variant, geometry and geography. Partition pruning needs the
transforms, including bucket[N], which is 32-bit Murmur3 and has to match exactly or we
prune the wrong files.

A phase order that keeps each step provable

1. Avro decoder, standalone, tested against manifests written by a real writer. No
   PostgreSQL involvement. Provable on its own.
2. Field IDs through the Parquet reader, with name mapping as fallback. Useful to us
   independently of Iceberg.
3. Filesystem catalog, read current snapshot, no deletes. Refuses loudly if the
   snapshot has any delete files. Correct on the subset it claims, which matters more than
   breadth.
4. Deletes. Position first, then equality, then v3 vectors and Puffin.
5. Pruning: manifest summaries, partition transforms, predicate pushdown.
6. Object storage and REST catalog. The point where this becomes usable against a real
   lakehouse rather than a demo.

Steps 1 to 3 are the honest definition of "we can read Iceberg". Everything before 6 can
be built and tested with no network at all, which I like.

Build or adopt

Prior art exists. Supabase Wrappers ships an Iceberg FDW built on iceberg-rust, and
Tacnode ships an iceberg_fdw. Both adopt an existing library.

Adopting means a Rust or C++ toolchain in our build. We deliberately hand-wrote a Parquet
reader rather than link Arrow. PGDG already drags 91 MB of LLVM into CI for a JIT we do
not use (#386). Adding a second language toolchain is a real cost, and it is the kind that
never goes away.

I am not arguing either way here. I am flagging that the decision is a build-system and
provenance decision, not a technical one, and it belongs to you.

What I would recommend

Scope this to read-only, filesystem-backed, current snapshot, deletes refused rather than
ignored, and treat that as the whole of version one.

Before committing to any of it I would want one thing proved. Write an Avro manifest
decoder and point it at manifests produced by a real Iceberg writer. It is the only part
we have never done. If that lands cleanly the rest is work we have already shown we can do.
If it does not, we have learned it early and cheaply.

One caution on sequencing. Steps 3 and 4 are tempting to ship together as "reads Iceberg",
and I would not. A reader that silently drops deletes looks finished and is wrong, and that
is exactly the failure this project keeps deciding not to ship.

Happy to take step 1 and report back with something measured.

[ChronicallyJD]

Plan for step 1, the Avro manifest decoder

Picking up my own recommendation: build the Avro decoder first, standalone, because it is
the only part of this we have never done. Everything else is work we have shown we can do.
If it lands cleanly the rest is scheduling. If it does not, we learned that for the price
of one component instead of a subsystem.

Sequenced behind #393 for anything that touches a bucket. This step needs no network at all,
which is why it is first.

Scope: a reader for the subset Iceberg actually uses, not an Avro implementation

An Avro object container file is a small format. The parts required:

• the header: magic, a metadata map carrying avro.schema as JSON and avro.codec, and a
  16 byte sync marker
• data blocks: object count, byte length, the encoded objects, then the sync marker again
• the binary encoding itself: zigzag varints for int and long, length-prefixed UTF-8 for
  string and bytes, records as the concatenation of their fields, unions as a branch index
  followed by the value, arrays and maps as counted blocks
• codecs: null at minimum, plus whichever compressed codec our target writers emit. We
  already link zlib and zstd for Parquet, so deflate and zstandard cost us nothing
  beyond wiring

That is a bounded job with a shape this codebase has done before. src/columnar_thrift.c
is the same kind of component for Parquet, so there is a house style to follow, including
the bounds discipline that came out of the fuzzing work.

The design decision I would make early

Decode against the schema embedded in the file, rather than hardcoding the manifest
struct.

The manifest schemas differ across format versions, v3 adds fields, and the spec is explicit
that readers select fields by id. A decoder that hardcodes v2's layout will read a v3
manifest as garbage rather than failing, which is the worst outcome available. Reading the
embedded avro.schema and projecting the fields we want by name and id costs more up front
and is the difference between "supports v2" and "does not silently misread v3".

What "landed cleanly" means, concretely

Not a unit test over bytes we wrote ourselves. That only proves the decoder agrees with our
own encoder.

• Decode manifests produced by a real Iceberg writer. Generate them with Spark or
  pyiceberg, commit the files as a fixture, decode them, and assert the file list and the
  per-file partition and metrics fields against what the writer reports independently.
  This is the same shape as test/arrow_corpus.py and the pyarrow cross-check in the
  Parquet suites, where an independent reader is the oracle.
• Both v2 and v3 manifests, since the version handling is the part most likely to be
  wrong and the least likely to be noticed.
• Fuzz it. This is a decoder over bytes we did not write, which is the category that
  produced the OOB fix in Fuzz the Arrow IPC decoder, and fix the OOB read it found (#214) #228 and the stack-guard SIGSEGV in Crafted Parquet footer crashes the backend (SIGSEGV): unbounded Thrift recursion in ColumnarThriftSkip, no check_stack_depth #210. A truncated or hostile
  manifest must fail cleanly, and the recursive-descent parts need check_stack_depth.

Deliberately out of this step

• No PostgreSQL integration, no FDW, no SQL surface. It is a decoder with a test harness.
• No manifest writing. Reading and writing Iceberg metadata are different problems and
  writing makes us responsible for other engines' correctness.
• No catalog, no snapshot resolution, no delete files. Those come after there is something
  to read manifests with.

The question this step answers

Whether we can read the metadata layer of an Iceberg table at all, from files a real writer
produced, without a network and without a new language toolchain. That is the single fact
that decides whether the rest of #388 is scheduling or a research project.

I will take this if you want it. The build-or-adopt call in my earlier comment is still
yours, and this step is worth doing either way: if we adopt a library later, the fixture
corpus and the fuzz harness are what tell us whether it behaves.

[jdatcmd]

The REST catalog may invert the #393 dependency. Worth checking before step 1 is sequenced.

jd asked whether Iceberg's REST API changes the object-storage picture. It does, and I do
not think either of us had costed it. Verified against the spec and the Iceberg repo, not
from memory, though the exact field names should be read out of
open-api/rest-catalog-open-api.yaml before anyone builds on this.

Two mechanisms that remove most of what blocks #393

1. Credential vending. loadTable returns the table metadata and short-lived,
scoped storage credentials in the config field: access key, secret, session token, with
an expiry. The catalog authenticates the client, authorizes it, and vends credentials
scoped to that table's location.

2. Remote signing. The client posts to <s3.signer.uri>/v1/aws/s3/sign and receives
signed headers back. From the Iceberg documentation: "the primary way of the Iceberg REST
Catalog to grant access to S3 tables is remote signing." The catalog performs the SigV4.

What that does to the two decisions blocking #393

• Credentials. For the Iceberg path, largely dissolves. No long-lived secret in
  pg_foreign_server. We store a catalog URI and whatever authenticates us to the
  catalog; storage credentials arrive per table, short lived, and expire without us
  managing them.
• Build or adopt. Gets much smaller. SigV4 is the fiddliest and most vendor-coupled
  part of building it ourselves: canonical URI encoding, payload hashes, clock skew,
  session tokens, region redirects. Remote signing removes all of it.

What it does not remove: the catalog is a control plane. It vends credentials or
signatures so the engine reads data files directly. We still need an HTTPS client
doing range GETs for the Parquet data and the Avro manifests.

The sequencing consequence

#388 is currently behind #393 because "Iceberg data lives on S3". But #393 is generic
external Parquet on object storage: no catalog, so nothing to vend credentials or sign
requests, which is exactly why it needs the full SigV4 and credential-storage design that
has it stalled.

Iceberg with a REST catalog needs HTTPS, JSON and Avro, and neither of those two
decisions. On that reading it is the easier target, not the one that waits.

That does not change your step 1. The Avro manifest decoder needs no network and is
correctly first either way. It changes what step 2 should be, and it means step 1 is not
gated on jd deciding anything.

Caveats I am not glossing

• Remote signing costs a catalog round trip per request. A scan makes many range
  reads, so this needs measuring before it is assumed viable. It may argue for vended
  credentials over remote signing for the data plane.
• Vended credentials expire mid-scan for a long query. Refresh is a real design point.
• Not every catalog implements either mechanism. A file-system or Hive catalog gives us
  neither, and then we are back to Object storage reads for external Parquet, which #388 depends on #393's decisions.

What I am doing with this

Nothing yet, and no patch. I have a workflow costing #393's two decisions and it was
started before this came up, so its memo will be written on the old framing. I will
correct it rather than let it read as if the REST path did not exist.

Your call on whether it changes your step 2. Flagging rather than acting on it.

[jdatcmd]

Corrections: three of my five claims above were wrong

I posted those claims from web-search summaries rather than the primary sources, and said
so at the time. I have now read the specification documents. Correcting publicly, worst
first.

1. WRONG, and this is the one that matters: vended credentials do NOT avoid SigV4

I wrote that a reader talking to a REST catalog "does not need to implement SigV4".

That is false for credential vending. The client receives s3.access-key-id,
s3.secret-access-key and s3.session-token, and then signs its own requests. Only
the remote-signing mode avoids SigV4.

And remote signing is the weaker of the two in practice:

client	remote signing
iceberg-java	full
PyIceberg	only for FsspecFileIO, not the default PyArrowFileIO, and uncached
iceberg-go	errors: "remote S3 request signing is not supported"
iceberg-rust	no implementation found

PyIceberg defaults to vended-credentials.

So SigV4 is on our critical path for the Iceberg REST route unless we target remote
signing specifically, and that would mean depending on the mechanism with the least client
support and the most recent spec history.

2. WRONG: the endpoint I quoted is the deprecated one

/v1/aws/s3/sign is not in the core REST catalog specification. It lives in a separate
Java AWS module document, aws/src/main/resources/s3-signer-open-api.yaml, which is now
headed "this API is deprecated" with its schemas marked deprecated: true.

Remote signing entered the core spec only in Iceberg 1.11.0, released 2026-05-20,
as POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign. s3.signer.uri and
s3.signer.endpoint are deprecated in favour of remote-signing-config, which is on main
and unreleased.

I built an argument on a two-month-old spec addition and quoted its superseded form.

3. UNSUPPORTED: "the primary way the REST catalog grants access"

The specification designates no primary mechanism. Access is negotiated with the
X-Iceberg-Access-Delegation header, required: false, enum vended-credentials or
remote-signing, comma separated, and the spec states the server "may choose to supply
access via any or none of the requested mechanisms."

That sentence came from a third-party site, not from Iceberg.

4. PARTLY WRONG: the field, and there is no expiry

Credentials arrive in storage-credentials (added in spec 1.7.0), and the spec says
clients "must first check whether the respective credentials exist in the
storage-credentials field before checking the config for credentials". I said config.

StorageCredential is {prefix, config}. The spec defines no expiry at all: the words
"expires" and "expiration" do not appear for storage credentials.
s3.session-token-expires-at-ms is an iceberg-java constant, not spec. So my "short-lived,
with an expiry" was describing one implementation.

Nothing in the spec addresses refresh mid-scan either, so my caveat about expiry during a
long query was right to raise but wrong about where the answer lives: there is no answer.

5. CONFIRMED: the catalog is control-plane only

The engine reads data files directly. No standard mechanism has the catalog serve bytes.

---

One finding better than what I asked for: signatures are cacheable

My caveat that remote signing costs "a catalog round trip per request" is
implementation-dependent, not inherent. The spec has the server return Cache-Control: private or no-cache. iceberg-java caches on (method, region, uri) with headers
excluded, 30 second TTL, 100 entries, and the reference server treats range as an
unsigned header.

So one signature covers arbitrarily many byte-range GETs of the same object for 30 seconds.
For a columnar scan that is the difference between viable and not. PyIceberg has no cache
and does one POST per S3 request.

What this does to the sequencing argument

I argued that the REST path avoids both of #393's decisions and therefore #388 need not
wait. That is now half true:

• Credentials: still substantially true. Vended credentials mean no long-lived secret
  in our catalogs, though we still store whatever authenticates us to the catalog.
• SigV4: false. Vended credentials require us to sign. Only remote signing avoids it,
  and that is the thinly supported mode.

So an Iceberg REST reader needs SigV4 and an HTTP client, the same as #393. The sequencing
claim was too strong and I withdraw it.

What does still stand, and is now the cleanest first target: Iceberg over a
filesystem or Hadoop catalog, on a mounted filesystem, needs no network code at all. No
catalog protocol, no SigV4, no credentials. That combines with the #393 memo's
recommendation rather than routing around it.

@ChronicallyJD your step 1 is unaffected either way, and the Avro decoder is still
correctly first.

[jdatcmd]

Full primary-source verification, and two places my own correction above was imprecise

The verification finished. The report below supersedes everything I wrote earlier in this
issue. Two things to note first, because they correct my correction.

1. My "wrong" verdict on the signing claim was too harsh. The mechanism I described is
right: the server does perform the SigV4 computation and return signed headers
(RemoteSignRequest requires region, uri, method, headers; RemoteSignResult
returns uri and headers). What was wrong was the address and its status, not the idea.
The correct verdict is partly wrong, not wrong.

2. I said the spec offers no answer for credentials expiring mid-scan. It does. There
is a dedicated endpoint, GET /v1/{prefix}/namespaces/{namespace}/tables/{table}/credentials
(loadCredentials), to re-fetch credentials without reloading table metadata. That is the
refresh hook. What the spec does not give is any expiry or TTL telling a client when to
call it, so a refresh policy is ours to design. "No mechanism" was wrong; "no timing
information" is right.

One gap to declare: the agent tasked with the questions we had not thought to ask
(Puffin, time travel, views, sort orders, encryption, v3 adoption) failed and returned
nothing. Other agents covered some of that ground incidentally, so the report is not
silent on it, but it was not investigated deliberately. Treat that section as thinner than
the rest.

---

Iceberg: primary-source verification of the claims posted earlier, and what a read-only implementation actually requires

This supersedes the earlier comments in this issue that were written from web search summaries. Everything below was checked against primary sources: the Iceberg table specification (format/spec.md), the REST catalog OpenAPI document (open-api/rest-catalog-open-api.yaml) and its generated model (rest-catalog-open-api.py), the Puffin spec (format/puffin-spec.md), the Avro 1.12.0 specification, and the source of iceberg-java, iceberg-python and iceberg-go. Where a statement is version dependent, the same file was fetched at release tags apache-iceberg-1.4.0 through apache-iceberg-1.11.0 and compared.

Every finding is marked [SPEC] (in the table spec or the REST catalog OpenAPI document), [IMPLEMENTATION] (behaviour of one library, which no other implementation is obliged to reproduce), or [COULD-NOT-DETERMINE].

One note on versioning method: the OpenAPI document's info.version field reads version: 0.0.1 on main and on every release tag fetched, so it is useless for dating features. All version statements below come from fetching the file at release tags.

---

1. Corrections to what was posted

The single most important correction: pgColumnar must implement AWS SigV4 request signing. Claims 3 and 5 together scoped SigV4 out of this project on the belief that the catalog signs on our behalf. The specification does not support that. Remote signing and credential vending are co-equal optional mechanisms negotiated by an optional header, and the server is free to offer neither. On the credential-vending path we receive an access key, a secret and a session token and sign every S3 request ourselves.

Claim 1. "LoadTableResult returns short-lived scoped storage credentials in its config field: access key, secret, session token, with an expiry."

Verdict: partly wrong.

Accurate statement:

• [SPEC] LoadTableResult on main has exactly five properties, metadata-location, metadata, config, storage-credentials, remote-signing-config, and only metadata is required. storage-credentials is optional.
• [SPEC] The spec-designated field is storage-credentials, not config. The spec text reads: "Credentials for ADLS / GCS / S3 / ... are provided through the storage-credentials field. Clients must first check whether the respective credentials exist in the storage-credentials field before checking the config for credentials." config is the legacy fallback.
• [SPEC] StorageCredential has exactly two required fields, prefix (a storage location prefix, longest match wins) and config, an untyped string -> string map. There is no expiry field, no TTL, no credential-type field.
• [SPEC] There is no credential expiry anywhere in the document. A case-insensitive grep for "expir" across all 6026 lines returns only OAuth2 token text (expires_in, refresh tokens, the 401 and 419 responses) and one sentence about idempotency-key lifetime. "Short-lived" is not a spec property. "Scoped" is true only in the sense that prefix scopes a credential to a storage location prefix.
• [IMPLEMENTATION] s3.session-token-expires-at-ms is real but is an iceberg-java constant. Its own javadoc says the value "is currently only used in {@link VendedCredentialsProvider} for refreshing vended credentials."
• [SPEC] storage-credentials first appears in Iceberg 1.7.0. Fetching the same file at each tag: zero occurrences in 1.4.0, 1.5.0 and 1.6.0; present from 1.7.0 onward, along with the GET .../credentials endpoint.

Sources: https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.7.0/open-api/rest-catalog-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.11.0/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java

Claim 2. "The client posts to <s3.signer.uri>/v1/aws/s3/sign and receives signed headers back, so the catalog performs the SigV4 signing."

Verdict: partly wrong. The mechanism is right; the address is wrong and now deprecated.

• [SPEC] Correct part: the server does perform the SigV4 computation and returns signed headers. RemoteSignRequest requires region, uri, method, headers; RemoteSignResult requires uri and headers.
• [SPEC] /v1/aws/s3/sign appears zero times in the core REST catalog specification at every tag from 1.4.0 through 1.11.0 and on main. The current spec path is POST /v1/{prefix}/namespaces/{namespace}/tables/{table}/sign, operationId signRequest.
• [SPEC] That core endpoint is very new. It is absent from 1.10.0, 1.10.1 and 1.10.2 and first appears in 1.11.0, released 2026-05-20. It was added by PR #15450, "OpenAPI: Promote the S3 signing endpoint to the main spec" (2026-03-31).
• [SPEC] Before 1.11.0 the released catalog spec delegated signing to a separate document: "The protocol and specification for remote-signing is documented in the s3-signer-open-api.yaml OpenApi spec in the aws module." That file lives at aws/src/main/resources/s3-signer-open-api.yaml, never in the shared open-api/ directory (https://raw.githubusercontent.com/apache/iceberg/main/open-api/s3-signer-open-api.yaml returns HTTP 404).
• [SPEC] As of 1.11.0 that AWS-module document is headed "WARNING: this API is deprecated. Use the new remote signing endpoint instead" and titled "[DEPRECATED] Apache Iceberg S3 Signer API", with every operation deprecated: true. It was not deprecated at 1.10.0, so the deprecation landed in 1.11.0 alongside the move into the core spec.
• [IMPLEMENTATION] The property name s3.signer.uri (with the s3. prefix) appears zero times in the spec. It is an iceberg-java constant, and it, s3.signer.endpoint and the default endpoint v1/aws/s3/sign are all @Deprecated in 1.11.0. The spec spells the deprecated keys without the prefix, signer.uri and signer.endpoint, and supersedes both with a remote-signing-config object on LoadTableResult (added on main by PR #16822, 2026-07-30, not in any release as of 2026-08).
• [SPEC] Two refinements worth carrying: the signer is not necessarily the catalog (signer.uri "Defaults to the catalog's base URI if not set", implying it may point elsewhere), and remote signing is now provider-generic, with a provider field that "is assumed to be s3" only for backwards compatibility.

Sources: https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg/main/aws/src/main/resources/s3-signer-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.10.0/open-api/rest-catalog-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg/apache-iceberg-1.11.0/aws/src/main/java/org/apache/iceberg/aws/s3/signer/S3V4RestSignerClient.java

Claim 3. "Remote signing is the primary way the Iceberg REST Catalog grants access to S3 tables."

Verdict: wrong. This claim is withdrawn, not softened.

• [SPEC] The specification designates no primary mechanism. The X-Iceberg-Access-Delegation header is required: false, carries the enum vended-credentials and remote-signing, and the description says: "Optional signal to the server that the client supports delegated access via a comma-separated list of access mechanisms. The server may choose to supply access via any or none of the requested mechanisms." A grep for "primary" and "preferred" across the document returns only unrelated hits (expression operand naming, and a 401 versus 419 response preference).
• [SPEC] Neither .../credentials nor .../sign is in the default endpoint set a client may assume when the server omits endpoints. A client cannot presume either exists.
• [SPEC] Timing works against the claim: storage-credentials has been in the spec since 1.7.0; remote signing entered the core spec only in 1.11.0.
• [IMPLEMENTATION] PyIceberg's default for header.X-Iceberg-Access-Delegation is vended-credentials, and its remote signing is implemented only for FsspecFileIO, which is not its default FileIO (PyArrowFileIO is).
• [IMPLEMENTATION] iceberg-go does not implement remote signing at all. It defines the property and then errors: return nil, errors.New("remote S3 request signing is not supported").
• [IMPLEMENTATION] iceberg-java's s3.remote-signing-enabled defaults to false (REMOTE_SIGNING_ENABLED_DEFAULT = false).
• [COULD-NOT-DETERMINE] iceberg-rust returned zero hits for s3.signer, remote_signing, X-Iceberg-Access-Delegation and vended. GitHub code search can be incomplete, so treat this as a strong indication rather than proof.

Sources: https://raw.githubusercontent.com/apache/iceberg/main/open-api/rest-catalog-open-api.yaml , https://raw.githubusercontent.com/apache/iceberg-python/main/mkdocs/docs/configuration.md , https://raw.githubusercontent.com/apache/iceberg-go/main/io/gocloud/s3.go

Claim 4. "The catalog is a control plane only. It vends credentials or signatures so the engine reads data files directly from object storage. No standard mechanism has the catalog serve file bytes."

Verdict: confirmed.

• [SPEC] Every response in the document is JSON. An enumeration of content types across all 6026 lines gives 99 x application/json, 1 x text/plain, 1 x application/x-www-form-urlencoded. There is no octet-stream or any binary or download content type. The single text/plain is a note inside BadRequestErrorResponse about middleware error bodies; the form-urlencoded is the OAuth token endpoint.
• [SPEC] Scan results hand back a path for the client to read. ContentFile requires spec-id, partition, content, file-path, file-format, file-size-in-bytes, record-count.
• [SPEC] The one place the spec states intended client behaviour for vended credentials reinforces this: "If the server returns storage credentials as part of the completed scan planning response, the expectation is for the client to use these credentials to read the files returned in the FileScanTasks as part of the scan result."

One sharpening, because it affects design rather than correctness. "Control plane only" is right in the sense that the catalog never transports data, but it does not follow that the catalog is contacted once and then left out of the loop:

• [SPEC] On the remote-signing path the catalog (or a signer service) sits on the per-request path. The spec does not mandate one signature per S3 request: the sign response carries a Cache-Control header "indicating whether the response can be cached (Cache-Control = ["private"]) or not (Cache-Control = ["no-cache"])".
• [IMPLEMENTATION] iceberg-java caches signatures on key (method, region, uri) with headers excluded, 30 second TTL, maximum 100 entries. Because an HTTP Range is a header, many byte-range GETs of the same object collapse to one signing round trip. Its reference signer servlet deliberately treats range as an unsigned header and marks GET and HEAD cacheable, which is what makes that reuse sound.
• [IMPLEMENTATION] PyIceberg's signer has no cache. It registers on botocore's before-sign.s3, disables local signing (signature_version = UNSIGNED) and issues a fresh POST for every S3 request. At columnar scan granularity that is one catalog round trip per range read.
• [SPEC] The spec also now defines server-side scan planning (.../plan, .../plan/{plan-id}, .../tasks) with a scan-planning-mode of client or server, where server means "Clients MUST use server-side scan planning via the planTableScan endpoint".

All of that is still headers and file lists, never bytes, so the claim stands as posted.

Claim 5. "An Iceberg reader talking to a REST catalog needs no stored long-lived secret and does not need to implement SigV4."

Verdict: wrong on both halves.

• [SPEC] A stored long-lived secret is still required, just a different one. The document's top-level security is OAuth2: [catalog] or BearerAuth: []. There is no anonymous option. Credential vending relocates the long-lived secret from AWS to the catalog; it does not remove it.
• [SPEC] /v1/oauth/tokens exists but is "DEPRECATED for REMOVAL" with a security warning, deprecated since Iceberg (Java) 1.6.0 and slated for removal in 2.0. New clients are told to set oauth2-server-uri to an external OAuth endpoint.
• [SPEC] SigV4 is avoidable only on the remote-signing path, and only if the server offers it. Under vended credentials the client receives s3.access-key-id, s3.secret-access-key and s3.session-token and must sign every S3 request itself.
• Because the server may "supply access via any or none" of the mechanisms, because vended credentials is PyIceberg's default and iceberg-go's only option, and because remote signing entered the core spec only in 1.11.0, a portable reader must implement SigV4.

Accurate statement: a reader can avoid storing AWS credentials, but it still stores a catalog credential, and it must implement SigV4 unless it is willing to work only against 1.11.0-or-newer catalogs that positively advertise and offer remote signing.

---

2. The specification versus implementation line

This is the distinction the original claims did not draw, and it is where most of the error came from.

In the REST catalog specification, therefore fair to build against:

• GET /v1/config returning CatalogConfig with required defaults and overrides, optional endpoints and idempotency-key-lifetime. Clients "should first call this route". Precedence is a three-layer merge: "Catalog configuration is constructed by setting the defaults, then client-provided configuration, and finally overrides." A warehouse query parameter selects which warehouse's config is returned.
• GET /v1/{prefix}/namespaces/{namespace}/tables/{table} (loadTable) returning LoadTableResult, whose only required field is metadata, the full table metadata "matching the table metadata JSON file".
• Optional storage-credentials on LoadTableResult (since 1.7.0) and a dedicated GET .../tables/{table}/credentials (loadCredentials, "Load vended credentials for a table from the catalog"), which takes an optional planId.
• Optional POST .../tables/{table}/sign (signRequest), since 1.11.0 only.
• The X-Iceberg-Access-Delegation header, optional, accepted on exactly five operations: createTable, planTableScan, fetchPlanningResult, registerTable, loadTable. Note it is not accepted on the credentials endpoint.
• Optional server-side scan planning, with scan-planning-mode telling the client which mode it must use.
• The S3 config keys named in the spec: client.region, s3.access-key-id, s3.secret-access-key, s3.session-token, s3.remote-signing-enabled, s3.cross-region-access-enabled.

Not in any specification, one library's behaviour only:

• /v1/aws/s3/sign as a path, and s3.signer.uri / s3.signer.endpoint as property names. iceberg-java, all @Deprecated as of 1.11.0.
• s3.session-token-expires-at-ms and the whole credential refresh policy around it. iceberg-java's VendedCredentialsProvider treats the credential as stale at expiry and prefetches five minutes early. Nothing obliges any other server to send that key or any other client to read it.
• gcs.oauth2.token and gcs.oauth2.token-expires-at. iceberg-java. No GCS or ADLS key is standardised anywhere in the OpenAPI document; the only mention of ADLS/GCS is the sentence deferring to storage-credentials, and "azure" does not occur at all.
• The 30 second / 100 entry signature cache and the exclusion of range from signed headers.
• Avro as the container format for the manifest list (see below).
• Manifest compression defaults.
• version-hint.text as the filesystem catalog pointer file.
• The Hive catalog's table_type / metadata_location table-parameter conventions.

---

3. What a read-only implementation must handle

3a. Correctness. These produce wrong answers, not errors

All [SPEC] from https://raw.githubusercontent.com/apache/iceberg/main/format/spec.md unless noted.

1. Reject a higher format version. "Implementations must throw an exception if a table's version is higher than the supported version." This is the cheapest guard we have.
2. Skip status = 2 (DELETED) manifest entries. "Deleted entries in data and delete manifests (those marked with status "DELETED") are not used in a scan." Reading them resurrects deleted files silently.
3. Sequence number inheritance is mandatory. "When reading, sequence numbers are inherited by replacing null with the manifest's sequence number from the manifest list." And: "The data and file sequence numbers are inherited only if the entry status is 1 (added). If the entry status is 0 (existing) or 2 (deleted), the entry must include both sequence numbers explicitly." Every delete scope rule is a sequence number comparison, so an error here silently applies or fails to apply deletes.
4. Never use file_sequence_number for delete pruning. It sits next to sequence_number in the same struct and is the obvious wrong field to grab: "The file sequence number can't be used for pruning delete files as the data within the file may have an older data sequence number."
5. Delete scope comparisons differ. Position deletes and deletion vectors apply when the data file's data sequence number is "less than or equal to" the delete's. Equality deletes apply when it is "strictly less than" the delete's. Position deletes therefore apply to data files from the same commit.
6. Equality deletes written with an unpartitioned spec are global. "Equality delete files stored with an unpartitioned spec are applied as global deletes. Otherwise, delete files do not apply to files in other partitions." A reader that always requires partition equality returns deleted rows.
7. A deletion vector suppresses matching position delete files. "If a DV is written for a data file, it must replace all previously written position delete files so that when a DV is present, readers can safely ignore matching position delete files."
8. v1 compatibility. Apply the v1 defaults (last-sequence-number, snapshot sequence-number, manifest list sequence_number and min_sequence_number, entry sequence_number and file_sequence_number all default to 0; manifest list content and data file content default to 0). Handle a v1 snapshot that carries an inline manifests list instead of a manifest-list file.
9. Be permissive about required fields in manifests. "Readers should be more permissive because v1 metadata files are allowed in v2 tables (or later)." Fields required in v2 must be read as optional in manifest and manifest-list files. Metadata JSON may be read strictly.
10. Project strictly by field ID, with the full four-step fallback. "Columns in Iceberg data files are selected by field id. The table schema's column names and order may change after a data file is written, and projection must be done using field ids." When a projected id is missing from a file, in this order: identity-partition value from the manifest, then schema.name-mapping.default, then the field's initial-default, then null. Skipping step 1 or step 3 returns nulls where real values exist. The spec's worked example is a good test case: file schema 1: a int, 2: b string, 3: c double projected as 3: measurement, 2: name, 4: a "must select file columns c (renamed to measurement), b (now called name), and a column of null values called a; in that order." A name-based reader returns the file's real a column instead of nulls.
11. Fail loudly on a non-Parquet file_format. Legal values are avro, orc, parquet, puffin. Silently skipping an ORC data file returns a wrong answer.
12. Partition pruning is inclusive, so residuals must still be evaluated. "This is called inclusive because rows that do not match the scan predicate may be included in the scan by the partition predicate." Treating partition pruning as sufficient returns extra rows.
13. An unknown partition transform projects to true, never false. Defaulting to false on an unrecognised transform silently drops files.
14. current-snapshot-id of -1 means no snapshot. "Java writes -1 for "no current snapshot" with V1 and V2 tables and considers this equivalent to omitted or null. This has never been formalized in the spec, but for compatibility, other implementations can accept -1 as null."
15. The day transform yields a date, but readers "must also accept int values", interpreted as days since 1970-01-01.
16. Duplicate file paths within one snapshot make scan results undefined. "Reader implementations may raise an error in this case, but are not required to do so." A cheap defensive check is legitimate here.

Two hazards that live inside otherwise optional features, and are correctness bugs if we use the feature at all:

17. Bound blobs do not record their type. "Iceberg's Avro manifest format does not store the type of lower and upper bounds, and type promotion does not rewrite existing bounds." A 4-byte bound on a long column was written as int; 4-byte on double means float; 4-byte on timestamp means date. The write-time type must be inferred from byte length. Decoding a 4-byte bound as the current 8-byte type produces garbage and therefore eliminates the wrong files. Encoding is Appendix D: int/float/date 4-byte little-endian, long/double/time/timestamp 8-byte little-endian, string as UTF-8 with no length prefix, decimal as unscaled two's-complement big-endian minimum bytes, uuid 16-byte big-endian. Note the endianness inconsistency between the numerics and decimal/uuid.
18. Bounds are inclusive and may be truncated, not exact min/max. They may be used only to eliminate files, never to answer a query such as min() or max() directly. v3 and earlier carry no tightness guarantee; v4 adds an explicit tight_bounds boolean precisely because of that.

And if transforms are implemented at all: bucket[N] is "the 32-bit Murmur3 hash, x86 variant, seeded with 0" with (hash & Integer.MAX_VALUE) % N, and truncate on negative integers is not C's %: "The remainder, v % W, must be positive. For languages where % can produce negative values, the correct truncate function is: v - (((v % W) + W) % W)". Conversion must use "the partition spec that was used to write the manifest file regardless of the current partition spec".

3b. Performance only. Omitting these costs time, never correctness

• Manifest-list partition field summaries (507 partitions, with contains_null, contains_nan, lower_bound, upper_bound) and the file and row counts. "Manifests that contain no matching files, determined using either file counts or partition summaries, may be skipped." Note partitions is optional in v1, v2 and v3, and the bounds within a field_summary are optional too, so a reader cannot depend on this level of pruning existing.
• All manifest column statistics (column_sizes, value_counts, null_value_counts, nan_value_counts, lower_bounds, upper_bounds). All optional in every version.
• Inclusive projection and every partition transform. Implementing zero transforms is spec-legal, because an unknown transform projects to true. In v1/v2 readers "should" ignore unknown transforms; in v3 this behaviour is required. Partition values are still compared bytewise for the delete-scope equality rules, and unknown transforms do not affect that comparison.
• 132 split_offsets, "all row group offsets in a Parquet file. Must be sorted ascending", directly usable for split planning.
• Delete-file statistics: "If metrics show that a delete file has no rows that match a scan predicate, it may be ignored just as a data file would be ignored."
• Server-side scan planning, unless scan-planning-mode is server.

3c. Format work implied, stated without effort claims

• [SPEC] Manifests are Avro: "A manifest is an immutable Avro file that lists data files or delete files". So an Avro object-container reader is unavoidable. Framing is small and fully specified: magic Obj\x01, a JSON metadata map containing avro.schema and optional avro.codec, a 16-byte sync marker, then blocks of object count, byte size, payload and sync (https://avro.apache.org/docs/1.12.0/specification/).
• [SPEC] Iceberg restricts Avro unions to the nullable two-branch case only: "Optional fields, array elements, and map values must be wrapped in an Avro union with null. This is the only union type allowed in Iceberg data files." No general union resolution is needed.
• [SPEC] Iceberg field IDs ride in the Avro schema JSON as custom attributes field-id, element-id, key-id, value-id, so the schema parser must retain unknown attributes.
• [SPEC] Each manifest carries its own schema and partition spec in the Avro file's key-value metadata (schema, partition-spec, partition-spec-id, format-version, content), and must be read before decoding rows, because the partition struct type varies per manifest.
• Codec, the load-bearing detail. [SPEC] The table spec never names an Avro codec; a grep of all 2142 lines for avro.codec, deflate, zstd, snappy returns nothing. [SPEC] Avro requires null and deflate, and Avro's deflate is raw RFC 1951: "Note that this format (unlike the "zlib format" in RFC 1950) does not have a checksum." That is inflateInit2(windowBits = -15), a different call from our existing Parquet GZIP path. [IMPLEMENTATION] iceberg-java's MANIFEST_COMPRESSION_DEFAULT = "gzip" and it maps GZIP to CodecFactory.deflateCodec(...), so the on-disk avro.codec value is the string deflate. PyIceberg makes the same mapping explicit: CODEC_MAPPING_ICEBERG_TO_AVRO = {"gzip": "deflate", "zstd": "zstandard"} and its Avro write default is also gzip. [IMPLEMENTATION] iceberg-java accepts UNCOMPRESSED, SNAPPY, GZIP, ZSTD; PyIceberg reads null, bzip2, snappy, zstandard, deflate. We link zlib, lz4 and zstd, so a snappy or bzip2 compressed manifest would be an unreadable table for us.
• Manifest list container format is a genuine spec gap. [SPEC] The spec says only "A manifest list is a valid Iceberg data file: files must use valid Iceberg formats, schemas, and column projection." This wording is identical at 1.4.0, 1.6.0, 1.9.0 and main. The only hint of Avro is an Appendix C example value "manifest-list": "s3://b/wh/.../s1.avro". [IMPLEMENTATION] iceberg-java hard-codes it: InternalData.write(FileFormat.AVRO, file) in ManifestListWriter.java and InternalData.read(FileFormat.AVRO, manifestList) in ManifestLists.java. Assuming Avro will work against Java-written tables, but that assumption rests on the reference implementation.
• Table metadata JSON. [SPEC] "Table metadata is stored as JSON" and "A metadata JSON file may be compressed with [GZIP]", with two suffix conventions in the wild (.gz.metadata.json, and Java additionally reads metadata.json.gz). [IMPLEMENTATION] METADATA_COMPRESSION_DEFAULT = "none", so uncompressed is the normal case. [SPEC] With a REST catalog we may never read the file at all, since metadata is a required field of LoadTableResult and metadata-location is the optional one.
• Parquet delta. [SPEC] "Column IDs are required to be stored as field IDs on the parquet schema" and "Lists must use the 3-level representation." [SPEC] On the Parquet side field_id is thrift field 9 of SchemaElement and is optional there (https://raw.githubusercontent.com/apache/parquet-format/master/src/main/thrift/parquet.thrift), so a general reader is not obliged to surface it and ours must plumb it through the footer parse.
• [IMPLEMENTATION] Iceberg's default Parquet codec is zstd, which we already link. brotli and lz4 are configurable alternatives; we link lz4 but not brotli (https://raw.githubusercontent.com/apache/iceberg/main/docs/docs/configuration.md).

---

4. Answers to the open questions

Deletes. v1 has none: "Row-level deletes are added by v2 and are not supported in v1." v2 adds position delete files and equality delete files. v3 adds Puffin deletion vectors and deprecates position delete files, but a v3 reader must still handle them: "Position delete files must not be added to v3 tables, but existing position delete files are valid."

Position deletes are cheap: keyed by (file_path, pos) starting at 0, sorted by file path then position, so they apply as an ordinal bitmap and can be streamed. Equality deletes are materially harder. Each delete row is an AND of equality predicates over the columns named in equality_ids, a null in a delete column means IS NULL rather than "no match", the delete may be global if its spec is unpartitioned, and: "If a delete column in an equality delete file is later dropped from the table, it must still be used when applying the equality deletes." That last rule forces column resolution against a historical schema. Position deletes cost a bitmap; equality deletes cost an anti-join.

Field IDs. Answered above: projection is by field ID only, Parquet field IDs are required by Iceberg, the four-step fallback is mandatory, and 3-level lists are required.

Format versions. "Versions 1, 2 and 3 of the Iceberg spec are complete and adopted by the community. Version 4 is under active development and has not been formally adopted." v3 adds nanosecond timestamps, unknown, variant, geometry, geography, column default values, multi-argument transforms, row lineage, binary deletion vectors and table encryption keys. v4 adds relative paths and a typed content_stats struct replacing the stats maps, and removes the filesystem commit scheme. Anything read about v4 today is a moving target.

Row lineage (v3) imposes reader MUSTs only if we expose the _row_id and _last_updated_sequence_number metadata columns. A reader that does not project them is unaffected. If it does, it must compute first_row_id by inheritance and derive _row_id as first_row_id + _pos.

Puffin. v3 deletion vectors are deletion-vector-v1 blobs from the Puffin spec (https://raw.githubusercontent.com/apache/iceberg/main/format/puffin-spec.md). Puffin framing is small: magic PFA1, blobs, then a footer located from the end. The DV is addressed directly by content_offset and content_size_in_bytes from the manifest entry, so the whole file need not be parsed. The DV blob layout is a 4-byte big-endian length, the 4-byte magic D1 D3 39 64, the serialized roaring bitmap, and a 4-byte big-endian CRC-32. The blob itself is never compressed ("Omit compression-codec; deletion-vector-v1 is not compressed"), though the Puffin footer may be LZ4-frame compressed, which we already link. Supporting v3 deletes therefore requires a Puffin reader plus a roaring bitmap decoder, not just Parquet.

Time travel. Reading an older snapshot is the same machinery pointed at a different snapshot ID: table metadata carries the snapshot list, and loadTable returns the full metadata. The -1 caveat above applies. I read the Snapshot References section only in passing, so branch and tag read semantics and the exact interaction of refs with current-snapshot-id are [COULD-NOT-DETERMINE] from this pass.

Encryption. v3 lists "Table encryption keys" as a capability, and manifest list entries carry 519 key_metadata. I did not read the encryption sections of the spec. The safe position is to detect encrypted tables and refuse them rather than risk misreading; the details are [COULD-NOT-DETERMINE].

Compatibility negotiation. The mechanisms are: the mandatory format-version check; the reader-permissiveness rule for v1 files inside v2+ tables; GET /v1/config with the three-layer merge; the optional endpoints array, whose documented default set (assumed when the field is absent) contains namespaces, tables, register, metrics, rename and transaction commit, and contains neither .../credentials nor .../sign; the optional X-Iceberg-Access-Delegation header, where the server may honour none of what we advertise; and scan-planning-mode. There is no version field to negotiate on: info.version is 0.0.1 everywhere.

---

5. What this means for sequencing, and #393

The earlier position, that the REST catalog path lets us defer #393's object-storage decisions, was overstated. What the sources support:

1. SigV4 is required work, not optional. It is needed on the vended-credentials path, which is the more established and more widely supported of the two mechanisms. It is also exactly what plain S3 object-storage reads need for Object storage reads for external Parquet, which #388 depends on #393, so the cost is shared rather than Iceberg-specific.
2. Build the vended-credentials path first; treat remote signing as a later optional addition. This reverses the priority implied by the earlier comments. Remote signing only entered the core spec at 1.11.0, iceberg-go refuses it, PyIceberg supports it only on its non-default FileIO, and iceberg-java disables it by default.
3. An HTTP and TLS client is required either way, and now for two purposes. The catalog itself is HTTP with mandatory authentication (OAuth2 client credentials or a bearer token), and data files still come from object storage over ranged GETs.
4. We must design our own credential refresh policy. The spec defines no expiry, no TTL and no refresh semantics for storage credentials. StorageCredential is {prefix, config} with an untyped string map. s3.session-token-expires-at-ms is an iceberg-java convention we would be copying, not a standard we can rely on. The spec does provide GET .../tables/{table}/credentials to re-fetch without reloading metadata, which is the natural refresh hook, but says nothing about when to call it.
5. What the REST catalog does buy us: no long-lived AWS key in configuration (replaced by a catalog credential), and one fewer object-store read, because metadata is returned inline by loadTable rather than fetched from metadata.json.
6. If we ever do implement remote signing, the round-trip cost depends entirely on server behaviour we have not verified. Cacheable signatures make it viable for a columnar scan; no-cache responses, or a server that signs the Range header, would make it one catalog round trip per range read.

---

6. What is still unknown

Listed honestly, because several of these bear directly on a build decision.

Vendor behaviour, none of which was checked. This is the largest gap.

• Whether Polaris, Unity, Lakekeeper, Gravitino, Nessie, AWS S3 Tables or Snowflake Open Catalog actually implement GET .../credentials or POST .../tables/{table}/sign, still only the deprecated /v1/aws/s3/sign, or neither, and which they default to. Only apache/iceberg sources were examined.
• Whether production catalogs emit Cache-Control: private for GET and HEAD sign responses. Only Iceberg's own reference test servlet was verified.
• Whether real servers exclude the Range header from signing the way the reference servlet does. If a server signs Range, cached signatures cannot be reused across byte ranges.

Specification gaps that are gaps, not lookup failures.

• Vended credential lifetime, expiry detection and refresh are entirely undefined by the spec.
• What a client must do when a catalog does not vend. The spec orders the lookup but never states a fallback to locally configured credentials, and defines no error for a non-vending catalog that receives X-Iceberg-Access-Delegation: vended-credentials.
• Whether s3.access-key-id and friends are formally the keys expected inside StorageCredential.config. The spec documents them under "AWS Configurations" for the table config map and separately says credentials come through storage-credentials, but never enumerates a StorageCredential's config map. Every implementation reuses the same names; the spec does not say so.
• No GCS or ADLS credential key is standardised at all.
• Behaviour against a server that sends no endpoints field and no credentials is unspecified beyond trial and error.

Version and stability questions.

• Whether released 1.11.0 types the delegation header as type: string (as 1.10.0 does) or type: array (as main does). The two reads of 1.11.0 disagreed and this was not re-checked. A reader must tolerate both shapes across catalog vintages regardless.
• Whether main's /sign endpoint and remote-signing-config schema will ship unchanged. PR #16822 is on main only as of 2026-08.
• The exact commit that introduced storage-credentials was not read; it was bracketed between tags 1.6.0 (absent) and 1.7.0 (present).

Format questions.

• Whether pyiceberg, iceberg-rust and iceberg-go also assume Avro for manifest lists. Only iceberg-java was confirmed. Since the spec does not mandate a format, this matters before relying on Avro as universal.
• What fraction of real tables are written with deflate versus zstd manifests. Only defaults are knowable from source. Trino, Flink, DuckDB, Snowflake and commercial catalogs were not audited and may set the property differently.
• Whether the manifest list uses write.manifest.compression-codec or falls back to write.avro.compression-codec. Both default to gzip/deflate so the practical answer is the same, but the precedence is unverified.
• Appendix B's per-type murmur3 input encodings (needed to actually compute bucket[N]) and Appendix A's Avro and ORC type mappings were not read in detail.
• No primary source gives a size distribution for metadata.json in practice. Only two anecdotal Apache issue reports were found: #5219 ("after hundreds of column addition operations, the metadata will have a size of several hundred MB") and #9734 (schema accumulation, "when a schema change is applied the entire previous schema(s) is reproduced in every subsequent metadata.json"). Widely repeated figures from vendor blog posts were not treated as primary and are excluded.

Scope not covered in this pass.

• The server-side scan planning path (POST .../plan, scan-planning-mode). If a catalog sets scan-planning-mode: server, the spec says clients MUST use it, which would change the manifest-reading requirements substantially. That needs its own pass.
• Branch and tag reads via refs.
• Encryption.
• Whether PostgreSQL's common/jsonapi.h is exported in the installed server headers for all of PG 15 through 19, and whether the incremental parser entry points exist before PG 17. That is a build-side check, not a source-side one.

Method caveats.

• The rendered pages at iceberg.apache.org/spec/ and iceberg.apache.org/puffin-spec/ returned site navigation only through WebFetch on repeated attempts. All spec quotes come from raw.githubusercontent.com. Spot checks against the published page matched for every sentence checked.
• Main-branch quotes include unreleased v4 material. Version-sensitive statements were pinned to release tags and are labelled as such.
• iceberg-rust's lack of remote signing rests on GitHub code search returning zero hits for four targeted queries. Code search indexing can be incomplete, so that is a strong indication, not a proof of absence.