[wip]feat: support column default values (initial-default / write-default)#731
Draft
huan233usc wants to merge 1 commit into
Draft
[wip]feat: support column default values (initial-default / write-default)#731huan233usc wants to merge 1 commit into
huan233usc wants to merge 1 commit into
Conversation
huan233usc
marked this pull request as draft
June 11, 2026 21:07
huan233usc
changed the title
huan233usc
force-pushed
the
feat/column-default-values
branch
5 times, most recently
from
June 12, 2026 07:24
bb5f2d0 to
1b2be81
Compare
huan233usc
commented
Jun 12, 2026
| return std::cref(*write_default_); | ||
| } | ||
|
|
||
| SchemaField SchemaField::WithInitialDefault(Literal initial_default) const { |
Contributor
Author
There was a problem hiding this comment.
Alternative to those with methods are make it in constructor
huan233usc
force-pushed
the
feat/column-default-values
branch
8 times, most recently
from
June 14, 2026 23:13
1613523 to
c1a3762
Compare
Implements Iceberg v3 column default values (apache#730, item 2 of apache#637): - Schema model: SchemaField carries optional `initial-default` and `write-default` literals, with validation that defaults are primitive and match the field type. - JSON serde: parse and write the two fields using single-value serialization (resolves the TODO in struct field serialization). - Read path: Project() maps a missing column with an initial-default to FieldProjection::Kind::kDefault carrying the literal (for both required and optional columns, per spec), and the Parquet and Avro readers materialize it as a constant column via a new Literal-to-Arrow helper. This resolves the default-value TODO in schema projection. - Schema evolution: Add*Column accept an optional default value (used as both initial-default and write-default); a required column with a default no longer needs AllowIncompatibleChanges(); RequireColumn() accepts columns added with a default in the same update (resolves the defaulted-add TODO); UpdateColumnDefault() updates the write-default of an existing column; doc/rename/type-promotion updates preserve defaults (promotion casts them to the new type). - Format version gating: Schema::Validate() rejects schemas with default values below v3, using the existing kMinFormatVersionDefaultValues. Writers consume complete Arrow arrays, so applying write-default to omitted columns remains the engine's responsibility (as in Java); the library stores, validates, and serializes it.
huan233usc
force-pushed
the
feat/column-default-values
branch
from
June 15, 2026 17:18
c1a3762 to
5feeec4
Compare
Contributor
Author
23 tasks
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 15, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 15, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 15, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 16, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 16, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 16, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 18, 2026
First of a multi-part split of column default value support (apache#730) — the schema foundation the read and evolution paths build on. Purely additive; no read/write behavior change on its own. - SchemaField carries `initial-default` / `write-default` (immutable std::shared_ptr<const Literal>) with copy-preserving WithInitialDefault / WithWriteDefault modifiers; getters return optional<reference_wrapper>. - JSON serde reads/writes `initial-default` / `write-default` via the existing single-value serialization. - Schema::Validate rejects default values below format v3 and validates they are non-null primitive literals matching the field type. - Generic schema projection maps a column missing from a data file with an initial-default to FieldProjection::Kind::kDefault. Read-path application (Parquet/Avro) and schema evolution follow in separate PRs. See apache#731 for the full end-to-end proof-of-concept.
wgtmac
pushed a commit
that referenced
this pull request
Jun 29, 2026
Part 1 of a multi-part split of #730 (column default values, item 2 of #637). The full end-to-end implementation is in #731, kept open as the proof-of-concept; this series lands it in reviewable pieces. This PR is the **schema foundation** — representing, serializing and validating v3 column default values. It is purely additive and changes no read or write behavior on its own. ## What's in this PR - **`SchemaField`** carries `initial-default` / `write-default`, stored as `std::shared_ptr<const Literal>` (immutable payload shared across copies, like the adjacent `type_`; the C++ analog of Java's `final Literal<?>`). They are set via the constructor. Getters return `std::optional<std::reference_wrapper<const Literal>>` for reading (the `Schema::FindFieldByName` idiom); `initial_default_ptr()` / `write_default_ptr()` expose the shared pointer so a rebuilt field (e.g. ID reassignment) shares the value instead of copying it. - **JSON serde**: parse/write `initial-default` / `write-default` using the existing single-value serialization (all primitive types). - **`Schema::Validate`**: version-gates the `initial-default` to format v3 (`kMinFormatVersionDefaultValues`) — it reinterprets how existing data files are read, so it requires the v3 reader contract. The `write-default` only affects values written going forward and is **not** version-gated (matching Java's `Schema.checkCompatibility`, which gates only the initial default). Both defaults are otherwise validated to be non-null primitive literals matching the field type. - **Generic projection**: a column missing from a data file with an `initial-default` maps to `FieldProjection::Kind::kDefault` carrying the literal (the per-format readers consume this in the follow-up PRs). ## Follow-ups (stacked on this PR) - read path — Parquet (`literal_util` + parquet projection/materialization) - read path — Avro - schema evolution (`UpdateSchema` add/update column defaults) ## Testing Added tests
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 29, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jun 30, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. The materializer wraps the storage array in the extension type for extension types such as `arrow.uuid` (compute::Cast has no storage->extension kernel). Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. The materializer wraps the storage array in the extension type for extension types such as `arrow.uuid` (compute::Cast has no storage->extension kernel). Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. The materializer wraps the storage array in the extension type for extension types such as `arrow.uuid` (compute::Cast has no storage->extension kernel). Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. The materializer wraps the storage array in the extension type for extension types such as `arrow.uuid` (compute::Cast has no storage->extension kernel). Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
…s (2/4) When a column is present in the read schema but missing from a Parquet data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Adds a shared Arrow materializer (arrow/literal_util) that turns a Literal into an Arrow scalar/array, and a kDefault projection branch in the Parquet schema/data projection paths. The materializer wraps the storage array in the extension type for extension types such as `arrow.uuid` (compute::Cast has no storage->extension kernel). Part 2 of the v3 column-default-values work (POC apache#731), built on the schema support merged in apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
AddColumn / AddRequiredColumn now accept an optional default value, used as both the initial-default and write-default of the new column; a non-null default also lets a required column be added (or an added column be made required) without AllowIncompatibleChanges(). UpdateColumnDefault sets or clears the write-default. Defaults are cast to the column type (rejecting uncastable or out-of-range values) and preserved across rename / doc / type updates and nested field-id reassignment. Part 4 of the v3 column-default-values work (POC apache#731), built on apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 1, 2026
AddColumn / AddRequiredColumn now accept an optional default value, used as both the initial-default and write-default of the new column; a non-null default also lets a required column be added (or an added column be made required) without AllowIncompatibleChanges(). UpdateColumnDefault sets or clears the write-default. Defaults are cast to the column type (rejecting uncastable or out-of-range values) and preserved across rename / doc / type updates and nested field-id reassignment. Part 4 of the v3 column-default-values work (POC apache#731), built on apache#746.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 2, 2026
AddColumn / AddRequiredColumn now accept an optional default value, used as both the initial-default and write-default of the new column; a non-null default also lets a required column be added (or an added column be made required) without AllowIncompatibleChanges(). UpdateColumnDefault sets or clears the write-default. Defaults are cast to the column type (rejecting uncastable or out-of-range values) and preserved across rename / doc / type updates and nested field-id reassignment. Part 4 of the v3 column-default-values work (POC apache#731), built on apache#746.
wgtmac
pushed a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 11, 2026
AddColumn / AddRequiredColumn now accept an optional default value, used as both the initial-default and write-default of the new column; a non-null default also lets a required column be added (or an added column be made required) without AllowIncompatibleChanges(). UpdateColumnDefault sets or clears the write-default. Defaults are cast to the column type (rejecting uncastable or out-of-range values) and preserved across rename / doc / type updates and nested field-id reassignment. Part 4 of the v3 column-default-values work (POC apache#731), built on apache#746.
wgtmac
pushed a commit
that referenced
this pull request
Jul 11, 2026
## What Part 3 of 4 of Iceberg v3 column default-value support (POC #731), built on the schema-layer support merged in #746. Independent of the read-path PRs (#792 and the Avro follow-up). Adds default-value handling to `UpdateSchema` (schema evolution). ## Changes - **`AddColumn` / `AddRequiredColumn`** take an optional `default_value`. When provided it is set as both the column's `initial-default` and `write-default`. A non-null default also lets a required column be added without `AllowIncompatibleChanges()` — rows written before the change read the default instead of null. - **`UpdateColumnDefault(name, default)`** (new) sets, or clears with `std::nullopt`, a column's `write-default`; the `initial-default` is fixed when the column is added. - Defaults are **cast to the column type** (rejecting uncastable or out-of-range values) and **preserved** across rename / doc / type-promotion updates and nested field-id reassignment. - `RequireColumn` may now mark a column that was added with a default required. The `SchemaField` constructor stores defaults verbatim (it does not coerce them), so the cast/promotion is performed explicitly at each evolution site — the same effect as Java, where `NestedField`'s constructor runs `castDefault`. Same-scale decimal precision widening is handled directly (the unscaled value is unchanged), since `Literal::CastTo` does not cast between decimal types. ## Tests 13 cases in `update_schema_test.cc`: add optional/required/nested column with a default, mismatched/narrowing rejection, `UpdateColumnDefault` (set / clear / cast-to-type / pre-existing column), require-after-default, and preservation across doc updates and type promotion — including same-scale decimal precision promotion. ## Stack 1. #746 — schema: represent / serialize / validate (merged) 2. #792 — read path: Parquet 3. read path: Avro (follows) 4. **this PR** — schema evolution: `addColumn` / `updateColumnDefault`
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 12, 2026
…3/4) When a column is present in the read schema but missing from an Avro data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Reuses the shared arrow/literal_util materializer (merged in apache#792) and adds AppendDefaultToBuilder for the row-by-row Avro decode paths, plus a kDefault projection branch in the Avro schema/data projection. Part 3 of the v3 column-default-values work (POC apache#731), built on the schema support in apache#746 and the Parquet read path in apache#792.
huan233usc
added a commit
to huan233usc/iceberg-cpp
that referenced
this pull request
Jul 20, 2026
…3/4) When a column is present in the read schema but missing from an Avro data file (written before the column existed), fill it with the column's v3 initial-default instead of null. Reuses the shared arrow/literal_util materializer (merged in apache#792) and adds AppendDefaultToBuilder for the row-by-row Avro decode paths, plus a kDefault projection branch in the Avro schema/data projection. Part 3 of the v3 column-default-values work (POC apache#731), built on the schema support in apache#746 and the Parquet read path in apache#792.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #730 (item 2 of #637).
This end-to-end proof of concept has been split into four smaller pull requests:
UpdateSchemaPlease review the split pull requests above. This PR is retained only as the original end-to-end proof of concept.