[IcebergIO] add test for column default values - #39739
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds integration test coverage for Iceberg V3 column default values in the Iceberg CDC incremental changelog reader, ensuring Beam’s IcebergIO correctly materializes defaulted values when reading older files after schema evolution.
Changes:
- Add a test verifying that an older data file (written before schema evolution) is read with the new column’s initial default value.
- Add a test verifying overwrite CDC semantics where the “before” row uses the initial default (old file) and the “after” row uses an explicitly written value (new file).
- Add a small formatting
DoFnused to assert defaulted vs explicit values in formatted CDC output.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #39739 +/- ##
============================================
+ Coverage 54.97% 58.17% +3.20%
- Complexity 1722 13104 +11382
============================================
Files 1071 2523 +1452
Lines 170836 265755 +94919
Branches 1262 10795 +9533
============================================
+ Hits 93910 154600 +60690
- Misses 74656 105336 +30680
- Partials 2270 5819 +3549
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Assigning reviewers: R: @kennknowles for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
Default column values is an Iceberg V3 feature: https://iceberg.apache.org/spec/#version-3-extended-types-and-capabilities
Add tests to verify that we support it (since we're using the latest Iceberg libraries)
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.