[oracle] Support row_id metadata column for Oracle CDC (with headers-loss bug fix) - #4486
Open
fightBoxing wants to merge 1 commit into
Open
[oracle] Support row_id metadata column for Oracle CDC (with headers-loss bug fix)#4486fightBoxing wants to merge 1 commit into
fightBoxing wants to merge 1 commit into
Conversation
Add support for extracting Oracle ROWID pseudo-column as a metadata
column via 'METADATA FROM row_id VIRTUAL' syntax in Flink SQL.
Changes:
1. OracleReadableMetaData: Add ROW_ID metadata enum that reads ROWID
from SourceRecord headers.
2. OracleScanFetchTask (snapshot phase):
- Rewrite SELECT SQL to append ROWID pseudo-column at the tail:
'SELECT * FROM tab' -> 'SELECT T0.*, ROWID FROM tab T0'.
- Manually construct ColumnArray excluding ROWID to bypass Debezium
ColumnUtils strict validation.
- Extract ROWID via OracleResultSet.getROWID(N+1) and inject into
SourceRecord headers by overriding
SnapshotChangeRecordEmitter#getEmitConnectHeaders.
3. JdbcSourceFetchTaskContext: Fix headers loss bug in
formatMessageTimestamp(). The previous 8-arg SourceRecord
constructor dropped headers when rewriting snapshot records; switch
to the 10-arg constructor to preserve timestamp and headers. This
fix benefits all JDBC-based Flink CDC connectors when they rely on
headers for downstream metadata.
Requirements:
- scan.incremental.snapshot.enabled=true (Debezium native snapshot
path is not covered by this change).
- Table supplemental log enabled for streaming ROWID capture.
Verified with Oracle 12c Standard Edition (12.1.0.2.0). Sample output:
+I[2, bbb, AABDuHAAGAAAAF3AAA]
+I[3, ccc, AABDuHAAGAAAAF0AAA]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概述
为 Oracle CDC 连接器新增
row_id元数据列支持,用户可通过METADATA FROM 'row_id' VIRTUAL在 Flink SQL 中获取 Oracle 表的 ROWID 伪列。动机
业务场景需要基于 Oracle ROWID 进行:
使用方式
输出示例:
改动清单
1.
OracleReadableMetaData.java— 新增 ROW_ID 元数据枚举从
SourceRecord.headers()中读取 key =ROWID的 header 值。2.
OracleScanFetchTask.java— 快照阶段实现SELECT * FROM tab→SELECT T0.*, ROWID FROM tab T0(ROWID 放末尾,保持物理列位置 1..N 不变)ColumnArray,只包含表原有列,避免ColumnUtils.toArray因 ROWID 列不在 schema 中而报错((OracleResultSet) rs).getROWID(N+1)SnapshotChangeRecordEmitter#getEmitConnectHeaders()3.
JdbcSourceFetchTaskContext.java— 关键 bug 修复 ⭐formatMessageTimestamp()在快照阶段重构 SourceRecord 时使用了 8 参构造函数,丢弃了原始 record 的 headers。改为 10 参构造函数保留timestamp和headers。此修复影响范围:不仅解决 Oracle row_id,也惠及所有基于 JDBC 的 Flink CDC 连接器(MySQL、PostgreSQL、SQL Server、Db2),使它们的增量快照模式下 SourceRecord headers 得以正确保留,为后续实现其他 headers 相关的元数据列(SCN、Position 等)打好基础。
4.
docs/ROWID_METADATA_IMPLEMENTATION.md— 方案说明文档前置要求
scan.incremental.snapshot.enabled = true(本 PR 未修改 Debezium 原生快照路径)ALTER TABLE <schema>.<table> ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS验证
mvn clean package -Dflink.version=1.16.3✅数据流