[cdc-connector][oracle] Support row_id metadata column and fix headers loss in JdbcSourceFetchTaskContext#4487
Open
fightBoxing wants to merge 1 commit into
Open
Conversation
…s loss in JdbcSourceFetchTaskContext
Add support for extracting Oracle ROWID pseudo-column as a metadata
column via 'METADATA FROM row_id VIRTUAL' syntax in Flink SQL.
Changes:
1. OracleReadableMetaData: Add ROW_ID metadata enum that reads ROWID
from SourceRecord headers.
2. OracleScanFetchTask (snapshot phase):
- Rewrite SELECT SQL to append ROWID pseudo-column at the tail:
'SELECT * FROM tab' -> 'SELECT T0.*, ROWID FROM tab T0'.
- Manually construct ColumnArray excluding ROWID to bypass Debezium
ColumnUtils strict validation on the extra pseudo-column.
- Extract ROWID via OracleResultSet.getROWID(N+1) and inject into
SourceRecord headers by overriding
SnapshotChangeRecordEmitter#getEmitConnectHeaders.
3. JdbcSourceFetchTaskContext (bug fix for all JDBC-based connectors):
Both rewriteOutputBuffer() and formatMessageTimestamp() were using
the 8-arg SourceRecord constructor and dropping the original
record's headers. Switch to the 10-arg constructor to preserve
timestamp and headers. Without this fix, any header-based metadata
(e.g. Oracle row_id) would be silently discarded when the fetcher
rewrites snapshot records.
Requirements to use row_id in Oracle CDC:
- scan.incremental.snapshot.enabled=true
- Table supplemental log enabled for streaming ROWID capture
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add support for extracting Oracle ROWID pseudo-column as a metadata column in Oracle CDC. Users can then access it in Flink SQL via:
Typical use cases: data lineage, exact-row identification, idempotent downstream writes, systems that rely on ROWID for reconciliation.
What is changed
1.
OracleReadableMetaData— newROW_IDenumReads the ROWID value from the
SourceRecordheaders (key =ROWID).2.
OracleScanFetchTask(snapshot phase)Oracle's
SELECT *does not include ROWID. So:SELECT * FROM tab→SELECT T0.*, ROWID FROM tab T0(ROWID placed at the tail keeps physical column positions unchanged).ColumnArrayfrom the table's own columns to bypass Debezium'sColumnUtils.toArray(rs, table)strict validation, which would otherwise reject the extra ROWID column.((OracleResultSet) rs).getROWID(N+1)and inject it intoSourceRecordheaders by overridingSnapshotChangeRecordEmitter#getEmitConnectHeaders.Streaming phase already carries ROWID in headers through Debezium's
LogMinerChangeRecordEmitter(fromV$LOGMNR_CONTENTS.ROW_ID), so no changes are needed there.3.
JdbcSourceFetchTaskContext— headers-loss bug fix ⭐While implementing the above, we discovered a latent bug affecting all JDBC-based CDC connectors (MySQL, PostgreSQL, SQL Server, Db2, Oracle):
Both
rewriteOutputBuffer()(used for PK-update path) andformatMessageTimestamp()(used for snapshot record normalization) were constructing newSourceRecords using the 8-argument constructor, silently discarding the timestamp and headers from the original record.Any header-based metadata — including the new Oracle
row_id, and any future header-based columns that may be added — would be silently dropped by these two rewrite paths.Fix: switch both call sites to the 10-argument
SourceRecordconstructor sotimestampandheadersare preserved.Verification
Compilation:
✅ BUILD SUCCESS
Runtime (Oracle Database 12c Standard Edition 12.1.0.2.0, back-ported to release-2.4 and verified end-to-end):
Both snapshot (READ) and streaming (INSERT/UPDATE/DELETE) rows carry a valid ROWID.
Data flow
User requirements
scan.incremental.snapshot.enabled=true(this PR does not touch Debezium's native snapshot path).Compatibility
row_idmetadata column see zero behavioral change (SQL rewrite still runs but only adds a single pseudo-column to the JDBC query, which is negligible).JdbcSourceFetchTaskContextfix is transparent — records that previously hadnull/absent headers still havenull/absent headers afterwards.