Skip to content

[FLINK-39765][cdc-pipeline/mysql] Implement lineage (FLIP-314) support for mysql source connectors#4415

Open
jsingh-yelp wants to merge 2 commits into
apache:masterfrom
jsingh-yelp:implement-flip-314-for-mysql-source
Open

[FLINK-39765][cdc-pipeline/mysql] Implement lineage (FLIP-314) support for mysql source connectors#4415
jsingh-yelp wants to merge 2 commits into
apache:masterfrom
jsingh-yelp:implement-flip-314-for-mysql-source

Conversation

@jsingh-yelp
Copy link
Copy Markdown

@jsingh-yelp jsingh-yelp commented May 26, 2026

Jira ticket: https://issues.apache.org/jira/browse/FLINK-39765

Summary: Adds Flink native lineage support (FLIP-314) for the MySQL CDC source.

Description:

  • Implement LineageVertexProvider on MySqlSource so that OpenLineage events report captured MySQL tables as input datasets with schema metadata.
  • This improves lineage for wildcard table patterns, including scan.binlog.newly-added-table.enabled jobs, for tables that already exist when the job is submitted. Tables created after job startup are captured at runtime but will only appear in lineage in the next job submission.
  • Add shared lineage utilities in flink-cdc-common for reuse by other source connectors

Testing:

{
      "namespace": "mysql://localhost:3306",
      "name": "testdb.users",
      "facets": {
        "schema": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.47.1/integration/flink",
          "_schemaURL": "https://openlineage.io/spec/facets/1-2-0/SchemaDatasetFacet.json#/$defs/SchemaDatasetFacet",
          "fields": [
            {
              "name": "id",
              "type": "int NOT NULL"
            },
            {
              "name": "name",
              "type": "varchar(100)"
            },
            {
              "name": "age",
              "type": "int"
            },
            {
              "name": "created_at",
              "type": "timestamp"
            }
          ]
        },
        "symlinks": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.47.1/integration/flink",
          "_schemaURL": "https://openlineage.io/spec/facets/1-0-1/SymlinksDatasetFacet.json#/$defs/SymlinksDatasetFacet",
          "identifiers": []
        },
        "config": {
          "_producer": "https://github.com/OpenLineage/OpenLineage/tree/1.47.1/integration/flink",
          "_schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/DatasetFacet",
          "type": "mysql-cdc"
        }
      }
    }

@jsingh-yelp
Copy link
Copy Markdown
Author

cc: @lvyanquan can I please have a review on this

@jsingh-yelp jsingh-yelp force-pushed the implement-flip-314-for-mysql-source branch from c38bd9e to 35a664d Compare May 26, 2026 16:25
@jsingh-yelp jsingh-yelp force-pushed the implement-flip-314-for-mysql-source branch from 35a664d to 8aac15a Compare May 27, 2026 13:17
@lvyanquan lvyanquan requested a review from Copilot May 28, 2026 10:09
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@jsingh-yelp jsingh-yelp force-pushed the implement-flip-314-for-mysql-source branch from a9b6e1e to 4010ed6 Compare May 28, 2026 17:06
@jsingh-yelp
Copy link
Copy Markdown
Author

@lvyanquan I have resolved comments from co-pilot, feel free to re-run or flag anything that needs more fixing thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants