[feature](data governance) Add data lineage documentation by seawinde · Pull Request #3997 · apache/doris-website

seawinde · 2026-07-17T09:51:50Z

What problem does this PR solve?

Issue Number: N/A

Related PR: N/A

Problem Summary:
The website does not provide a complete technical reference for Apache Doris
data lineage or an end-to-end guide that developers can follow to implement,
package, deploy, and verify a lineage plugin.

Change Summary:

Add English and Chinese data lineage technical guides under Data Governance
for dev and 4.x, covering the collection flow, event semantics, supported DML,
FE configuration, queue behavior, runnable SQL examples, and troubleshooting.
Add English dev and 4.x Key Features pages with a self-contained, runnable
audit-log lineage quick start and links to the detailed technical guide.
Add an English and Chinese source-tree tutorial for a logging lineage plugin,
including Maven layout, Java implementations, service registration,
packaging, deployment, verification, rollback, and external governance
integration options such as OpenMetadata and OpenLineage.
Add localized architecture and event-model diagrams.
Add dev, 4.x, Community, and Chinese Community navigation entries.

Design rationale:
The Data Governance guide explains the user-facing behavior and lineage model,
the Key Features page provides a concise entry point, and the Community guide
provides the implementation workflow. Dev and 4.x pages remain aligned except
for version-specific availability wording, and executable code blocks are
aligned between their English and Chinese counterparts.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason

Manual validation:

Executed all unique documented SQL examples against a local Doris instance
and verified the query and audit-log results.
Compiled the Java snippets against the public Apache Doris 4.x source tree,
built the plugin JAR, and verified that the directory plugin loader loaded
and executed it.
Validated all Shell snippets with bash -n and parsed all XML blocks.
Verified English and Chinese routes, images, and page layout locally.
Ran changed-scope link, SEO, i18n, feature, and whitespace checks.
Completed the full English production build successfully. It reports only
pre-existing broken-link and Markdown warnings outside the data lineage pages.
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes. This PR adds the documentation.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: N/A Related PR: N/A Problem Summary: The website did not provide a complete reference for Doris data lineage or an end-to-end guide for developing a lineage plugin. Change Summary: - Add English and Chinese data lineage technical guides for dev and 4.x, including event semantics, configuration, examples, operations, and troubleshooting. - Add an English and Chinese source-tree tutorial for a logging lineage plugin and external governance integration. - Add localized architecture diagrams and navigation entries. ### Release note None ### Check List (For Author) - Test: Manual test - Validated SQL examples, Java compilation and plugin loading. - Validated shell and XML snippets, page routes, and images. - Ran changed-scope links, SEO, i18n, and feature checks. - Behavior changed: No. - Does this need documentation: Yes. This is the documentation PR.

seawinde · 2026-07-17T09:52:04Z

run buildall

morrySnow · 2026-07-17T10:43:24Z

+| `INSERT OVERWRITE ... SELECT` | Overwrite 成功后生成一个事件。 |
+| `CREATE TABLE AS SELECT` | 内部 Insert 成功后生成一个事件。 |
+
+以下操作不会生成事件：`SELECT`、`UPDATE`、`DELETE`、导入任务、仅包含 `VALUES` 的 Insert，以及目标表为 `__internal_schema` 的写入。


UPDATE、DELETE 也不会吗？特意排除了？

morrySnow · 2026-07-17T10:48:51Z

@@ -0,0 +1,336 @@
+---
+{
+    "title": "数据血缘原理与使用",


这个为什么在sql操作手册下面呢？可能放在 docs/data-governance 下比较合适？

morrySnow

Thanks for this comprehensive data lineage documentation! I've cross-referenced it against the actual code implementation at fe/fe-core/src/main/java/org/apache/doris/nereids/lineage/. The documentation is largely accurate, but I found a few issues that should be addressed:

Summary of Findings

JAR name inconsistency between user guide and developer guide (see inline comment).
INSERT OVERWRITE syntax is missing the TABLE keyword throughout (see inline comments).
PluginContext property access description is imprecise (see inline comment).
"FE trims surrounding spaces" claim for activate_lineage_plugin parsing should be verified or removed.
"Every active plugin" phrasing is slightly misleading — the worker dispatches to every loaded plugin, and eventFilter() decides per-plugin processing.

Verified as Correct ✅

LineagePlugin / LineagePluginFactory interface signatures match the code exactly
LineageInfo data structure (direct, indirect, dataset lineage maps) matches code
LineageContext fields (sourceCommand, queryId, user, clientIp, database, catalog, state, timestampMs, durationMs, externalCatalogProperties) all match
eventFilter() is indeed called twice — once in query thread (via hasActivePlugins() before extraction) and once in worker thread (before exec())
Queue size default of 50000 (Config.lineage_event_queue_size) matches code
Best-effort delivery semantics (no rollback, no retry, no persistence) match code
UPDATE, DELETE, SELECT, load jobs do not generate lineage events — confirmed by checking InsertIntoTableCommand, InsertOverwriteTableCommand, CreateTableCommand only
VALUES-only inserts and __internal_schema target tables are skipped — confirmed in LineageUtils.shouldSkipLineage()
Worker uses eventQueue.poll(5, SECONDS) with 5-second timeout — matches description
Empty activate_lineage_plugin instantiates ALL discovered factories — confirmed in discoverPlugins() logic
Plugin discovery only at FE startup, no dynamic reload — confirmed
Single daemon worker thread — confirmed (new Thread(new Worker(), "LineageEventProcessor"))
Log keywords in troubleshooting table all match the actual code
CTE resolution and UNION branch expansion are performed — confirmed in LineageInfoExtractor
Plugin extends AutoCloseable with close() default method — matches code
LineagePluginFactory.create(context) defaults to delegating to create() — matches code

morrySnow · 2026-07-17T11:01:26Z

+
+`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`.
+
+| Part | API | Meaning |


The phrase "PluginContext contains plugin.name and plugin.path" is imprecise. Looking at the PluginContext class (fe-extension-spi/.../PluginContext.java), it only exposes getProperties() which returns a Map<String, String>. The properties "plugin.name" and "plugin.path" are stored as map entries (set in discoverPlugins() via props.put("plugin.path", ...) and props.put("plugin.name", ...)). A developer reading this would look for context.getPluginName() or context.pluginName which don't exist. Please clarify: accessed via context.getProperties().get("plugin.name") and context.getProperties().get("plugin.path"). Same issue exists in the Chinese version at the corresponding position.

morrySnow · 2026-07-17T11:01:26Z

+
+`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`.
+
+At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through `plugin.path`, and initialize its resources there.


Same INSERT OVERWRITE issue as the user guide — the TABLE keyword is missing. Standard Doris syntax is INSERT OVERWRITE TABLE ... SELECT .... The command class name InsertOverwriteTableCommand also confirms this. Please use the precise syntax throughout (applies to all files: EN user guide, ZH user guide, ZH dev guide, and all versioned copies).

morrySnow · 2026-07-17T11:01:26Z

+  <text x="96" y="226" class="label">Supported DML</text>
+  <text x="96" y="262" class="body">INSERT INTO SELECT</text>
+  <text x="96" y="290" class="body">INSERT OVERWRITE SELECT</text>
+  <text x="96" y="318" class="body">CREATE TABLE AS SELECT</text>


The SVG text uses INSERT OVERWRITE SELECT without the TABLE keyword. The standard Doris SQL syntax is INSERT OVERWRITE TABLE ... SELECT .... Please add TABLE to match the actual syntax. Affects both lineage-architecture.svg and its zh-CN variant.

Move the user guide under the top-level data governance section. Align INSERT OVERWRITE syntax and example JAR names across guides and diagrams. Clarify PluginContext property access, plugin filtering, configuration parsing, and UPDATE/DELETE event behavior. Synchronize the related dev and 4.x Key Features descriptions.

morrySnow · 2026-07-21T02:29:09Z

+| Target | `getTargetTable()`, `getTargetColumns()` | The target table and output columns of the write. |
+| Source tables | `getTableLineageSet()` | Tables referenced by the analyzed logical plan. |
+| Direct lineage | `getDirectLineageMap()` | A map from each output Slot to source expressions, classified as `IDENTITY`, `TRANSFORMATION`, or `AGGREGATION`. |
+| Dataset indirect lineage | `getDatasetIndirectLineageMap()` | Expressions that affect the complete result set: `JOIN`, `FILTER`, `GROUP_BY`, and `SORT`. |
+| Output indirect lineage | `getOutputIndirectLineageMap()` | Dependencies that affect a specific output column: `WINDOW` and `CONDITIONAL`. |
+| Query context | `getContext()` | Query ID, SQL text, user, client IP, session database and catalog, execution state, timestamp, duration, and sanitized external catalog properties. |


要先有一个summary，类似你这个。然后要有一个detail，每个接口逐一介绍，写清楚接口的输入输出类型。可以看看 javadoc 接口是怎么写的，比如：https://docs.oracle.com/javase/8/docs/api/java/util/HashMap.html#containsKey-java.lang.Object-

morrySnow · 2026-07-21T02:31:40Z

+
+At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` exposes an immutable property map through `getProperties()`; FE stores the plugin name and directory in the `plugin.name` and `plugin.path` entries. Access them with `context.getProperties().get("plugin.name")` and `context.getProperties().get("plugin.path")`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through the `plugin.path` entry, and initialize its resources there.
+
+## Lineage event model


这个章节叫做 event model，下面的接口介绍，看起来和他无关。应当单独放置？

morrySnow · 2026-07-21T02:31:53Z

+| Output indirect lineage | `getOutputIndirectLineageMap()` | Dependencies that affect a specific output column: `WINDOW` and `CONDITIONAL`. |
+| Query context | `getContext()` | Query ID, SQL text, user, client IP, session database and catalog, execution state, timestamp, duration, and sanitized external catalog properties. |
+
+Direct lineage describes how an output value is produced. A plain source-column reference is `IDENTITY`, for example `target.customer_id <- source.customer_id`. A calculation, function, window expression, or conditional expression without an aggregate function is `TRANSFORMATION`, for example `UPPER(source.region)`. An expression that contains an aggregate function is `AGGREGATION`, for example `SUM(source.amount)`. Indirect lineage records expressions that affect the output without directly becoming an output value, including Join keys, `WHERE` and `HAVING` predicates, grouping keys, and sorting keys. `WINDOW` records window partitioning and ordering inputs. `CONDITIONAL` records the effect of `CASE`, `IF`, or `COALESCE` on a specific output column.


表格描述

seawinde temporarily deployed to Production July 17, 2026 09:51 — with GitHub Actions Inactive

morrySnow reviewed Jul 17, 2026

View reviewed changes

seawinde temporarily deployed to Production July 19, 2026 06:13 — with GitHub Actions Inactive

[doc](fe) Make data lineage quick start runnable

4bdfd98

seawinde had a problem deploying to Production July 20, 2026 12:53 — with GitHub Actions Failure

morrySnow changed the title ~~[doc](fe) Add data lineage documentation~~ [doc](data governance) Add data lineage documentation Jul 21, 2026

morrySnow changed the title ~~[doc](data governance) Add data lineage documentation~~ [feature](data governance) Add data lineage documentation Jul 21, 2026

morrySnow reviewed Jul 21, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feature](data governance) Add data lineage documentation#3997

[feature](data governance) Add data lineage documentation#3997
seawinde wants to merge 3 commits into
apache:masterfrom
seawinde:docs/data-lineage

seawinde commented Jul 17, 2026 •

edited

Loading

Uh oh!

seawinde commented Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 17, 2026

Uh oh!

morrySnow Jul 21, 2026

Uh oh!

morrySnow Jul 21, 2026

Uh oh!

morrySnow Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		`LineageInfo` contains a target table, target output columns, source tables, direct lineage, indirect lineage, and a `LineageContext`.

		\| Part \| API \| Meaning \|


		`eventFilter()` is called once before lineage extraction on the DML query path and again before worker dispatch. Return `false` when the plugin should not receive events. The method is called concurrently from query threads and the worker thread, so it must be thread-safe. `exec()` is called by one worker thread, but can run concurrently with `eventFilter()`.

		At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` contains `plugin.name` and `plugin.path`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through `plugin.path`, and initialize its resources there.


		At FE startup, ServiceLoader first discovers each `LineagePluginFactory`. FE then calls the Factory's `create(context)` method and calls `initialize(context)` on the returned plugin. The default `LineagePluginFactory.create(context)` implementation delegates to the no-argument `create()`, so the minimal example in this guide implements only `create()`. `PluginContext` exposes an immutable property map through `getProperties()`; FE stores the plugin name and directory in the `plugin.name` and `plugin.path` entries. Access them with `context.getProperties().get("plugin.name")` and `context.getProperties().get("plugin.path")`. A production plugin that reads a configuration file from its plugin directory can override `initialize(context)`, locate the file through the `plugin.path` entry, and initialize its resources there.

		## Lineage event model

Uh oh!

Conversation

seawinde commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

seawinde commented Jul 17, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

morrySnow left a comment

Choose a reason for hiding this comment

Summary of Findings

Verified as Correct ✅

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

seawinde commented Jul 17, 2026 •

edited

Loading