[feature](tso) Add global monotonically increasing Timestamp Oracle(TSO)#61199
[feature](tso) Add global monotonically increasing Timestamp Oracle(TSO)#61199morningman merged 35 commits intoapache:masterfrom
Conversation
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
996adea to
66a8f8e
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
66a8f8e to
fa48c2a
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
fa48c2a to
dc6a3a6
Compare
|
run buildall |
…TSO) Signed-off-by: Jingzhe Jia <AntiTopQuark1350@outlook.com>
dc6a3a6 to
9c9dec0
Compare
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
run buildall |
FE UT Coverage ReportIncrement line coverage |
Signed-off-by: Jingzhe Jia <AntiTopQuark1350@outlook.com>
|
run buildall |
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
|
PR approved by at least one committer and no changes requested. |
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Test Coverage AnalysisThe core TSO logic (TSOTimestamp + TSOService) has solid unit test coverage. However, the integration seams — where TSO meets the transaction commit path, DDL operations, BE publish pipeline, and edit log replay — are under-tested. What's Well Covered
GapsHigh Priority:
Medium Priority:
Low Priority:
|
…riptions to ensure clarity of the descriptive information.
9a463a7 to
be4bbfe
Compare
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 29077 ms |
TPC-DS: Total hot run time: 179179 ms |
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
…SO) (apache#61199) ### What problem does this PR solve? Issue Number: close apache#61198 Related apache#57921 Problem Summary: ### Release note - Implement a **global monotonically increasing Timestamp Oracle (TSO)** service that generates unique, monotonically increasing timestamps for transactions. The service calibrates its initial timestamp at startup and periodically updates it to maintain a **time window**. A **TSO timestamp** encodes the physical time and a logical counter; it is assembled and extracted by the new **TSOTimestamp** class. - Introduce **TSOService**, a master-only daemon that manages global timestamps. The service exposes two main methods: - `getTSO()` – returns a new TSO timestamp for transaction commits. - `getCurrentTSO()` – returns the current TSO without bumping the logical counter. - Add multiple configuration properties to control the behavior of the TSO feature: - `experimental_enable_feature_tso` – enables/disables the TSO feature. - `tso_service_update_interval_ms` – interval in milliseconds for the TSO service to update its window. - `max_update_tso_retry_count` and `max_get_tso_retry_count` – retry limits for updating and obtaining TSOs. - `tso_service_window_duration_ms` – length of the time window allocated by the TSO service. - `tso_time_offset_debug_mode` – debug offset for the physical time. - `enable_tso_persist_journal` and `enable_tso_checkpoint_module` – persistence switches for edit log and checkpoint. - **Table property**: Introduce `enable_tso` which can be configured in `CREATE TABLE` or modified via `ALTER TABLE`. Only tables with `enable_tso = true` generate commit TSO for transactions; when disabled, commit_tso remains `-1`. - **Transaction and commit integration**: - During commit, `TransactionState` now fetches a commit TSO from `TSOService` when TSO is enabled and stores it in the transaction state and `TableCommitInfo`. - The commit TSO is recorded per partition (via `TPartitionVersionInfo.commit_tso`), and is persisted with each rowset (see next item). - **Rowset and meta changes**: - `Rowset::make_visible` now accepts a `commit_tso` parameter and writes it to `RowsetMeta`. - `RowsetMetaPB` adds a new field `commit_tso` to persist commit timestamps. - `information_schema.rowsets` introduces a new column **`COMMIT_TSO`** allowing users to query the commit timestamp for each rowset. - Pending publish tasks, asynchronous publish tasks and other internal structures have been extended to carry commit TSO. - **External interface**: A new REST endpoint `/api/tso` is added for retrieving current TSO information. It returns a JSON payload containing: - `window_end_physical_time` – end of the current TSO time window. - `current_tso` – the current composed 64‑bit TSO. - `current_tso_physical_time` and `current_tso_logical_counter` – the decomposed physical and logical parts of the current TSO. This API does **not** increment the logical counter. - **Metrics & observability**: New metrics counters (e.g., `tso_clock_drift_detected`, `tso_clock_backward_detected`, `tso_clock_calculated`, `tso_clock_updated`) expose state and health of the TSO service. - **Regression & unit tests**: New unit tests verify `TSOTimestamp` bit manipulation, `TSOService` behavior, commit TSO propagation, and the `/api/tso` endpoint. Regression tests verify that rowset commit timestamps are populated when TSO is enabled and that the API returns increasing TSOs. ### Impact and Compatibility - **Experimental**: the TSO feature is currently guarded by `experimental_enable_feature_tso`. It is disabled by default and can be enabled in front-end configuration. When enabled, old FE versions without this feature cannot replay edit log entries containing TSO operations; therefore upgrade all FEs before enabling. - **Table compatibility**: tables created before enabling TSO remain unaffected unless explicitly modified to set `enable_tso` to `true`. Tables with TSO enabled will produce commit TSO for each rowset and may require downstream consumers to handle the new `commit_tso` field. - **Client API**: clients can call `/api/tso` to inspect current TSO values. No existing API is modified. Signed-off-by: Jingzhe Jia <AntiTopQuark1350@outlook.com>
What problem does this PR solve?
Issue Number: close #61198
Related #57921
Problem Summary:
Release note
The service calibrates its initial timestamp at startup and periodically updates it to maintain a time window.
A TSO timestamp encodes the physical time and a logical counter; it is assembled and extracted by the new TSOTimestamp class.
The service exposes two main methods:
getTSO()– returns a new TSO timestamp for transaction commits.getCurrentTSO()– returns the current TSO without bumping the logical counter.experimental_enable_feature_tso– enables/disables the TSO feature.tso_service_update_interval_ms– interval in milliseconds for the TSO service to update its window.max_update_tso_retry_countandmax_get_tso_retry_count– retry limits for updating and obtaining TSOs.tso_service_window_duration_ms– length of the time window allocated by the TSO service.tso_time_offset_debug_mode– debug offset for the physical time.enable_tso_persist_journalandenable_tso_checkpoint_module– persistence switches for edit log and checkpoint.enable_tsowhich can be configured inCREATE TABLEor modified viaALTER TABLE. Only tables withenable_tso = truegenerate commit TSO for transactions; when disabled, commit_tso remains-1.TransactionStatenow fetches a commit TSO fromTSOServicewhen TSO is enabled and stores it in the transaction state andTableCommitInfo.TPartitionVersionInfo.commit_tso), and is persisted with each rowset (see next item).Rowset::make_visiblenow accepts acommit_tsoparameter and writes it toRowsetMeta.RowsetMetaPBadds a new fieldcommit_tsoto persist commit timestamps.information_schema.rowsetsintroduces a new columnCOMMIT_TSOallowing users to query the commit timestamp for each rowset.A new REST endpoint
/api/tsois added for retrieving current TSO information. It returns a JSON payload containing:window_end_physical_time– end of the current TSO time window.current_tso– the current composed 64‑bit TSO.current_tso_physical_timeandcurrent_tso_logical_counter– the decomposed physical and logical parts of the current TSO. This API does not increment the logical counter.New metrics counters (e.g.,
tso_clock_drift_detected,tso_clock_backward_detected,tso_clock_calculated,tso_clock_updated) expose state and health of the TSO service.New unit tests verify
TSOTimestampbit manipulation,TSOServicebehavior, commit TSO propagation, and the/api/tsoendpoint. Regression tests verify that rowset commit timestamps are populated when TSO is enabled and that the API returns increasing TSOs.Impact and Compatibility
experimental_enable_feature_tso. It is disabled by default and can be enabled in front-end configuration. When enabled, old FE versions without this feature cannot replay edit log entries containing TSO operations; therefore upgrade all FEs before enabling.enable_tsototrue. Tables with TSO enabled will produce commit TSO for each rowset and may require downstream consumers to handle the newcommit_tsofield./api/tsoto inspect current TSO values. No existing API is modified.Check List (For Author)
Test
Behavior changed:
Does this need documentation?
docs: add TSO-related functions description and documentation. doris-website#3454
Check List (For Reviewer who merge this PR)