fix: use epoch-based monotonic message ID for BanyanDB writes - #13987
Open
waterWang wants to merge 1 commit into
Open
fix: use epoch-based monotonic message ID for BanyanDB writes#13987waterWang wants to merge 1 commit into
waterWang wants to merge 1 commit into
Conversation
Replace System.nanoTime() with a message ID generator that uses the Unix epoch as its base. The generator is backed by an AtomicLong that guarantees strictly increasing values even under concurrent writes or a wall-clock rollback, making message IDs comparable across OAP hosts and process restarts. For MeasureWrite, the generated value is also set as DataPointValue.version so that BanyanDB does not need to derive it from a host-local source. Fixes apache#13986
Member
|
This approach could ease the issue, but thr time clock adjusting could still be an issue. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #13986
Problem
MeasureWriteandStreamWriteuseSystem.nanoTime()as themessage_idfor BanyanDB writes.System.nanoTime()has a host-specific origin and is only meaningful for measuring elapsed time within one JVM — values are not comparable across hosts or host reboots.In a multi-node OAP deployment, a host with a shorter uptime generates a smaller value. When
DataPointValue.versionis zero, BanyanDB assignsmessage_idto the persisted data-point version, so a genuinely newer update can be treated as an older version and rejected. Mutable measure fields such aslast_pingstop advancing even though Kafka consumption and writes appear successful.Observed ordering in production:
Fix
Added
AbstractWrite.nextMessageId(), an epoch-based, JVM-monotonic message ID generator:System.currentTimeMillis() * 1_000_000), so current values are comparable across hosts and are larger than legacy uptime-basedSystem.nanoTime()values in normal deployments.AtomicLongwithupdateAndGet(Math.max(prev + 1, now))guarantees strictly increasing values under concurrent writes and during temporary wall-clock rollback.MeasureWrite.build()/buildValues()now callnextMessageId(), explicitly set the same value asDataPointValue.version, and use it asmessage_id— matching the fix direction suggested in the issue.StreamWrite.build()/buildValues()usenextMessageId()formessage_idas well.Tests
New
MeasureWriteMessageIdTestcovers:message_id == DataPointValue.versionin bothbuild()andbuildValues()