Skip to content

fix: use epoch-based monotonic message ID for BanyanDB writes - #13987

Open
waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/13986-epoch-based-message-id
Open

fix: use epoch-based monotonic message ID for BanyanDB writes#13987
waterWang wants to merge 1 commit into
apache:masterfrom
waterWang:fix/13986-epoch-based-message-id

Conversation

@waterWang

Copy link
Copy Markdown

Fixes #13986

Problem

MeasureWrite and StreamWrite use System.nanoTime() as the message_id for BanyanDB writes. System.nanoTime() has a host-specific origin and is only meaningful for measuring elapsed time within one JVM — values are not comparable across hosts or host reboots.

In a multi-node OAP deployment, a host with a shorter uptime generates a smaller value. When DataPointValue.version is zero, BanyanDB assigns message_id to the persisted data-point version, so a genuinely newer update can be treated as an older version and rejected. Mutable measure fields such as last_ping stop advancing even though Kafka consumption and writes appear successful.

Observed ordering in production:

previous persisted version: 129399194960299109
newly generated version:     34229098907069822

Fix

Added AbstractWrite.nextMessageId(), an epoch-based, JVM-monotonic message ID generator:

  • Base is the Unix epoch (System.currentTimeMillis() * 1_000_000), so current values are comparable across hosts and are larger than legacy uptime-based System.nanoTime() values in normal deployments.
  • An AtomicLong with updateAndGet(Math.max(prev + 1, now)) guarantees strictly increasing values under concurrent writes and during temporary wall-clock rollback.
  • MeasureWrite.build() / buildValues() now call nextMessageId(), explicitly set the same value as DataPointValue.version, and use it as message_id — matching the fix direction suggested in the issue.
  • StreamWrite.build() / buildValues() use nextMessageId() for message_id as well.

Tests

New MeasureWriteMessageIdTest covers:

  • message_id == DataPointValue.version in both build() and buildValues()
  • strictly increasing message IDs across 10,000 sequential calls
  • uniqueness across 8 threads × 2,000 concurrent calls

Replace System.nanoTime() with a message ID generator that uses the Unix
epoch as its base. The generator is backed by an AtomicLong that guarantees
strictly increasing values even under concurrent writes or a wall-clock
rollback, making message IDs comparable across OAP hosts and process
restarts.

For MeasureWrite, the generated value is also set as DataPointValue.version
so that BanyanDB does not need to derive it from a host-local source.

Fixes apache#13986
@wu-sheng

Copy link
Copy Markdown
Member

This approach could ease the issue, but thr time clock adjusting could still be an issue.
Also, changing the version could make the existing deployment unable to update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] BanyanDB measure updates may become stale because message ID uses System.nanoTime()

2 participants