Skip to content

otap: implement Schema/Dictionary/Record codec from data_model.md - #5

Open
zzylol wants to merge 1 commit into
mainfrom
feat/schema-dictionary-record-codec
Open

otap: implement Schema/Dictionary/Record codec from data_model.md#5
zzylol wants to merge 1 commit into
mainfrom
feat/schema-dictionary-record-codec

Conversation

@zzylol

@zzylol zzylol commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Summary

An audit against docs/data_model.md found that the OTAP Strategy-B wire codec (encode_batch/decode_batch) never actually implemented the doc's Schema/Dictionary/Record economics: every row inlined metric, labels, sketch_type, agg_id, schema_version, and encoding, repeated on every RECORD row, every window, forever — the exact anti-pattern the doc's rationale exists to avoid. The design only ever existed as a field-categorization scheme, not an implemented protocol behavior.

This PR implements it for real.

What changed

  • New otap::dictionary module: SeriesDictionary (sender) assigns stable series_ids and emits SCHEMA/DICTIONARY/LABELS rows only the first time an agg_id/series is seen; RECORD always carries just series_id + window bounds + envelope/value. SeriesDictionaryDecoder (receiver) retains that state across a continuous stream and reconstructs full SketchEnvelopes by joining RECORD rows back against it — an unresolvable series_id/agg_id is a hard decode error, not a silent partial result, per the doc's own statefulness caveat.
  • sketch_size is resolved from PrecomputeConfig::sketch_params via a new Precompute::active_config() trait method.
  • hash_seed/hash_function resolve asap_sketchlib's HashSpec down to the one canonical seed position (seed_list[canonical_seed_index]) rather than carrying the full 20-entry seed table — checked against asap_sketchlib's own self-describing wire-format doc, since one SCHEMA row (one agg_id, one sketch_type) only ever needs one seed position. SeriesDictionaryDecoder::schema_for() exposes the resolved value on the receiver side without fabricating a lossy reconstructed HashSpec.
  • StubPlugin and AsapSketchesPlugin now use this codec for tick/drain (encode) and inbound-envelope (decode), each with persistent dictionary state across calls.
  • encode_batch/decode_batch/records::{flatten,lift} are unchanged — they solve a different problem (disguising a payload as OTAP-Metrics-shaped so it can transit a generic OTAP pipeline hop) that docs/data_model.md was never about (its own first line scopes it to the asap_sketches-to-asap_sketches hop).
  • New examples/sketch_pipeline_demo.rs: a runnable three-stage demo — sketch creation processor → receive processor (merges via observe_envelope, queries via estimate mode) → Prometheus text exposition. Run with cargo run --example sketch_pipeline_demo --features otap. The printed output makes the dictionary economics directly visible: window 0 sends all four batches, every later window sends RECORD only.

Testing

  • cargo test --features otap: 149/149 passing (14 new tests covering first-window-vs-repeat-window batch shapes, distinct series_id assignment, full encode→decode round trips, hash-seed resolution, and decode error handling for unknown series_id/agg_id).
  • cargo fmt --check and cargo clippy --features otap --tests --examples: clean.
  • cargo run --example sketch_pipeline_demo --features otap: verified end to end — window 0 emits schema=1 dictionary=1 labels=1 record=1, windows 1-3 emit schema=0 dictionary=0 labels=0 record=1, and the receiver's queried p99 gauge tracks the injected latency drift correctly.

🤖 Generated with Claude Code

The OTAP Strategy-B wire codec (encode_batch/decode_batch) emitted one
fully self-describing row per envelope: metric name, labels, sketch_type,
agg_id, schema_version, and encoding all repeated on every RECORD row,
every window, forever -- exactly the anti-pattern docs/data_model.md's
Schema/Dictionary/Record split exists to avoid. Nothing in the codec
implemented the doc's "sent once, referenced by index thereafter"
economics; the design existed only as a field-categorization scheme, not
an implemented protocol behavior.

Add otap::dictionary with SeriesDictionary (sender) / SeriesDictionaryDecoder
(receiver):

- SeriesDictionary::encode assigns stable series_ids and emits SCHEMA /
  DICTIONARY / LABELS rows only the first time an agg_id / series is seen;
  RECORD always carries just series_id + window bounds + envelope/value.
- SeriesDictionaryDecoder retains SCHEMA/DICTIONARY/LABELS state across a
  continuous stream and reconstructs full SketchEnvelopes by joining RECORD
  rows back against it, per the doc's own statefulness caveat -- an unknown
  series_id/agg_id is a hard decode error, not a silent partial result.
- sketch_size is resolved from PrecomputeConfig::sketch_params via the new
  Precompute::active_config() trait method.
- hash_seed/hash_function resolve HashSpec down to the one canonical seed
  position (seed_list[canonical_seed_index]) rather than carrying
  asap_sketchlib's full 20-entry seed table, per its own self-describing
  wire-format doc; SeriesDictionaryDecoder::schema_for() exposes the
  resolved value without fabricating a lossy reconstructed HashSpec.

StubPlugin and AsapSketchesPlugin now use this codec for tick/drain
(encode) and inbound-envelope (decode), each with persistent dictionary
state across calls. encode_batch/decode_batch/records::{flatten,lift}
are kept as-is -- they solve a different problem (disguising a payload as
OTAP-Metrics-shaped to transit a generic OTAP pipeline hop) that
docs/data_model.md was never about (its own first line scopes it to the
asap_sketches-to-asap_sketches hop).

Also adds examples/sketch_pipeline_demo.rs: a runnable three-stage demo
(sketch creation processor -> receive processor that merges + queries via
estimate mode -> Prometheus text exposition) showing the dictionary
economics directly in the output -- window 0 sends all four batches,
every later window sends RECORD only.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant