otap: implement Schema/Dictionary/Record codec from data_model.md - #5
Open
zzylol wants to merge 1 commit into
Open
otap: implement Schema/Dictionary/Record codec from data_model.md#5zzylol wants to merge 1 commit into
zzylol wants to merge 1 commit into
Conversation
The OTAP Strategy-B wire codec (encode_batch/decode_batch) emitted one
fully self-describing row per envelope: metric name, labels, sketch_type,
agg_id, schema_version, and encoding all repeated on every RECORD row,
every window, forever -- exactly the anti-pattern docs/data_model.md's
Schema/Dictionary/Record split exists to avoid. Nothing in the codec
implemented the doc's "sent once, referenced by index thereafter"
economics; the design existed only as a field-categorization scheme, not
an implemented protocol behavior.
Add otap::dictionary with SeriesDictionary (sender) / SeriesDictionaryDecoder
(receiver):
- SeriesDictionary::encode assigns stable series_ids and emits SCHEMA /
DICTIONARY / LABELS rows only the first time an agg_id / series is seen;
RECORD always carries just series_id + window bounds + envelope/value.
- SeriesDictionaryDecoder retains SCHEMA/DICTIONARY/LABELS state across a
continuous stream and reconstructs full SketchEnvelopes by joining RECORD
rows back against it, per the doc's own statefulness caveat -- an unknown
series_id/agg_id is a hard decode error, not a silent partial result.
- sketch_size is resolved from PrecomputeConfig::sketch_params via the new
Precompute::active_config() trait method.
- hash_seed/hash_function resolve HashSpec down to the one canonical seed
position (seed_list[canonical_seed_index]) rather than carrying
asap_sketchlib's full 20-entry seed table, per its own self-describing
wire-format doc; SeriesDictionaryDecoder::schema_for() exposes the
resolved value without fabricating a lossy reconstructed HashSpec.
StubPlugin and AsapSketchesPlugin now use this codec for tick/drain
(encode) and inbound-envelope (decode), each with persistent dictionary
state across calls. encode_batch/decode_batch/records::{flatten,lift}
are kept as-is -- they solve a different problem (disguising a payload as
OTAP-Metrics-shaped to transit a generic OTAP pipeline hop) that
docs/data_model.md was never about (its own first line scopes it to the
asap_sketches-to-asap_sketches hop).
Also adds examples/sketch_pipeline_demo.rs: a runnable three-stage demo
(sketch creation processor -> receive processor that merges + queries via
estimate mode -> Prometheus text exposition) showing the dictionary
economics directly in the output -- window 0 sends all four batches,
every later window sends RECORD only.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
An audit against
docs/data_model.mdfound that the OTAP Strategy-B wire codec (encode_batch/decode_batch) never actually implemented the doc's Schema/Dictionary/Record economics: every row inlinedmetric,labels,sketch_type,agg_id,schema_version, andencoding, repeated on everyRECORDrow, every window, forever — the exact anti-pattern the doc's rationale exists to avoid. The design only ever existed as a field-categorization scheme, not an implemented protocol behavior.This PR implements it for real.
What changed
otap::dictionarymodule:SeriesDictionary(sender) assigns stableseries_ids and emitsSCHEMA/DICTIONARY/LABELSrows only the first time anagg_id/series is seen;RECORDalways carries justseries_id+ window bounds +envelope/value.SeriesDictionaryDecoder(receiver) retains that state across a continuous stream and reconstructs fullSketchEnvelopes by joiningRECORDrows back against it — an unresolvableseries_id/agg_idis a hard decode error, not a silent partial result, per the doc's own statefulness caveat.sketch_sizeis resolved fromPrecomputeConfig::sketch_paramsvia a newPrecompute::active_config()trait method.hash_seed/hash_functionresolveasap_sketchlib'sHashSpecdown to the one canonical seed position (seed_list[canonical_seed_index]) rather than carrying the full 20-entry seed table — checked againstasap_sketchlib's own self-describing wire-format doc, since oneSCHEMArow (oneagg_id, onesketch_type) only ever needs one seed position.SeriesDictionaryDecoder::schema_for()exposes the resolved value on the receiver side without fabricating a lossy reconstructedHashSpec.StubPluginandAsapSketchesPluginnow use this codec for tick/drain (encode) and inbound-envelope (decode), each with persistent dictionary state across calls.encode_batch/decode_batch/records::{flatten,lift}are unchanged — they solve a different problem (disguising a payload as OTAP-Metrics-shaped so it can transit a generic OTAP pipeline hop) thatdocs/data_model.mdwas never about (its own first line scopes it to theasap_sketches-to-asap_sketcheshop).examples/sketch_pipeline_demo.rs: a runnable three-stage demo — sketch creation processor → receive processor (merges viaobserve_envelope, queries via estimate mode) → Prometheus text exposition. Run withcargo run --example sketch_pipeline_demo --features otap. The printed output makes the dictionary economics directly visible: window 0 sends all four batches, every later window sendsRECORDonly.Testing
cargo test --features otap: 149/149 passing (14 new tests covering first-window-vs-repeat-window batch shapes, distinct series_id assignment, full encode→decode round trips, hash-seed resolution, and decode error handling for unknown series_id/agg_id).cargo fmt --checkandcargo clippy --features otap --tests --examples: clean.cargo run --example sketch_pipeline_demo --features otap: verified end to end — window 0 emitsschema=1 dictionary=1 labels=1 record=1, windows 1-3 emitschema=0 dictionary=0 labels=0 record=1, and the receiver's queried p99 gauge tracks the injected latency drift correctly.🤖 Generated with Claude Code