Skip to content

serialization.md

Latest commit

 

History

History
273 lines (224 loc) · 16.8 KB

serialization.md

File metadata and controls

273 lines (224 loc) · 16.8 KB
Raw

Serialization benchmarks

The measurement instrument for PRD-serialization.md. Directions A–D in the PRD are accepted or rejected on the numbers this harness produces, not on argument.

What it measures (PRD §10)

For each subject × size × format:

  • serialized size (bytes on the wire) and compression ratio vs JSON;
  • serialize and deserialize time (median + spread over --repeat runs), the round-trip total, and throughput (MB of wire per second) so formats are comparable across sizes — this is how the impact of zipping / protobuf / Arrow gets read;
  • peak memory during deserialize (tracemalloc);
  • round-trip fidelity: exact equality of __data__ and of the format-independent canonical_hash — where a lossy profile (e.g. protobuf float32) will show up.

Subjects (PRD §10.1)

Subject Payload Attributes
mesh jittered grid, float64 verts + int faces none
mesh_attrs same + a per-vertex float per-element (hybrid-layout cost)
pointcloud N float64 points none¹
graph jittered grid, nodes (x,y,z) + edges per-node coords
points, vectors, lines, frames, planes, boxes, spheres, circles a list of N primitives/shapes —
polylines, polygons, beziers, polyhedrons a list of N compound objects (each holds several points) —
arcs, ellipses, parabolas, hyperbolas a list of N conics —
cylinders, cones, capsules, toruses a list of N solids —
transformations, translations, rotations, scales, shears, reflections, projections, quaternions a list of N transforms/quaternions —

¹ The COMPAS Pointcloud type has no per-point attribute slot, so the "with per-element attributes" case lives on mesh_attrs.

The primitive/shape subjects are collections (a Python list of N objects, a common real-world payload — e.g. a list of frames as robot targets). The compound subjects each hold several points, exercising compas_pb's flat repeated double point arrays. The corpus now benchmarks all 31 of compas_pb's native serializable types (the report shows a live coverage banner). All round-trip losslessly except frames/planes, which show a ~1e-16 discrepancy for every format including JSON — COMPAS re-normalizes their axes on construction (normalize(normalize(v)) ≠ normalize(v) at the last ULP), a geometry quirk the fidelity check surfaces, not a serialization defect.

Fixtures are seeded (fixtures.py) so runs are comparable and reused across every format; the harness measures a list or a single Data object transparently. Each format gets one untimed dump/load warm-up before measurement so plugin discovery, imports, and lazy codec initialization cannot make the first format look slower than formats measured later. Cold-start latency should be measured separately in a fresh process rather than mixed into the steady-state medians.

Formats under test

Registered in formats.py:

  • json — compact text (the default, lossless);
  • json_zip — zip-compressed JSON (size baseline);
  • compas_pb — protobuf binary via the compas_pb plugin (optional dep). The optimizations described below (double precision, flat packed coordinate arrays, columnar vertex attributes) shipped in compas_pb 1.1.4. Skipped automatically if not installed. The harness still quantifies any residual error (max_abs_error / rms_error).
  • compas_pb_zip — the above, zip-compressed (DEFLATE);
  • compas_pb_zstd — the above, zstandard-compressed (optional zstandard dep; similar ratio to zip at ~3× faster compression).
  • compas_msgpack — MessagePack over the JSON-shape tree, the approach from the Kumiki project (which uses msgspec), applied to COMPAS types via an msgspec enc_hook on __jsondump__ (no Mesh subclass needed). Optional msgspec dep. Binary but row-oriented / schemaless — a midpoint between JSON and the columnar compas_pb.
  • compas_msgpack_zstd — the above, zstandard-compressed.

An Arrow/columnar prototype registers the same way; the runner picks up any registered, available format automatically and skips those whose optional dependency is missing (available=False).

Running

Every run writes a CSV and a self-contained, theme-aware HTML report next to it (results/<name>.html) — open it in a browser for a readable, per-subject view with grouped bars (round-trip time, wire size) and a full table. The CSV is the machine record; the HTML is the human one. Each format serializes a fresh fixture, so one format's dump (JSON accesses .guid) can't mutate what a later format encodes.

It also writes encoded-format samples to results/samples/ — tiny, fully-inspectable fixtures encoded with the three main formats, so you can see the shape of each encoding: <subject>.json, <subject>.pb + <subject>.pb.json (the protobuf bytes rendered back to JSON, showing the flat/columnar wire structure), and <subject>.msgpack + <subject>.msgpack.json (the row-oriented tree). Regenerate standalone with uv run python -m compas_benchmarks.serialization.samples; skip during a run with --no-samples.

# Quick baseline (small sizes, fast) — writes results/baseline_quick.{csv,html} + samples/
uv run python -m compas_benchmarks.serialization.run

# Full PRD corpus (large; slow, memory-hungry)
uv run python -m compas_benchmarks.serialization.run --preset full --out results/baseline_full.csv

# Subset
uv run python -m compas_benchmarks.serialization.run --subjects mesh pointcloud --formats json

Run from the repository root — results are written to results/ relative to the working directory. Sync the environment first (uv sync --all-extras); this pulls in a working numpy (COMPAS geometry imports it at load time) along with the optional binary formats.

Each batch is reported on stderr as it completes (subject, size, then each format's wire size and round-trip time), so a long run is never silent; the results table goes to stdout, and --quiet suppresses the progress. The reporting happens strictly between measurements — the timed regions live inside metrics.measure — so it cannot perturb the timings.

On CI

The serialization-benchmark GitHub Actions workflow (.github/workflows/benchmark.yml) runs the benchmark and publishes the whole fresh run directory (index + HTML report + CSV + encoded samples) to GitHub Pages — the report is interactive, so it is published as a page rather than flattened into a Markdown job summary. Pull requests do not publish (the site keeps showing main); they upload the identical site as a downloadable artifact, linked from the run summary. The page is assembled by site.py, which also runs locally: uv run python -m compas_benchmarks.site results --out site.

Inputs (Actions → serialization-benchmark → Run workflow) let you pick the preset, the repeat count, the formats, and the versions of the subjects under test: compas_ref and compas_pb_ref each take either a PyPI specifier (==2.15.1) or a git ref (git:main, git:some-branch) — compas_pb commits its generated _pb2 modules, so no protoc is needed.

Pushes to main and pull requests run the quick preset automatically. Their size and losslessness numbers are deterministic and meaningful; the timing numbers are only indicative on a noisy shared runner — use workflow_dispatch (or a dedicated machine) when the exact timings matter.

Results

Generated reports under results/ are deliberately ignored by Git. Run the commands above or download a CI artifact to produce baseline_quick.{csv,html} and the encoded samples. The HTML report opens with an executive summary (headline stat tiles + a per-subject winners table) so the conclusion is visible before any detail. The findings recorded below come from the reference runs: with double precision + flat coordinate arrays + inline attribute maps, compas_pb goes from larger and slower than JSON (as shipped) to the smallest and fastest lossless option on numeric-heavy data:

subject (largest size) JSON json_zip compas_pb compas_pb_zip
mesh @10k, wire 864 KB 238 KB 320 KB 158 KB
mesh @10k, round-trip 66 ms 87 ms 50 ms 58 ms
mesh_attrs @10k, wire 1.1 MB 339 KB 398 KB 233 KB
mesh_attrs @10k, round-trip 67 ms 97 ms 52 ms 65 ms
pointcloud @100k, wire 5.5 MB 2.7 MB 2.3 MB 2.2 MB
pointcloud @100k, round-trip 343 ms 551 ms 186 ms 254 ms

On the bulk-numeric types every format is lossless and compas_pb is smallest+fastest (_zip/_zstd smallest on the wire, raw compas_pb fastest to load). Applied optimizations:

  • Precision fixed (float → double): coordinate error is 0 everywhere.
  • Flat coordinate arrays replaced a PointData message per vertex/point (each of which carried a per-point UUID + name) with packed repeated double triplets — the dominant size/speed win, for Mesh, Pointcloud, Polyline, Polygon, Bezier, Polyhedron.
  • Inline map<string, AnyData> attributes (Mesh, Graph) dropped the DictData wrapper and skip empty/default entries.
  • Int/float distinction preserved — AnyData gained explicit int64/double arms instead of routing numbers through google.protobuf.Value, so 0.0 no longer comes back as 0. This makes Mesh/Graph fully lossless (canonical hash matches).
  • CSR face storage — faces are a flat packed face_vertices index array + a face_sizes length array, instead of one FaceList message per face.
  • No Any wrapper on plain dicts/lists — AnyData gained explicit dict_value / list_value arms, so every nested dict/list no longer carries a ~44-byte google.protobuf.Any type_url. Helps Graph, fallback, and any nested container.
  • Columnar attributes (mesh vertices + graph nodes/edges) — each attribute name is stored once with a packed value array (typed double/int/bool, generic fallback, dense columns skip indices), instead of a dict per element. mesh_attrs@10k: 1.1 MB → 398 KB / 130 ms → 56 ms; graph@10k: 931 KB → 360 KB / 379 ms → 87 ms — both now smaller and faster to load than JSON.
  • guid/name only when explicitly set — auto-generated guids and default names are no longer written (an object serializes its guid only if _guid was set). This removes ~40 bytes per object, including every Point/Vector nested in Lines/Frames/shapes. boxes@10k 3.4 MB → 1.6 MB (now < JSON 2.7 MB). Contract change: auto guids no longer round-trip (they are session-local uuid4s); explicit guids still do.

What the expanded corpus shows

The corpus also covers graph and collections of primitives/shapes, plus a third format: compas_msgpack (MessagePack over the JSON-shape tree — the Kumiki approach). Findings:

  • compas_pb is the smallest on 11/12 subjects (bulk-numeric and primitive/shape lists) after the guid fairness fix (see below). It is also fastest to load on the bulk-numeric types (mesh, mesh_attrs, pointcloud, graph). For small-primitive lists (points/vectors/lines/…) it is still smallest but loads slightly slower than JSON — each element goes through registry dispatch + message unpack. See Pending #1.
  • compas_msgpack sits between JSON and compas_pb: binary and ~30–45% smaller than JSON with very fast encode, but row-oriented — so its load speed is JSON-like (no columnar/bulk win). A cheap JSON upgrade that does not fix deserialization.
  • frames/planes show lossless=no for every format including JSON (~1e-16 error): Frame/Plane re-normalize their vectors on construction, so normalize(normalize(v)) differs at the last ULP. A COMPAS geometry quirk the fidelity check surfaces, not a serialization defect.

Fairness fix. Each format serializes a fresh fixture. Serializing to JSON accesses .guid (forcing it onto the object); when all formats shared one object, compas_pb then re-serialized those forced guids, overstating its size on primitive/shape lists by ~30%. With independent objects, compas_pb's smallest-on-11/12 result stands.

The HTML report opens with a three-pill toggle (Uncompressed / Compressed / All) that hides non-matching formats, re-normalizes the bars, and re-bases the summary — so you can read json vs compas_pb vs compas_msgpack raw, or the compressed variants, in isolation. The summary tiles report median ratios with their range across subjects (not a single best), plus how many subjects compas_pb wins on size/speed.

To inspect the actual encodings, see results/samples/ (per subject: .json, .pb + .pb.json, .msgpack + .msgpack.json) — *.pb.json shows the columnar/flat wire shape, *.msgpack.json the row-oriented tree.

The optimizations were developed on the benchmark/double-precision branch of the external compas_pb repo and released in compas_pb 1.1.4, which is what the formats extra installs. To benchmark unreleased work, install that repo's branch over it (uv pip install "compas_pb @ git+https://github.com/gramaziokohler/compas_pb.git@<ref>", or pass compas_pb_ref: git:<ref> on CI).

N1 — binary-vs-JSON scaling (--preset full)

results/baseline_full.{csv,html} sweeps the bulk-numeric subjects across 1e3 → 1e6 elements. compas_pb vs JSON at 1e6 elements:

subject wire load time peak memory
mesh 2.8× smaller (100 → 35 MB) 1.3× faster 1.9× lower
mesh_attrs 3.0× smaller (128 → 43 MB) 1.3× faster 1.8× lower
pointcloud 2.4× smaller (55 → 23 MB) 1.2× faster 1.2× lower
graph 2.1× smaller (85 → 40 MB) 3.2× faster 1.7× lower

Wire and peak-memory ratios are ~flat with size; the load-time advantage grows with size (compas_pb has fixed per-call overhead — it can be slower than JSON at 1e3, but is amortized by 1e5 and clearly ahead by 1e6). Graph shows the biggest load win (3.2×) because the columnar node layout skips rebuilding N per-node dicts.

N1 is met, with a proposed acceptance bar (holds for all four subjects at ≥ 1e5 elements): wire ≥ 2× smaller, load ≥ 1.2× faster, peak memory ≥ 1.2× lower than compact JSON.

The preset caps at 1e6 because beyond that the tracemalloc peak-memory probe (it traces every allocation on deserialize) dominates runtime — measuring memory at PRD scale needs an RSS-sampling probe instead. A one-off spot check at 5e7 points (size + time only, no tracemalloc) confirms the ratios are flat with size and it does not OOM (RSS ≈ 9 GB of 36):

pointcloud @ 5e7 json compas_pb ratio
wire 2.90 GB 1.20 GB 2.42× smaller
dump 208 s 24 s 8.6× faster
load 188 s 155 s 1.21× faster

(The wire/load ratios match the 1e6 numbers almost exactly; the dump gap widens because JSON spends ~200 s building a 2.9 GB string while compas_pb writes packed doubles.)

Pending optimizations

  1. Batched primitive lists — collections of small primitives load slightly slower than JSON because each element goes through registry lookup + message unpack. A batched/columnar encoding for homogeneous primitive lists (e.g. N points/frames as flat arrays) would close it, the same way the mesh/graph rewrites did.
  2. Peak memory at PRD scale (5e6/5e7) — size and load-time ratios are already confirmed flat at 5e7 (spot check above), so this is a productization stress test, not an N1 gap. The only thing unmeasured at those sizes is peak memory, which needs an RSS-sampling probe (the tracemalloc probe is too slow past ~1e6).

Status

Phase 1: baseline harness + JSON numbers, plus canonical_hash() decoupling object identity from the JSON text (COMPAS core's Data.sha256(), which hashes the guid-bearing JSON, is unusable for cross-format comparison and is left untouched). Phase 2: compas_pb measured and optimized — double precision, flat coordinate arrays, columnar attributes (mesh vertices/faces/edges, graph nodes/edges), explicit int/float, CSR faces, guid/name omission, msgpack comparison. Now a lossless binary mode, smaller and faster than JSON on numeric data (N1 met — see the full-preset table). All 31 serializable types benchmarked. The wire-version check is a hard gate (PRD N4) — the version bump that activates rejection of older data happens at release.