Bit packing serialization - #758
Open
James Ryan Carr (jryancarr) wants to merge 11 commits into
Open
Conversation
Ciphertext and key data consist of 64-bit words storing integers modulo primes much smaller than the word size. The significant bits are close to uniformly random and hence essentially incompressible; only the always-zero high bits are redundant, and a general-purpose compressor cannot remove partial bytes of them. The new mode splits the serialized stream into 4096-byte blocks and packs each block's run of 64-bit words using only as many bits per word as the largest word in the block requires, so each word begins immediately after the last significant bit of the previous one. A per-block phase (0-7 verbatim leading bytes) aligns the packed run with the data's natural word grid, since serialized metadata is not always a multiple of eight bytes. On a BFV ciphertext at poly_modulus_degree 8192 this saves 17% over Zstandard (31% over uncompressed). Loading is streamed one block at a time through a pull-based streambuf, mirroring the zlib/zstd paths, so hostile size claims cannot drive unbounded allocation. The default compression mode and the wire format of all existing modes are unchanged; the new mode requires no external dependency and is always available. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…24 bytes. A block-size sweep across real objects shows the optimum scales with sqrt(object size / number of unpackable runs): roughly 512 bytes for a small ciphertext, 2-4 KB for multi-megabyte keys. No single constant is optimal, and the previous 4096 was baked into the decoder, silently making it part of the wire format. The block size (as its base-2 logarithm, validated to 64 B - 64 KB on load) is now recorded in the encoded stream, so future encoders can tune it per object without a format change. The encoder default moves to 1024 bytes, which is within 0.3% of the per-object optimum on large objects and about 15% smaller than 4096-byte blocks on small ones. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The maximum useful phase is one byte less than a word: a phase of bytes_per_word reproduces the alignment of phase zero while wasting a verbatim word. Spelling the bound as bytes_per_word - 1 also distinguishes it from the bit-within-a-byte masks in the packing loops, whose 7s are bits-per-byte quantities and intentionally unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
For a final block shorter than a word, every phase up to the block length encodes zero packed words at identical cost, splitting the bytes between the verbatim phase prefix and tail; the encoder's tie-break settles on phase zero. The decoder must nevertheless accept all of the equivalent encodings and reject a phase beyond the block length, which would underflow the word count. A new test pins both behaviors with hand-crafted streams. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sts. The compr_mode_type::bitpack enumerator (C++ and .NET) now states that bit-packing, unlike ZLIB and Zstandard, performs no integrity checking of the data; users migrating from zlib would otherwise silently lose Adler-32's accidental-corruption detection. Removed a stray blank line left in the enum documentation by an earlier edit. Added GaloisKeys and RelinKeys bit-packed round-trip tests covering both expanded keys (data equality and a size win over the unpacked form) and seeded keys, whose nested seeded frames must load to keys identical to the same seeded object saved uncompressed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree company="Enveil" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds a new serialization mode,
compr_mode_type::bitpack(native) /ComprModeType.BitPack(dotnet), that assumes the serialized bytes contain arrays ofuint64_twhere some number of high bits are always 0. It breaks the input stream into blocks of 1024 bytes, and then uses a simple heuristic for guessing where in the block the array starts, inferring what the always-0 high bits are, and packing away the always-0 bits in the output stream (a more detailed overview including diagrams and a worked example is provided in the overview comment at the top ofnative/src/seal/util/bitpack.h). For blocks of the input stream that contain arrays ofuint64_t, this approach achieves near-optimal compression. For blocks of the input stream that don't contain such arrays (e.g. blocks containing metadata), this approach probably won't compress the data at all, but these blocks are rare enough that when serializing actual SEAL objects the approach beats bothzstdandzlibon size, save speed, and load speed.The advantages of this approach are:
zstdandzlibwhen serializing all SEAL cryptographic payloads (i.e. everything exceptPlaintexts), while also having faster save and load performance. This PR adds new benchmarks to the existing benchmark suite so reviewers can verify this themselves if desired. Here are a few lines as a representative sample. Thecompr_mode_typeis at the end of the benchmark name, e.g.SaveCiphertextBFVNoneusescompr_mode_type::none;bitpackis the last two lines in each section.zliborzstd, giving users a compression option that does not require third-party libraries.Security
I've done my best to ensure this new serialization scheme follows the recent improvements to serialization security:
SEALHeader.size, which the existing 4.3.3 guard has already verified against the input actually available (or refused to trust, on non-seekable streams).BitUnpackGetBufferis a pull-basedstreambufmodeled directly onztools::InflateGetBuffer: blocks are decoded on demand in constant memory, work is bounded by what the parser's own validated metadata requests, and remaining data in the stream after parsing is handled using the existing guards fromzlib/zstd.bitpackruns inside the existing hostile-input test loops, and adds per-field tamper tests including hostile size claims in both directions.bitpackdoes not provide integrity checking likezlibandzstddo, and this is noted in the headers. However, those integrity checks don't protect against malicious alteration of the payload, only accidental corruptions, sobitpack's security posture against malicious adversaries is the same as the other schemes in this regard.Compatibility
IsSupportedComprModecheck (the same behavior as a zstd stream in a build without zstd).#ifdefin the enum).Other notes
Development was AI-assisted (Claude Code with model Fable 5); I manually reviewed all code and documentation. The validation / benchmarks above were run on my Apple M-series laptop.
I hope the maintainers find this contribution useful, and would be happy to work with them on any requested changes!