Accepted - 2026-06-09
serialize is a single-header C++ bit-packing library by Glenn Fiedler (Mas Bandwidth LLC,
BSD-3-Clause). Reference implementation: serialize.h, ~2100 lines including tests.
Core features:
- Bit-granularity packing - encode an integer value in exactly the minimum number of bits required
to represent its range
[min, max]. A boolean uses 1 bit, a value in[0, 31]uses 5 bits, etc. - Unified read/write interface - a single templated
Serializemethod handles both directions; the stream type (WriteStream,ReadStream,MeasureStream) determines behavior at compile time. - Compressed floats - quantize a float in
[min, max]to an integer with a given resolution, pack into the minimum bits required. - Relative integer encoding - delta-encode an integer relative to a previous value using a tiered bit scheme (1, 3, 5, 9, 13, 17, 32 bits) for small deltas.
- Safety checks on read -
ReadStreamvalidates range and buffer bounds, returningfalseon any violation without throwing exceptions.
serialize is NOT a message framing or transport layer. It only provides the bit-packing primitives. The caller is responsible for buffer management and framing.
| Component | Responsibility |
|---|---|
popcount, log2, bits_required |
Compute minimum bits for range [min, max] |
bswap, host_to_network, network_to_host |
Little-endian byte-swap (no-op on x86/ARM LE) |
signed_to_unsigned, unsigned_to_signed |
Zig-zag encode: 0,-1,+1,-2,+2... <-> 0,1,2,3,4... |
clamp |
Clamp value to [a, b] |
BitWriter |
Write N-bit unsigned integers into a buffer via 64-bit scratch word |
BitReader |
Read N-bit unsigned integers from a buffer via 64-bit scratch word |
BaseStream |
Context and allocator pointer holder |
WriteStream |
Wraps BitWriter; IsWriting=1, IsReading=0 |
ReadStream |
Wraps BitReader; IsWriting=0, IsReading=1 |
MeasureStream |
Counts bits without writing; IsWriting=1, IsReading=0 |
Macros serialize_* |
Unified read/write/measure via stream template dispatch |
serialize_int_relative |
Delta-encode integer relative to previous value |
serialize_ack_relative |
Delta-encode uint16 ack relative to sequence |
fields: uint32_t* data, uint64_t scratch, int scratchBits, int wordIndex, int numBits, int bitsWritten
writeBits(value, bits):
scratch |= uint64(value) << scratchBits
scratchBits += bits
if scratchBits >= 32:
data[wordIndex++] = uint32(scratch & 0xFFFFFFFF) // little-endian flush
scratch >>= 32
scratchBits -= 32
bitsWritten += bits
flushBits():
if scratchBits > 0:
data[wordIndex++] = uint32(scratch & 0xFFFFFFFF)
scratchBits = 0
Values are packed right-to-left within the 64-bit scratch; flushed left-to-right as 32-bit little-endian words. The result is a compact little-endian bit stream.
fields: uint32_t* data, uint64_t scratch, int scratchBits, int wordIndex, int numBits, int bitsRead
readBits(bits):
bitsRead += bits
if scratchBits < bits:
scratch |= uint64(data[wordIndex++]) << scratchBits // refill from buffer
scratchBits += 32
output = scratch & ((1 << bits) - 1)
scratch >>= bits
scratchBits -= bits
return output
Mirror of BitWriter. A ReadStream must read in exactly the same sequence as the
corresponding WriteStream wrote - there is no field framing in the bit stream.
- Buffer size passed to
BitWriterMUST be a multiple of 4 bytes (writes 32-bit words). - Buffer allocated for
BitReadermust be rounded up to the next 4-byte boundary in memory even if the byte count passed to the constructor is not a multiple of 4. writeAlign()/readAlign(): pad/skip to next byte boundary; padding bits must be zero (checked on read).writeBytes()/readBytes(): requiresgetAlignBits() == 0; performs amemcpy-style bulk copy for efficiency.
delta = max - min
values = delta / resolution
maxIntegerValue = ceil(values)
bits = bits_required(0, maxIntegerValue)
write:
normalized = clamp((value - min) / delta, 0.0, 1.0)
integerValue = floor(normalized * maxIntegerValue + 0.5)
writeBits(integerValue, bits)
read:
integerValue = readBits(bits)
normalized = integerValue / float(maxIntegerValue)
value = normalized * delta + min
| Delta range | Bits on wire |
|---|---|
| == 1 | 1 (flag bit only) |
| [2, 6] | 1 + 3 = 4 |
| [7, 23] | 2 + 4 = 6 |
| [24, 280] | 3 + 9 = 12 |
| [281, 4377] | 4 + 13 = 17 |
| [4378, 69914] | 5 + 17 = 22 |
| > 69914 | 6 + 32 = 38 |
Requires previous < current (monotonically increasing integers only).
write:
ack_delta = (ack < sequence) ? sequence - ack : sequence + 65536 - ack
ack_in_range = (ack_delta <= 64)
if ack_in_range: write ack_delta in [1, 64] (6 bits + 1 flag bit = 7 bits total)
else: write raw ack in 16 bits
read:
if ack_in_range: ack = (sequence - ack_delta) & 0xFFFF
else: ack = raw 16-bit value
Used by the reliable layer for ack compression; exposed here as a utility.
C++ uses reference parameters (int32_t & value) for unified read/write. Java has no
pass-by-reference for primitives. Two options were evaluated:
- (A)
int[1]holder array passed in - allocation risk if not pre-allocated; awkward call site. - (B) Separate
writeInt(value, min, max)onWriteStreamandreadInt(min, max)onReadStream- zero allocation, clean call site, direct JIT optimization.
Decision: Option B - separate write/read methods.
WriteStream, ReadStream, and MeasureStream each expose their own strongly-typed methods.
Unified Serializable interface implementations call the correct method based on which stream
type they receive. No macros, no reflection, no holder arrays.
Because writeAlign() pads to the next byte boundary and the number of padding bits depends on
the current bit position in the final stream (not known at measure time), MeasureStream
always counts 7 bits for any writeAlign() call. This is the worst-case estimate from the C
reference and is correct: the actual padding is in [0, 7] so the measurement is always >=
the real cost. Callers pre-size buffers using MeasureStream.getBytesProcessed() and are
guaranteed the buffer is large enough.
BaseStream in C++ carries a void* context and void* allocator passed through to
Serialize methods via stream.GetContext(). In Java:
- Allocation is never needed in serialize code (no
mallocequivalent). - Context is injected via constructor of the codec object owning the stream instance.
- Removing it eliminates a raw pointer in the API and avoids any temptation to allocate through the stream.
Decision: no context or allocator on any stream class.
ReadStream methods (readInt, readBits, readFloat, etc.) return the decoded value
directly (clean call site). When a read would exceed the buffer or a value falls outside
[min, max], the stream sets an internal boolean error flag and returns a safe sentinel (0,
0.0f, etc.). The caller checks readStream.isError() once after a batch of reads. This is:
- Zero allocation.
- Branch-predictable (error is cold path).
- Compatible with hot-path coding style (no exceptions for flow control).
C++ requires the caller to ensure getAlignBits() == 0 before calling writeBytes/readBytes.
WriteStream.SerializeBytes in the C reference already calls WriteAlign() internally. The
Java port follows the stream-level behavior: WriteStream.writeBytes() and
ReadStream.readBytes() call writeAlign()/readAlign() internally. BitWriter.writeBytes()
and BitReader.readBytes() retain the assertion for direct use.
New package: net.ztrust.serialize
src/main/java/net/ztrust/serialize/
BitUtils.java - bitsRequired, signedToUnsigned, unsignedToSigned, clamp
BitWriter.java - bit-level write into MutableDirectBuffer
BitReader.java - bit-level read from DirectBuffer
WriteStream.java - serialize-direction stream (IsWriting=true)
ReadStream.java - deserialize-direction stream with range validation
MeasureStream.java - bit-count estimation stream
Serializable.java - interface for objects that support all three stream types
src/test/java/net/ztrust/serialize/
BitUtilsTest.java
BitPackerTest.java
StreamRoundTripTest.java
MeasureStreamTest.java
src/jmh/java/net/ztrust/serialize/
BitPackerBenchmark.java
public final class BitUtils {
private BitUtils() {}
/**
* Returns the number of bits required to represent values in [min, max].
* Returns 0 if min == max (nothing to encode).
* @param min unsigned lower bound (>= 0)
* @param max unsigned upper bound (>= min)
*/
public static int bitsRequired(final int min, final int max) {
return (min == max) ? 0 : 32 - Integer.numberOfLeadingZeros(max - min);
}
/**
* Zig-zag encodes a signed int to an unsigned int.
* 0, -1, +1, -2, +2 ... -> 0, 1, 2, 3, 4 ...
*/
public static int signedToUnsigned(final int n) {
return (n << 1) ^ (n >> 31);
}
/**
* Zig-zag decodes an unsigned int to a signed int.
* 0, 1, 2, 3, 4 ... -> 0, -1, +1, -2, +2 ...
*/
public static int unsignedToSigned(final int n) {
return (n >>> 1) ^ -(n & 1);
}
public static int clamp(final int value, final int a, final int b) {
return (value < a) ? a : (value > b ? b : value);
}
public static float clamp(final float value, final float a, final float b) {
return (value < a) ? a : (value > b ? b : value);
}
}public final class BitWriter {
private MutableDirectBuffer data;
private int baseOffset; // offset of word 0 within data
private long scratch;
private int scratchBits;
private int wordIndex;
private int numBits;
private int bitsWritten;
/** Creates an unwrapped writer. Must call wrap() before any write. */
public BitWriter() {}
/**
* Wraps this writer to write into buf[offset .. offset+bytes-1].
* bytes MUST be a multiple of 4.
*/
public void wrap(MutableDirectBuffer buf, int offset, int bytes);
/**
* Writes the low 'bits' bits of value into the stream.
* bits in [1, 32]. value must be in [0, (1 << bits) - 1].
* Returns this writer for chaining.
*/
public BitWriter writeBits(int value, int bits);
/**
* Pads the stream to the next byte boundary with zero bits.
* No-op if already byte-aligned.
*/
public void writeAlign();
/**
* Bulk-copies bytes from src into the stream.
* Requires getAlignBits() == 0 before calling.
* src[srcOffset .. srcOffset+bytes-1] are copied.
*/
public void writeBytes(DirectBuffer src, int srcOffset, int bytes);
/**
* Flushes any remaining bits in scratch to the underlying buffer.
* MUST be called after all writeBits calls and before reading back the data.
*/
public void flushBits();
/** Number of bits written including any align padding, excluding unflushed scratch. */
public int getBitsWritten();
/** (getBitsWritten() + 7) / 8 - effective packet size after flush. */
public int getBytesWritten();
/** Bits available before buffer is full. */
public int getBitsAvailable();
/** Zero-pad bits needed to reach the next byte boundary: result in [0, 7]. */
public int getAlignBits();
}public final class BitReader {
private DirectBuffer data;
private int baseOffset;
private long scratch;
private int scratchBits;
private int wordIndex;
private int numBits;
private int bitsRead;
public BitReader() {}
/**
* Wraps this reader to read from buf[offset .. offset+bytes-1].
* The allocated buffer must be rounded up to the next 4-byte boundary.
*/
public void wrap(DirectBuffer buf, int offset, int bytes);
/** Returns true if reading 'bits' more bits would exceed the buffer. */
public boolean wouldReadPastEnd(int bits);
/**
* Reads 'bits' bits from the stream and returns them as an unsigned int.
* bits in [1, 32].
*/
public int readBits(int bits);
/**
* Reads and discards align padding to the next byte boundary.
* Returns false if any padding bit is non-zero (malformed stream).
*/
public boolean readAlign();
/**
* Bulk-reads bytes from the stream into dst.
* Requires getAlignBits() == 0 before calling.
*/
public void readBytes(MutableDirectBuffer dst, int dstOffset, int bytes);
public int getBitsRead();
public int getBitsRemaining();
public int getAlignBits();
}public final class WriteStream {
public static final boolean IS_WRITING = true;
public static final boolean IS_READING = false;
private final BitWriter writer = new BitWriter();
public void wrap(MutableDirectBuffer buf, int offset, int bytes);
/** Encodes value in [min, max] using bitsRequired(min, max) bits. */
public void writeInt(int value, int min, int max);
/** Writes the low 'bits' bits of value. bits in [1, 32]. */
public void writeBits(int value, int bits);
/** Writes all 64 bits of value as two 32-bit words (lo then hi). */
public void writeUint64(long value);
public void writeBool(boolean value);
/** Reinterprets float bits as int32 and writes 32 bits. */
public void writeFloat(float value);
/**
* Quantizes value in [min, max] with given resolution and writes
* bitsRequired(0, ceil((max-min)/res)) bits.
*/
public void writeCompressedFloat(float value, float min, float max, float res);
/** Reinterprets double bits as int64 and writes 64 bits. */
public void writeDouble(double value);
/**
* Aligns to byte boundary, then bulk-copies bytes.
* Internally calls writeAlign() before the copy.
*/
public void writeBytes(DirectBuffer src, int srcOffset, int bytes);
public void writeAlign();
/**
* Encodes current relative to previous using the tiered delta scheme.
* Requires previous < current.
*/
public void writeIntRelative(int previous, int current);
/**
* Encodes ack relative to sequence using 7-bit or 16-bit encoding.
* sequence and ack are unsigned uint16 values passed as int.
*/
public void writeAckRelative(int sequence, int ack);
/** Flushes scratch to buffer. Call once after all writes. */
public void flush();
public int getBitsProcessed();
public int getBytesProcessed();
public int getAlignBits();
}public final class ReadStream {
public static final boolean IS_WRITING = false;
public static final boolean IS_READING = true;
private final BitReader reader = new BitReader();
private boolean error;
public void wrap(DirectBuffer buf, int offset, int bytes);
/** Resets the error flag. Call wrap() or reset() before reuse. */
public void reset();
/**
* Returns true if any previous read operation failed
* (out-of-bounds or value outside [min, max]).
*/
public boolean isError();
/**
* Reads bitsRequired(min, max) bits, reconstructs value in [min, max].
* Sets error flag and returns 0 if buffer exhausted or value out of range.
*/
public int readInt(int min, int max);
/** Reads 'bits' bits as unsigned int. Sets error flag on buffer exhaustion. */
public int readBits(int bits);
/** Reads 64 bits as two 32-bit words (lo then hi). */
public long readUint64();
/** Reads 1 bit. Returns false on error. */
public boolean readBool();
/** Reads 32 bits, reinterprets as float. */
public float readFloat();
/** Reads compressed float encoded by writeCompressedFloat. */
public float readCompressedFloat(float min, float max, float res);
/** Reads 64 bits, reinterprets as double. */
public double readDouble();
/**
* Aligns to byte boundary (validates zero padding), then bulk-reads bytes.
* Sets error flag if alignment padding is non-zero or buffer exhausted.
*/
public void readBytes(MutableDirectBuffer dst, int dstOffset, int bytes);
/**
* Reads and validates alignment padding.
* Sets error flag if padding is non-zero.
*/
public void readAlign();
/**
* Reads an integer encoded relative to previous by writeIntRelative.
* Returns previous + decoded delta, or 0 on error.
*/
public int readIntRelative(int previous);
/**
* Reads ack encoded relative to sequence by writeAckRelative.
* Returns decoded ack (uint16 range), or 0 on error.
*/
public int readAckRelative(int sequence);
public int getBitsProcessed();
public int getBytesProcessed();
public int getAlignBits();
}public final class MeasureStream {
public static final boolean IS_WRITING = true;
public static final boolean IS_READING = false;
private int bitsWritten;
public void reset();
public void writeInt(int value, int min, int max); // += bitsRequired(min, max)
public void writeBits(int value, int bits); // += bits
public void writeUint64(long value); // += 64
public void writeBool(boolean value); // += 1
public void writeFloat(float value); // += 32
public void writeCompressedFloat(float value, float min, float max, float res);
public void writeDouble(double value); // += 64
/**
* += 7 (worst-case alignment) + bytes * 8.
* Conservative: actual align padding is in [0, 7].
*/
public void writeBytes(DirectBuffer src, int srcOffset, int bytes);
/** += 7 (worst-case). */
public void writeAlign();
public void writeIntRelative(int previous, int current);
public void writeAckRelative(int sequence, int ack);
public int getBitsProcessed();
/** (getBitsProcessed() + 7) / 8 - always >= actual WriteStream.getBytesProcessed(). */
public int getBytesProcessed();
public int getAlignBits(); // always returns 7
}/**
* Implemented by any object that can serialize itself to/from a stream.
*
* <p>Implementations MUST be deterministic, allocation-free, and produce identical
* bit sequences for identical state across write and read directions.
*
* <p>Pattern:
* <pre>
* // Write
* WriteStream ws = ...; ws.wrap(buf, 0, size);
* object.write(ws);
* ws.flush();
*
* // Read
* ReadStream rs = ...; rs.wrap(buf, 0, bytesWritten);
* object.read(rs);
* if (rs.isError()) { ... }
*
* // Measure
* MeasureStream ms = ...; ms.reset();
* object.measure(ms);
* int bufferNeeded = ms.getBytesProcessed();
* </pre>
*/
public interface Serializable {
void write(WriteStream stream);
void read(ReadStream stream);
void measure(MeasureStream stream);
}| Operation | Target |
|---|---|
writeBits (single field, 5-bit int) |
< 5 ns |
readBits (single field, 5-bit int) |
< 5 ns |
writeCompressedFloat |
< 15 ns |
readCompressedFloat |
< 15 ns |
Full TestObject write (14 fields) |
< 100 ns |
Full TestObject read (14 fields) |
< 100 ns |
MeasureStream full object |
< 50 ns |
| Allocation per write/read | 0 bytes |
Measured with JMH @BenchmarkMode(AverageTime), -prof gc to assert zero allocation.
Regression > 10% vs baseline requires explicit justification.
flowchart TD
A[Application Game State] -->|write/read/measure| B[Serializable objects]
B -->|WriteStream| C[Bit-packed payload bytes]
C -->|ReliableEndpoint.sendPacket| D[reliable layer]
D -->|CONNECTION_PAYLOAD encryption| E[NetcodeClient / NetcodeServer]
E -->|DatagramChannel.send| F[UDP wire]
F -->|DatagramChannel.receive| G[NetcodeClient / NetcodeServer]
G -->|CONNECTION_PAYLOAD decrypt| H[reliable layer]
H -->|ReliableEndpoint.receivePacket -> processHandler| I[Bit-packed payload bytes]
I -->|ReadStream| J[Application Game State]
serialize sits above the reliable layer - it encodes/decodes the application-level payload
that travels inside CONNECTION_PAYLOAD datagrams. It does NOT interact with:
PacketHeaderCodec/FragmentCodecinnet.ztrust.reliable(byte-level, fixed format).BufferWriter/BufferReaderinnet.ztrust.netcode.codec(byte-aligned, netcode headers).NetcodeCrypto(operates on opaque byte buffers; serialize output is the plaintext payload).
| Step | File | Notes |
|---|---|---|
| 1 | BitUtils.java |
No dependencies |
| 2 | BitWriter.java |
Depends on Agrona MutableDirectBuffer / DirectBuffer |
| 3 | BitReader.java |
Depends on Agrona DirectBuffer / MutableDirectBuffer |
| 4 | WriteStream.java |
Depends on BitWriter, BitUtils |
| 5 | ReadStream.java |
Depends on BitReader, BitUtils |
| 6 | MeasureStream.java |
Depends on BitUtils |
| 7 | Serializable.java |
Depends on all three streams |
| 8 | BitUtilsTest.java |
Port test_bits_required from C |
| 9 | BitPackerTest.java |
Port test_bitpacker and test_endian from C |
| 10 | StreamRoundTripTest.java |
Port test_stream with full TestObject equivalent |
| 11 | MeasureStreamTest.java |
Verify measure >= write bytes for all field types |
| 12 | BitPackerBenchmark.java |
JMH: single field, full object, compressed float |
| Risk | Mitigation |
|---|---|
Java has no unsigned 32-bit type; writeBits receives int |
Document contract: caller passes unsigned value as int; (1 << bits) - 1 mask applied internally; & 0xFFFFFFFFL used when promoting to long for scratch OR |
BitReader refill reads a 32-bit word past the declared byte count |
Buffer allocated by caller must be rounded up to 4-byte boundary (documented in wrap Javadoc); wouldReadPastEnd check guards application-level reads |
readAlign false-positive on non-zero padding |
ReadStream.readAlign() sets error flag; caller detects via isError(); never throws |
writeIntRelative requires previous < current |
Validated by assertion in debug builds; documented precondition in Javadoc; violation is a programming error, not a wire error |
MeasureStream over-estimates bytes (7-bit align) |
By design and documented: caller receives an upper bound, always safe to allocate that many bytes |
Off-by-one in bitsRequired for power-of-two ranges |
bits_required(0, 4) == 3, not 2 (value 4 requires 3 bits); port uses 32 - Integer.numberOfLeadingZeros(max - min) which matches C __builtin_clz result; covered by table test |