Slick Dynamic Buffer

slick::dynamic_buffer<BufferT> is a header-only, template Boost.Asio DynamicBuffer_v1 adapter over any slick buffer backend. It is designed as a drop-in replacement for boost::beast::flat_buffer: network bytes received by boost::asio / boost::beast are written directly into the backend ring, and publishing a complete message to consumer threads — or other processes via shared memory — requires zero copies.

Supported backends (any type satisfying slick::buffer_backend):

Backend	Use case
slick::stream_buffer	Single-producer SPMC ring; consumers call stream_buffer.read(cursor)
slick::stream_buffer_multiplexer::producer_buffer	Fans into an MPMC shared queue so multiplexer consumers receive the record

The library has no hard dependency on either backend — bring your own and include the relevant header alongside <slick/dynamic_buffer.h>.

How it works

The adapter exposes the familiar dynamic-buffer interface (prepare / commit / consume / data / size), with one twist:

• prepare(n) returns a contiguous writable region — asio writes received bytes there
• commit(n) moves bytes into the readable area — the app parses them in place
• consume(n) does not discard bytes: it publishes them to consumers as one discrete message record

 network ──asio──▶ prepare/commit ──▶ [ data ring ] ──consume(n)──▶ record {offset, len}
                                                                        │
                                              consumer A (own cursor) ◀─┤  zero-copy reads
                                              consumer B (own cursor) ◀─┤  (threads or
                                              process C (shared memory)◀┘   processes)

Each consumer owns an independent cursor and reads whole messages zero-copy directly from the underlying buffer.

Features

• Boost.Asio DynamicBuffer adapter usable with boost::beast / boost::asio read operations
• Zero-copy fan-out of received network data to threads and processes
• Template backend: works with slick::stream_buffer (SPMC) and producer_buffer (MPMC)
• No hard backend dependency — the library only requires Boost.Asio
• Drop-in replacement for boost::beast::flat_buffer (including clear())
• Cross-platform — Windows, Linux, macOS
• Modern C++20, header-only

Requirements

• C++20 compatible compiler
• Boost.Asio — only the buffer types are used (boost/asio/buffer.hpp)
• A slick buffer backend — slick-stream-buffer and/or slick-stream-buffer-multiplexer (not fetched automatically; include the backend header and link its target yourself)

Installation

Header-only. Add the include directory to your include path:

#include <slick/dynamic_buffer.h>
// also include your chosen backend:
#include <slick/stream_buffer.h>               // for stream_buffer backend
#include <slick/stream_buffer_multiplexer.h>   // for producer_buffer backend

Using CMake FetchContent

include(FetchContent)

set(BUILD_SLICK_DYNAMIC_BUFFER_TESTS OFF CACHE BOOL "" FORCE)
FetchContent_Declare(
    slick-dynamic-buffer
    GIT_REPOSITORY https://github.com/SlickQuant/slick-dynamic-buffer.git
    GIT_TAG v1.0.0
)
FetchContent_MakeAvailable(slick-dynamic-buffer)

# Link your backend(s) separately:
target_link_libraries(your_target PRIVATE
    slick::dynamic_buffer
    slick::stream_buffer)               # or slick::stream_buffer_multiplexer

Usage

With slick::stream_buffer (SPMC)

#include <slick/dynamic_buffer.h>
#include <slick/stream_buffer.h>

// 64 MB data ring, 64K message records; named -> shared memory, nullptr -> local
slick::stream_buffer stream(1ull << 26, 1u << 16, "market_data");
slick::dynamic_buffer dyn(stream);   // CTAD deduces dynamic_buffer<stream_buffer>

for (;;) {
    std::size_t n = socket.read_some(dyn.prepare(64 * 1024));
    dyn.commit(n);

    // parse the readable area; publish every complete package
    while (std::size_t package_size = find_complete_package(dyn.data())) {
        dyn.consume(package_size);   // publishes one record — no copy
    }
}

Consumers read from the underlying buffer directly:

// same process:
slick::stream_buffer& stream = dyn.buffer();
// another process (shared memory):
slick::stream_buffer stream("market_data");

uint64_t cursor = stream.initial_reading_index();   // or 0 to replay history
for (;;) {
    auto [data, length] = stream.read(cursor);
    if (data == nullptr) continue;
    handle_package(data, length);   // zero-copy pointer into the ring
}

With slick::stream_buffer_multiplexer::producer_buffer (MPMC)

#include <slick/dynamic_buffer.h>
#include <slick/stream_buffer_multiplexer.h>

slick::stream_buffer_multiplexer mux(1024);
auto pb = mux.add_producer(0, 1ull << 26, 1u << 16);  // shared_ptr<producer_buffer>

// shared_ptr constructor (owning) — pb stays alive as long as dyn does
slick::dynamic_buffer dyn(pb);   // CTAD deduces dynamic_buffer<producer_buffer>

for (;;) {
    std::size_t n = socket.read_some(dyn.prepare(64 * 1024));
    dyn.commit(n);
    while (std::size_t package_size = find_complete_package(dyn.data())) {
        dyn.consume(package_size);   // publishes to producer ring AND fans into shared queue
    }
}

The adapter also works with composed operations such as boost::asio::read(socket, dyn, ...), boost::beast::http::read(...) and websocket::stream::read(...).

API Overview

slick::buffer_backend concept

template<typename T>
concept buffer_backend = requires(T& b, const T& cb, std::size_t n) {
    { b.prepare(n) } -> std::same_as<std::pair<uint8_t*, std::size_t>>;
    { b.commit(n) };
    { b.consume(n) };
    { b.discard() };
    { cb.data() }    -> std::same_as<const uint8_t*>;
    { cb.size() }    -> std::convertible_to<std::size_t>;
    { cb.capacity() }-> std::convertible_to<std::size_t>;
};

slick::dynamic_buffer<BufferT>

// Shared-ownership constructor (natural for shared_ptr backends like producer_buffer)
explicit dynamic_buffer(std::shared_ptr<BufferT> ptr,
                        std::size_t max_size = /* unlimited */) noexcept;

// Non-owning reference constructor (backward-compatible; caller manages lifetime)
explicit dynamic_buffer(BufferT& buffer,
                        std::size_t max_size = /* unlimited */);

Both constructors clamp max_size to buffer.capacity(). The adapter is a cheap copyable handle — asio composed operations copy DynamicBuffer_v1 objects by value; copies share the underlying shared_ptr.

• mutable_buffers_type prepare(std::size_t n) — contiguous writable region of n bytes; throws std::length_error if size() + n > max_size()
• void commit(std::size_t n) — make n prepared bytes readable
• auto consume(std::size_t n) — publish the first n readable bytes as one message record; return type is deduced from BufferT::consume() — a published_record for both slick backends (asio/beast callers may ignore the return value)
• void clear() — drop the readable bytes and any prepared region without publishing, matching beast::flat_buffer::clear()
• const_buffers_type data() / std::size_t size() — the readable (committed, unconsumed) region
• std::size_t max_size() / std::size_t capacity() — limits, as required by DynamicBuffer_v1
• BufferT& buffer() / const BufferT& buffer() const — access the underlying backend
• std::shared_ptr<BufferT> buffer_ptr() — shared-ownership handle to the backend

Important Constraints

Single producer. All adapter (producer) methods must be called from one thread. Consumers are lock-free and independent.

Lossy semantics. The producer never blocks; if it laps a slow consumer, the consumer skips ahead and the loss is counted. Size the rings so this cannot happen in normal operation.

Pointer invalidation. prepare() may relocate the readable region to keep it contiguous when the ring wraps; pointers previously returned by data()/prepare() are invalidated — the same rule as flat_buffer reallocation.

Record granularity. Every consume(n) call produces exactly one consumer-visible record. If a protocol layer consumes incrementally (e.g. the beast HTTP parser), records correspond to those increments; call consume() yourself on package boundaries when you need strict framing.

Disconnects mid-message. If the connection drops after a partial message was committed, call clear() before reconnecting so the leftover bytes are not prepended to the new connection's data. clear() does not publish a record.

Building and Testing

cmake -S . -B build
cmake --build build --config Debug
ctest --test-dir build -C Debug --output-on-failure

Boost.Asio and at least one slick backend are required to build the tests.

License

slick-dynamic-buffer is released under the MIT License.

Made with ⚡ by SlickQuant