FastQueue2

FastQueue2 is a rewrite of FastQueue. It supports 8-byte transfers only.

But first

• Is this queue memory efficient?

  No. This queue aims for speed, not memory efficiency.

• The queue is ‘dramatically under-synchronized’

  Write a test and prove it (you can use FastQueueIntegrityTest.cpp as a boilerplate). Don’t just say stuff out of the blue, prove it!

• Why not use partial specialization for pointers since that's all you support?

  This queue supports the transport of 8 bytes from a producer to a consumer. It might be a pointer and it might not be, so that’s why no specialization is implemented. However, if we gain speed by specializing for pointers, then let’s implement that. I did not see any gain in my tests, and this queue is all about speed.

Background

When I was benchmarking SPSC queues, deaod’s and Dro’s were strong baselines. Rigtorp, Folly, moodycamel, and boost are other established implementations. My previous attempt (FastQueue) reached the top tier but did not lead every measured workload. It also implements stopQueue, which the comparison implementations do not all provide.

This project targets measured use cases rather than universal queue rankings: 64-bit x86_64 and arm64 CPUs, one producer, one consumer, and 8-byte transfers. Pointers are common payloads, but any 8-byte value fits.

In the general SPSC queue implementation there is a circular buffer where push checks whether it’s possible to push an object by looking at the distance between the tail and head pointer/counter. The same goes for popping an object: if there is a distance between tail and head, there is at least one object to pop. That means that if the push runs on one CPU and the pop runs on another CPU, you share the tail/head counters and the object itself between the CPUs.

The above picture is taken from Deaod’s repo

The first version of this queue used the object slot itself as the full/empty flag: pop cleared the slot to nullptr, push waited for nullptr. Elegant, and it means the CPUs only ever share the object. The problem only shows up when you measure the pure queue (no per-message malloc masking it): that scheme bounces a whole cache line both directions for every single element (producer publishes the pointer → consumer reads it → consumer writes the nullptr back → producer reads that back), so it moves an order of magnitude more coherence traffic than a packed ring. Against a benchmark dominated by new/delete it looked great; as a raw queue it was several times slower than the titans.

So this version keeps FastQueue API (push, pop, stopQueue, 8-byte objects) and uses data/control layouts selected per architecture and measured workload:

• Cached indices. Each side keeps a private copy of the other side's index and only re-reads the shared atomic when its cache says the queue is full (producer) or empty (consumer). In steady state the producer only writes its own write-index line plus the data; the consumer only writes its own read-index line. The control lines stop ping-ponging.
• One-directional, packed data. The slot is never written back, so a single cache line carries many elements flowing producer → consumer instead of one element bouncing both ways.
• x86 throughput profiles. -march=znver2 selects wrapped phase indexes plus six-item cushion: measured best Zen2 throughput. Other x86 targets select monotonic indexes plus immediate drain: avoids phase-mask work and batching penalty on Haswell. Both remain user-overridable at compile time.
• Architecture-specific ring placement. fast_queue_arm64.h selects queue-owned inline contiguous storage by default (FQ_ARM_RING_INLINE=1); define it as 0 to test separately allocated ARM storage. fast_queue_x86_64.h keeps ring storage inline. Both layouts preserve control-line isolation. ARM has no compile-time throughput profile; x86 profile selection is described below.

If the tail catches the head there's nothing to pop; if the head catches the tail the buffer is full and push waits. Same contract as before, very different cost.

The need for speed

A word on measuring first, because it bit me hard: the original benchmark ran each queue back-to-back in a fixed order, and on a laptop that means the queue that runs first gets the cold/turbo advantage and looks fastest. It also does new/delete per message, and that allocator cost (cross-thread free is expensive) dominates the loop and hides the queue entirely. So the numbers below come from a rewritten benchmark that rotates the order every round, reports the median, and runs two passes: a heap pass (new/delete per message, the classic FastQueue benchmark, allocator-bound) and a pooled pass (pre-allocated objects — this is what actually measures the queue).

Pooled pass, fixed transaction count, higher is better, median of 12 rotated rounds. Absolute numbers differ by machine, compiler, clocks, and scheduler; compare queues within one row.

Machine	FastQueue	Deaod	Dro	David V5
Apple M5, macOS arm64	396.473M	165.428M	77.379M	154.271M
ARM Cortex-X925 + Cortex-A725, Linux arm64, X925 CPUs 5/6	84.478M	92.927M	87.410M	85.326M
AMD EPYC 7702, Zen2 dual socket, CPUs 1/3	123.935M	90.443M	107.959M	102.075M
AMD EPYC 7702P, Zen2, CPUs 1/3	118.629M	75.078M	90.129M	79.699M
Intel Xeon E5-2630L v3, Haswell, CPUs 1/3	117.951M	28.725M	31.869M	27.067M

Measurement method

Every table row uses joined workers, atomic start gate, exact transfer count, sequence validation, four-way order rotation, pooled pointers, and a 12-round median. Compare queues only within one row.

Apple M5 arm64 row uses 5,000,000 fixed transfers. FastQueue is row winner at 396.473M/s; Deaod is 165.428M/s, David V5 is 154.271M/s, and Dro is 77.379M/s. FQ_ARM_RING_INLINE=1 uses queue-owned contiguous storage with peer-index caches and release/acquire publication. The 100,000,000-transfer confirmation measured FastQueue 405.951M/s versus Deaod 176.884M/s, David V5 155.166M/s, and Dro 72.701M/s. macOS affinity is a scheduler hint, not hard physical-core pinning; P-core/E-core placement adds variance.

Cortex-X925 Linux row uses 100,000,000 fixed transfers, physical X925 CPUs 5 and 6, performance governor, chrt -f 90, and g++ -O3 -DNDEBUG -march=native. Host has two 5-core Cortex-X925 clusters (CPUs 5–9 and 15–19) plus Cortex-A725 cores (CPUs 0–4 and 10–14); CPUs 5/6 are distinct X925 cores in one cluster at 3.9 GHz. Deaod wins this workload at 92.927M/s; Dro, David V5, and FastQueue measure 87.410M/s, 85.326M/s, and 84.478M/s. 5,000,000-transfer confirmation under identical controls measured Dro 85.788M/s, Deaod 85.255M/s, David V5 86.804M/s, and FastQueue 84.088M/s. FastQueue 3-second two-slot integrity test completed 355,855 transactions.

Linux x86 rows use pooled pointers, physical-core pinning, performance governor, chrt -f 90, g++ -O3 -DNDEBUG -march=native, exact sequence validation, and fixed transfer counts. FastQueue is row winner on all three listed x86 hosts: +14.8% versus Dro on dual-socket Zen2, +31.6% on single-socket Zen2, and +270.1% on this Haswell. Linux controls improve repeatability; they do not make Linux and macOS absolute throughput directly comparable.

David V5 comes from David Álvarez Rosa's ring-buffer analysis Results identify row winners only for stated machine/workload conditions, not universal #1 across every CPU, compiler, capacity, or latency workload.

Heap pass is allocator-bound. Use pooled objects, rotated order, joined worker threads, fixed-work sequence validation, and median distributions for queue comparisons.

Per-architecture tuning

fast_queue_arm64.h selects an inline contiguous ring by default (FQ_ARM_RING_INLINE=1), which is measured fastest for Apple M5 pooled throughput. Define FQ_ARM_RING_INLINE=0 to restore separately allocated ARM storage for experiments. fast_queue_x86_64.h selects an x86 profile from GCC/Clang -march macros:

• -march=znver2: FQ_WRAPPED_INDICES=1, FQ_CONSUMER_CUSHION=6.
• Other x86 targets: FQ_WRAPPED_INDICES=0, FQ_CONSUMER_CUSHION=0.
• Define either macro before including header to override profile.
• FQ_OCCUPANCY_INSTRUMENT=1 records empty-refresh occupancy and perturbs throughput; diagnosis only.

Fixed-work reproduction:

g++ -O3 -DNDEBUG -std=c++20 -march=native -DPOOLED_ONLY=1 \
  -DTRANSFER_COUNT=100000000 -DROUNDS=12 -DCONSUMER_CPU=1 -DPRODUCER_CPU=3 \
  -I. -Ideaod_spsc -Idro main.cpp -o bench
sudo cpupower frequency-set -g performance
sudo chrt -f 90 ./bench

Usage

See the original FastQueue (the link above).

(Just copy the header file for your architecture into your project.) fast_queue_arm64.h / fast_queue_x86_64.h

Build and run the tests

git clone https://github.com/andersc/fastqueue2.git
cd fastqueue2
mkdir build
cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
cmake --build .

(Run the benchmark against Deaod and Dro)

./fast_queue2

(Run the integrity test)

./fast_queue_integrity_test

Some thoughts

There are a couple of findings that puzzled me.

1. Cache-line separation is not one-size-fits-all, but on both families the right answer is "keep the two hot index lines out of each other's prefetch window". x86's L2 spatial prefetcher pulls 128-byte (two-line) pairs, so the write index and read index must live in different 128-byte pairs — 128-byte separation measured ~18% faster than 64 on AMD Zen (with 64, fetching one index dragged the other core's index line along). On ARM (Apple M-series and Cortex-X925) the streaming prefetcher reaches even further and 256-byte separation was best. So the alignment is set per-architecture in the two headers.
2. Heap-allocating the ring (instead of embedding it in the object) is worth ~3x on Apple silicon. Sitting next to the indices, the ring got hoovered up by the prefetcher into the wrong core. Moving it out is the single biggest M-series win.
3. Memory ordering is not free and not uniform: making the slot itself an std::atomic with acquire/release (LDAR/STLR per access) was ~2.6x slower on Apple silicon than plain loads/stores ordered by one release/acquire pair on the index. Measure, don't assume.
4. Micro-benchmarks lie. The order you run competitors in, whether you malloc per message, and how warm the machine is all swing the numbers more than the code does. The benchmark here rotates order and reports medians for that reason — the results should still be 'considered with a grain of salt'.
5. macOS and Linux expose different measurement controls. On macOS, thread_policy_set with THREAD_AFFINITY_POLICY is an affinity tag, not hard core pinning; scheduler placement between P- and E-cores can change results. Run several rounds and use medians. On Linux, pthread_setaffinity_np, a performance governor, and chrt can constrain placement, frequency policy, and scheduling policy; results still depend on CPU model, thermals, and system load. These controls affect measurement repeatability, not correctness.

Each benchmark row reports its own setup. Compare queues within a row; do not use absolute M/s values to compare different machines or operating systems.