LatencyBook

A low-latency limit order book & alpha signal engine, instrumented down to OS internals

LatencyBook reconstructs a full limit order book from raw exchange messages, runs three research-backed alpha signals over it, and trades those signals through a queue-position-aware simulated exchange — while measuring, at every stage, exactly which operating-system mechanism the code is leaning on and what that mechanism costs. It is not just an order book with comments that mention pool allocators, lock-free queues, and huge pages; every one of those is implemented, benchmarked against a naive baseline, and the delta is reported below with real numbers from an actual run, not placeholders.

---

🎯 Key Results

All numbers below are measured, not estimated — reproduce them yourself with ./scripts/run_benchmarks.sh. They come from a single-core sandbox VM (Ubuntu 24.04, GCC 13.3.0, 2.1GHz, no perf, no multi-socket NUMA); rows marked ⚠️ specifically need ≥2 physical cores to show the effect they're designed to demonstrate, and the benchmarks say so themselves when run on one core.

Metric	Before	After	Improvement
Order-node allocation cost (per alloc+free)	malloc/free: 9.81 ns · new/delete: 14.3 ns	PoolAllocator<T>: 1.76 ns	5.6×–8.1× faster
Allocations on the OrderBook hot path (steady state, 5000 add/cancel cycles)	not verified / typically >0 with a map-based book	0 (assertion-verified, LB_TRACK_ALLOCATIONS)	100% eliminated
Queue push+pop, uncontended	MutexQueue (mutex+condvar): 38.4 ns	SPSCRingBuffer (lock-free): 1.05 ns	36.6× faster
OrderBook add+cancel cycle (warmed-up book, 2000 resting orders/side)	—	21.5 ns (≈10.75 ns/op)	O(1) confirmed
OrderBook best-bid/best-ask read	—	0.525 ns	O(1) confirmed
Cold first-touch of a 16MB arena	Regular 4KB pages: 4.74 ms	2MB huge pages: 0.75 ms	6.3× faster
Pre-faulted access, same 16MB arena	Regular 4KB pages: 470 µs	2MB huge pages: 39.8 µs	11.8× faster
Per-message wait latency	Blocking read() on a pipe: 232 ns	Busy-poll SPSCRingBuffer: 21 ns	~11× faster
Tick-to-trade latency, full pipeline (p50 / p99 / p99.9)	—	312 ns / 23.6 µs / 42.0 µs	—
Cross-core queue throughput (padded vs. false-sharing-prone) ⚠️	514k msgs/s (unpadded)	517k msgs/s (padded)	needs ≥2 cores — see caveat
Context-switch / scheduling wakeup, pinned+SCHED_FIFO vs. unpinned ⚠️	p50 1.66 µs	p50 2.05 µs	needs ≥2 cores — see caveat
Unit tests / sanitizer runs	—	40/40 passing, ThreadSanitizer clean, ASan+UBSan clean	—

📌 A note on this machine

Two rows above are marked ⚠️ because this sandbox has exactly one logical CPU. Cross-core false sharing and core-pinning/real-time-scheduling jitter are both effects that only exist between cores — on one core, "pinned" and "unpinned" threads still share the same core, so the code correctly detects this at runtime, prints a warning, and (for the demo binary) refuses to enable SCHED_FIFO at all, because a busy-polling real-time thread with nowhere else to run would starve every other thread on the box rather than speed anything up. Every other number in the table is single-core-representative and will hold up on any machine.

---

🧠 System Architecture

Thread / data-flow pipeline

      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  Feed Handler    │ │   LOBSTER CSV, custom .lbbf binary,
    │  (Thread 1)      │╱    or the zero-dependency Poisson generator
    └─────────────────┘
             │
             ▼   SPSC Lock-Free Ring Buffer (65,536 slots, cache-padded)
      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  OrderBook +     │ │   Direct-indexed price ladder, pool-allocated
    │  Alpha Signals   │╱    L3 orders, OFI / BookImbalance / MicropriceMomentum
    │  (Thread 2)      │
    └─────────────────┘
             │
             ├──────────────► BacktestHarness (walk-forward hit rate / IC / PnL)
             ▼
      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  RiskEngine +    │╱    Position + rate limits, then queue-position-
    │  SimulatedExchange│     aware fills (not "always fills at the touch")
    └─────────────────┘

OS-concept mapping

      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  Feed Handler    │ │   🧵 std::thread (ThreadAffinity::pin_to_cpu, ≥2 cores)
    │                  │╱
    └─────────────────┘
             │
             ▼   🔒 Lock-free SPSC ring buffer — alignas(64) head_/tail_,
                    no mutex, no futex, no syscall on the hot path
      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  OrderBook +     │ │   📦 PoolAllocator + FixedHashMap (zero malloc,
    │  Alpha Signals   │╱     verified) · optional 📄 MAP_HUGETLB arena
    │  (RT candidate)  │      🧵 SCHED_FIFO + pinned, ≥2 cores only
    └─────────────────┘
             │
             ▼
      ┌─────────────────┐
     ╱                 ╱│
    ┌─────────────────┐ │
    │  Risk + OMS      │╱    std::deque / std::unordered_map — deliberately
    │                  │      OFF the zero-allocation hot path (see comments)
    └─────────────────┘

Mermaid version

flowchart TD
    A["Feed Handler Thread<br/>LOBSTER CSV / .lbbf / Synthetic Generator"] -->|"SPSC lock-free ring buffer"| B["OrderBook<br/>L2/L3, pool allocator, O(1) top-of-book"]
    B --> C["Alpha Signals<br/>OFI / BookImbalance / MicropriceMomentum"]
    C --> D["RiskEngine<br/>position + rate limits"]
    D --> E["SimulatedExchange<br/>queue-position-aware fills"]
    C --> F["BacktestHarness<br/>hit rate / IC / naive PnL"]
    B -.->|"-DUSE_MULTIPROCESS"| G["POSIX shared memory<br/>(shm_open + mmap)"]
    G -.-> H["Strategy Process<br/>(separate address space)"]

Loading

---

📁 File Structure

LatencyBook/
├── 📁 include/
│   ├── 📁 orderbook/
│   │   ├── 📄 OrderBook.hpp          # Direct-indexed price ladder, O(1) top-of-book
│   │   ├── 📄 PoolAllocator.hpp      # Fixed-size freelist arena, zero malloc after warmup
│   │   └── 📄 FixedHashMap.hpp       # Open-addressing order_id -> Order* (no per-insert heap)
│   ├── 📁 feed/
│   │   ├── 📄 Message.hpp            # Fixed-size POD event struct (32 bytes, no strings)
│   │   ├── 📄 FeedHandler.hpp        # LOBSTER CSV parser + custom "LBBF" binary format
│   │   └── 📄 SyntheticGenerator.hpp # Superposed-Poisson order flow, zero external deps
│   ├── 📁 alpha/
│   │   ├── 📄 IAlphaSignal.hpp       # CRTP static interface (no vtable on the hot path)
│   │   ├── 📄 OrderFlowImbalance.hpp # Cont/Kukanov/Stoikov OFI
│   │   ├── 📄 BookImbalance.hpp      # Multi-level geometrically-weighted imbalance
│   │   └── 📄 MicropriceMomentum.hpp # Short-horizon microprice deviation/momentum
│   ├── 📁 concurrency/
│   │   ├── 📄 SPSCRingBuffer.hpp     # Lock-free, cache-padded (+ Unpadded twin for the demo)
│   │   └── 📄 MutexQueue.hpp         # Baseline mutex+condvar comparison queue
│   ├── 📁 os/
│   │   ├── 📄 ThreadAffinity.hpp     # sched_setaffinity / pthread_setaffinity_np
│   │   ├── 📄 RealtimeSched.hpp      # SCHED_FIFO / SCHED_RR wrappers
│   │   ├── 📄 HugePageArena.hpp      # MAP_HUGETLB arena, graceful fallback
│   │   └── 📄 NumaInfo.hpp           # libnuma topology reporting, graceful degradation
│   ├── 📁 latency/
│   │   ├── 📄 TscClock.hpp           # RDTSCP + linear-regression wall-clock calibration
│   │   └── 📄 LatencyRecorder.hpp    # Log-scale histogram, percentiles, self-overhead
│   ├── 📁 oms/
│   │   ├── 📄 RiskEngine.hpp         # Max position + sliding-window order-rate limit
│   │   └── 📄 SimulatedExchange.hpp  # Queue-position-aware fills, partial fills
│   ├── 📁 backtest/
│   │   └── 📄 BacktestHarness.hpp    # Walk-forward hit rate, IC, naive PnL, CSV export
│   ├── 📁 multiprocess/
│   │   └── 📄 SharedMemoryIpc.hpp    # shm_open + mmap IPC (-DUSE_MULTIPROCESS only)
│   └── 📁 util/
│       └── 📄 AllocationCounter.hpp  # Global operator new/delete override for verification
├── 📁 src/                            # .cpp implementations mirroring include/, plus:
│   └── 📄 main.cpp                   # Demo driver — wires the full pipeline end to end
├── 📁 tests/                          # 40 GoogleTest cases: unit + concurrency stress
├── 📁 benchmarks/                     # Google Benchmark + custom real-thread microbenchmarks
├── 📁 scripts/
│   ├── 🐍 plot_latency_histogram.py
│   ├── 🐍 plot_backtest_results.py
│   └── 🖥️ run_benchmarks.sh
├── 📁 data/
│   └── 📄 sample_ticks.csv           # Small bundled LOBSTER-format sample (self-generated)
├── 📁 .github/workflows/
│   └── ⚙️ ci.yml
├── 📄 CMakeLists.txt
├── 📄 README.md
└── 📄 LICENSE

Note on scope beyond the original spec: FixedHashMap.hpp, NumaInfo.hpp, backtest/, multiprocess/, and util/ are small additions layered onto the original design as the implementation revealed genuine need for them (a NUMA-reporting class distinct from the arena, a dedicated backtest module, an isolated allocation-tracking utility). Every file above is real, compiling, non-stub code — nothing here is a placeholder.

---

🧩 OS Concepts Map

OS Concept	Where Used	What Was Measured	Result
Process & thread management	std::thread feed + strategy pipeline; optional fork() + POSIX shared memory (-DUSE_MULTIPROCESS)	Both variants built and run end to end; multiprocess demo moves 20,000 messages parent→child via shared memory	Both work; see Future Work for a formal fork() vs pthread_create() cost comparison
CPU scheduling	SCHED_FIFO/SCHED_RR via sched_setscheduler; sched_setaffinity/pthread_setaffinity_np	Semaphore ping-pong round-trip latency: unpinned vs. pinned vs. pinned+SCHED_FIFO	p50 ≈1.6–2.0 µs on this single-core box (⚠️ needs ≥2 cores for the intended effect — code detects and reports this)
Memory management	PoolAllocator<T> vs malloc/new; HugePageArena (MAP_HUGETLB); NumaInfo (libnuma)	Alloc/free cost; cold vs. pre-faulted access, regular vs. huge pages	Pool: 1.76 ns vs malloc 9.81 ns (5.6×). Huge pages: 6.3× faster cold, 11.8× pre-faulted
Synchronization / concurrency	SPSCRingBuffer (lock-free, alignas(64)-padded) vs MutexQueue; UnpaddedSPSCRingBuffer false-sharing twin	Uncontended push+pop cost; padded vs. unpadded cross-thread throughput	Lock-free: 1.05 ns vs mutex 38.4 ns (36.6×). False-sharing demo needs ≥2 cores to show separation here
I/O and interrupts	Blocking read() on a pipe vs. busy-poll on SPSCRingBuffer	Per-message wait latency (bench_io_blocking)	read(): 232 ns p50 vs busy-poll: 21 ns p50 (~11×)
Inter-process communication	POSIX shared memory (shm_open+mmap) carrying an SPSCRingBuffer between a forked feed process and a strategy process	Functional correctness of the multiprocess demo	20,000 messages transferred correctly; not yet a dedicated cross-process latency benchmark (see Future Work)

---

📊 Alpha Signal Methodology

Order Flow Imbalance (primary signal)

Based on Cont, Kukanov & Stoikov (2014), "The Price Impact of Order Book Events", Journal of Financial Econometrics. At each book-changing event n, define the bid- and ask-side contributions from the change in the best quote versus the previous event:

delta_bid_n = q_bid_n                if P_bid_n  > P_bid_(n-1)
            = q_bid_n - q_bid_(n-1)  if P_bid_n == P_bid_(n-1)
            = -q_bid_(n-1)           if P_bid_n  < P_bid_(n-1)

delta_ask_n = -q_ask_(n-1)           if P_ask_n  > P_ask_(n-1)
            = q_ask_n - q_ask_(n-1)  if P_ask_n == P_ask_(n-1)
            = q_ask_n                if P_ask_n  < P_ask_(n-1)

e_n = delta_bid_n - delta_ask_n

OrderFlowImbalance reports a rolling sum of e_n over a configurable event window — positive means net buy-side pressure, negative net sell-side pressure. See include/alpha/OrderFlowImbalance.hpp for the full derivation in comments.

Multi-level weighted book imbalance

weighted_bid = Σ_i  decay^i * bid_qty_i
weighted_ask = Σ_i  decay^i * ask_qty_i
imbalance    = (weighted_bid - weighted_ask) / (weighted_bid + weighted_ask)     ∈ [-1, +1]

The touch (i=0) is weighted most heavily; deeper levels are geometrically discounted (decay, default 0.7), since resting liquidity further from the touch is real but less informative about imminent price movement — it can be pulled or re-priced before ever being reached.

Microprice momentum

The microprice ((P_bid·q_ask + P_ask·q_bid) / (q_bid+q_ask), leaning toward the side with less resting size) is tracked over a short lookback window via a fixed-size circular buffer; the signal reports microprice_now - microprice_(lookback events ago).

(Sample chart placeholder — run python3 scripts/plot_backtest_results.py ofi_backtest.csv --title OrderFlowImbalance after ./latencybook_demo to generate a real one from your own run.)

Honesty note on backtest numbers: the bundled SyntheticGenerator is a memoryless zero-intelligence Poisson model (see its header for the full derivation and citation) — it has no real informational structure, so hit rates near 50% and IC near 0 on synthetic data are the expected, correct result, not a bug. Point --lobster path/to/real_data.csv at a real LOBSTER dataset for a backtest that means something economically.

---

⚙️ Build & Run

Requirements

• CMake ≥ 3.20, a C++20 compiler (GCC ≥ 11 or Clang ≥ 14), Linux (the OS-specific code targets Linux syscalls/APIs throughout)
• No dependencies required for the core build — GoogleTest and Google Benchmark are fetched automatically via FetchContent only if you enable tests/benchmarks
• Optional: libnuma-dev for real NUMA topology reporting (degrades gracefully to "unavailable" without it); Python 3 + matplotlib for the plotting scripts

Basic build (synthetic data, no deps)

cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j"$(nproc)"
./build/latencybook_demo                       # runs with 200,000 synthetic messages
./build/latencybook_demo --messages 1000000    # more messages
./build/latencybook_demo --lobster data/sample_ticks.csv   # real(-format) data instead

Multiprocess (fork + shared memory) variant

cmake -S . -B build-mp -DCMAKE_BUILD_TYPE=Release -DLB_USE_MULTIPROCESS=ON
cmake --build build-mp -j"$(nproc)"
./build-mp/latencybook_demo --multiprocess

Verifying the zero-allocation claim yourself

cmake -S . -B build-alloc -DCMAKE_BUILD_TYPE=Debug -DLB_TRACK_ALLOCATIONS=ON
cmake --build build-alloc -j"$(nproc)"
./build-alloc/tests/lb_tests --gtest_filter="OrderBook.ProcessingIsAllocationFreeAfterWarmup"

Benchmarks

cmake -S . -B build -DLB_BUILD_BENCHMARKS=ON -DLB_BUILD_TESTS=OFF
cmake --build build -j"$(nproc)"
./scripts/run_benchmarks.sh build     # runs all 6, writes benchmark_results_<timestamp>.txt

All CMake options

Option	Default	Effect
LB_BUILD_TESTS	ON	Fetch GoogleTest, build tests/lb_tests
LB_BUILD_BENCHMARKS	ON	Fetch Google Benchmark, build benchmarks/bench_*
LB_USE_MULTIPROCESS	OFF	Build the fork()+shared-memory IPC variant
LB_USE_HUGEPAGES	ON	Attempt MAP_HUGETLB (runtime-falls-back if unavailable regardless)
LB_WARNINGS_AS_ERRORS	ON	-Werror on top of -Wall -Wextra -Wpedantic -Wshadow -Wconversion -Wsign-conversion
LB_ENABLE_TSAN	OFF	Build with -fsanitize=thread
LB_ENABLE_ASAN	OFF	Build with -fsanitize=address,undefined
LB_TRACK_ALLOCATIONS	OFF	Global operator new/delete override for allocation verification

---

🧪 Testing

cmake -S . -B build -DLB_BUILD_TESTS=ON && cmake --build build -j"$(nproc)"
ctest --test-dir build --output-on-failure

40 tests across 13 suites, all passing:

• OrderBook (8 tests): add/cancel/modify/trade against known expected states, price-range/duplicate/pool-exhaustion rejection, mid/microprice, L2 snapshot ordering, plus the isolated LB_TRACK_ALLOCATIONS-guarded proof that steady-state processing is allocation-free
• PoolAllocator / FixedHashMap (5 tests): capacity/exhaustion/freelist-reuse; collision survival across 1000 keys; tombstone accumulation being invisible to size() but caught by non_empty_fraction(), and cleared by compact()
• SPSCRingBuffer / MutexQueue (4 tests): single-threaded FIFO+capacity correctness, and a real two-thread concurrent producer/consumer stress test (200,000 items) run under ThreadSanitizer in CI
• Alpha signals (4 tests), BacktestHarness (3 tests), FeedHandler (4 tests: LOBSTER field mapping, missing-file handling, binary round-trip, bad-magic rejection), SyntheticGenerator (5 tests: determinism, timestamp monotonicity, message-type scope), RiskEngine / SimulatedExchange (7 tests: position/rate limits, queue-position-aware partial fills)

CI (.github/workflows/ci.yml) runs the full suite under plain Release, under LB_TRACK_ALLOCATIONS, under ThreadSanitizer, and under AddressSanitizer+UBSan, plus a smoke test of the multiprocess variant — all on every push.

---

🚀 Performance Optimization Journey

#	Baseline	Change	Measured result
1	std::map-based order book (considered)	Direct-indexed price ladder (price_ticks - min_price_ticks_ as array index)	O(1) top-of-book reads; O(1) amortized updates (documented O(range) worst case only when a side empties completely)
2	malloc/new for every order node	PoolAllocator<Order> — one arena allocation at construction, intrusive freelist thereafter	9.81 ns → 1.76 ns (5.6×) per alloc+free
3	std::unordered_map<order_id, Order*>	Custom FixedHashMap — one fixed-capacity vector, open addressing, no per-insert heap node	Removed the last hidden allocation source in add_order/cancel_order
4	(bug found during benchmarking) naive add+cancel-every-iteration benchmark measured 69,000 ns/cycle	Root-caused to FixedHashMap's documented tombstone accumulation: an ever-emptying book kept hitting OrderBook's own documented O(range) worst-case rescan. Fixed the benchmark to reflect realistic steady-state load (a permanently-populated book), and added compact_order_index() + non_empty_fraction() so this is diagnosable instead of mysterious	69,000 ns → 21.5 ns per add+cancel cycle once measuring the representative case
5	std::mutex + std::condition_variable queue between feed and strategy threads	Lock-free SPSCRingBuffer (single-producer/single-consumer, power-of-two capacity)	38.4 ns → 1.05 ns (36.6×) uncontended push+pop
6	Adjacent head_/tail_ atomics sharing a cache line	alignas(64) padding (kept UnpaddedSPSCRingBuffer as a permanent before/after twin)	Isolates the false-sharing cost as a standing, re-runnable comparison (needs ≥2 cores to show separation on top of the mutex-vs-lockfree win above)
7	Regular 4KB pages for a large order-book arena	MAP_HUGETLB 2MB huge pages, with automatic runtime fallback if unavailable	6.3× faster cold first-touch, 11.8× faster once pre-faulted (pure TLB-miss reduction)
8	Blocking read() for message ingestion	Busy-poll SPSCRingBuffer::try_pop()	232 ns → 21 ns (~11×) per-message wait, at the cost of a fully spinning core
9	"We don't allocate on the hot path" as an unverified comment	Global operator new/delete override (LB_TRACK_ALLOCATIONS) + a GoogleTest assertion scoped tightly around the steady-state loop	0 allocations, machine-verified, not asserted by hand-waving
10	Default (SCHED_OTHER) scheduling and no core affinity for the strategy thread	sched_setaffinity pinning + SCHED_FIFO wrappers, with a runtime guard that disables both on a single-core host (a busy-polling real-time thread there would starve everything else rather than help)	Implemented and correctness-tested; the jitter-reduction benefit needs ≥2 cores to observe (see Key Results caveat)

---

🔮 Future Work

• Kernel-bypass networking (DPDK / Solarflare OpenOnload): real HFT feed handlers avoid the kernel network stack entirely — user-space NIC polling eliminates interrupt and syscall overhead this project's read() vs busy-poll comparison only approximates with a pipe. Not implemented here; a natural next step once real exchange connectivity exists.
• Multi-symbol support: the current OrderBook is single-instrument; a multi-symbol engine needs per-symbol books behind a dispatch layer plus cross-symbol risk aggregation in RiskEngine.
• Exchange connectivity (FIX / ITCH): FeedHandler currently parses offline LOBSTER files and a custom binary format; a real deployment needs a live ITCH or FIX session decoder feeding the same OrderMessage struct.
• NUMA-aware multi-socket scaling: NumaInfo detects topology and degrades gracefully today; a multi-socket machine would let this project actually implement NUMA-local allocation (numa_alloc_onnode) for the order book arena and measure the cross-node access penalty directly, not just report node count.
• Formal fork() vs pthread_create() cost comparison: the multiprocess variant is functionally verified but doesn't yet have a dedicated benchmark quantifying process-creation and cross-process wakeup cost against the thread-based pipeline's numbers.
• Backward-shift deletion for FixedHashMap: would remove the tombstone-accumulation tradeoff entirely (see compact()'s header comment) at the cost of more intricate deletion logic.
• Price-changing modify: currently modeled as cancel+add at the feed-handler level, matching every venue's actual wire semantics; a unified in-place price-and-quantity modify could shave one hash-map round-trip.

---

📚 References

• Cont, R., Kukanov, A., & Stoikov, S. (2014). The Price Impact of Order Book Events. Journal of Financial Econometrics, 12(1), 47–88. — Order Flow Imbalance signal.
• Cont, R., Stoikov, S., & Talreja, R. (2010). A Stochastic Model for Order Book Dynamics. Operations Research, 58(3), 549–563. — Basis for SyntheticGenerator's superposed-Poisson zero-intelligence model.
• Huang, W., & Polak, T. (2011). LOBSTER: Limit Order Book Reconstruction System. — Message-file format FeedHandler::load_lobster_csv parses. Dataset: lobsterdata.com.
• Tene, G. — HdrHistogram. — Design inspiration (simplified in this project) for LatencyHistogram's log-scale bucketing scheme.
• Drepper, U. (2007). What Every Programmer Should Know About Memory. — Background for the huge-pages/TLB and cache-line/false-sharing sections.
• Data used: the bundled data/sample_ticks.csv is self-generated by this project's own SyntheticGenerator (see the file header) — it is not third-party data, and is included purely so the LOBSTER-format code path has something to parse out of the box in addition to the always-available synthetic fallback.

---

📄 License

MIT — see LICENSE.