A lightweight, persistent key-value database built from scratch in C++

Redis-inspired · event-driven · cross-platform · containerized

---

About

ByteForge is a Redis-inspired key-value store that supports basic CRUD operations over a TCP connection. It is designed around a phonebook use case where string keys map to string values.

This is a personal learning project. The goal is not to build a production database, but to understand how databases, servers, networking, and concurrency actually work under the hood.

The server runs cross-platform on both Windows and Linux using platform-specific networking abstractions.

---

Table of Contents

• Features
• Commands
• Architecture
• Concurrency Model
• Rate Limiting
• Getting Started
• Build Locally
• Run With Docker
• Project Structure
• Roadmap
• License

---

Features

Feature	Description
🧩 Custom HashMap	Hand-rolled hash table with separate chaining for collision resolution and dynamic growth, protected by a std::shared_mutex for concurrent read/write safety
⚡ Event-Driven TCP Server	Readiness-based event loop supporting many concurrent connections without one thread per client — epoll on Linux, IOCP on Windows (adapted via zero-byte WSARecv), both behind a shared IEventLoop interface selected automatically at compile time
🧵 Thread Pool	Fixed pool of 3 worker threads executing actual command work (GET/INSERT/DELETE), decoupled from connection count
📝 Command Parser	Parses INSERT, GET, and DELETE commands from raw TCP bytes
🕒 Snapshot Scheduler	Background thread that automatically triggers RDB snapshots on a fixed interval
💾 Hybrid Persistence	AOF logs every write in real time; RDB snapshots dump the full database every 5 minutes; AOF resets after each snapshot
🚦 Rate Limiting	Per-IP and global request throttling using the Token Bucket algorithm
🛑 Graceful Shutdown	Type stop to cleanly flush data and join all threads
🐳 Docker Support	Multi-stage containerized build targeting Linux
🔐 Authentication	In progress

---

Commands

Connect to the server using ncat or any TCP client:

ncat 127.0.0.1 6625

Command	Description	Example
INSERT key value	Inserts or updates a key-value pair	INSERT Ahmed 51020651
GET key	Retrieves the value for a key	GET Ahmed
DELETE key	Removes a key-value pair	DELETE Ahmed

---

Architecture

Client (ncat / custom client)
        │
        │ TCP
        ▼
 Cross-Platform TCP Server
 (Winsock2 / Linux sockets)
        │
        ▼
 Event Loop (readiness-based)
 epoll (Linux) / IOCP (Windows)
 behind a shared IEventLoop interface
        │  watches all client sockets;
        │  reports which ones are ready
        ▼
        ├──▶ Rate Limiter (Token Bucket)
        │    per-IP + global request cap
        │         │
        │         ▼
        │    Command Parser
        │         │
        │         ▼
        │    Thread Pool (3 workers)
        │    executes actual GET/INSERT/DELETE
        │         │
        │         ▼
        │    HashMap (in-memory store)
        │    shared_mutex: readers run concurrently,
        │    writers get exclusive access
        │
        └──▶ Persistence Layer
                  ├── appendonly.log  (real-time AOF log)
                  └── snapshot.log    (periodic RDB snapshot)
                            ▲
                   SnapshotScheduler
                   (background thread, every 5 min)

---

Concurrency Model

Component	Protection	Strategy
Event Loop	Single dedicated thread	Watches all sockets for readiness; never blocks on I/O
Client jobs	ThreadPool + std::condition_variable	Only dispatched once a socket is actually readable
Connections	busySockets guard (std::mutex)	Prevents duplicate recv() jobs for the same socket
HashMap	std::shared_mutex	Multiple readers, exclusive writers
AOF stream	std::mutex	Single writer at a time
Snapshot	SnapshotScheduler thread	Sleeps on interval, wakes on shutdown
Rate limiter	std::mutex per map + global	Per-IP and global window isolated

---

Rate Limiting

ByteForge uses the Token Bucket algorithm for rate limiting — the same approach used by Stripe, GitHub, and AWS.

• Each IP gets a bucket of tokens (default: 10)
• Each request consumes one token
• Tokens refill at a fixed rate (default: 5/sec)
• A global cap limits total requests per second across all IPs (default: 1000)
• Blocked requests are dropped immediately without consuming a worker thread

---

Getting Started

Prerequisites

• C++17 compiler
• CMake 3.15+
• Docker (optional)
• nmap/ncat for testing

For local builds:

• Windows: MSVC / MinGW with Winsock2
• Linux: GCC / Clang with POSIX sockets

---

Build Locally

cmake -S . -B build
cmake --build build

Run:

./build/Debug/KV_Database.exe   # Windows
./build/KV_Database             # Linux

The server starts on port 6625 by default. Type stop to shut it down cleanly.

---

Run With Docker

Build the image:

docker build -t byteforce:latest .

Run the container:

docker run -d -p 6625:6625 --name my-byteforce-db byteforce-db:latest

Connect using:

ncat 127.0.0.1 6625

---

Project Structure

kv-db/
├── src/
│   ├── main.cpp
│   ├── Server/
│   │   ├── Server.h
│   │   └── Server.cpp
│   ├── Networking/
│   │   ├── NetworkTypes.h
│   │   ├── EventLoop.h
│   │   ├── EpollEventLoop.h / .cpp
│   │   ├── IocpEventLoop.h / .cpp
│   │   └── EventLoopFactory.h
│   ├── UserSession/
│   │   ├── UserSession.h / .cpp
│   │   └── Connection.h
│   ├── RAM/
│   │   ├── HashMap.h
│   │   └── HashMap.cpp
│   ├── Storage/
│   │   ├── Persistence.h
│   │   └── Persistence.cpp
│   ├── Worker/
│   │   ├── ThreadPool.h
│   │   ├── ThreadPool.cpp
│   │   ├── SnapshotScheduler.h
│   │   └── SnapshotScheduler.cpp
│   ├── Limit/
│   │   ├── Limiter.h
│   │   └── Limiter.cpp
│   └── Tests/
│       ├── test_hashmap.cpp
│       └── test_threadpool.cpp
├── Dockerfile
├── .dockerignore
├── CMakeLists.txt
└── README.md

---

Roadmap

• Custom HashMap with separate chaining
• TCP server with Winsock2
• Cross-platform networking support (Windows/Linux)
• Command parser (INSERT, GET, DELETE)
• AOF persistence with real-time logging
• RDB snapshots every 5 minutes
• AOF reset after each snapshot
• Thread pool for concurrent client sessions
• HashMap thread safety with shared_mutex
• Persistence thread safety with mutex
• Snapshot scheduler background thread
• Clean server shutdown
• Data recovery on restart (RDB + AOF replay)
• Unit tests with GoogleTest (HashMap + ThreadPool)
• Docker containerization (multi-stage Linux build)
• Rate limiting with Token Bucket algorithm
• Cross-platform event-driven architecture (epoll / IOCP)
• Route idle-session socket cleanup through Server's connection teardown path
• TCP fragmentation / message framing (LineBuffer)
• Authentication (username + password)

---

License

This project is licensed under the MIT License. See LICENSE for details.

---

Built from scratch as a self-learning project to understand C++, networking, persistence, and systems programming.