Overview

CopperDB is a simple b+tree database built in Rust. It was born from

• Reading the book "Database Internals" by Alex Petrov
• Needing a pet-project to learn Rust

Near the beggining of the year I bought the book "Database Internals" after finishing "Designing Data Intensive Application" (the distributed systems "bible"). The main motivation was to peer more into LSM Tree implementations as I was interested in being able to speak about the difference between using Cassandra or Postgres and why one might be better for write intesive applications. However, as I started the book, I realized how little I actually really knew/understood BTrees and their many offshoots.

Around the same time, my Medium feed was being filled with "Rust is Amazing" blog posts. Just as any good sheep would (baa), my interest started peaking and I wanted to dive in (head first). Naturally, I realized that I had two short comings that could serve eachother. "Why not try to build a b+tree backed data base in Rust!?". Another aspect of the times that aided in this venture were LLMs. Likley, I might have started smaller or given up after trying to binary encode an in memory page to disk (thanks bincode), but with LLMs, its really the best time ever to learn new things in a more seamless way. Thus, CopperDB was born.

Features

The feature set for Copper started (and remains) very simple.

• A b+tree implementation
• Supports split and merge operations to accomodate insert and deletes
• Alows for table (row based) or K/V pair formats - only supports simple value types
• Persists tables per data file to binary using bincode
• API operations are implemented via gRPC. And CLI client has been started to talk to the DB server

B+tree Specific Features

Through building this, Ive been able to come across quite a few interesting opinions and sources around database implementations. Ive taken the following appraoches

• Use slotted cell approach - This was explained in the "Database Internals" book and I quite liked the idea.
• No mmap-ing - A few sources Ive come across talk about the (controversial) antipattern of attempting to mmap memory to disk due to clashes with the kernel. What Ive come to learn is, "the kernel is not your friend when implementing databases." Despite this, many large and production worthy projects use mmap.
• Iterative approach to split/merge - As opposed to recursive. This was an artifact of Rust specifically and might be artificial (due to my lack of depth with Rust). I was unable to borrow mutable self multiple times (through a recursive call stack). My design might also be flawed shrug.
• Rightmost Pointer - This is an optimization for the b+tree internals.

Roadmap

Its highly possible little to none of these things will actually get implemented in any significant capacity, but I wanted to note these down anyway

• WAL - some sort of write ahead log to better account for unplanned restarts. This will also improve the current checkpointing logic.
• Compaction - The database allows for both inserts and deletes and supports the slotted page approach. With that, eventually gaps exist within the disk layout. Some form of vaccum/cleanup would be interesting.
• Compression - possibly some form of compression on disk to eek out more performance.
• Interactive CLI - running commands one-by-one is not great. It would be sleek to be able to enter into a session or run scripts.
• Performance/Metrics - One main goal was to try to realize why ppl like Rust. In this, it would be cool to do some performance and implement metrics for fruther analysis. It may also be cool to explore Linux kernel stuff via perf or eBPF.
• Concurrency control / Transactions / Locking / Latching

References and Inspiration

• "Database Internals" by Alex Petrov - As I previously mentioned this book was a huge driver/inspiration here
• WiredTiger - I spent hours/days reading the source code. This is MongoDB's data layer. It is implemented in C and is a feature rich B+tree implementation. As an aside, it is interesting that even though B+trees are typically associated with SQL based projects, it provides the data layer for MongoDB (NoSQL). What I later learned more is the distinguishing factor isnt so much SQL vs NoSQL, but read vs write intensive workloads and how an implementation impacts this.
• CMU Intro to Database Systems - This YouTube series (through Carnegie Mellon) is amazing! The instructor has a great precense (and sense of humor). I really enjoyed the lectures.