Writing an LSM-tree in Go

Most databases you reach for to handle a firehose of writes (Cassandra, RocksDB, LevelDB, ScyllaDB) are LSM-trees underneath. I built one from scratch in Go because reading about compaction and implementing compaction are very different amounts of understanding.

Here’s the core idea, and the trade-off it’s built around.

The trick: never write randomly

Disks (and SSDs) hate random writes. A B-tree updates data in place, which means seeking all over the file. An LSM-tree refuses to do that. Every write is appended to a write-ahead log for durability, and inserted into an in-memory sorted structure, the memtable. When the memtable fills, it’s flushed to disk sequentially as an immutable SSTable (sorted string table).

You never modify an SSTable; you only ever write new ones.

func (t *Tree) Put(key, val []byte) error {
    if err := t.wal.Append(key, val); err != nil { // durability first
        return err
    }
    t.mem.Insert(key, val)                          // sorted in-memory
    if t.mem.Size() >= t.threshold {
        t.flush()                                   // -> new SSTable on disk
    }
    return nil
}

Writes are now sequential and fast. But we’ve pushed the cost somewhere: reads.

Reads pay the bill

A key might be in the memtable, or in any of the SSTables on disk, newest first. A naive read checks all of them, and that’s a lot of disk I/O. Two structures rescue it.

• Bloom filters: a tiny probabilistic set per SSTable. Ask “might this key be here?” and get back definitely not or maybe. Most SSTables answer definitely not in memory, so you never touch their disk blocks.
• Sparse block index: SSTables are stored as sorted, prefix-compressed blocks; a small in-memory index jumps you to the right block.

A Bloom filter can say “maybe” when the answer is “no” (a false positive), but it never says “no” when the answer is “yes.” That asymmetry is exactly what a read path wants.

The WAL and crash recovery

The memtable lives in RAM, so a crash between a write and the next flush would lose data, except that the write-ahead log already has it. On startup, the engine replays the WAL to rebuild the memtable exactly as it was, then discards the log segments whose data has since been flushed to SSTables.

The correctness detail that bites everyone: the WAL append has to be durable before you acknowledge the write (an fsync, or a batched group commit), and each record needs a checksum so a torn write from a mid-crash append is detected and truncated on replay rather than silently corrupting the memtable. Get the ordering wrong and you have a database that’s fast and occasionally lying.

Compaction: the real trade-off

SSTables pile up. Old versions of keys and tombstones (deletes) accumulate. Compaction merges SSTables in the background, dropping shadowed values, reclaiming space and keeping reads fast.

The strategy you pick is the write-vs-read-vs-space trade-off:

Strategy	Optimises for	Cost
Leveled	reads + space	higher write amplification
Size-tiered	writes	more space, worse reads
Time-window	time-series	assumes append-mostly data

I made all three live-switchable, which turned an abstract table like the one above into something I could actually feel by watching write amplification move.

Where it got interesting

The single-node engine was the warm-up. Adding Raft for quorum replication, and then multi-Raft to shard the keyspace across groups, is where “I read about this” became “I understand this.” Consensus has a way of punishing every fuzzy assumption you made about ordering and failure.

The full engine is on GitHub. Next I’ll break down the on-disk SSTable format, block by block.