RocksDB tuning: block cache, bloom filters, write stalls

What it is

RocksDB is an embedded LSM key-value store: a library you link into your process, not a server you connect to. That framing matters because it changes what tuning means. There is no DBA, no query planner, no separate machine to blame. RocksDB's performance is a property of your process's memory and disk budget, and the knobs are yours.

You meet it more often than you might expect. It backs Kafka Streams state stores, Flink's EmbeddedRocksDBStateBackend (see Flink state and checkpoints), MyRocks under MySQL, CockroachDB's older storage layer, TiKV, and a large number of in-house services that needed an ordered on-disk map.

Three subsystems account for nearly all real tuning:

Subsystem	Governs	The failure it causes when wrong
Block cache	Read path memory	Every read hits disk; p99 latency 10-100x worse
Bloom filters	Skipping files on point reads	Reads check every level; read amplification
Write buffer + compaction	Write path and background merging	Write stalls: the engine deliberately stops accepting writes

What it is confused with: RocksDB's memory usage is not block_cache_size. The total is block cache plus memtables plus index and filter blocks plus pinned metadata, and the last two are frequently larger than people expect. Sizing a container from the block cache alone is the standard route to an out-of-memory kill, and it is covered below.

The problem it solves

The general problem is the LSM one (see LSM trees vs B-trees) and the specific problem RocksDB tuning addresses is that the defaults are conservative and workload-neutral, which for an embedded database means they are wrong for almost every specific deployment.

Concretely, three defaults cause most trouble:

block_cache_size defaults to 8 MB. On a machine with 64 GB of RAM serving a 200 GB dataset, that is close to no cache at all, so every read that misses the OS page cache is a disk read plus a decompression.

Bloom filters are off by default in the base configuration (BlockBasedTableOptions with no filter policy). Without them, a point read for a key that does not exist must check every level, which for a 7-level tree is 7 index lookups and up to 7 block reads to return "not found."

Write stalls are on by default and are invisible until they fire. RocksDB will deliberately slow or stop writes when compaction falls behind, and the application sees a Put() that takes 4 seconds with no error. Teams report this as "the database randomly hangs."

Mechanics

The block cache

The block cache holds uncompressed data blocks (and optionally index and filter blocks) in memory. It sits above the OS page cache, which holds compressed blocks, so you have two caches with different contents.

BlockBasedTableOptions table_options;
table_options.block_cache = NewLRUCache(8ULL << 30);       // 8 GB, NOT the 8 MB default
table_options.cache_index_and_filter_blocks = true;         // count them against the budget
table_options.pin_l0_filter_and_index_blocks_in_cache = true;
table_options.block_size = 16 * 1024;                       // default 4 KB

options.table_factory.reset(NewBlockBasedTableFactory(table_options));

cache_index_and_filter_blocks = true is the setting that prevents surprise OOMs, and it is worth understanding why. By default, index and filter blocks live outside the block cache and grow without bound with the dataset. Setting this to true puts them inside the budget, so total memory is bounded by block_cache_size rather than by the data size. The cost is that index and filter blocks can now be evicted, which is why you pin the L0 ones (they are checked on every read).

Sizing index and filter blocks is arithmetic worth doing:

100 million keys, 10 bits/key bloom filter:
  filter blocks   = 100e6 x 10 bits / 8   = 125 MB
  index blocks    ≈ one entry per data block
                  = (200 GB / 16 KB) x ~40 bytes ≈ 500 MB
                                                  ────────
  metadata total  ≈ 625 MB, before a single data block is cached

block_size is the other lever with a real trade. Larger blocks mean fewer index entries (less metadata memory) and better compression, and each read pulls more bytes from disk. 16 KB to 32 KB suits scan-heavy workloads; 4 KB to 8 KB suits point lookups. The default 4 KB is the point-lookup end.

Bloom filters

// 10 bits per key: ~1% false positive rate. The standard choice.
table_options.filter_policy.reset(NewBloomFilterPolicy(10, false));
//                                                         ^ false = full filter
//                                                           (per-file, not per-block)

The false-positive rate against bits per key:

Bits/key	False positive	Memory for 100M keys
6	5.6%	75 MB
10	1.0%	125 MB
16	0.05%	200 MB
20	0.01%	250 MB

10 is the standard because the curve flattens: going from 10 to 20 halves nothing that matters while doubling memory. Going below 10 degrades quickly.

Ribbon filters (RocksDB 6.15+) give the same false-positive rate for roughly 30 percent less memory, at higher CPU cost to construct. Use them on lower levels where data is cold and memory matters, and bloom on the hot upper levels:

// Ribbon below L2, bloom above: memory saving where it is worth the CPU.
table_options.filter_policy.reset(NewRibbonFilterPolicy(10, /*bloom_before_level=*/2));

The limitation, which is the same one as on the LSM page: filters answer "is key K present" and therefore do nothing for range scans. prefix_extractor with prefix bloom filters partially addresses this when your scans share a prefix:

// If keys are "userid:timestamp" and you scan by userid, a prefix filter helps.
options.prefix_extractor.reset(NewFixedPrefixTransform(8));
table_options.whole_key_filtering = true;   // keep whole-key filters too

Write stalls: the failure that looks like a hang

RocksDB throttles or stops writes when it judges that continuing would make reads unusable. There are four independent triggers and the diagnosis differs for each.

// 1. Too many L0 files (L0 files overlap, so every read checks all of them)
options.level0_slowdown_writes_trigger = 20;    // start throttling
options.level0_stop_writes_trigger     = 36;    // stop entirely

// 2. Too many immutable memtables awaiting flush
options.max_write_buffer_number = 6;            // stall when all are full

// 3. Pending compaction bytes too large
options.soft_pending_compaction_bytes_limit = 64ULL << 30;    // 64 GB: throttle
options.hard_pending_compaction_bytes_limit = 256ULL << 30;   // 256 GB: stop

// 4. Estimated space amplification exceeded (universal compaction only)

Each trigger names its cause, which is the useful part:

L0 file count: flushes are outpacing L0-to-L1 compaction. Not enough compaction throughput, or the memtable is too small so flushes are too frequent.
Immutable memtable count: flush cannot keep up with writes. Usually disk-bound, or max_background_flushes too low.
Pending compaction bytes: compaction as a whole is behind. Usually the write rate exceeds what the disk can sustain at this write amplification.

The stall is reported in db->GetProperty("rocksdb.is-write-stopped") and in the STALL lines of the LOG file, and it must be a monitored metric, because from the application's perspective a stall is indistinguishable from a slow disk:

// Kafka Streams / Flink: expose these or you are flying blind.
long stalled = db.getLongProperty("rocksdb.is-write-stopped");
long l0Files = db.getLongProperty("rocksdb.num-files-at-level0");
long pending = db.getLongProperty("rocksdb.estimate-pending-compaction-bytes");

The levers, in the order I would try them:

// A. More compaction throughput. The first thing to check.
options.max_background_jobs = 8;                 // was 2 by default in older versions
options.max_subcompactions   = 4;                // parallelise ONE compaction job

// B. Bigger memtables: fewer, larger flushes, less L0 churn.
options.write_buffer_size = 256 << 20;           // 256 MB, default 64 MB
options.max_write_buffer_number = 6;
options.min_write_buffer_number_to_merge = 2;    // merge 2 before flushing

// C. Bigger L1, so the L0->L1 ratio is sane.
options.max_bytes_for_level_base = 1ULL << 30;   // 1 GB, default 256 MB
options.target_file_size_base = 128 << 20;       // 128 MB SST files

// D. Rate-limit compaction so it does not starve foreground reads.
options.rate_limiter.reset(NewGenericRateLimiter(200 << 20));   // 200 MB/s

max_bytes_for_level_base deserves a note, because it is the one people get wrong. RocksDB compacts L0 into L1, and if L1 is small relative to the L0 file size, that merge rewrites L1 constantly. The guidance is for max_bytes_for_level_base to be roughly write_buffer_size * min_write_buffer_number_to_merge * level0_file_num_compaction_trigger, so L1 can absorb an L0 batch without thrashing.

Compression, per level

options.compression_per_level = {
    kNoCompression,     // L0: hot, compaction rewrites it constantly
    kNoCompression,     // L1: same
    kLZ4Compression,    // L2+: good ratio, very fast
    kLZ4Compression,
    kZSTD,              // deepest levels: cold, best ratio, slower is fine
};
options.bottommost_compression = kZSTD;

The top levels are rewritten repeatedly by compaction, so compressing them costs CPU repeatedly for data that will be rewritten soon. The bottom level holds most of the data and is rarely rewritten, so the best ratio pays off. This one setting commonly cuts disk usage by 40 to 60 percent against uniform LZ4, at negligible cost.

A worked example: a Kafka Streams job that stalled every four hours

A Kafka Streams application maintaining a session store: 180 million keys, roughly 400 bytes each (about 72 GB of user data), 240,000 updates per second, running on 12 instances with 32 GB RAM and 2 TB NVMe each.

Symptoms:

Every ~4 hours, on one instance at a time:
  - processing latency p99: 40ms -> 9s
  - consumer lag grows to ~2M messages
  - after 3-8 minutes it recovers on its own
  - no errors, no exceptions, nothing in the application log

Roughly every four hours the instance appeared to hang and then recovered. Kafka's own metrics showed the consumer alive and heartbeating, so it was not a rebalance (see consumer rebalancing). The team had spent two weeks on Kafka configuration.

The diagnosis came from exposing RocksDB properties, which Kafka Streams does not expose by default:

public class StoreMetricsListener implements RocksDBConfigSetter {
    @Override
    public void setConfig(String storeName, Options options, Map<String, Object> cfg) {
        // ... plus a scheduled task reading properties every 10s:
        //   rocksdb.is-write-stopped
        //   rocksdb.num-files-at-level0
        //   rocksdb.estimate-pending-compaction-bytes
        //   rocksdb.actual-delayed-write-rate
    }
}

During an incident:
  rocksdb.is-write-stopped:              1        <- there it is
  rocksdb.num-files-at-level0:           41       <- trigger is 36
  rocksdb.actual-delayed-write-rate:     2,097,152  (2 MB/s, from unlimited)
  rocksdb.estimate-pending-compaction-bytes: 88 GB

The engine was deliberately stopping writes, and the four-hour period was how long it took L0 to back up from a clean state at that write rate.

The root causes were three defaults compounding:

write_buffer_size:              64 MB   (Kafka Streams default)
max_background_jobs:            2       (RocksDB default at that version)
max_bytes_for_level_base:       256 MB
level0_file_num_compaction_trigger: 4

At 240,000 updates per second across 12 instances, each instance wrote about 8 MB/s of user data, flushing a 64 MB memtable roughly every 8 seconds. Two background jobs had to handle both flushes and compaction. L0-to-L1 compaction merged 4 files of 64 MB into a 256 MB L1, meaning it rewrote the entire L1 on every L0 compaction, because L1 was exactly the size of one L0 batch.

The changes:

@Override
public void setConfig(String storeName, Options options, Map<String, Object> cfg) {
    BlockBasedTableConfig table = new BlockBasedTableConfig();

    // Shared cache across all stores on this instance: bound total memory ONCE.
    table.setBlockCache(SHARED_CACHE);              // 8 GB LRU, static
    table.setCacheIndexAndFilterBlocks(true);       // metadata inside the budget
    table.setPinL0FilterAndIndexBlocksInCache(true);
    table.setFilterPolicy(new BloomFilter(10, false));
    table.setBlockSize(16 * 1024);
    options.setTableFormatConfig(table);

    // Fewer, larger flushes.
    options.setWriteBufferSize(256 * 1024 * 1024);
    options.setMaxWriteBufferNumber(4);
    options.setMinWriteBufferNumberToMerge(2);

    // L1 sized to absorb an L0 batch: 256 MB x 2 x 4 = 2 GB.
    options.setMaxBytesForLevelBase(2L * 1024 * 1024 * 1024);
    options.setTargetFileSizeBase(128 * 1024 * 1024);

    // Enough background capacity to keep up.
    options.setMaxBackgroundJobs(6);
    options.setMaxSubcompactions(2);

    // Compression where it pays.
    options.setCompressionType(CompressionType.LZ4_COMPRESSION);
    options.setBottommostCompressionType(CompressionType.ZSTD_COMPRESSION);
}

The shared block cache across all stores on an instance is the memory-safety change. Kafka Streams creates one RocksDB instance per store per partition, so an instance owning 8 partitions with 3 stores has 24 RocksDB instances. With per-instance caches, memory is 24 x cache_size and unpredictable as partitions move during rebalancing. One static shared cache bounds it regardless of assignment.

Measured over the following month:

                                  before        after
write stalls per day              ~6            0
p99 processing latency            9,000ms       31ms
p99 (steady state, no stall)      40ms          31ms
consumer lag (peak)               2.1M          negligible
disk used per instance            310 GB        128 GB    (compression)
RSS per instance                  27-31 GB      19 GB     (bounded, predictable)
L0 file count (steady)            18-41         2-5
compaction I/O                    ~180 MB/s     ~95 MB/s

Zero stalls, and memory became predictable, which was the second win. The instances had been running at 27 to 31 GB of a 32 GB limit and were occasionally OOM-killed, which the team had attributed to a leak. There was no leak: index and filter blocks lived outside the cache budget and grew with the dataset, and per-store caches multiplied with partition count.

The number worth carrying: two weeks were spent on Kafka configuration for a problem that was entirely inside RocksDB, and the diagnosis took one metric. Nothing in the application or Kafka logs mentioned RocksDB. rocksdb.is-write-stopped is a boolean that says exactly what is happening, and it was not exposed.

Production evidence

RocksDB was built at Facebook from LevelDB for server workloads, and the tuning guide on the RocksDB wiki is written by the team that operates it under MyRocks, their MySQL storage engine. Their published MyRocks comparisons against InnoDB (roughly half the storage, substantially lower write amplification) are the clearest production numbers available.

The RocksDB wiki explicitly warns that the default options are not tuned for performance and directs readers to the tuning guide, which is unusual candour for project documentation and reflects that an embedded database cannot guess its workload.

Kafka Streams exposes RocksDBConfigSetter precisely because the defaults do not suit every state store, and Confluent's documentation on memory management recommends a shared block cache across stores for exactly the bounding reason above.

Flink's EmbeddedRocksDBStateBackend ships predefined option sets (SPINNING_DISK_OPTIMIZED, SPINNING_DISK_OPTIMIZED_HIGH_MEM, FLASH_SSD_OPTIMIZED) because the Flink team concluded that most users would not tune RocksDB themselves and needed reasonable starting points per hardware profile.

CockroachDB moved off RocksDB to Pebble, their own Go reimplementation, and their published reasoning is instructive: they wanted control over the exact feature set and to avoid the CGo boundary, not because RocksDB was slow. Pebble implements the same LSM design and much of the same tuning surface.

The debate

Should you tune RocksDB at all, or use a preset? For Flink, start with a predefined option set matched to your hardware and tune from there; for Kafka Streams, you have to write a RocksDBConfigSetter because there is no equivalent preset, and the defaults are demonstrably wrong for large state. My position: always set the block cache, always enable bloom filters, always set cache_index_and_filter_blocks, and always expose is-write-stopped as a metric. Those four are close to universally correct. Beyond that, tune when you have a measured problem.

Block cache versus OS page cache. They hold different things: block cache holds uncompressed blocks, page cache holds compressed ones. A large block cache gives faster hits (no decompression) and less effective total caching (uncompressed data is bigger). A small block cache leaves more RAM for the page cache, which caches more data in less space. The RocksDB guidance, which I would follow, is roughly one third of available memory to the block cache and let the OS use the rest, unless you have measured that decompression CPU is a bottleneck.

Is cache_index_and_filter_blocks = true always right? It bounds total memory, which is what you want in a container with a hard limit, and it makes index and filter blocks evictable, so a read can now pay to reload an index block. In a container, memory predictability is worth more than that tail latency, and being OOM-killed is worse than any latency. Outside a container with generous RAM, leaving it false is defensible. In Kubernetes it should be true, and pinning L0 filters and indexes recovers most of the latency cost.

Should you rate-limit compaction? Yes, if read latency matters. Unrestricted compaction saturates disk bandwidth in bursts, and foreground reads queue behind it, so p99 read latency correlates with compaction activity. A rate limiter smooths it at the cost of compaction taking longer, which raises the risk of falling behind. The honest framing is that you are choosing between predictable moderate latency and mostly-good latency with periodic spikes, and for anything user-facing the first is better.

When is RocksDB the wrong choice? When you need range scans as your primary access pattern and the data is not naturally clustered, since that is the LSM weakness and no tuning removes it. When you need multi-key transactions with real isolation, because RocksDB's transaction support is limited compared to a proper database. And when your working set fits comfortably in memory, where a simpler in-memory structure with a write-ahead log is less machinery for the same result.

Follow-up Q&A

"Your service using RocksDB has periodic multi-second latency spikes with no errors. What do you check?"

rocksdb.is-write-stopped first, because that is a boolean saying exactly whether the engine is deliberately refusing writes, and a stall is indistinguishable from a hang from outside. Then which trigger: num-files-at-level0 against level0_stop_writes_trigger, the immutable memtable count, and estimate-pending-compaction-bytes against the hard limit. Each names its cause. If it is L0 backup, the levers are compaction throughput, a bigger memtable so flushes are less frequent, and max_bytes_for_level_base so L1 can absorb an L0 batch without rewriting itself.

"How much memory does RocksDB actually use?"

Block cache, plus memtables (write_buffer_size * max_write_buffer_number, per instance), plus index and filter blocks, plus table reader overhead. Index and filter blocks are the ones that surprise people: they default to living outside the block cache and growing with the dataset, so 100 million keys can be 600 MB of metadata alone. Setting cache_index_and_filter_blocks = true brings them inside the budget. And in Kafka Streams or Flink, remember there is one RocksDB instance per store per partition, so a shared cache is how you bound the total rather than multiplying.

"How many bits per key for a bloom filter?"

10, giving about a 1 percent false-positive rate, and the curve flattens fast: 20 bits gets you 0.01 percent for double the memory, which is rarely worth it, while 6 bits gives 5.6 percent, which starts costing real disk reads. Use ribbon filters on the lower levels if memory is tight: same false-positive rate for roughly 30 percent less memory, at more CPU to build, which is fine for cold data.

"Why does compression per level matter?"

The top levels are rewritten constantly by compaction, so compressing them spends CPU repeatedly on data that is about to be rewritten. The bottom level holds most of the data and is rarely rewritten, so the best ratio pays off once and keeps paying. No compression at L0 and L1, LZ4 in the middle, ZSTD at the bottom is the standard shape and it commonly cuts disk 40 to 60 percent against uniform LZ4.

"What does max_bytes_for_level_base do and why does it matter?"

It is L1's target size, and every level below is 10x the one above. If L1 is small relative to the L0 batch being merged into it, every L0-to-L1 compaction rewrites essentially all of L1, which is a large multiplier on write amplification. Size it as roughly write_buffer_size * min_write_buffer_number_to_merge * level0_file_num_compaction_trigger so L1 can absorb one L0 batch comfortably. In the worked example L1 was 256 MB and each L0 batch was 256 MB, so it was rewriting all of L1 every time.

"Point lookups are slow. What is your list?"

Bloom filters enabled and at 10 bits per key. Block cache large enough that the working set fits, checked with rocksdb.block.cache.hit and .miss counters rather than assumed. cache_index_and_filter_blocks with L0 pinned, so index lookups are not going to disk. Block size smaller (4 to 8 KB) for point lookups rather than the scan-oriented 16 to 32 KB. And if it is still slow, check the L0 file count, because L0 files overlap and every read checks all of them, so a backed-up L0 makes reads slow as well as stalling writes.

What does bloom_filter_bits_per_key actually buy, and how do you choose it? A bloom filter per SST file lets a read skip that file entirely when the filter says the key is absent, which is the difference between one disk read and one per level. The parameter is a direct space-versus-false-positive trade with a well known curve: 10 bits per key gives roughly a 1 percent false positive rate, and each additional 5 bits or so cuts that rate by about an order of magnitude, at the cost of memory that competes with the block cache. The default of 10 is right for most workloads and the interesting decision is when to deviate. Raise it when reads are dominated by keys that do not exist, which is the shape of a read-through cache or an idempotency-key check, because there every false positive is a wasted disk read and nothing else. Lower it or disable it on the largest level when the working set is small enough that the block cache is a better use of the same bytes. The second knob worth naming is optimize_filters_for_hits, which drops the filter on the bottommost level on the grounds that a key reaching the bottom level usually exists; that saves the majority of the filter memory, since the bottom level holds most of the data, and it is the wrong choice for exactly the non-existent-key workloads above.

Common misconceptions

"RocksDB memory equals block cache size." It is block cache plus memtables plus index and filter blocks plus reader overhead, and in Kafka Streams or Flink it is that multiplied by the number of stores times partitions. Sizing a container from the block cache alone is the standard route to an OOM kill misdiagnosed as a leak.

"A write stall is a bug." It is deliberate backpressure: the engine judges that accepting more writes would make reads unusable, so it slows or stops them. The bug is not exposing it as a metric, which turns a diagnosable condition into an unexplained hang.

"Bloom filters speed up all reads." Point lookups only. They answer "is key K present," and a range scan asks a different question, so scans get nothing. Prefix bloom filters help when your scans share a key prefix, and only then.

"The defaults are reasonable." The RocksDB documentation says explicitly that they are not tuned for performance. An 8 MB block cache and no bloom filters are starting points, not defaults in the sense of "fine unless you have a reason."

"More background threads always helps." They compete with foreground work for CPU and disk. Beyond the point where compaction keeps up, more threads make read latency worse. max_background_jobs of 4 to 8 is the usual useful range, and a rate limiter is often the better tool than more parallelism.

Interview delivery note

Say this verbatim: "A write stall is RocksDB deliberately refusing writes because compaction has fallen behind, and from outside the process it is indistinguishable from a hang, so rocksdb.is-write-stopped has to be a metric. Once you have it, the trigger tells you the cause: L0 file count means flushes are outpacing compaction, pending compaction bytes means the write rate exceeds what the disk sustains at this amplification." Diagnosis, mechanism and the specific metric, which is the answer that would have saved the team in the example two weeks.

The senior-versus-staff separator is knowing that RocksDB memory is not the block cache. A senior engineer sizes the block cache correctly. A staff engineer knows that index and filter blocks default to living outside the budget and grow with the dataset, that Kafka Streams and Flink create one instance per store per partition so the total multiplies with assignment, and that a shared cache plus cache_index_and_filter_blocks is how you make container memory predictable. That is the difference between a service that gets OOM-killed and one that does not.

The second signal is max_bytes_for_level_base sized against the L0 batch. It is an obscure knob and getting it wrong means every L0-to-L1 compaction rewrites all of L1, which is a multiplier on write amplification that no amount of extra compaction threads fixes.

The Staff & Team Lead Interview Reference