Lucene segments: refresh, flush and merge

What it is

A Lucene index is not one structure. It is a set of segments, each a complete, self-contained, immutable mini-index with its own postings lists, term dictionary, stored fields and doc values. A search runs against every segment and merges the results.

The three operations people conflate, and they do genuinely different things:

Operation	What it does	Makes documents searchable?	Makes them durable?
refresh	Opens a new searcher over the in-memory buffer, creating a new segment	Yes	No
flush	Writes the in-memory buffer to disk as a segment and truncates the translog	Yes (as a side effect)	Yes
fsync / commit	Forces the OS page cache to stable media	No	Yes
merge	Combines several segments into one larger one	No	No

Commonly confused: refresh is about visibility, flush is about durability, and merge is about efficiency. Elasticsearch's naming makes this worse because flush in its API means "commit and truncate the translog", while Lucene's IndexWriter.flush means something narrower.

Also commonly confused: an updated document is not modified in place. Segments are immutable, so an update writes a new document and marks the old one deleted in a bitset. The old copy stays on disk until a merge removes it, which is the root of several operational problems below.

The problem it solves

An inverted index is expensive to modify. Adding one document to a postings list of ten million entries would mean rewriting it, so a naive mutable index makes indexing throughput collapse as the index grows.

Segments solve it by never modifying anything. New documents accumulate in memory and are written as a new small segment. Searches read all segments. This makes indexing an append, which is fast and constant-time regardless of index size.

The costs it introduces, which is what the rest of the design is about:

Many segments -> every search touches every segment
                 -> search latency grows with segment count
Deletes accumulate -> deleted documents still occupy disk and
                 still cost time to skip during search
Nothing is durable until it is written and fsynced

Merging is the compaction that pays these costs down, and refresh interval is the knob that trades visibility latency against segment count.

Mechanics

The write path, end to end

  index request
       │
       ▼
  ┌─────────────────────┐        ┌──────────────────┐
  │ IN-MEMORY BUFFER    │───────►│ TRANSLOG          │  appended
  │ (not searchable)    │        │ (durability)      │  and fsynced
  └──────────┬──────────┘        └──────────────────┘  per request
             │                                          (by default)
             │  REFRESH  (default every 1 s if searched)
             ▼
  ┌─────────────────────┐
  │ NEW SEGMENT          │  in the OS page cache, SEARCHABLE,
  │ (searchable, not     │  and not yet fsynced
  │  necessarily durable)│
  └──────────┬──────────┘
             │  FLUSH / COMMIT (translog size or time threshold)
             ▼
  ┌─────────────────────┐
  │ SEGMENT ON DISK      │  fsynced, translog truncated
  └──────────┬──────────┘
             │  MERGE (background, by policy)
             ▼
  ┌─────────────────────┐
  │ FEWER, LARGER        │  deleted docs physically removed
  │ SEGMENTS             │
  └─────────────────────┘

The translog is what makes the gap safe. A document is in the in-memory buffer and not in any segment, so a crash would lose it. The translog is an append-only log written and fsynced per request (by default), so recovery replays it. That is the same write-ahead-log reasoning as any database, and the same fsync trade-off applies.

index.translog.durability: request   (default) fsync per request.
                                     Safe. Costs an fsync per write.
                         : async     fsync every sync_interval (5 s).
                                     Much faster. Loses up to 5 s of
                                     acknowledged writes on a node
                                     crash.

Refresh: the visibility knob

Near-real-time search means "visible within one refresh interval", not "immediately". A document indexed at t=0 with a 1-second refresh interval becomes searchable somewhere in [0, 1] seconds.

refresh_interval: 1s      default. One new segment per second per
                          shard, if there is anything to write.
                : 30s     fewer, larger segments; 30 s visibility lag
                : -1      disabled. No automatic refresh at all.

The arithmetic that motivates tuning it:

Bulk load: 100M documents at 50,000/sec = ~33 minutes.

refresh_interval = 1s
  ~2,000 refreshes -> ~2,000 small segments created
  Each must later be merged, repeatedly, up the tiers.
  Merge write amplification: a document is typically written
  5-10 times as it migrates up the merge tiers.

refresh_interval = -1 during the load, then restore
  Segments are created only on flush, so far fewer and larger.
  Measured effect on large bulk loads is commonly a 2-3x
  throughput improvement, primarily from avoided merge work.

And the second half of that optimisation: set number_of_replicas: 0 during the bulk load. Every replica performs the same indexing and merging work independently, so one replica doubles it. Restore both settings before serving traffic, and the replica rebuild is a segment copy rather than a re-index, which is much cheaper than the indexing it replaced.

Since Elasticsearch 7.0, an index with no search traffic for 30 seconds stops refreshing automatically and refreshes on demand when a search arrives. That default means an idle index is not paying refresh cost, which is worth knowing before tuning it manually.

Merge: the compaction

Lucene's TieredMergePolicy groups segments into size tiers and merges within a tier once enough segments accumulate.

Key parameters:
  max_merged_segment          5 GB default. Segments above this are
                              never merged again, which bounds the
                              cost of any single merge.
  segments_per_tier           10 default. How many segments of
                              similar size before merging them.
  deletes_pct_allowed         33% default (ES 7.5+). Merge is
                              triggered when deleted docs exceed
                              this share of the index.

Merging is expensive and it is I/O, not CPU. Merging ten 1 GB segments reads 10 GB and writes ~10 GB, and it competes with indexing and search for the same disk. Elasticsearch throttles it (indices.store.throttle.max_bytes_per_sec, and on modern versions an adaptive scheme), and on spinning disks the throttle is essential while on NVMe it is often the wrong default and worth raising.

Force-merge is the operation people misuse:

POST /index/_forcemerge?max_num_segments=1

Correct use:   a READ-ONLY index. A time-based index whose day has
               passed, merged to one segment: best search latency,
               all deletes physically purged, smallest disk usage.

Wrong use:     an index still being written. It produces one huge
               segment above max_merged_segment, which is therefore
               NEVER merged again, so its deleted documents can
               never be reclaimed. The index degrades permanently
               and the only fix is a reindex.

That failure is worth naming precisely, because it is common and it is irreversible without a reindex. Someone force-merges a hot index to "clean it up", the resulting 40 GB segment exceeds the 5 GB ceiling, and six months later a third of the index is deleted documents that nothing will ever remove.

Deletes and updates

Delete doc 4711:
  -> set bit 4711 in the segment's .liv (live docs) bitset
  -> the document's terms remain in the postings lists
  -> searches skip it at query time, paying the cost of skipping

Update doc 4711:
  -> mark old as deleted (as above)
  -> index a NEW document in the current buffer
  -> both copies exist until a merge

Consequence: an index with heavy updates carries a large
population of deleted documents, which cost disk AND search
time, until merges reclaim them.

GET /index/_stats/docs

"docs": { "count": 8420000, "deleted": 4110000 }

33% deleted. Every search is skipping over 4 million tombstoned
documents. Either merges are not keeping up (check the merge
throttle and disk I/O) or the index was force-merged and its
segments are above the ceiling.

deleted / (count + deleted) is the health metric, and a sustained value above roughly a third means merge policy or I/O is the problem, not the query.

Why segment count matters for search

A query executes against every segment and the results are merged:

Query cost ~ (per-segment fixed cost x segment_count)
           + (postings traversal, roughly independent of
              segmentation)

The fixed per-segment cost is real: term dictionary lookup,
skip-list setup, and per-segment heap for the query.

  50 segments  -> 50 term dictionary lookups per query term
   5 segments  ->  5

For a query with 8 terms that is 400 lookups versus 40.

This is why a time-based index that has stopped receiving writes should be force-merged to one segment, and why an index with hundreds of segments has a latency problem that no query tuning will fix.

Production evidence

Lucene's TieredMergePolicy is the default in Elasticsearch, OpenSearch and Solr, and its documented parameters (max_merged_segment at 5 GB, segments_per_tier at 10) are the basis for the numbers above. Mike McCandless's write-ups on merge policy visualisation are the canonical explanation of why tiered merging behaves the way it does.

Elasticsearch's documented bulk-indexing guidance explicitly recommends disabling the refresh interval and setting replicas to zero for large initial loads, then restoring both, which is direct vendor confirmation of the arithmetic above.

Elasticsearch 7.0's automatic refresh suspension for indices with no search traffic in 30 seconds is documented behaviour and reflects that refresh cost on idle indices was a real production problem.

The force-merge warning is in the official documentation: force-merging an index that is still being written produces segments larger than max_merged_segment which are then never merged, and the documentation recommends it only for indices that are no longer written to.

Elasticsearch's translog durability settings (request versus async) mirror the synchronous_commit trade in PostgreSQL, and the documented consequence of async is losing up to sync_interval of acknowledged writes on a node crash.

The debate

The case for a short refresh interval: users expect what they just wrote to be findable. A one-second interval makes the system feel real-time and matches what people assume search does.

The case for a long refresh interval: every refresh creates a segment, every segment must eventually be merged, and merge write amplification means each document is written five to ten times as it migrates up the tiers. On a write-heavy index a 30-second interval can be a substantial throughput gain for a visibility delay nobody notices.

The case for force-merging aggressively: fewer segments is faster search and less disk, and for read-only data it is unambiguously right.

My position: default the refresh interval to the largest value the product can tolerate, and treat force-merge as an operation that applies only to indices that will never be written again.

On refresh, the question I would ask the product owner is "how stale can a newly created document be before someone complains", and the answer is usually far more than one second: for a document search, thirty seconds is invisible; for a chat message index it is not. The default of one second is chosen for safety rather than because it is right, and on a write-heavy index it costs real throughput.

On force-merge I would state the rule as a hard one, because the failure is irreversible: never force-merge an index that still receives writes. The resulting segment exceeds max_merged_segment, is never merged again, and its deleted documents can never be reclaimed, so the index degrades permanently and the only fix is a full reindex. I have seen this done as routine "maintenance" and discovered months later.

And for bulk loading, disable refresh and set replicas to zero, then restore both. It is the standard advice, it is commonly a two to three times throughput improvement, and the reason it works is worth being able to state: you are avoiding merge write amplification and avoiding duplicating all of that work on every replica. The replica rebuild afterwards is a segment copy, which is far cheaper than the indexing it replaced.

Where I would push back on the framing: "search is slow" is often a segment count problem rather than a query problem, and the first thing to check is _stats for segment count and deleted-document percentage, before touching the query. A query tuned against an index with four hundred segments is tuned against the wrong problem.

Follow-up Q&A

"What is the difference between refresh and flush?" Refresh is about visibility: it opens a new searcher over the in-memory buffer so recently indexed documents become findable, and it creates a segment in the page cache that is not necessarily durable. Flush is about durability: it writes the buffer to disk, fsyncs, and truncates the translog. They are independent, which is why a document can be searchable but not yet durable, and the translog is what covers that gap by being fsynced per request.

"Why are segments immutable?" Because modifying an inverted index in place is expensive: adding a document to a postings list of ten million entries would mean rewriting it. Immutability makes indexing an append, which is constant-time regardless of index size. The costs are that searches must touch every segment, and that an update writes a new document while the old one lingers as a tombstone until a merge removes it.

"How would you speed up a bulk load of 100 million documents?" Disable the refresh interval and set replicas to zero, then restore both afterwards. Refresh at one second on a 33-minute load creates around two thousand small segments, each of which has to be merged repeatedly up the tiers, and merge write amplification means each document gets written five to ten times. Replicas double all of that work because each one indexes and merges independently. Restoring replicas afterwards is a segment copy, which is much cheaper than the indexing it replaced. Commonly a two to three times improvement.

"When should you force-merge?" Only on an index that will never be written to again, a time-based index whose window has closed. On an index still receiving writes it is actively harmful: it produces one segment larger than max_merged_segment, which the merge policy then never touches again, so the deleted documents inside it can never be reclaimed. The index degrades permanently and the only fix is a reindex. It is a common piece of well-intentioned maintenance and its damage shows up months later.

"An index reports 33 percent deleted documents. What is happening?" Merges are not reclaiming them, and there are two likely causes. Either merge is being throttled or starved of I/O, so check the merge throttle setting and disk utilisation, and note that the default throttle is tuned for spinning disks and is often wrong on NVMe. Or someone force-merged the index while it was still being written, so its segments exceed the ceiling and will never be merged again. The second case needs a reindex.

"Why does segment count affect search latency?" Because a query runs against every segment and merges results, so there is a fixed per-segment cost per query term: a term dictionary lookup and skip-list setup. An eight-term query against fifty segments does four hundred term dictionary lookups; against five segments it does forty. Which is why "search is slow" is often a segment-count problem, and I would check _stats for segment count and deleted percentage before tuning the query.

"What does the translog actually protect against?" The window between a document being acknowledged and being in a durable segment. It is append-only and fsynced per request by default, so a node crash replays it on recovery. Setting durability to async fsyncs every five seconds instead, which is much faster and loses up to five seconds of acknowledged writes on a crash. That is the same trade as synchronous_commit in PostgreSQL, and the same question applies: if we lose five seconds, what replays it?

"How do you choose the refresh interval?" By asking the product owner how stale a newly created document can be before someone complains, which is usually much more than one second. Thirty seconds is invisible for a document search and unacceptable for a chat index. The one-second default is chosen for safety rather than because it is right, and on a write-heavy index raising it is one of the cheapest throughput gains available. Also worth knowing: since 7.0, an index with no search traffic for thirty seconds stops refreshing automatically, so an idle index is not paying for it.

Common misconceptions

"Elasticsearch is real-time." It is near-real-time: visible within one refresh interval, which by default is up to a second and can be much longer.

"Deleting a document frees space." It sets a bit. The terms stay in the postings lists and the space is reclaimed only when a merge rewrites the segment.

"Force-merge is routine maintenance." On a written index it permanently damages the merge policy's ability to reclaim deletes.

"Flush makes documents searchable." Refresh does. Flush makes them durable. The two are independent and Elasticsearch's naming actively encourages the confusion.

"More shards means faster." More shards means more segments in total and more per-query fixed cost. Over-sharding is one of the most common causes of poor search latency.

Interview delivery note

Separate the three operations immediately, because conflating them is the tell: "Refresh is visibility, flush is durability, merge is efficiency. A document can be searchable and not durable, which is why there's a translog fsynced per request covering that gap. And they're independent, which Elasticsearch's naming actively obscures."

Give the immutability reason, because it explains everything downstream: "Segments are immutable because modifying an inverted index in place means rewriting a postings list of ten million entries. Immutability makes indexing an append. The cost is that searches touch every segment, and that an update leaves the old copy as a tombstone until a merge removes it."

The bulk-load answer with its arithmetic is a reliable signal: "Disable refresh and set replicas to zero, then restore both. At a one-second interval a 33-minute load creates about two thousand small segments, each merged repeatedly up the tiers, and merge write amplification means each document gets written five to ten times. Replicas double all of it. Typically two to three times faster."

State the force-merge rule as a hard one, because the failure is irreversible: "Never force-merge an index that still receives writes. You get one segment above the five-gigabyte ceiling, the merge policy never touches it again, and its deleted documents can never be reclaimed. The index degrades permanently and the only fix is a reindex. I've seen it done as routine maintenance and discovered months later."

And the diagnostic reframe: "when someone says search is slow, I check segment count and deleted-document percentage in _stats before I look at the query, because a query tuned against an index with four hundred segments is tuned against the wrong problem."

The Staff & Team Lead Interview Reference