Where fsync fits in a durability guarantee

"Where does fsync fit in a durability guarantee, and what happens if it fails?"

What it is

fsync(fd) is the system call that tells the kernel: take every dirty page belonging to this file, push it to the storage device, wait until the device says it is on stable media, and only then return. Without it, a successful write() means the data is in the kernel's page cache, which is RAM, which is gone on power loss or kernel panic.

Three calls, three different guarantees:

Call	Returns when	Survives
`write()`	Data is copied into the page cache	Process crash. Not power loss or kernel panic
`fsync()`	Data and metadata are on stable media	Power loss, if the device is honest
`fdatasync()`	Data plus only metadata needed to read it back	Same, and it is faster because it skips `mtime` updates

Commonly confused with O_DIRECT, which bypasses the page cache but does not promise the drive has flushed its own volatile write cache. Bypassing one cache is not the same as flushing all of them. Also confused with "the write returned, so it is saved", which is the belief this entire question exists to correct.

The problem it solves

A durability claim is a promise to a user: once I have acknowledged your transaction, it will still be there after any single failure I have promised to survive. Every layer between the application and the platter has a volatile buffer, and each one needs an explicit instruction to give it up.

Application buffer   (userspace, lost on process crash)
        |  write()
Kernel page cache    (RAM, lost on power loss or panic)
        |  fsync()  -> writeback + FLUSH/FUA
Device write cache   (volatile DRAM on the SSD/HDD, lost on power loss
        |             unless the device has power-loss protection)
Stable media         (NAND / platter)

Skip the fsync and you have an application that is fast and occasionally loses acknowledged transactions after a power cut. That failure is invisible in testing because testing rarely pulls the power cord mid-write.

Mechanics

Where it sits in a database commit

Every durable database does the same thing, whatever it calls it:

BEGIN
  ... changes accumulate in memory and in WAL buffers ...
COMMIT
  1. Append the commit record to the write-ahead log      write()
  2. Force the log to stable storage                      fsync()  <-- durability point
  3. Acknowledge the commit to the client
  ... data pages are flushed later, lazily, at a checkpoint ...

Step 2 is the durability boundary. Everything before it can be lost; everything after it is promised. This is why the write-ahead log exists at all: one sequential fsync on an append-only log is far cheaper than random fsyncs across every data page the transaction touched, and it is enough, because the log can reconstruct the pages during recovery.

The cost, measured:

Device	Approximate `fsync` latency
Spinning disk	5 to 10 ms (a rotation)
Consumer SSD, no power-loss protection	0.5 to 2 ms
Enterprise NVMe with power-loss protection	20 to 100 µs
Cloud network block storage	0.5 to 2 ms, plus network variance

An enterprise drive with a capacitor-backed cache can acknowledge a flush as soon as the data is in its own DRAM, because the capacitor guarantees that DRAM reaches NAND even if power is cut. That is the entire reason the enterprise drive is faster at this one operation by an order of magnitude, and it is why "the same NVMe part number, but the datacenter SKU" costs what it does.

Group commit: how you avoid one fsync per transaction

If every transaction paid a 1 ms fsync, throughput would cap at 1,000 commits per second per log. Group commit removes that ceiling: transactions that commit within a small window share a single flush.

t=0.0ms  txn A commits -> appends to WAL, waits
t=0.2ms  txn B commits -> appends to WAL, waits
t=0.4ms  txn C commits -> appends to WAL, waits
t=0.5ms  one fsync() covers A, B and C
t=1.5ms  all three acknowledged

Three durable commits for one flush. PostgreSQL exposes this as commit_delay and commit_siblings; MySQL's InnoDB does it automatically in its two-phase binlog commit; every serious engine has a version. Throughput scales; per-transaction latency does not improve and may slightly worsen. That trade is almost always correct.

The knobs people turn, and what they cost

PostgreSQL:

synchronous_commit = on        -- default: fsync WAL before acknowledging
                   = off       -- acknowledge first, flush within wal_writer_delay
                                  (default 200 ms). Transactions are still ATOMIC
                                  after a crash; you just lose the last ~200 ms
                                  of committed ones. No corruption.
                   = local     -- fsync locally, do not wait for replicas
                   = remote_write / on / remote_apply  -- with synchronous replicas
fsync = off                    -- never do this in production. Corruption, not just
                                  loss, because data pages and WAL can be reordered.

The distinction between synchronous_commit = off and fsync = off is the one worth knowing. The first trades a bounded window of committed transactions for speed and keeps the database consistent. The second abandons crash safety entirely. One is a legitimate tuning decision for a workload that can replay lost writes; the other is a benchmarking-only setting.

MySQL/InnoDB:

innodb_flush_log_at_trx_commit = 1   -- fsync per commit; ACID
                               = 2   -- write to OS cache per commit, fsync each
                                        second; survives mysqld crash, not power loss
                               = 0   -- flush each second; loses up to 1s on any crash
sync_binlog = 1                      -- fsync binlog per commit; needed for
                                        replication safety, doubles the flush cost

innodb_flush_log_at_trx_commit = 1 plus sync_binlog = 1 is the fully durable configuration, and it costs two flushes per commit, which is why so many production systems quietly run 2 and 1000 and have not thought carefully about what that means.

fsyncgate: what happens when fsync fails

This is the part that separates a good answer from a complete one.

In 2018 the PostgreSQL developers discovered that on Linux, if the kernel's writeback of a dirty page fails (a transient device error, a thin-provisioned volume that ran out of space, a network block device that blipped), the kernel reports the error to one fsync caller and then, in some versions, marks the pages clean anyway. A subsequent fsync on the same file returns success. The data is gone and the application has been told everything is fine.

Worse, the error may be reported to whichever process happens to call fsync next, which need not be the process that issued the write.

The consequences, which are now the standard practice:

PostgreSQL 12 and later panic on fsync failure by default (data_sync_retry = off). A crash and a WAL replay is the only safe response, because the in-memory state can no longer be reconciled with what is on disk. Deliberately crashing is the correct behaviour.
Retrying fsync after a failure is unsafe. The second call can succeed while the data remains lost. Any code that does while (fsync(fd) < 0) retry; is wrong.
The behaviour differs across kernels and filesystems, which is why the answer is "crash and recover from the log" rather than "handle the error".

Naming fsyncgate is a strong signal in an interview, because it demonstrates that you understand durability as an end-to-end property that can be broken by a layer you do not control.

Durability is not only fsync

fsync gets you durability against process crash, kernel panic and power loss on one machine. It does nothing about the machine dying, the rack losing power, or the disk failing permanently. For those you need replication, and the two compose:

Local durability      fsync on the primary            ~0.1 to 2 ms
Replicated durability fsync + acknowledgement from N replicas   + 1 RTT

Raft / Paxos commit latency  =  local fsync + network RTT to a quorum

That formula is the reason consensus systems are latency-sensitive to both disk and network, and the reason etcd is famously unhappy on slow disks: every Raft log append is an fsync, and a 10 ms fsync puts a hard ceiling on the cluster's write rate regardless of how fast the network is.

Kafka is the instructive counter-example: it does not fsync per message by default. It relies on replication to N brokers plus the page cache, and flush.messages/flush.ms are left effectively unbounded. Durability comes from acks=all with min.insync.replicas=2, which survives a broker failing but not a simultaneous power loss across the whole rack. That is a deliberate, documented trade, not an oversight, and quoting it shows you understand that fsync is one strategy for durability rather than the definition of it.

A worked example

A payments service acknowledges a transaction, the datacenter loses power, and after recovery three transactions that returned 200 OK are missing. Walk the layers.

1. Was synchronous_commit on?
   -> It was set to 'off' six months ago during a latency push.
      Window of loss: wal_writer_delay = 200 ms.
      3 transactions in the final 200 ms is exactly consistent with this.

2. Was fsync reaching the device?
   -> Check whether the volume is backed by a drive with a volatile write cache
      and whether write barriers are enabled:
        cat /sys/block/nvme0n1/queue/write_cache   -> "write back"
      A "write back" cache without power-loss protection means fsync must issue
      a FLUSH, and the filesystem must not be mounted with nobarrier.
        mount | grep ' / '   -> check for nobarrier / barrier=0

3. Was the storage layer honest?
   -> Some virtualised and consumer devices acknowledge FLUSH without flushing.
      diskchecker.pl and the fio --sync tests exist to detect this; the honest
      answer in an interview is that you verify it with a power-cut test on
      representative hardware, because you cannot take the datasheet's word.

4. Did fsync ever fail silently?
   -> dmesg for I/O errors around the incident window, and the database log for
      any fsync error. On PG 12+ a failure would have panicked, which is itself
      evidence: no panic means no reported failure.

Root cause: synchronous_commit = off. The fix is to turn it back on and measure what it actually costs, which for a well-configured NVMe with group commit is usually much less than the team feared when they turned it off. And the durable follow-up: the setting was changed with no record of the trade being accepted, so the real fix is that a durability-affecting setting requires a documented decision.

Production evidence

The PostgreSQL fsyncgate thread (pgsql-hackers, March 2018) and the follow-up paper by Rebello et al., "Can Applications Recover from fsync Failures?" (USENIX ATC 2020), which tested PostgreSQL, LMDB, LevelDB, SQLite and Redis against injected fsync failures and found data loss or corruption in several. The paper is the citation to reach for, because it is systematic rather than anecdotal.

PostgreSQL's response, data_sync_retry defaulting to off and a PANIC on fsync failure since version 12, is the durable industry answer: crash and replay rather than attempt to recover in place.

Kafka's design, documented in its own durability section, deliberately relies on replication rather than per-message fsync, and states the resulting failure mode (correlated power loss) explicitly.

etcd's hardware recommendations specify low fsync latency (they publish wal_fsync_duration_seconds as a primary health metric and recommend p99 under 10 ms) because Raft's commit path is an fsync per append.

Enterprise SSD power-loss protection (capacitor-backed write caches) exists as a product category precisely because honest fsync is expensive without it, which is useful physical evidence that this is a real constraint rather than a theoretical one.

The debate

The case for always fsyncing on commit: durability is a promise, and a system that acknowledges data it can lose is lying to its users. Financial, medical and legal systems have no room here. Modern NVMe with power-loss protection makes the cost around 50 µs, and group commit amortises it further, so the argument that it is too expensive is often based on decade-old hardware numbers.

The case for relaxed durability: many workloads can replay. If the source of truth is an upstream event log, losing 200 ms of a derived store costs a replay rather than data. Kafka's design and every analytics pipeline built on it depend on this. Insisting on per-commit fsync in a derived system buys durability you already have elsewhere, at real throughput cost.

My position: fsync on commit is the default, and relaxing it requires naming the recovery path. The question I ask is not "can we afford the latency" but "if we lose the last 200 milliseconds, what specifically replays it?" If the answer is an upstream log or a reconciliation job that already exists, relaxing is a legitimate engineering decision and I would write it down. If the answer is "nothing, but it's only 200 milliseconds", the setting is wrong, because the size of the window is not the point; whether anything reconstructs it is. And I would separate the two PostgreSQL knobs explicitly, since synchronous_commit = off is a bounded loss of committed transactions while fsync = off is corruption, and teams conflate them.

Follow-up Q&A

"Where does fsync fit in a durability guarantee?" It is the boundary. A write() returns when the data is in the kernel page cache, which is RAM, so it survives a process crash and nothing else. fsync pushes those pages to the device and issues a cache flush, so the data survives power loss. In a database the commit path is: append the commit record to the WAL, fsync the WAL, then acknowledge the client. Everything before the fsync can be lost; everything after it is promised. And the WAL exists so that one sequential flush covers a transaction that touched many random pages.

"What happens if fsync fails?" This is fsyncgate, from 2018. On Linux, a writeback failure is reported to one caller and the dirty pages may then be marked clean, so a retried fsync returns success while the data is gone. The error can even be delivered to a process that did not issue the write. The consequence is that retrying is unsafe, and PostgreSQL's answer since version 12 is to panic on fsync failure and recover from the WAL. Deliberately crashing is correct here, because in-memory state can no longer be reconciled with the disk.

"fsync vs fdatasync vs O_DIRECT?" fdatasync skips metadata that is not needed to read the data back, mainly timestamps, so it can save a metadata write per call and is what most databases use for the WAL. O_DIRECT bypasses the page cache but makes no promise about the device's own volatile cache, so it is not a durability mechanism, it is a caching-policy mechanism, and code using it still needs fsync unless the device has power-loss protection and the filesystem is configured to trust it.

"How does group commit change the arithmetic?" Without it, throughput is capped at one over the fsync latency, so a 1 ms flush means 1,000 commits per second. Group commit lets transactions arriving within a small window share one flush, so three transactions in 0.5 ms cost one flush and throughput rises with concurrency. Per-transaction latency does not improve and can get marginally worse, which is almost always the right trade.

"Why is etcd sensitive to disk latency?" Every Raft log append is an fsync before the follower can acknowledge, so commit latency is local flush plus a network round trip to the quorum. A 10 ms fsync puts a hard ceiling on cluster write throughput no matter how fast the network is, which is why etcd publishes wal_fsync_duration_seconds as a primary health metric and why running it on shared or network storage causes leader elections under load.

"Kafka doesn't fsync per message. Is that a bug?" No, it is a documented trade. Kafka gets durability from replication: acks=all with min.insync.replicas=2 means the write is in the page cache of multiple brokers before acknowledgement. That survives any single broker failing, including a hard crash, because the other brokers still have it. What it does not survive is correlated power loss across the whole replica set, which is why rack and availability-zone spread is part of the durability story rather than an availability nicety.

"How would you verify the storage layer is honest?" Not from the datasheet. A power-cut test on representative hardware: write a known sequence with fsync after each record, cut power physically, and check for gaps on reboot. Tools like diskchecker.pl automate the pattern. In a cloud environment you cannot pull the cord, so you rely on the provider's durability statement and on /sys/block/*/queue/write_cache plus checking that the filesystem is not mounted nobarrier.

Common misconceptions

"The write returned, so it's saved." It is in RAM. This is the misconception the whole topic exists to correct.

"O_DIRECT means durable." It bypasses the page cache and says nothing about the device cache.

"If fsync fails, retry it." Unsafe. The retry can succeed against lost data. Crash and replay.

"synchronous_commit = off risks corruption." It does not. It risks losing a bounded window of committed transactions and leaves the database consistent. fsync = off is the one that risks corruption.

"Durability means fsync." fsync gives you durability against one machine failing in one specific way. Machine loss and disk loss need replication, and the two compose rather than substitute.

Interview delivery note

Lead with the boundary, because that is the actual question: "fsync is the durability boundary. A write returns when the data is in the page cache, which is RAM, so it survives a process crash and nothing else. fsync pushes it to the device and flushes the device cache. In a database the commit path is append to the WAL, fsync the WAL, then acknowledge, and the WAL exists so one sequential flush covers a transaction that touched many random pages."

Then give the cost, because numbers make it concrete: "That's five to ten milliseconds on a spinning disk, one to two on a consumer SSD, and under a hundred microseconds on an enterprise NVMe with a capacitor-backed cache. Group commit amortises it, so several transactions share one flush."

The depth signal is fsyncgate: "and the thing worth knowing is what happens when it fails. Since the 2018 PostgreSQL fsyncgate work we know that on Linux a writeback failure can be reported once and the pages marked clean, so a retried fsync returns success against data that's gone. That's why PostgreSQL 12 onwards panics on fsync failure rather than retrying. Deliberately crashing is the safe response."

Close by widening the frame: "and fsync only covers one machine. Machine loss needs replication, which is why Raft commit latency is local flush plus a quorum round trip, and why Kafka deliberately doesn't fsync per message and relies on acks=all instead."

The Staff & Team Lead Interview Reference