Iceberg table lifecycle: snapshots, hidden partitioning, small files

What it is

Apache Iceberg is a table format: a specification for what set of files constitutes a table at a point in time, plus the metadata to make that set atomically replaceable. It sits between the storage layer (Parquet files on S3) and the engines (Spark, Trino, Flink, Snowflake, DuckDB), and its job is to give object storage the semantics people expect from a table.

The thing it replaces is Hive table format, where a table was "whatever files are under this directory prefix." That definition has no atomicity (a reader mid-write sees partial data), no schema evolution beyond adding columns at the end, and requires listing directories to plan a query, which on S3 is slow and eventually consistent.

Iceberg's structure is a tree of metadata, and understanding the levels is most of understanding Iceberg:

catalog  ->  points to the CURRENT metadata file for a table
                │
   metadata.json  (schema, partition spec, snapshot list, current snapshot id)
                │
   manifest list  (one per snapshot: which manifests, with partition ranges)
                │
   manifest file  (a list of data files, with per-file column stats: min, max, nulls)
                │
   data files     (Parquet / ORC / Avro)

A commit is a single atomic swap of the catalog pointer to a new metadata file. That one property is what gives Iceberg serialisable isolation, time travel, and rollback, and everything else on this page follows from it.

What it is confused with: Iceberg is not a query engine and not a storage system. It does not run queries and does not store bytes. Comparing "Iceberg vs Spark" is a category error; the comparison that makes sense is Iceberg against Delta Lake and Apache Hudi, which are the other table formats.

The problem it solves

Four concrete failures of directory-as-table, each of which Iceberg addresses with a specific mechanism.

No atomicity. A Spark job rewriting a partition deletes files and writes new ones. A query running concurrently sees some of each. There is no way to make a multi-file change appear at once, so "rewrite yesterday's partition" is a window of wrong answers.

Query planning by directory listing. Hive planning lists directories to find partitions, and on S3 a LIST on a prefix with 200,000 objects is many paginated calls taking tens of seconds before any data is read. Iceberg reads a manifest instead: one file describing every data file with statistics.

Partition columns leak into every query. In Hive, if a table is partitioned by dt=2026-08-03, then a query filtering WHERE event_time >= '2026-08-03' gets no pruning, because event_time and dt are different columns as far as the planner is concerned. Every user must know the physical layout and filter on the partition column explicitly, and forgetting is a full scan.

Schema and layout changes are migrations. Renaming a column in Hive breaks readers, because columns are matched by name and position. Changing the partitioning requires rewriting the table.

Mechanics

Hidden partitioning: the headline feature

Iceberg stores a partition transform in the table metadata: a function from a column to a partition value. The user filters on the source column and Iceberg applies the transform to prune.

CREATE TABLE events (
    event_id    BIGINT,
    event_time  TIMESTAMP,
    user_id     BIGINT,
    payload     STRING
) USING iceberg
PARTITIONED BY (days(event_time), bucket(16, user_id));

-- Prunes to one day's partitions. No dt column, no knowledge of the layout.
SELECT * FROM events WHERE event_time >= '2026-08-03' AND event_time < '2026-08-04';

-- Also prunes, to one of the 16 buckets.
SELECT * FROM events WHERE user_id = 4711;

Available transforms: year, month, day, hour, bucket(N, col), truncate(W, col), and identity. The user never writes a partition predicate, which removes an entire class of accidental full scans and, more importantly, means the physical layout is no longer part of the table's public interface.

Which leads to the second half:

Partition evolution. Because the partition spec is metadata and each data file records which spec it was written under, you can change partitioning without rewriting anything:

-- Volume grew: switch from daily to hourly partitions. No rewrite.
ALTER TABLE events REPLACE PARTITION FIELD days(event_time) WITH hours(event_time);

Old files keep their daily partitioning, new files are hourly, and Iceberg plans across both. In Hive that change is a full table rewrite, so teams pick a partitioning scheme early, get it wrong, and live with it. Iceberg removes that one-way door.

Snapshots, time travel and rollback

Every write creates a new snapshot. The old ones remain valid until expired.

-- What snapshots exist?
SELECT snapshot_id, committed_at, operation, summary
  FROM events.snapshots ORDER BY committed_at DESC LIMIT 5;

     snapshot_id      |      committed_at      | operation
----------------------+------------------------+-----------
  8841029384710294857 | 2026-08-03 14:22:01    | append
  4410293847102948571 | 2026-08-03 13:22:00    | overwrite
  2938471029485710294 | 2026-08-03 12:22:04    | append

-- Read the table as it was.
SELECT * FROM events VERSION AS OF 4410293847102948571;
SELECT * FROM events TIMESTAMP AS OF '2026-08-03 13:00:00';

-- Undo a bad write: a metadata pointer change, not a data restore.
CALL catalog.system.rollback_to_snapshot('db.events', 2938471029485710294);

Rollback is instant regardless of table size, because it swaps a pointer. A 40 TB table rolls back in the time it takes to write one metadata file. That is the strongest practical argument for Iceberg over Hive: a bad backfill is a one-command undo instead of a restore from backup.

The cost is storage. Snapshots retain their data files, so a table with 90 days of snapshots retains every file version from 90 days.

-- Expire old snapshots and DELETE the data files only they referenced.
CALL catalog.system.expire_snapshots(
    table => 'db.events',
    older_than => TIMESTAMP '2026-07-27 00:00:00',
    retain_last => 10);

This is not optional maintenance. Without expiry, storage grows without bound and every query plans over a longer snapshot history. Retention is a real decision: long enough to recover from a bad write discovered late, short enough that storage is bounded. Seven days is a reasonable default, and it should be matched to how long it takes you to notice a data quality problem.

The small-files problem

This is the operational issue that dominates Iceberg in practice, and it is caused by streaming ingestion.

A Flink or Spark Structured Streaming job committing every minute writes at least one file per partition per commit:

Commit interval:    1 minute
Partitions written: 12 (a bucketed partition spec)
Files per day:      1440 x 12 = 17,280 files/day
File size:          ~2 MB each

Compare with the target of 128 to 512 MB per file, and the costs are:

Planning. Every file is an entry in a manifest with statistics. A million small files means a large manifest to read and scan before any data is touched.
Read overhead. Each Parquet file has a footer to read and a schema to parse. Fixed per-file cost dominates when files are small.
S3 request cost. One GET per file, and object-store requests are billed and rate-limited per prefix.
Compression. Parquet's encodings (dictionary, run-length) work on row groups; a 2 MB file compresses far worse than a 512 MB one.

The fix is compaction, run as a scheduled job:

-- Rewrite small files into larger ones. Does not change table contents.
CALL catalog.system.rewrite_data_files(
    table => 'db.events',
    strategy => 'binpack',
    options => map(
        'target-file-size-bytes', '536870912',      -- 512 MB
        'min-input-files', '5',
        'max-concurrent-file-group-rewrites', '10'));

-- Sort within files, so column stats prune better. More expensive than binpack.
CALL catalog.system.rewrite_data_files(
    table => 'db.events',
    strategy => 'sort',
    sort_order => 'user_id ASC NULLS LAST');

binpack just combines files; sort also orders rows so that per-file min/max statistics become narrow and therefore useful for pruning. A table sorted by user_id lets a query on one user skip files whose min/max range excludes it. An unsorted table's files each contain a wide range of user IDs, so no file can be skipped.

zorder is the multi-column variant, clustering on several columns at once, useful when queries filter on different combinations.

Manifests need compaction too, and this is frequently forgotten:

CALL catalog.system.rewrite_manifests('db.events');

-- And orphan files: data written by a failed job that no snapshot references.
CALL catalog.system.remove_orphan_files(
    table => 'db.events',
    older_than => TIMESTAMP '2026-08-01 00:00:00');

The full maintenance set is four procedures, and a production Iceberg table needs all four on a schedule: expire_snapshots, rewrite_data_files, rewrite_manifests, remove_orphan_files.

Copy-on-write versus merge-on-read

For updates and deletes, Iceberg v2 offers two strategies, and choosing wrong is the second most common performance problem:

ALTER TABLE events SET TBLPROPERTIES (
    'write.delete.mode' = 'merge-on-read',
    'write.update.mode' = 'merge-on-read',
    'write.merge.mode'  = 'merge-on-read');

Copy-on-write rewrites every data file containing an affected row at write time. Writes are expensive, reads are unaffected. Right for infrequent large batch updates.

Merge-on-read writes a small delete file recording which rows are deleted, and readers apply it. Writes are cheap, every subsequent read pays to merge. Right for frequent small updates, and it requires regular compaction or read performance degrades steadily as delete files accumulate.

That last clause is the trap: merge-on-read without scheduled compaction is a table that gets slower every day, and the degradation is gradual so nothing alerts.

A worked example: 40-second query planning on a streaming table

A clickstream platform. Flink writing to Iceberg on S3, queried by Trino for analytics. About 2.4 billion events a day.

Configuration:

Flink checkpoint interval:  60 seconds  (each checkpoint commits to Iceberg)
Partition spec:             days(event_time), bucket(32, session_id)
Table age:                  8 months
No maintenance jobs configured.

Symptoms:

Trino query planning time:       38-44 seconds     <- before reading any data
Trino query execution:            6-9 seconds
total data files:                 41.2 million
average file size:                1.8 MB
manifest files:                   184,000
metadata directory size:          human-noticeable in S3 console
S3 GET requests/month:            ~2.1 billion
storage:                          78 TB (of ~19 TB live data)

Planning took five times longer than execution. That is the signature of a small-files problem: the query is fast once it knows which files to read, and finding out is the expensive part.

Arithmetic on how it got there:

Commits per day:     1440 (one per 60s checkpoint)
Partitions per commit: up to 32 buckets x 1 day
Files per day:       ~46,000
Over 8 months:       ~11 million... plus retained snapshot versions -> 41 million

Every snapshot from eight months was retained, so files superseded by later rewrites were still stored and still referenced by old manifests.

The fixes, and what each contributed.

-- 1. Expire snapshots. Retain 7 days.
CALL catalog.system.expire_snapshots(
    table => 'clicks.events',
    older_than => TIMESTAMP '2026-07-27 00:00:00',
    retain_last => 20);

storage:        78 TB -> 24 TB
data files:     41.2M -> 12.8M

Fifty-four terabytes of files referenced only by snapshots older than a week.

-- 2. Compact data files. Sort by session_id so column stats prune.
CALL catalog.system.rewrite_data_files(
    table => 'clicks.events',
    strategy => 'sort',
    sort_order => 'session_id ASC',
    options => map('target-file-size-bytes','536870912',
                   'partial-progress.enabled','true',
                   'max-concurrent-file-group-rewrites','20'));

data files:      12.8M -> 68,400
avg file size:   1.8 MB -> 486 MB
storage:         24 TB -> 19.2 TB     (better compression at larger row groups)

-- 3. Compact manifests.
CALL catalog.system.rewrite_manifests('clicks.events');

manifest files:  184,000 -> 412

-- 4. Remove orphans from failed Flink jobs.
CALL catalog.system.remove_orphan_files(
    table => 'clicks.events',
    older_than => TIMESTAMP '2026-08-01 00:00:00');

recovered:  1.9 TB

Plus the change that stops it recurring:

Flink checkpoint interval:  60s -> 300s      (5x fewer commits)
Plus a scheduled maintenance job (Airflow, hourly compaction, daily expiry).

Final:

                              before        after
query planning time           38-44s        0.8s        (~50x)
query execution time          6-9s          2.1s        (larger files, better pruning)
total query time              ~46s          ~2.9s       (~16x)
data files                    41.2M         68,400      (600x fewer)
average file size             1.8 MB        486 MB
manifest files                184,000       412
storage                       78 TB         19.2 TB
S3 GET requests/month         ~2.1B         ~34M        (62x fewer)
S3 request cost/month         ~$8,400       ~$140

Two things about this deserve emphasis.

Execution time improved too, from 6 to 9 seconds down to 2.1, and not only from larger files. Sorting by session_id narrowed each file's min/max range so Trino could skip files entirely on a session filter. Unsorted files each spanned the whole ID range, so no file could ever be pruned. binpack would have fixed planning and left execution where it was; the sort is what improved both.

Nothing here was an Iceberg defect. Every number was the predictable result of running a streaming writer against a table with no maintenance. The table format provides the procedures; it does not run them, and there is no default that runs them for you. That is the single most important operational fact about Iceberg and the thing teams coming from a managed warehouse do not expect.

Production evidence

Iceberg was created at Netflix by Ryan Blue and Dan Weeks specifically to fix Hive's problems at their scale: their published motivation cites S3 listing costs, the lack of atomic commits, and users needing to know the physical partitioning. Netflix's tables were large enough that directory listing alone dominated query planning.

Apple, Adobe, LinkedIn and Airbnb are among the contributors and production users, and Iceberg is a top-level Apache project. Snowflake, Databricks, AWS (Athena, Glue, EMR), Google BigQuery, Trino, Dremio and ClickHouse all read Iceberg, which is the strongest argument for it over the alternatives: it is the format with the broadest engine support, and that interoperability is often the deciding factor rather than any technical difference.

Databricks acquired Tabular (the company founded by Iceberg's creators) in 2024 and has committed to interoperability between Delta Lake and Iceberg, which was a significant signal that the format war was resolving toward Iceberg as the neutral option.

AWS Glue and EMR ship Iceberg maintenance as managed operations, and S3 Tables (announced 2024) provides Iceberg tables with automatic compaction and snapshot expiry as a service. That a managed offering exists whose main feature is running the maintenance procedures is direct evidence that the small-files problem is the dominant operational burden.

The Iceberg specification is versioned (v1, v2 with row-level deletes, v3 in progress) and published, which is what lets independent implementations exist across Java, Python (PyIceberg), Rust and Go.

The debate

Iceberg, Delta Lake or Hudi? All three solve the same problem with similar architecture. Delta Lake has the deepest Databricks integration and the strongest tooling if you are on Databricks. Hudi has the best incremental-processing and upsert story, having been built at Uber for CDC ingestion. Iceberg has the broadest engine support and the most neutral governance, which for most teams is the deciding factor because it avoids a vendor coupling on the storage layer. My default is Iceberg unless you are Databricks-native (Delta) or your primary workload is streaming upserts (Hudi is worth evaluating).

Is the maintenance burden acceptable? It is real, and it is four scheduled procedures that nobody tells you about. Teams arriving from Snowflake or BigQuery expect the table to look after itself and get 40-second planning times eight months later. My position: budget the maintenance jobs as part of adopting Iceberg, before the first table goes to production, and monitor file count and average file size per table as first-class metrics. Managed offerings (S3 Tables, Glue optimisation) remove this and cost more.

How often should you compact? Frequently enough that file count stays bounded, which for a streaming table means hourly. The trade is that compaction rewrites data, costing compute and creating new snapshots, and it conflicts with concurrent writers (Iceberg resolves this optimistically, and a compaction competing with a heavy writer can retry repeatedly). Hourly binpack with a nightly sort is a reasonable shape.

Copy-on-write or merge-on-read? Copy-on-write for infrequent bulk updates, where write cost is paid once and reads stay clean. Merge-on-read for frequent small updates, particularly CDC ingestion. The critical caveat is that merge-on-read requires compaction on a schedule, or delete files accumulate and every read gets slower. A merge-on-read table without compaction degrades gradually with nothing alerting, which is the same failure shape as unvacuumed Postgres and unrepaired Cassandra.

Should you stream directly into Iceberg? It works and it is the main source of the small-files problem, because the commit interval determines the file count. The alternatives are a longer checkpoint interval (fewer, larger commits, at the cost of freshness), or landing in Kafka and micro-batching into Iceberg every few minutes. Freshness and file count are directly traded, and choosing a 60-second checkpoint because it was the default is choosing 1,440 commits a day without deciding to.

Follow-up Q&A

"What is hidden partitioning and why does it matter?"

Iceberg stores a transform (days(event_time), bucket(16, user_id)) in table metadata and applies it during planning, so users filter on the source column and get pruning automatically. In Hive, partitioning by a derived dt column means a query filtering on event_time gets no pruning at all, so every user must know the physical layout. The deeper consequence is that layout is no longer part of the table's interface, which is what makes partition evolution possible: you can switch daily to hourly partitions without rewriting data, because each file records the spec it was written under.

"How does Iceberg give atomic commits on S3?"

A commit writes new metadata and data files, then atomically swaps the catalog's pointer to the new metadata file. Readers resolve the table through the catalog, so they see either the old snapshot or the new one and never a mixture. The atomicity requirement is pushed down to the catalog: with a Hive Metastore or JDBC catalog it is a database transaction, with the AWS Glue catalog it is a conditional update, and with a filesystem catalog it needs an atomic rename, which is why filesystem catalogs on S3 were historically problematic before conditional writes.

"Why do small files hurt so much?"

Three costs. Planning: every file is a manifest entry with statistics that must be read and evaluated before any data is touched, so a million files means a large manifest scan per query. Read: each Parquet file has a footer and schema to parse, a fixed cost that dominates when the file is 2 MB. And object-store requests: one GET per file, billed and rate-limited. In one case planning took 38 seconds against 6 seconds of execution, which is the signature.

"What maintenance does a production Iceberg table need?"

Four procedures on a schedule, and nothing runs them for you. expire_snapshots to bound storage and snapshot history. rewrite_data_files to compact small files into 128 to 512 MB targets. rewrite_manifests to compact the metadata layer, which is separately forgotten. And remove_orphan_files for data written by failed jobs that no snapshot references. Monitor file count and average file size per table as metrics, because the degradation is gradual.

"binpack or sort when compacting?"

binpack just combines files into larger ones, which fixes planning time and request cost. sort additionally orders rows so per-file min/max statistics are narrow, which lets the engine skip whole files on a filtered query. An unsorted table's files each span the full value range, so no file is ever prunable. Sort is more expensive to run, so a common pattern is hourly binpack for file count and a nightly sort on the column queries filter by most.

"Copy-on-write or merge-on-read?"

Copy-on-write rewrites affected data files at write time: expensive writes, clean reads, right for infrequent bulk updates. Merge-on-read writes delete files that readers apply: cheap writes, and every read pays to merge, right for frequent small updates like CDC. The condition on merge-on-read is scheduled compaction, because delete files accumulate and reads get gradually slower with nothing alerting.

Common misconceptions

"Iceberg is a database." It is a table format specification. It stores no data and runs no queries; engines do both. The comparison set is Delta Lake and Hudi, not Spark or Trino.

"Iceberg handles maintenance automatically." It provides procedures and runs none of them. A streaming table with no scheduled maintenance accumulates millions of small files and unbounded snapshot storage, and the failure is gradual.

"Time travel is free." Snapshots retain their data files, so retention is storage. Ninety days of snapshots on a frequently-rewritten table can be several times the live data size, which in one case was 78 TB holding 19 TB of live data.

"Hidden partitioning means you do not think about partitioning." It means users do not have to. You still choose the transform and granularity, and choosing daily partitions for a table that grows 100x still produces partitions too large to prune usefully. What Iceberg gives you is the ability to change it later without a rewrite.

"Compaction changes the data." It rewrites files without changing table contents, and it does create a new snapshot. Queries against older snapshots still resolve, which is why compaction and snapshot expiry interact: compaction's old files are retained until the snapshots referencing them expire.

Interview delivery note

Say this verbatim: "Iceberg's core trick is that a commit is an atomic swap of a metadata pointer, which gives you serialisable isolation, instant rollback on a table of any size, and hidden partitioning, so users filter on the source column and the layout stops being part of the table's interface. The operational cost is four maintenance procedures that nothing runs for you." The mechanism, what it buys, and the cost that teams discover late.

The senior-versus-staff separator is naming the small-files problem as the dominant operational burden and knowing that planning time is where it shows up. A senior engineer explains snapshots and hidden partitioning correctly. A staff engineer says "planning was 38 seconds and execution was 6, which is definitionally a file-count problem," and then distinguishes binpack from sort: one fixes planning, the other also narrows per-file statistics so execution improves through pruning.

The second signal is connecting the commit interval to the file count. Saying "a 60-second Flink checkpoint is 1,440 commits a day, times the partitions touched, and that is your file count, so freshness and file count are the same knob" shows you understand the cause rather than the symptom.

The Staff & Team Lead Interview Reference