Iceberg table maintenance

When RisingWave streams data into Iceberg, it generates many small data files, fragmented manifests, and frequent snapshots. Over time, this can degrade query performance and increase storage costs. To maintain healthy tables, RisingWave provides both automatic and manual maintenance features for compaction, manifest rewrite, and snapshot expiration.

Compaction: Merges small data files and delete files into larger, optimized files to improve read performance.
Manifest Rewrite: Merges fragmented data manifests into fewer, better-sized manifests to reduce metadata overhead during planning.
Snapshot Expiration: Removes old, unneeded snapshots and their associated data files to reclaim storage space.

Automatic maintenance

Version notes

RisingWave introduced Iceberg automatic maintenance (default) in v2.5.0 and added compaction.type with the small-files and files-with-delete compaction types in v2.7.0.
Newly created internal Iceberg tables (CREATE TABLE ... ENGINE = iceberg) using Iceberg format V2 enable enable_manifest_rewrite by default. External Iceberg sinks remain opt-in, and Iceberg format V3 does not support manifest rewrite.
compaction.write_parquet_compression and compaction.write_parquet_max_row_group_rows were added in v2.9.0. The compaction.target_file_size_mb parameter now also controls the output file size for sink writes (previously only applied to compaction).
Parameters prefixed with compaction are currently in technical preview stage and may change in future releases.

You can enable automatic maintenance to run periodically in the background for your Iceberg sinks and internal tables. Periodic manifest rewrite and snapshot expiration run in the meta-node maintenance loop, rather than using the per-sink compaction_interval_sec parameter. Configure the interval with iceberg_gc_interval_sec in the [meta] section of risingwave.toml (default: 3600 seconds).

Dedicated compactor required for data file compactionAutomatic data file compaction requires a dedicated compactor node. Before enabling enable_compaction = true, ensure your cluster has at least one compactor node deployed. Manifest rewrite and snapshot expiration run on the meta node and do not require a dedicated compactor. For deployment instructions and sizing guidelines, see Deploy a dedicated Iceberg compactor.

Compaction triggers

When automatic data file compaction is enabled, a new compaction becomes eligible when either condition is met:

The number of pending snapshot commits reaches compaction.trigger_snapshot_count, even if the time interval has not elapsed.
The compaction_interval_sec interval has elapsed and there is at least one pending snapshot commit, even if the snapshot-count threshold has not been reached.

The snapshot-count threshold allows an early trigger; the interval limits how long the scheduler waits for that threshold when there are pending commits. It is not a minimum delay between compactions, and elapsed time alone does not trigger a new compaction without pending commits. Actual execution depends on scheduling and compactor availability. For example, with compaction_interval_sec = 600 and compaction.trigger_snapshot_count = 10, ten pending commits can trigger compaction before 600 seconds elapse. If only one commit is pending when the interval elapses, compaction is still eligible. Setting the threshold to 1 makes each new commit sufficient to trigger compaction without waiting for the interval.

Compaction types

RisingWave supports three compaction types for Iceberg tables. You can specify the type using the compaction.type parameter.

Compaction type	Description
`full`	Rewrites all data files. This is the default type.
`small-files`	Only compacts files smaller than a specified threshold. Use the `compaction.small_files_threshold_mb` parameter to set the threshold.
`files-with-delete`	Only compacts data files that have associated delete files. Use the `compaction.delete_files_count_threshold` parameter to set the minimum number of delete files to trigger compaction.

The small-files and files-with-delete compaction types are only supported in Merge-on-Read mode. Copy-on-Write mode only supports the full compaction type.

Parameters

Configure automatic maintenance by specifying the following parameters in the WITH clause of a CREATE SINK or CREATE TABLE ... ENGINE = iceberg statement.

General parameters

Parameter	Description
`enable_compaction`	Optional. Set to `true` to enable automatic data file compaction. Default: `false`.
`compaction_interval_sec`	Optional. The time-based trigger interval in seconds for data file compaction. When it elapses, compaction becomes eligible if there is at least one pending snapshot commit. Reaching `compaction.trigger_snapshot_count` can trigger compaction earlier. It does not control manifest rewrite or snapshot expiration. Default: `3600`.
`enable_snapshot_expiration`	Optional. Set to `true` to enable snapshot expiration. Default: `true`.
`snapshot_expiration_max_age_millis`	Optional. The maximum age (in milliseconds) for a snapshot to be retained. If unspecified, RisingWave uses the default retention behavior. To keep only the latest snapshot, set this to `0`.
`snapshot_expiration_retain_last`	Optional. The minimum number of snapshots to retain, regardless of their age. If unspecified, RisingWave uses the default retention behavior.

Manifest rewrite parameters

Parameter	Description
`enable_manifest_rewrite`	Optional. Periodically rewrites fragmented Iceberg data manifests. Newly created internal Iceberg tables using Iceberg format V2 enable this by default unless explicitly set to `false`. External Iceberg sinks remain disabled by default.
`manifest_rewrite_target_size_bytes`	Optional. The target size in bytes for rewritten manifests. Must be greater than `0`. Default: `8388608` (8 MiB).
`manifest_rewrite_min_count_to_merge`	Optional. The minimum number of manifests required to rewrite an under-filled batch. Must be greater than `0`. Default: `100`.

Compaction parameters

Parameter	Description
`compaction.type`	Optional. The compaction strategy: `full`, `small-files`, or `files-with-delete`. Default: `full`.
`compaction.max_snapshots_num`	Optional. The maximum number of snapshots allowed since the last rewrite operation. If set, the sink will pause if this number is exceeded until compaction completes. If `enable_compaction = true` and this parameter is not set, RisingWave uses `1000` by default. If `enable_compaction = false`, the gate remains disabled unless you explicitly set a positive value. `0` is not allowed.
`compaction.trigger_snapshot_count`	Optional. The number of pending snapshot commits that triggers data file compaction early, without waiting for `compaction_interval_sec`. The conditions use OR logic: compaction can also be triggered when the interval elapses with at least one pending snapshot commit. See Compaction triggers.
`compaction.target_file_size_mb`	Optional. The target output file size in MB. Applies to both sink writes and compaction. Default: `1024`.
`compaction.write_parquet_compression`	Optional. The Parquet compression codec for output files. Accepted values: `uncompressed`, `snappy`, `gzip`, `lzo`, `brotli`, `lz4`, `zstd`. Default: `snappy`. Supports dynamic updates via `ALTER SINK`.
`compaction.write_parquet_max_row_group_rows`	Optional. The maximum number of rows per Parquet row group. Must be greater than 0. Default: `122880`. Supports dynamic updates via `ALTER SINK`.
`compaction.small_files_threshold_mb`	Optional. For `small-files` compaction type, the threshold size in MB below which files will be compacted.
`compaction.delete_files_count_threshold`	Optional. For `files-with-delete` compaction type, the minimum number of delete files associated with a data file required to trigger compaction.

Deprecated storage config: The node-level storage config iceberg_compaction_write_parquet_max_row_group_rows is deprecated. Use the sink-level parameter compaction.write_parquet_max_row_group_rows instead.

Examples

Full compaction (default)

The following example enables automatic compaction with the default full compaction type:

CREATE TABLE my_iceberg_table (
    id INT PRIMARY KEY, 
    name VARCHAR
) WITH (
    enable_compaction = true, 
    compaction_interval_sec = 1800, 
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
) ENGINE = iceberg;

Manifest rewrite

For Iceberg sinks with fragmented metadata, enable manifest rewrite to periodically merge data manifests:

CREATE SINK my_iceberg_sink FROM my_mv
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_db',
    table.name = 'my_table',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    s3.access.key = 'your-key',
    s3.secret.key = 'your-secret',
    enable_manifest_rewrite = true,
    manifest_rewrite_target_size_bytes = 8388608,
    manifest_rewrite_min_count_to_merge = 100
);

Manifest rewrite is not supported for Iceberg format version 3 tables because the current rewrite path cannot preserve row lineage.

Small files compaction

For Merge-on-Read tables with many small files, use the small-files compaction type to only compact files smaller than a threshold:

CREATE SINK my_iceberg_sink FROM my_mv
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_db',
    table.name = 'my_table',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    s3.access.key = 'your-key',
    s3.secret.key = 'your-secret',
    write_mode = 'merge-on-read',
    -- Maintenance settings
    enable_compaction = true, 
    compaction_interval_sec = 1800,
    compaction.type = 'small-files',
    compaction.small_files_threshold_mb = 128,
    compaction.target_file_size_mb = 512,
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
);

Files with delete compaction

For Merge-on-Read tables with accumulated delete files, use the files-with-delete compaction type to only compact data files that have associated delete files:

CREATE SINK my_iceberg_sink FROM my_mv
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'id',
    warehouse.path = 's3://my-bucket/warehouse',
    database.name = 'my_db',
    table.name = 'my_table',
    catalog.type = 'rest',
    catalog.uri = 'http://rest-catalog:8181',
    s3.access.key = 'your-key',
    s3.secret.key = 'your-secret',
    write_mode = 'merge-on-read',
    -- Maintenance settings
    enable_compaction = true, 
    compaction_interval_sec = 1800,
    compaction.type = 'files-with-delete',
    compaction.delete_files_count_threshold = 100,
    compaction.trigger_snapshot_count = 10,
    enable_snapshot_expiration = true,
    snapshot_expiration_retain_last = 10
);

Manual maintenance

In addition to automatic background maintenance, you can trigger maintenance manually using the VACUUM command. VACUUM rewrites eligible data manifests, then expires eligible snapshots. Manifest rewrite runs only when enable_manifest_rewrite = true and the configured rewrite thresholds select an eligible batch. Snapshot expiration runs only when enable_snapshot_expiration = true. VACUUM FULL first compacts data files, then performs the same manifest rewrite and snapshot expiration steps, subject to their enable flags and thresholds. The manifest rewrite and snapshot expiration steps do not override their corresponding maintenance settings.

-- Rewrite eligible manifests, then expire eligible snapshots
VACUUM my_iceberg_table;

-- Compact files, then rewrite eligible manifests and expire eligible snapshots
VACUUM FULL my_iceberg_table;

VACUUM FULL waits for the manual compaction task to finish before returning. If the target already has an automatic or manual compaction task queued or running, the new VACUUM FULL request returns an error instead of scheduling another task.

Interact with Apache Iceberg

Nimtable: Iceberg o11y platform

Iceberg table maintenance

Automatic maintenance

Compaction triggers

Compaction types

Parameters

General parameters

Manifest rewrite parameters

Compaction parameters

Examples

Full compaction (default)

Manifest rewrite

Small files compaction

Files with delete compaction

Manual maintenance

​Automatic maintenance

​Compaction triggers

​Compaction types

​Parameters

​General parameters

​Manifest rewrite parameters

​Compaction parameters

​Examples

​Full compaction (default)

​Manifest rewrite

​Small files compaction

​Files with delete compaction

​Manual maintenance

Automatic maintenance

Compaction triggers

Compaction types

Parameters

General parameters

Manifest rewrite parameters

Compaction parameters

Examples

Full compaction (default)

Manifest rewrite

Small files compaction

Files with delete compaction

Manual maintenance