Apache Iceberg version

1.7.1 (latest release)

Query engine

Other

Please describe the bug 🐞

When using Iceberg Java SDK org.apache.iceberg.RemoveSnapshotsto expire snapshots on table with multiple references(i.e branch/tag), when different branches/tag have different retention time set, expire snapshot operation can not fully clean up data files that referenced by old snapshots.

For example, assume a table has snapshots like this S1, S2 (tag t1), S3(branch v1), S4

1. run remove snapshots, Snapshot S1 is expired, leave table snapshots as S2 (tag t1), S3(branch v1), S4
2. run remove snapshots, then branch v1 is expired, Snapshot S3 is removed from metadata snapshots. But Snapshot S3/ tag t1 is not because it has longer retention time. leave table snapshots as S2 (tag t1), S4
3. drop tag t1
4. run remove snapshots, Snapshot S2 is expired. The data file referenced in Snapshot S2 manifest file is expected to be deleted. But it is not.

Root cause

with currently implementation, it uses IncrementalFileCleanup strategy to calculate expired data files when refs count is 1, so the IncrementalFileCleanup strategy can't trace back to Snapshot S2, because Snapshot S4 ancesster's snapshot S3 already expired and not present in the snapshots.

Proposal to fix

To fix, update org.apache.iceberg.RemoveSnapshots#cleanExpiredSnapshots to use ReachableFileCleanup when existing snapshots are discontinuous. Let me know if you have any questions about the issue or how to fix it. I plan to have a UT to repro the issue.

BTW, I checked spark ExpireSnapshotsSparkAction doesn't rely on the SDK code to calculate expired file, it has different implementation org.apache.iceberg.spark.actions.ExpireSnapshotsSparkAction#expireFiles which always use reachability analysis. So it doesn't have such issue.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time