Core, Spark: guard DeleteOrphanFiles against shared table locations#17254
Open
terrytlu wants to merge 1 commit into
Open
Core, Spark: guard DeleteOrphanFiles against shared table locations#17254terrytlu wants to merge 1 commit into
terrytlu wants to merge 1 commit into
Conversation
terrytlu
force-pushed
the
delete-orphan-shared-location
branch
8 times, most recently
from
July 17, 2026 01:39
d4c1156 to
450230f
Compare
When two Iceberg tables share the same underlying location (for example a test table created by copying SHOW CREATE TABLE but forgetting to change the LOCATION), running orphan-file cleanup for one table walks the whole location and deletes every file not reachable from that single table's metadata, silently corrupting the other table. Add a locationConflictMode(LocationConflictMode) option to DeleteOrphanFiles, defaulting to ERROR. Before deleting anything, it detects whether another table shares the metadata location by comparing the immutable table-uuid stored in each metadata.json (also handling .metadata.json.gz). A missing or unreadable uuid (legacy file without a uuid, corrupt, or compressed) is always treated as a conflict (fail-safe). In ERROR mode it aborts when a conflict is detected; in IGNORE mode the cleanup is skipped entirely and the other table's files are preserved; in DELETE mode cleanup proceeds even if a conflict is detected (legacy behavior), and should be used only when you are certain no other table shares the location. The detection helper lives in core (OrphanFileUtils.hasOtherTableInLocation) and is wired into DeleteOrphanFilesSparkAction for Spark 3.5/4.0/4.1. The Spark remove_orphan_files procedure also exposes a location_conflict_mode parameter, and docs/spark-procedures.md documents it. Core and Spark procedure tests cover the conflict / no-conflict / null-uuid / compressed / corrupt cases. Closes apache#17213 Signed-off-by: terrytlu <terrytlu@tencent.com> Generated-by: Claude Code
terrytlu
force-pushed
the
delete-orphan-shared-location
branch
from
July 17, 2026 02:14
450230f to
854aa5e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix #17213
Approach
Every Iceberg
metadata.json(including.metadata.json.gz) carries the creating table's immutabletable-uuid. So, before deleting anything, we inspect the table's metadata directory:metadata.jsonthat is not part of this table's own history (current +previousFiles()), andtable-uuiddiffers from, or cannot be read for, this table's uuid,→ we conclude that another table shares the location and stop before any file is deleted.
Constraint — requires
SupportsPrefixOperations: the detection relies ontable.io().listPrefix(...)to walk the metadata directory. This means the table'sFileIOmust implement theSupportsPrefixOperationsinterface.Fail-safe bias: a missing or unreadable uuid (legacy file without a uuid, corrupt file, or compressed file) is always treated as a conflict (preferring false positives over silently deleting another table's data).
Changes
OrphanFileUtils.hasOtherTableInLocation+readTableUuid(the latter@VisibleForTesting, transparently handling gzip); newTestOrphanFileUtilscovering conflict / no-conflict / null-uuid / compressed / corrupt / unsupported-FileIO cases.DeleteOrphanFilesinterface extension (above).DeleteOrphanFilesSparkActionextractsshouldSkipDueToLocationConflict()(called at the top ofdoExecute, keeping the main flow clean);RemoveOrphanFilesProcedureexposes alocation_conflict_modeparameter and forwards it.docs/spark-procedures.mddocuments the new parameter.API changes
In
api'sDeleteOrphanFilesinterface:enum LocationConflictMode { ERROR, IGNORE, DELETE }plusfromString(...)(consistent with the existingPrefixMismatchMode).Mode semantics:
ERROR(default)ValidationException; protects the other table (recommended default).IGNOREDELETEFileIOlacksSupportsPrefixOperations)DELETE.Tests
TestOrphanFileUtils(foreign metadata with same/foreign uuid, missing uuid, compressed, corrupt, unsupported-FileIO throws).TestRemoveOrphanFilesProcedureexercises all three modes across catalog variants.AI Disclosure
Model: hy3
Platform/Tool: CodeBuddy
Human Oversight: fully reviewed
Prompt Summary: Implement a shared-location guard for DeleteOrphanFiles (core detection helper + api LocationConflictMode enum + Spark 3.5/4.0/4.1 action/procedure wiring) and fix the resulting CI checkstyle/test failures.