Skip to content

HDDS-13173: Clean up temporary checkpoint directories on snapshot defragmentation failure#10774

Draft
cchung100m wants to merge 2 commits into
apache:masterfrom
cchung100m:HDDS-13173
Draft

HDDS-13173: Clean up temporary checkpoint directories on snapshot defragmentation failure#10774
cchung100m wants to merge 2 commits into
apache:masterfrom
cchung100m:HDDS-13173

Conversation

@cchung100m

@cchung100m cchung100m commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this pull request?

Root Cause: When snapshot defragmentation fails due to transient errors, the temporary checkpoint directories are not cleaned up. Over 3+ days of continuous operations, these orphaned directories accumulate and exhaust disk space, causing OM startup failures.

Solution: Implement multi-layer exception handling with an explicit defragSuccessful flag.

  1. Add success flag defragSuccessful: Initialize to false, set to true only after atomicSwitchSnampshotDB() succeeds.
  2. Protect close(): Wrap in try-catch to prevent cleanup bypass
  3. Protect old checkpoint deletion: Wrap in try-catch but continue with flag set
  4. Add cleanup in finally block: Delete checkpoint directory if !defragSuccessful
  5. Add metrics: Track cleanup failures via incNumSnapshotDefragFails()

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-13173

How was this patch tested?

  • Existing unit and integration tests validate the current change cause unexpected impacts
  • Add new test case testCheckpointCleanupOnDefragFailure to validate the exception handling

@cchung100m cchung100m marked this pull request as ready for review July 15, 2026 15:33
@adoroszlai

Copy link
Copy Markdown
Contributor

Thanks @cchung100m for working on this.

  • Stack trace in the Jira issue indicates root cause of OM startup failure was No space left on device. Can you please explain how a change in SnapshotDefragService solves that problem?
  • I'm not sure allowing OM to start with No space left on device is useful/desired.
  • Please describe testing in more detail. "Existing tests" does not cut it, since this is supposed to fix a bug, but existing tests are already passing without the patch.

@cchung100m cchung100m marked this pull request as draft July 15, 2026 16:45
@cchung100m cchung100m changed the title HDDS-13173: Fix OM start failed with exception java.io.IOException HDDS-13173: Clean up temporary checkpoint directories on snapshot defragmentation failure Jul 15, 2026
@adoroszlai adoroszlai requested a review from smengcl July 16, 2026 11:32
@adoroszlai adoroszlai added the snapshot https://issues.apache.org/jira/browse/HDDS-6517 label Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

snapshot https://issues.apache.org/jira/browse/HDDS-6517

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants