Skip to content

[K8s] Fix failed application cancellation state race#4459

Open
johntomcat7408-cmyk wants to merge 1 commit into
apache:devfrom
johntomcat7408-cmyk:fix/4332-k8s-failing-cancel
Open

[K8s] Fix failed application cancellation state race#4459
johntomcat7408-cmyk wants to merge 1 commit into
apache:devfrom
johntomcat7408-cmyk:fix/4332-k8s-failing-cancel

Conversation

@johntomcat7408-cmyk

@johntomcat7408-cmyk johntomcat7408-cmyk commented Jul 25, 2026

Copy link
Copy Markdown

What changes were proposed in this pull request

Issue Number: close #4332

Allow Kubernetes application jobs in FAILING state to be canceled and make cancellation persistence robust to asynchronous watcher events both before and after terminal convergence.

The persistence rules are deliberately scoped:

  • While the database row is CANCELLING with option state CANCELLING or SAVEPOINTING, non-terminal watcher snapshots may refresh runtime metrics but cannot replace the main or option state.
  • Once cancellation has converged to CANCELED, the mapper atomically rejects a late non-terminal metrics update. Queued FAILING, RUNNING, or CANCELLING snapshots therefore cannot revert the terminal state or write their stale snapshot fields.
  • Terminal watcher snapshots remain authoritative. An explicit restart first moves the row to STARTING, while interrupted CANCELLING/NONE and recoverable LOST states can still advance normally.
  • An ordinary standalone savepoint can still clear SAVEPOINTING.

Brief change log

  • Show the Cancel action for both RUNNING and FAILING applications.
  • Preserve the main and option states while cancel or cancel-with-savepoint is active.
  • Reject late non-terminal snapshots after cancellation has reached CANCELED.
  • Keep terminal convergence, explicit restart, Console restart recovery, LOST recovery, and standalone savepoint completion intact.
  • Add regression coverage for the active-cancel race and the completed-cancel race.

Verifying this change

This change added tests and was verified as follows:

  • FlinkApplicationManageServiceTest: 12 tests run, 0 failures, 0 errors, and 1 pre-existing disabled integration test.
  • FlinkJobStatusWatcherTest: 2 tests passed.
  • Maven Spotless checks passed for the affected Console and Kubernetes modules.
  • Maven Checkstyle passed for the affected Console module.
  • Kubernetes Reactor verify compiled main and test sources; Scalastyle processed 39 files with 0 errors and 0 warnings.
  • git diff --check passed.
  • The final mapper behavior was exercised through Spring, MyBatis, and H2 MySQL compatibility mode. The active-cancellation conditional-update matrix was also checked on MySQL 8.0.39 and H2 PostgreSQL compatibility mode.

Does this pull request potentially affect one of the following parts

  • Dependencies (does it add or upgrade a dependency): no

Allow failed Kubernetes applications to be canceled while protecting active and completed cancellation from stale watcher snapshots. Keep the database guard scoped so interrupted cancellation recovers after Console restart, explicit restarts and terminal corrections remain authoritative, and standalone savepoint completion remains unaffected.

Refs apache#4332
@johntomcat7408-cmyk
johntomcat7408-cmyk force-pushed the fix/4332-k8s-failing-cancel branch from 73be3be to de2697c Compare July 26, 2026 07:19
@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flink job canceled by user,but status is failing and cannot be cancelled and stopped correctly

1 participant