maintainer: prevent stale spans from reentering scheduler state (#6073)

[ti-chi-bot]

This is an automated cherry-pick of #6073

What problem does this PR solve?

Issue Number: close #6072

What is changed and how it works?

This PR prevents a stale SpanReplication from being reintroduced into the scheduler’s Absent set after it has already been removed, replaced, or rebound to another node.

Previously, terminal dispatcher status handling performed the following operations separately:

1. Look up the SpanReplication by dispatcher ID.
2. Process the dispatcher’s Stopped or Removed status.
3. Check that no operator exists.
4. Call MarkSpanAbsent with the previously obtained span pointer.

A concurrent split could call ReplaceReplicaSet between these operations. The split removed the old span and created its replacement spans, but the terminal-status handler still held a pointer to the old span. Because MarkSpanAbsent did not verify that the span was still part of the controller’s desired state, it inserted the obsolete span into the scheduler’s Absent set.

This created a ghost Absent span: the scheduler repeatedly tried to create an Add operator for a dispatcher that no longer existed in the span controller, resulting in continuous add operator failed, span not found errors and a stalled checkpoint.

This PR makes the transition safe in two ways:

• MarkSpanAbsent now checks, while holding the span controller mutex, that the dispatcher ID is still registered in allTasks and that the registered value is the exact same SpanReplication instance. If the span has already been removed or replaced, the operation returns without changing scheduler or checkpoint-tracker state.

• The terminal-status fallback uses MarkSpanAbsentIfCurrent, which additionally verifies under the same lock that the span is still bound to the node that reported the terminal status. This prevents a delayed Stopped status from an old owner from marking a span Absent after it has already been moved to another node.

ReplaceReplicaSet and the new validation use the same span controller mutex. Therefore, removal/replacement and the Absent transition now have a deterministic order:

ReplaceReplicaSet first
-> old span is no longer the current task
-> MarkSpanAbsent is skipped

MarkSpanAbsent first
-> the span is still current and owned by the reporting node
-> the valid Absent transition completes

The terminal-status warning is now emitted only when the span is actually transitioned to Absent. Stale terminal statuses that lose the atomic validation are ignored without producing a misleading state-change log.

Check List

Tests

• Unit test
• Integration test

Questions

Will it cause performance regression or break compatibility?

Do you need to update user documentation, design documentation or monitoring documentation?

Release note

Fix a race condition that could stall the changefeed checkpoint and downstream replication after dispatcher split or move operations.

Summary by CodeRabbit

• Bug Fixes
  • Improved handling of terminal dispatcher statuses when no operator is available.
  • Prevented stale, removed, or reassigned spans from being incorrectly marked absent.
  • Ensured rescheduling occurs only when a span remains assigned to the reporting node.
  • Improved reliability during node removal and reassignment scenarios.
  • Increased synchronization wait time in split-and-merge integration checks.
  • Expanded MySQL test coverage across all test shards.

[ti-chi-bot]

This cherry pick PR is for a release branch and has not yet been approved by triage owners.
Adding the do-not-merge/cherry-pick-not-approved label.

To merge this cherry pick:

1. It must be LGTMed and approved by the reviewers firstly.
2. For pull requests to TiDB-x branches, it must have no failed tests.
3. AFTER it has lgtm and approved labels, please wait for the cherry-pick merging approval from triage owners.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[ti-chi-bot]

@wk989898 This PR has conflicts, I have hold it.
Please resolve them or ask others to resolve them, then comment /unhold to remove the hold label.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign 3aceshowhand for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

@ti-chi-bot: ## If you want to know how to resolve it, please read the guide in TiDB Dev Guide.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4d2c2046-d88a-4172-bb3c-c2f90fcc65b7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}