LAC: smoothing tokens request and keep in high level (#10997)

[ti-chi-bot]

This is an automated cherry-pick of #10997

What problem does this PR solve?

Issue Number: close #10996

Summary

This change improves TiFlash Local Admission Controller token refill behavior to keep the local token bucket near a high watermark without requesting a large amount of tokens in a single GAC request.

Problem

The previous acquire calculation was based only on predicted consumption:

acquire_tokens = max(smoothed_speed * 5s * 1.1 - remaining_tokens, 0)

When the smoothed consumption speed was underestimated, a small positive token balance could make acquire_tokens zero. The local balance would then remain low and could be exhausted by a traffic burst, causing unexpected throttling.

Always refilling directly to the full bucket capacity would avoid this problem, but could transfer and retain too many tokens in TiFlash at once, reducing the tokens available to other clients such as TiDB.

Changes

• Added a proactive refill watermark at 80% of the local high watermark.
• Added a one-second refill check interval in normal mode.
• Included proactive refill checks in addition to the existing low-token and consumption-report triggers.
• Added incremental token acquisition for normal refills:

deficit = high_watermark - remaining_tokens

fallback_batch = min(
    5000,
    high_watermark * 20%
)

incremental_batch = max(
    smoothed_consumption_speed * 1s * 1.1,
    fallback_batch
)

acquire_tokens = min(deficit, incremental_batch)

• Preserved emergency refill behavior when the bucket reaches the existing low-token threshold. In that case, the incremental limit is bypassed to avoid request throttling.
• Before the first GAC token response, the Resource Group fill_rate is used as the local high watermark.
• After the first GAC response, the capacity assigned by GAC to the current client is used as the high watermark.
• Added has_gac_capacity state to distinguish the global Resource Group burst limit from the capacity assigned to the local client.
• Added a read-only TokenBucket::getCapacity() accessor.
• Kept the low-token threshold based on the actual post-grant token balance. This prevents a capacity increase from immediately classifying the bucket as low-token and triggering a large emergency refill.
• Preserved the existing five-second consumption reporting period and GAC target request period.
• Did not change RU accounting, token deduction, GAC grant handling, or trickle-mode semantics.

Resulting Behavior

• TiFlash starts refilling before the local bucket reaches a critically low balance.
• Normal refill requests are spread across smaller requests instead of immediately filling the entire capacity.
• High-throughput workloads can still request approximately one second of predicted consumption per refill.
• Low-token conditions retain an emergency path that prioritizes avoiding unexpected query throttling.
• A newly started TiFlash instance does not use the global Resource Group burst limit as its initial local refill target.
• Unused tokens are less likely to be transferred from GAC to TiFlash in one large request, reducing the impact on other clients sharing the Resource Group.

##Test
During bench tpch workload, after acquire tokens from GAC, the remaining_tokens keeps close to the high watermark.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes
  • Improved resource admission control token refilling for more accurate capacity tracking.
  • Reduced the risk of under-refilling or over-refilling resources during startup and normal operation.
  • Improved handling of low-token conditions to help maintain smoother request throughput.
  • Refined refill behavior when predicted resource consumption changes.
• Tests
  • Added coverage for startup, incremental refill, predicted consumption, and low-token scenarios.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (3)

• release-8.5
• release-7.5
• release-8.1

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 91f4cbc7-1025-489b-8a8c-c023cdb201ca

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, yongman

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,yongman]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-23 02:47:51.335441008 +0000 UTC m=+1459457.371536065: ☑️ agreed by JaySon-Huang.
• 2026-07-23 02:48:08.196795305 +0000 UTC m=+1459474.232890361: ☑️ agreed by yongman.