LAC: smoothing tokens request and keep in high level

[yongman]

What problem does this PR solve?

Issue Number: close #10996

Summary

This change improves TiFlash Local Admission Controller token refill behavior to keep the local token bucket near a high watermark without requesting a large amount of tokens in a single GAC request.

Problem

The previous acquire calculation was based only on predicted consumption:

acquire_tokens = max(smoothed_speed * 5s * 1.1 - remaining_tokens, 0)

When the smoothed consumption speed was underestimated, a small positive token balance could make acquire_tokens zero. The local balance would then remain low and could be exhausted by a traffic burst, causing unexpected throttling.

Always refilling directly to the full bucket capacity would avoid this problem, but could transfer and retain too many tokens in TiFlash at once, reducing the tokens available to other clients such as TiDB.

Changes

• Added a proactive refill watermark at 80% of the local high watermark.
• Added a one-second refill check interval in normal mode.
• Included proactive refill checks in addition to the existing low-token and consumption-report triggers.
• Added incremental token acquisition for normal refills:

deficit = high_watermark - remaining_tokens

fallback_batch = min(
    5000,
    high_watermark * 20%
)

incremental_batch = max(
    smoothed_consumption_speed * 1s * 1.1,
    fallback_batch
)

acquire_tokens = min(deficit, incremental_batch)

• Preserved emergency refill behavior when the bucket reaches the existing low-token threshold. In that case, the incremental limit is bypassed to avoid request throttling.
• Before the first GAC token response, the Resource Group fill_rate is used as the local high watermark.
• After the first GAC response, the capacity assigned by GAC to the current client is used as the high watermark.
• Added has_gac_capacity state to distinguish the global Resource Group burst limit from the capacity assigned to the local client.
• Added a read-only TokenBucket::getCapacity() accessor.
• Kept the low-token threshold based on the actual post-grant token balance. This prevents a capacity increase from immediately classifying the bucket as low-token and triggering a large emergency refill.
• Preserved the existing five-second consumption reporting period and GAC target request period.
• Did not change RU accounting, token deduction, GAC grant handling, or trickle-mode semantics.

Resulting Behavior

• TiFlash starts refilling before the local bucket reaches a critically low balance.
• Normal refill requests are spread across smaller requests instead of immediately filling the entire capacity.
• High-throughput workloads can still request approximately one second of predicted consumption per refill.
• Low-token conditions retain an emergency path that prioritizes avoiding unexpected query throttling.
• A newly started TiFlash instance does not use the global Resource Group burst limit as its initial local refill target.
• Unused tokens are less likely to be transferred from GAC to TiFlash in one large request, reducing the impact on other clients sharing the Resource Group.

##Test
During bench tpch workload, after acquire tokens from GAC, the remaining_tokens keeps close to the high watermark.

Check List

Tests

• Unit test
• Integration test
• Manual test (add detailed scripts or steps below)
• No code

Side effects

• Performance regression: Consumes more CPU
• Performance regression: Consumes more Memory
• Breaking backward compatibility

Documentation

• Affects user behaviors
• Contains syntax changes
• Contains variable changes
• Contains experimental features
• Changes MySQL compatibility

Release note

None

Summary by CodeRabbit

• Bug Fixes
  • Improved resource admission control token refilling for more accurate capacity tracking.
  • Reduced the risk of under-refilling or over-refilling resources during startup and normal operation.
  • Improved handling of low-token conditions to help maintain smoother request throughput.
  • Refined refill behavior when predicted resource consumption changes.
• Tests
  • Added coverage for startup, incremental refill, predicted consumption, and low-token scenarios.

[coderabbitai]

📝 Walkthrough

Walkthrough

ResourceGroup now uses high-watermark-based token refill decisions, tracks GAC capacity availability, limits normal-mode refill batches, and triggers GAC requests through the new refill predicate. Tests cover startup, incremental, predicted-consumption, and low-token refill behavior.

Changes

Token Refill Control

Layer / File(s)	Summary
Refill state and calculation dbms/src/Flash/ResourceControl/LocalAdmissionController.h, dbms/src/Flash/ResourceControl/TokenBucket.h, dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp	Adds refill constants and runtime capacity state, exposes token-bucket capacity, and recalculates high-watermark and acquisition amounts.
Request refill integration dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp	Uses the refill interval for acquisition sizing, records positive normal/trickle capacities, and allows shouldRefillToken to trigger GAC requests.
Refill behavior validation dbms/src/Flash/ResourceControl/tests/gtest_local_admission_controller.cpp	Updates request-info preconditions and adds coverage for startup, incremental, predicted-consumption, and low-token refill paths.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant LocalAdmissionController
  participant ResourceGroup
  participant TokenBucket
  LocalAdmissionController->>ResourceGroup: shouldRefillToken(current_tick)
  ResourceGroup->>TokenBucket: getCapacity()
  TokenBucket-->>ResourceGroup: configured capacity
  ResourceGroup-->>LocalAdmissionController: refill decision
  LocalAdmissionController->>ResourceGroup: buildRequestInfoIfNecessary()
  ResourceGroup-->>LocalAdmissionController: acquire_tokens

Loading

Suggested labels: approved, lgtm

Suggested reviewers: jayson-huang

Poem

I’m a rabbit with tokens to spare,
High-water marks float through the air.
Small refills hop in line,
Low buckets get mine,
And GAC requests bloom everywhere!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address #10996 by triggering earlier refills, smoothing normal requests, and preserving the low-token emergency path.
Out of Scope Changes check	✅ Passed	The modified code and tests are focused on the reported Local Admission Controller refill behavior with no clear unrelated scope creep.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Title check	✅ Passed	The title is related to the main change, describing smoother token requests and keeping the bucket at a higher level.
Description check	✅ Passed	The description covers the required sections and includes the issue number, change summary, tests, and release note.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (1)

dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp (1)

505-506: 🚀 Performance & Scalability | 🔵 Trivial

Refill trigger increases GAC request cadence; confirm GAC-side load headroom.

With shouldRefillToken gating on an 80% high-watermark and a 1s REFILL_TOKEN_INTERVAL, every normal-mode group whose bucket sits below 80% will now emit a token request each mainLoop tick (~1s), versus the prior low-token-only fetch. Under steady consumption groups tend to stay below the watermark, so per-group GAC request frequency rises materially with the number of active resource groups. Worth confirming GAC can absorb the aggregate rate at your expected group count, and consider a metric/alert on type_request_gac_count to watch for request storms.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp` around lines 505
- 506, Review the refill condition in the local admission-control main loop,
specifically the shouldRefillToken call combined with
local_keyspace_low_token_resource_groups. Confirm the resulting per-group GAC
request cadence is within expected capacity at the maximum active resource-group
count, and add monitoring or an alert for type_request_gac_count to detect
request storms.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp`:
- Around line 505-506: Review the refill condition in the local
admission-control main loop, specifically the shouldRefillToken call combined
with local_keyspace_low_token_resource_groups. Confirm the resulting per-group
GAC request cadence is within expected capacity at the maximum active
resource-group count, and add monitoring or an alert for type_request_gac_count
to detect request storms.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 32727b95-3f40-472e-8dad-bd7f3ac23edd

📥 Commits

Reviewing files that changed from the base of the PR and between 3d986e3 and c228b29.

📒 Files selected for processing (4)

• dbms/src/Flash/ResourceControl/LocalAdmissionController.cpp
• dbms/src/Flash/ResourceControl/LocalAdmissionController.h
• dbms/src/Flash/ResourceControl/TokenBucket.h
• dbms/src/Flash/ResourceControl/tests/gtest_local_admission_controller.cpp

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JaySon-Huang, Lloyd-Pottiger

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [JaySon-Huang,Lloyd-Pottiger]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-07-22 12:24:05.507281226 +0000 UTC m=+1407631.543376282: ☑️ agreed by Lloyd-Pottiger.
• 2026-07-23 02:41:25.104714596 +0000 UTC m=+1459071.140809652: ☑️ agreed by JaySon-Huang.

[yongman]

/cherry-pick release-nextgen-202603

[ti-chi-bot]

@yongman: new pull request created to branch release-nextgen-202603: #10998.

Details

In response to this:

/cherry-pick release-nextgen-202603

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@yongman: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-sanitizer-tsan	c228b29	link	false	/test pull-sanitizer-tsan
pull-unit-test	c228b29	link	unknown	/test pull-unit-test

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[JaySon-Huang]

/cherry-pick release-8.5

[ti-chi-bot]

@JaySon-Huang: new pull request created to branch release-8.5: #11015.
But this PR has conflicts, please resolve them!

Details

In response to this:

/cherry-pick release-8.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.