Reduce memory use of the waiting_tasks metric - #8009
Open
aKlimau wants to merge 1 commit into
Open
Conversation
aKlimau
changed the title
aKlimau
changed the title
aKlimau
force-pushed
the
tasks-in-memory
branch
from
August 25, 2026 13:48
a68db73 to
21abd31
Compare
Contributor
Author
|
After discussing all the suggested solutions in #7986 with claude, it looks like solutions 2 and 3 would have a positive impact on memory usage with no behavioral change. METRIC_HEARTBEAT_INTERVAL is a separate change that would make metrics not as fresh, so we'd need to decide weather it's worth. And using Redis cache doesn't save us much, since the computation is always done by a single worker and runs once per cycle. So introducing a round trip and another potential point of failure doesn't seem worth it to me. |
Contributor
Author
|
@decko by any chance do we have a way to measure the performance impact of such changes? (Memory allocation during count_waiting_tasks_for_metric() call) |
dralley
reviewed
Aug 25, 2026
| """ | ||
| Split a ``reserved_resources_record`` array into (exclusive, shared) resource names. | ||
|
|
||
| Shared reservations are stored with a ``"shared:"`` prefix; exclusive ones are bare. |
Contributor
There was a problem hiding this comment.
Get rid of the double backticks please
The metric scans every waiting/running task but only needs each task's reserved resources. Fetch just those instead of building a model object per row. ref pulp#7986 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aKlimau
force-pushed
the
tasks-in-memory
branch
from
August 26, 2026 10:40
21abd31 to
72bfa27
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
count_waiting_tasks_for_metric() now streams reserved_resources_record via values_list().iterator() instead of materializing deferred Task instances.
The "shared:" split is moved into redis_locks._split_reserved_resources().
ref #7986
📜 Checklist
See: Pull Request Walkthrough