Backport #24554 to 7.82.x: keep broker_timestamps in memory and persist every 5 minutes

[piochelepiotr]

What does this PR do?

Backports #24554 to the 7.82.x branch. Keeps the DSM broker_timestamps history in memory across check runs (loaded from the persistent cache only once, on the first run) and persists it back to disk at most once every 5 minutes, instead of loading and dumping the whole structure on every run.

Motivation

With Data Streams enabled, the broker_timestamps cache grows toward its per-cluster entry cap. Loading it from disk and re-dumping it on every run created a large transient allocation each cycle that scales with the cache size, fragmenting the heap and driving agent RSS up. Holding the structure in memory removes the per-run load/dump churn; periodic persistence (every 5 min) preserves restart durability while keeping write churn low. On agent restart at most a few minutes of timestamp history is lost, which is acceptable for lag-in-seconds interpolation.

See #incident-57455 for the impact on memory fragmentation.

Note: the 7.82.x branch still persists broker_timestamps via json.dumps/json.loads, since the switch to base64+marshal serialization landed on master in a separate, later commit not covered by #24554. This backport applies the same in-memory/periodic-persistence behavior on top of the existing json-based read/write, and does not pull in the marshal/base64 change.

Review checklist (to be filled by reviewers)

• Feature or bugfix MUST have appropriate tests (unit, integration, e2e)
• Add qa/required if this PR needs QA validation, or qa/skip-qa if it does not. Exactly one of the two is required.
• If you need to backport this PR to another branch, you can add the backport/<branch-name> label to the PR and it will automatically open a backport PR once this one is merged

[temporal-github-worker-1]

Backport evaluation: View Slack thread

[datadog-prod-us1-5]

 

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 91.81%

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 7410371 | Docs | Datadog PR Page | Give us feedback!}

[dd-octo-sts]

Validation Report

All 21 validations passed.

Show details

Validation	Description	Status
agent-reqs	Verify check versions match the Agent requirements file	✅
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
dep	Verify dependency pins are consistent and Agent-compatible	✅
http	Validate integrations use the HTTP wrapper correctly	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
labeler	Validate PR labeler config matches integration directories	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
license-headers	Validate Python files have proper license headers	✅
licenses	Validate third-party license attribution list	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
openmetrics	Validate OpenMetrics integrations disable the metric limit	✅
package	Validate Python package metadata and naming	✅
qa-label	Validate the pull request declares whether it needs QA for the next Agent release	✅
readmes	Validate README files have required sections	✅
saved-views	Validate saved view JSON file structure and fields	✅
version	Validate version consistency between package and changelog	✅

View full run