optimise prometheus scraping workflow

[DaanHoogland]

Description

This PR...

Fixes: #13586

1. Removes alertManager.recalculateCapacity() from PrometheusExporterImpl.updateMetrics(). This is the actual fix — it stops the exporter from forcing a full, thread-pool-churning, all-zone capacity recalculation on every single scrape. The exporter should just read CapacityDao, which AlertManagerImpl's own timer already keeps current. This alone should flatten the scrape_duration curve.
2. Reuses a single long-lived ExecutorService in AlertManagerImpl instead of creating/tearing one down on every recalculateHostCapacities()/recalculateStorageCapacities() call. Cheap, low-risk, and removes the thread-churn contributor even for the timer-driven path.

this should solve the issue, but claude suggested the following improvements as well:

3. Give the exporter's HttpServer an explicit bounded executor (httpServer.setExecutor(Executors.newFixedThreadPool(2))) so one slow scrape can't serialize/queue all others.
4. Add a short TTL/in-flight guard around updateMetrics() (e.g., skip recompute if last run was < N seconds ago, or synchronize so concurrent scrapes share one in-progress computation) so scrape frequency can never multiply backend load.
5. Instrument: log/measure updateMetrics() wall-clock time so the reporter (and CI) can confirm which sub-metric collector is actually slow and verify the fix closes the growth.

if anyone wants to, these are nice to haves/good first issue ;)

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[codecov]

Codecov Report

❌ Patch coverage is 70.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 16.26%. Comparing base (e8df87e) to head (4261c3d).

Files with missing lines	Patch %	Lines
...rc/main/java/com/cloud/alert/AlertManagerImpl.java	70.00%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff            @@
##               4.20   #13650   +/-   ##
=========================================
  Coverage     16.26%   16.26%           
- Complexity    13435    13437    +2     
=========================================
  Files          5667     5667           
  Lines        500731   500732    +1     
  Branches      60803    60804    +1     
=========================================
+ Hits          81430    81436    +6     
+ Misses       410197   410189    -8     
- Partials       9104     9107    +3     

Flag	Coverage Δ
uitests	4.14% <ø> (ø)
unittests	17.11% <70.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[DaanHoogland]

@blueorangutan package

[blueorangutan]

@DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
46.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18617

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests