Skip to content

fix(service-operation): honor max_retries in uptime service checks#232

Open
Flegma wants to merge 1 commit into
operacle:developfrom
Flegma:fix/service-operation-max-retries
Open

fix(service-operation): honor max_retries in uptime service checks#232
Flegma wants to merge 1 commit into
operacle:developfrom
Flegma:fix/service-operation-max-retries

Conversation

@Flegma

@Flegma Flegma commented Jul 8, 2026

Copy link
Copy Markdown

What

Make the uptime checker honor the per-service Retry Attempts setting (services.max_retries). A service is now only marked down after all attempts in a check cycle fail, instead of after the first failed probe.

Closes #231. Fixes the behavior reported in #180 (closed as duplicate of #176) and complements the notification strategy discussed in #176.

Why

max_retries is configurable in the UI, stored by the frontend, and already deserialized into the Go Service struct, but nothing under server/ ever read it. performCheck ran exactly one attempt per cycle, so a single slow or dropped probe flapped the service to down and fired notifications. The legacy frontend checker honored the field (httpChecker.ts: service.max_retries || 3), so this restores parity lost in the migration to the Go microservice.

Real-world impact: on our production instance (interval 300s, Retry Attempts 5, 10s timeout) one probe crossing the timeout during a nightly backup produced a false "DOWN, then UP 4.5 minutes later" alert, with exactly one probe request visible in the origin access log per incident.

How

In server/service-operation/monitoring/checker.go:

  • performCheck now calls runCheckAttempts, which executes up to max_retries attempts (2s apart, per-attempt timeout unchanged at 10s) and returns on the first success, otherwise the final attempt's outcome.
  • The single-attempt switch is extracted unchanged into executeCheck (a plain function, so it is directly testable). Unknown service types keep the exact current behavior: log and return without a status update.
  • normalizeMaxRetries falls back to 3 (the UI default and legacy frontend default) when the field is unset or invalid, and clamps to 10 defensively.
  • The status determination, paused-service double-check, status update, and metrics saving are unchanged and operate on the final result.

Notes on behavior and scope:

  • Checks run in the per-service monitor goroutine (startMonitor), so retry sleeps only delay that service's own loop, never other services. Worst case with the UI maximum (5 attempts): about 58s inside one cycle.
  • Behavior change for existing installs: previously every service was effectively single-shot; with this PR an unset/default max_retries means 3 attempts, which is what the UI has claimed all along. Users who want single-shot semantics can set Retry Attempts to 1.
  • No UI or schema changes needed; the field and its persistence already exist.
  • Response time recorded for a down result is the final attempt's, matching current semantics.

Testing

Added monitoring/checker_test.go (first tests in the module):

  • max_retries normalization (unset, negative, in-range, above cap).
  • Service recovers within the retry budget: httptest server fails twice then succeeds; result is up and the server saw exactly 3 requests.
  • Retry Attempts = 1 keeps strict single-shot behavior (exactly 1 request).
  • All attempts failing yields down after exactly max_retries requests.
  • Unsupported service types return the sentinel error from both executeCheck and runCheckAttempts.

Ran with Go 1.21 (matching go.mod):

$ go build -o /tmp/service-operation .   # same target the Docker image builds
$ go vet ./monitoring/
$ go test ./monitoring/ -v
=== RUN   TestNormalizeMaxRetries
--- PASS: TestNormalizeMaxRetries (0.00s)
=== RUN   TestRunCheckAttemptsRecoversWithinRetryBudget
    ⚠️ flaky check failed (attempt 1/3), retrying in 10ms
    ⚠️ flaky check failed (attempt 2/3), retrying in 10ms
    ✅ flaky succeeded on attempt 3/3
--- PASS: TestRunCheckAttemptsRecoversWithinRetryBudget (0.03s)
=== RUN   TestRunCheckAttemptsSingleAttemptStaysSingleShot
--- PASS: TestRunCheckAttemptsSingleAttemptStaysSingleShot (0.00s)
=== RUN   TestRunCheckAttemptsMarksDownAfterAllAttemptsFail
--- PASS: TestRunCheckAttemptsMarksDownAfterAllAttemptsFail (0.02s)
=== RUN   TestExecuteCheckUnsupportedServiceType
--- PASS: TestExecuteCheckUnsupportedServiceType (0.00s)
PASS
ok      service-operation/monitoring

Changed files are gofmt clean. Heads-up unrelated to this PR: go build ./... on develop already fails in ping/icmp.go (a pre-existing type error in a package nothing imports); the main binary target builds fine before and after this change, so I left that package untouched to keep this PR focused.

The services collection stores max_retries (UI "Retry Attempts") and the
Go Service struct already deserializes it, but performCheck ran exactly
one attempt and marked the service down on the first failure. The field
was effectively write-only: notifications fired on a single slow or
dropped probe.

Retry the check up to max_retries times (2s apart, per-attempt timeout
unchanged) and only mark the service down after every attempt fails.
This mirrors the legacy frontend checker (httpChecker.ts used
service.max_retries || 3). Unset or invalid values fall back to 3;
values above 10 are clamped.

Checks run in the per-service monitor goroutine, so retry sleeps never
delay other services. Adds the module's first tests
(monitoring/checker_test.go).
@Flegma Flegma requested a review from tolaleng as a code owner July 8, 2026 13:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: "Retry Attempts" (max_retries) is ignored by the service checker, one failed probe marks a service down

1 participant