Continuous monitor: follow-ups
Not urgent, none of it blocks the monitor doing its job. Picking up when metrics are needed.
Fix the latency histogram buckets. They're 0.5/1/2/3/5/8/15/30/60/120 seconds, but every round we've measured lands between 4 and 7, so the whole distribution sits in one bucket. A percentile query would return an interpolated guess and couldn't show a slowdown from 6.4s to 7.5s. Needs half-second steps from 4 to 7, then wider steps that separate one missed block from two. No results so far are affected — the summary percentiles are computed in Go from exact measurements, and metrics were off during the 26h soak.
Turn metrics on. Needs a vmagent.yaml scrape config, plus REMOTE_WRITE_URL and an instance label in .env. Set the labels via vmagent flags rather than in the config file so no host name is committed to a public repo.
Add the alert. last_success_timestamp_seconds going stale is the one to alert on: it fires whether the API stopped serving keys or the monitor itself died. Lives with the vmalert config, so it's a change in that repo. Don't alert on latency — a slow round is a missed block, and a round slow enough to matter already counts as a failure.
Write per-round results to a file. CSV or JSONL with exact values, so analysis doesn't depend on docker logs, which are lost when the container is recreated. This is the useful idea from the old decryption monitor in shutter-tests.
Retire shutter-tests/tests/decryptionmonitor.go. Superseded — it checks that a key came back but never that it decrypts. Merge the rest of that branch separately.
Decide on restart behaviour for standing-monitor mode. Currently no restart policy, which is right for a bounded soak but means a monitor doesn't survive a reboot. unless-stopped is safe once the duration is unbounded, since it never exits on its own.
Continuous monitor: follow-ups
Not urgent, none of it blocks the monitor doing its job. Picking up when metrics are needed.
Fix the latency histogram buckets. They're 0.5/1/2/3/5/8/15/30/60/120 seconds, but every round we've measured lands between 4 and 7, so the whole distribution sits in one bucket. A percentile query would return an interpolated guess and couldn't show a slowdown from 6.4s to 7.5s. Needs half-second steps from 4 to 7, then wider steps that separate one missed block from two. No results so far are affected — the summary percentiles are computed in Go from exact measurements, and metrics were off during the 26h soak.
Turn metrics on. Needs a
vmagent.yamlscrape config, plusREMOTE_WRITE_URLand an instance label in.env. Set the labels via vmagent flags rather than in the config file so no host name is committed to a public repo.Add the alert.
last_success_timestamp_secondsgoing stale is the one to alert on: it fires whether the API stopped serving keys or the monitor itself died. Lives with the vmalert config, so it's a change in that repo. Don't alert on latency — a slow round is a missed block, and a round slow enough to matter already counts as a failure.Write per-round results to a file. CSV or JSONL with exact values, so analysis doesn't depend on docker logs, which are lost when the container is recreated. This is the useful idea from the old decryption monitor in shutter-tests.
Retire
shutter-tests/tests/decryptionmonitor.go. Superseded — it checks that a key came back but never that it decrypts. Merge the rest of that branch separately.Decide on restart behaviour for standing-monitor mode. Currently no restart policy, which is right for a bounded soak but means a monitor doesn't survive a reboot.
unless-stoppedis safe once the duration is unbounded, since it never exits on its own.