EPIC: Shutter API — Production Readiness for Continuous Time-Based Triggers

[ylembachar]

Driver: game customer, ~1-min rounds, time-based triggers on the internal set (1003).
Load: ~1,440 registrations/day and ≥1,440 key reads/day, sustained.

• Chiado rotated onto a new signer, verified end to end (register → key → decrypt), event registry ownership transferred, 8 consecutive rounds at 1/min passing
• Handed to the customer with an integration note
• Gas measured: 46,860 per registration
• Chiado alerting coverage verified (API and keypers both on InstanceDown, KeyperSyncStalled covers the crash-loop case)
• Issued mainnet key with higher limits and added to the config (see API config)
• Opened port to fallback Keyper on 1003 API
• Mainnet soak on 1003: 10 consecutive rounds at 1/min, 10/10 passing, key released within ~2s of the decryption timestamp (eon 1 live)
• Continuous test for 24h
• Per-key rate limit tiers (standard/premium), limits in limits.yaml rather than env variables (tested/not deployed)
• Deploy per-key rate limit tiers to 1003 / make sure same keys are kept / current tiers are hardcoded
• One API PR: nonce serialisation + receipt confirmation + gas bump in a shared sender (crypto.go:520/:536, same bugs in eventtrigger.go:406/:422), gauges→counters, add keys-served / key-miss / signer balance.
• Soak + overlapping-registration stress on 1003, and pass it.

Done when

• 60 rounds at 1/min on 1003, a key every round
• No silently dropped registrations under overlapping load
• Their key on the raised limits
• Alerts fire on keys-stopped and low signer balance
• Signer funded to a documented runway with a named owner

Notes

1. 200 OK from register_identity doesn't mean registered — reproduced live: two concurrent calls both returned 200, second tx reverted AlreadyRegistered(), success metric counted both.
2. Caddy rate-limit counters are in-memory and the snippet is read only at startup, so any restart resets everyone's window.
3. The Go suite never touches Caddy, so it cannot test rate limits. Use curl against the deployed host.
4. tests/integration spends real gas and runs migrations on whatever DB_URL you point it at.

[ylembachar]

Continuous test results:

2026/08/01 17:38:56 done rounds=1437 passed=1437 failed=0 elapsed=25h59m43s p50=6.368s p95=6.443s max=14.813s
--- PASS: TestContinuous (93582.69s)