docs: record F8 (silent config dropout) and F9 (stale kit) from the 7-site run

[Ultimate-Storm]

Records two failure modes found while running the first 7-site ODELIA swarm test today. Both cost real debugging time, and both are invisible unless you go looking — so they belong in the runbook, not a chat log.

F8 — silent client dropout during configuration ⚠️ the important one

The swarm controller stops waiting once configure_min_clients sites have answered swarm_config, then advances. A site whose job cell needs another second or two to come up receives the p2p key_exchange before its child cell exists:

CoreCell ERROR ... CH=credential_manager TP=key_exchange ... cannot forward req: no path

…and is dropped. Its training subprocess then idles until the 900 s pipe timeout and exits with No best checkpoint found — having trained nothing. The run continues and reports success on a subset.

Observed today with the default min_clients=5 on a 7-site run: RUMC_1 and UKA_1 were dropped at 5.8 seconds. Re-submitting the identical job with --min-clients 7 --configure-min-clients 7 configured all seven. Neither site needed any change — they were healthy, correctly provisioned, data loading fine. It was purely a server-side scheduling parameter.

The sharp edge: the default drops the slowest sites, which on a real consortium are the largest ones — exactly the data you least want to lose. For a benchmark, configure_min_clients must equal the participating-site count.

F9 — stale startup kit, client "healthy" but never registers

A site whose kit predates the server's provisioning generation loops on ClientConnectorCertificateError while Docker reports Up N weeks (healthy), so nobody notices. RSH_1 had been in that state for three weeks. Docker health means the process is alive; it says nothing about FL registration.

Also

Extends Before every run with a post-submit participation check — registered != configured != training, and only the middle one is visible in the job log:

grep -c 'successfully configured client ' <server-kit>/<job_id>/log.txt   # must equal site count

Docs-only; no CI impact.

🤖 Generated with Claude Code

[oleschwen]

ad F8: For a more flexible swarm, I would expect that one can configure a minimal number of clients and that there is some mechanism that would allow additional clients to register within a short time frame. For our current purposes, however, I agree that we want to know that all sites are on board; we can reduce the number if some are known to be absent.

ad F9: We could consider making a TLS error visible to users as a failure and exit the client container if we want that. However, this would not always show up when starting a client—we actually want to be able to start a client and try to connect until the server is available later, or continue a client container to be running even if the server is temporarily unavailable. It is not feasible to check this on the client side, so I agree that this needs to be checked when submitting a job.

[oleschwen]

Looks good.
Documentation only, no change in code.