Survive losing the connection to the results database

[adrpo]

A run opens its connection to the results database, claims its libraries, and then tests for hours before writing anything: every model's results go into a single transaction at the very end, after the reports have been generated and uploaded.

That leaves the connection idle for the whole run — and an idle connection is exactly what a firewall, a NAT or a restarted server drops. When that happened, the first INSERT at the end threw

psycopg2.OperationalError: server closed the connection unexpectedly

test.py died, and a night of testing was lost. Nothing looked wrong until that moment either: the heartbeat that holds the job claims opens a connection of its own every minute, so the claims kept being refreshed while the connection that mattered was already dead.

What this does

• Asks for TCP keepalives (keepalives=1 keepalives_idle=60 …), so an idle connection is kept alive and a genuinely dead one is noticed early rather than at the end.
• Reconnects and replays if it is lost anyway: the statement that discovers the broken connection reconnects, replays the statements this transaction had written so far, and carries on.

Replaying is safe because of what #295 already put in place: everything a run writes is an insert guarded by a unique key — (date, libname, model) for a branch table — and the inserts use ON CONFLICT DO NOTHING. Replaying can only produce the rows that were lost, never a duplicate. Queries are not remembered, so the tens of thousands a run makes do not fill the buffer.

Nothing in test.py changes; the recovery lives in resultsdb.py and the sqlite3 backend is untouched.

How it was checked

Against a real PostgreSQL, with a run's transaction half written, killing its backend the way a firewall would:

run's connection is backend 4117692
killing backend 4117692 after 250 rows...
  pg_terminate_backend -> t
Lost the connection to the database (SSL connection has been closed unexpectedly); reconnecting
Replayed 250 statements of the interrupted transaction
rows written: 500, rows in the database: 500 -> PASS
usable afterwards: PASS

Also, that the paths this must not disturb still work: a full configs/sanityCheck.json run on sqlite3 with the CI assertion passing, and a run against PostgreSQL with two FMI simulators filling v1.27-fmi and v1.27-fmi-fmpy and releasing its claim.

What this is not

It makes losing the connection survivable; it does not make the results arrive any earlier. A run that is killed still loses everything it has tested, because it has written nothing yet.

Writing the results per library, as each one finishes — in the same transaction that marks that library's claim done — would fix that properly: a lost connection or a crash would then cost one library instead of a night, every library in the database would be whole rather than half-tested, and a rerun would resume where the last one stopped. It needs the results to be gathered per library rather than accumulated to the end, so it is worth doing as its own change.

---

Generated by Claude Code.