sqlite3 to postgresql - migration

[adrpo]

Towards #295: the results move from a sqlite3 file per test machine to one shared PostgreSQL database, and the machines coordinate through it instead of overwriting each other.

Today every job downloads its machine's sqlite3.db, tests, and copies the whole file back. Two machines running the same job both write a full copy, and whichever finishes last wins - the other one's results are gone. That is the thing this PR removes.

What is in the database

omdb on openmodelica.org now holds all 160,247,359 rows of both machines: ripper1's 34 tables and ripper2's 54. 50 GB, or 20 GB on disk after the ZFS lz4 the tablespace sits on.

The layout is a 1:1 mirror of the sqlite3 one - one table per branch, same column names, only integer/real becoming bigint/double precision - so the test scripts push new results with the statements they already use.

What the mirror adds is a key per table, which is what makes a shared database possible at all:

table	key
<branch>	(date, libname, model)
omcversion	(date, branch)
libversion	(date, branch, libname, confighash)

They hold across all 160 million migrated rows, and they stop a run that is pushed twice from being stored twice.

Job claiming

Before testing a library, test.py claims it in job_claim, keyed by exactly the question the run already asks - which library, in which version, against which compiler and configuration:

CREATE TABLE job_claim (
  branch text, libname text, libversion text, omcversion text, confighash bigint,
  host text NOT NULL, state text NOT NULL,
  claimed_at timestamptz NOT NULL DEFAULT now(),
  heartbeat  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (branch, libname, libversion, omcversion, confighash)
);

Only one machine can win the claim. The others print

Skipping Buildings_9.1.0 as ripper2 has been testing it since 2026-08-10 22:14:03

and move on to the next library. The winner refreshes heartbeat every minute and sets state='done' when the results are written, so a machine that dies parks its jobs for 30 minutes rather than forever.

Using it

resultsdb.py puts both backends behind one interface. Every script takes --db, which defaults to the LIBTEST_DB environment variable:

./test.py --branch=master --db=postgresql://om@openmodelica.org/omdb configs/conf.json

The password comes from PGPASSWORD or ~/.pgpass, never from the URL or a command line. The scripts write the same statements for either backend, with ? as the placeholder, and ask the connection where the dialects genuinely differ: quote(), tableExists(), groupConcat(), countIf(), likeNoCase(), insertIgnore().

The Jenkinsfile gets a postgres parameter, on by default, which points the pipeline at omdb and drops both the wget of sqlite3.db and the sshPublisher that copied it back. Untick it and the old behaviour is exactly what it was. The password is a Jenkins secret file credential, omdb-pgpass, bound to PGPASSFILE at pipeline level.

Migrating and catching up

export PGPASSFILE=~/.pgpass
DB=/var/www/libraries.openmodelica.org/sqlite3
./sqlite2postgres.py --host localhost --sqlite $DB/ripper1/sqlite3.db --source ripper1
./sqlite2postgres.py --host localhost --index
./sqlite2postgres.py --host localhost --sqlite $DB/ripper2/sqlite3.db --source ripper2 --skip-existing
./sqlite2postgres.py --host localhost --index
./sqlite2postgres.py --host localhost --sqlite $DB/ripper1/sqlite3.db --verify

ripper1 goes in first, then ripper2 with --skip-existing, so ripper1 wins whenever a key collides. On the current data the priority never actually fires - not one (branch, date) is shared between the two machines, not even for the four branches both of them test - so the merge is a plain union. The rule matters for re-runs and for two machines pushing later on.

The script needs nothing but the standard library and the psql client. It streams through COPY in batches and commits each batch together with its progress row in migration_progress, so an interrupted migration continues where it stopped instead of duplicating rows. Run: ~50 minutes per machine at 48k rows/s, on the server itself.

--catch-up copies the runs a machine wrote after the migration read its database, which any job still using its sqlite3 file keeps doing until it is switched over. It picks the runs by date rather than continuing from the rowid it stopped at, because VACUUM renumbers the rowids of these tables and clean-empty-omcversion-dates.py runs one after every test - and because two machines write into the same table for the branches they both test. Two to three minutes per machine, and repeating it is harmless.

How it was checked

• --verify on both machines: 0 of 34 and 0 of 54 tables incomplete; every table's row count matches its source, and the shared tables are clean unions (libversion 310901 + 452246 = 763147, omcversion 11271 + 8776 = 20047).
• Round-trip fidelity: empty strings, NULLs, tabs, newlines, backslashes, a literal \N and unicode in model names all come back byte-identical. This is why COPY uses its text format and not CSV - an empty CSV field reads back as NULL, which would have silently turned the empty libversion strings of the old data into NULLs.
• Interrupting a migration: killed mid-run, restarted, no gaps and no duplicates.
• resultsdb.py self-test, both backends, 15 checks: two machines racing for one claim, the loser refused, a released claim taken over.
• Report queries against both databases: the same queries run on ripper1's sqlite file and on the migrated data return the same rows - phase counts, per-phase sums and regression rows. Checked on master, whose table holds 43 million of them (92 libraries, 17 regression rows), and on heavy_tests, conversion (252 regression rows) and basemodelica_jl_master. The two only render the doubles with a different number of digits, and the report parses them back with float().

That last test earned its keep: it caught a %-versus-+ precedence slip in the ported regression query that would have thrown TypeError on the first all-reports.py run.

Before enabling it

• ripper1 and ripper2 need psycopg2 - pip3 install psycopg2-binary. It is in requirements.txt and in .CI/build-dep, but the non-docker node stages run test.py with the node's python.
• test.py has not been run end-to-end against PostgreSQL here, because that needs omc and the libraries. The claim path and the insert path are verified against synthetic and migrated data, not a live run. Worth ticking postgres on one small job first - heavy_tests or generateSymbolicJacobian - and checking that job_claim fills in and the new rows land.
• Run --catch-up once more right after the switch, to sweep up whatever the last sqlite-era jobs wrote in the meantime.

Still open in #295: #78 (build each FMU once and simulate it with several tools) and dockerising the remaining Jenkins jobs.

---

Generated by Claude Code.

[adrpo]

@AnHeuermann and @sjoelund maybe you can have a quick look at this before I activate it so we can run all jobs on all the library testing computers.

[adrpo]

I will merge this to check on the actual computers if this works properly. We'll sort the issues afterwards.