You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Let the testing use a shared PostgreSQL database instead of sqlite3 files
Second part of issue #295: the test machines can now write their results to one
network database and coordinate through it, instead of each copying a sqlite3
file in, testing, and copying it back - the step that made two machines
overwrite each other depending on which one finished last.
resultsdb.py holds both backends behind one interface. The scripts write the
same statements for either, with "?" as the placeholder, and ask the connection
where the dialects genuinely differ: quoting a branch name, testing whether a
table exists, concatenating a group, counting a condition, matching a branch
without case, skipping a row that is already there. Every script takes --db,
which defaults to the LIBTEST_DB environment variable, so Jenkins sets the
database once for the whole pipeline rather than on forty invocations.
A machine claims a job in [job_claim] before testing a library, keyed by
exactly the question the run already asks: which library, in which version,
against which compiler and configuration. Only one machine can win the claim,
and the others skip that library and move on. The winner refreshes a heartbeat
every minute and marks the claim done when the results are written, so a
machine that dies parks its jobs for STALE_CLAIM_MINUTES rather than forever.
A local sqlite3 file has a single writer, so there claim() always says yes.
Two PostgreSQL specifics were needed for the reports: GROUP_CONCAT relies on
sqlite keeping the order of the subquery that feeds it, so the ported query
orders inside the aggregate, and COUNT(x or null) becomes COUNT(*) FILTER.
Running the report queries against ripper1's sqlite file and against the
migrated database gives the same rows, down to the phase counts, the per-phase
sums and the regression rows: checked on master, whose table holds 43 million
of them, and on heavy_tests, conversion and basemodelica_jl_master. The two
only render the doubles with a different number of digits, and the report
parses them back with float().
The Jenkinsfile gets a "postgres" parameter, on by default, which selects the
database and drops the download and the publishing of sqlite3.db. Unticking it
restores the old behaviour unchanged. The password comes from a Jenkins secret
file credential, omdb-pgpass, bound to PGPASSFILE, so it never reaches a
command line or the build log.
---
Generated by Claude Code.
Signed-off-by: Adrian Pop <adrian.pop@liu.se>
Copy file name to clipboardExpand all lines: .CI/Jenkinsfile
+40-15Lines changed: 40 additions & 15 deletions
Original file line number
Diff line number
Diff line change
@@ -3,6 +3,8 @@ pipeline {
3
3
parameters {
4
4
booleanParam(name: 'OLDLIBS', defaultValue: false, description: 'Also test some outdated libraries')
5
5
6
+
booleanParam(name: 'postgres', defaultValue: true, description: 'Store the results in the shared PostgreSQL database (omdb on openmodelica.org) rather than in the per-machine sqlite3 file. Machines coordinate through it, so two of them no longer overwrite each other. Untick to go back to the sqlite3 files.')
cursor.execute('''CREATE INDEX IF NOT EXISTS [idx_%s_date] ON [%s](date)'''% (branch,branch))
115
+
db.createDateIndex(branch)
116
116
libs= {}
117
-
for (date,libname,total,frontend,backend,simcode,template,compile,simulate,verify) incursor.execute("""SELECT date,libname,COUNT(finalphase),COUNT(finalphase>=1 or null),COUNT(finalphase>=2 or null),COUNT(finalphase>=3 or null),COUNT(finalphase>=4 or null),COUNT(finalphase>=5 or null),COUNT(finalphase>=6 or null),COUNT(finalphase>=7 or null)
118
-
FROM [%s]
117
+
for (date,libname,total,frontend,backend,simcode,template,compile,simulate,verify) incursor.execute("""SELECT date,libname,COUNT(finalphase),%s
# Order by date so we can select and know which is the older and which is the newer value... for finalphase, and the execution times
169
-
# Note: GROUP_CONCAT returns both values as a string... So you need to split it later
170
-
query="""SELECT model,libname,GROUP_CONCAT(finalphase),GROUP_CONCAT(frontend),GROUP_CONCAT(backend),GROUP_CONCAT(simcode),GROUP_CONCAT(templates),GROUP_CONCAT(compile),GROUP_CONCAT(simulate) FROM
171
-
(SELECT model,libname,finalphase,frontend,backend,simcode,templates,compile,simulate FROM [%s] WHERE date IN (?,?) AND libname IN (%s) ORDER BY date)
172
-
GROUP BY model,libname HAVING
169
+
# Note: the group concatenation returns both values as a string... So you need to split it
170
+
# later. The order is the one of the dates, which PostgreSQL only guarantees when the
171
+
# aggregate says so, hence the date column in the inner query.
(SELECT model,libname,date,finalphase,frontend,backend,simcode,templates,compile,simulate FROM %%s WHERE date IN (?,?) AND libname IN (%%s) ORDER BY date) AS phases
176
+
GROUP BY model,libname HAVING"""%concat+"""
173
177
(MIN(finalphase) <> MAX(finalphase)) OR
174
178
((MIN(finalphase) >= ?) AND
175
179
(MAX(frontend) > ?*MIN(frontend) AND MAX(frontend) > ?) OR
cursor.execute("SELECT libversion,confighash FROM [libversion] WHERE branch LIKE ? COLLATE NOCASE AND date<=? AND libname=? ORDER BY date DESC LIMIT 1", (branch,d1,libname))
235
+
cursor.execute("SELECT libversion,confighash FROM libversion WHERE %s AND date<=? AND libname=? ORDER BY date DESC LIMIT 1"%db.likeNoCase("branch"), (branch,d1,libname))
232
236
(lv1,lh1) =cursor.fetchone()
233
237
lv1=lv1.strip()
234
-
cursor.execute("SELECT libversion,confighash FROM [libversion] WHERE branch LIKE ? COLLATE NOCASE AND date<=? AND libname=? ORDER BY date DESC LIMIT 1", (branch,d2,libname))
238
+
cursor.execute("SELECT libversion,confighash FROM libversion WHERE %s AND date<=? AND libname=? ORDER BY date DESC LIMIT 1"%db.likeNoCase("branch"), (branch,d2,libname))
0 commit comments