Test every library of an FMI job with FMI

[adrpo]

The master-fmi job of build 11369 ran for seven hours and twenty minutes and then threw all of it away:

Traceback (most recent call last):
  File "./test.py", line 1153, in <module>
    (lambda filename_prefix, errPrefix, diff:
  File "./test.py", line 1158, in <lambda>
    ("%s (%d verified)" % (timeSeconds(diff.get("time")), diff.get("numCompared"))) if s[3]["phase"]>=7 else (...
AttributeError: 'NoneType' object has no attribute 'get'

A model reported that it had been verified while having nothing to show for the comparison. Chasing that turned up an older bug underneath it.

Four libraries have not been testing FMI at all

The configuration loop prepares a reference-file directory once and lets the libraries that share it reuse the result:

if c["referenceFiles"]["destination"] in preparedReferenceDirs:
  (c["referenceFiles"],c["referenceFilesURL"]) = preparedReferenceDirs[destination]
  continue

That continue skips the rest of the loop body — and the only thing after it is the line that turns FMI on:

if allTestsFmi:
  c["fmi"] = "2.0"

So in configs/conf.json four libraries have been running as ordinary native models inside the FMI jobs, each of them the second library to want a directory another had already cloned:

library	shares the reference files of
PowerGrids_symb_jac	PowerGrids
PowerGrids_dev	PowerGrids
ClaRa_dev	ClaRa
ScalableTestSuite_noopt	ScalableTestSuite

The results say so plainly. Each of the four verifies exactly the same number of models in master and in master-fmi, while every library that prepares its own directory drops as an FMI job should:

library	master	master-fmi
PowerGrids_symb_jac	66	66	never built an FMU
PowerGrids_dev	19	19	never built an FMU
ClaRa_dev	72	72	never built an FMU
ScalableTestSuite_noopt	244	244	never built an FMU
PowerGrids	65	31
ScalableTestSuite	242	176
ClaRa	72	62

The same three lines also looked the directory up under destination — a variable assigned after the lookup, so it held whatever the previous library left behind. It has been reading the right entry only because every destination in the configuration happens to be spelled in normalised form.

Both are fixed by computing the normalised destination first and putting the preparation in an else branch, so nothing in the loop body is unreachable any more.

Why it started crashing

Before #297 this only mistested those libraries quietly. Now a job can run several simulators against one FMU, and a model that was never built as an FMU has no per-simulator results at all — so the second simulator's branch fell back to the phase the first simulator reached. That is 7, verified, with no comparison to go with it, and the report generator trusts phase >= 7 to mean there is one.

Two changes:

• a simulator that never ran a model now reports the phase the shared build reached, never a phase another simulator went on to reach with the same FMU. The database row and the report agree, because both go through the same helper.
• a model with nothing to compare renders as an empty cell whatever phase it claims. A mismatch should not be able to throw away a finished run.

Testing

The report cell was evaluated directly, before and after, by extracting the real expression from test.py:

before   diff=None, phase=7 -> AttributeError: 'NoneType' object has no attribute 'get'
before   verified          -> '1.50 (12 verified)'
before   failed vars       -> '1.50 (<a href="files/L_M.diff.html">2/12 failed</a>)'
after    diff=None, phase=7 -> '&nbsp;'
after    verified          -> '1.50 (12 verified)'
after    failed vars       -> '1.50 (<a href="files/L_M.diff.html">2/12 failed</a>)'

The reference-file loop was executed as it stands, with git and omc stubbed, over three libraries sharing one directory. Before, the third lost fmi; after, all three keep it, and each still ends up with the same prepared reference files.

resultValues and dataForSimulator were exercised over the shapes that matter: a simulator with no results of its own (phase capped at the build, no diff, no simulation time, and the row records the same), a build that failed (keeps its own lower phase), a simulator with its own results (untouched), one that ran and failed (reports its own failure, not the other's success), and the primary branch (passed through unchanged).

Related to #297, which turned this from a silent mistest into a crash.

---

Generated by Claude Code.