Skip to content

FMI build once simulate many #297

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adrpo merged 6 commits into from
Aug 11, 2026
Merged

FMI build once simulate many #297

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
c257beb
[FMI] Let --fmisimulator be given more than once
adrpo Aug 11, 2026
9c0c528
[FMI] Describe the FMI simulators in a configuration file
adrpo Aug 11, 2026
9db9fe3
[FMI] Build each FMU once and simulate it with every tool asked for
adrpo Aug 11, 2026
2d80b2d
[FMI] Publish the results of every simulator to its own branch directory
adrpo Aug 11, 2026
df84808
[FMI] Run each pair of FMI jobs as one job
adrpo Aug 11, 2026
1f8b458
[FMI] Document testing one FMU with several simulators
adrpo Aug 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
118 changes: 43 additions & 75 deletions .CI/Jenkinsfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@ pipeline {
booleanParam(name: 'v1_27', defaultValue: false, description: 'maintenance/v1.27 branch (ryzen-5950x-1)')
booleanParam(name: 'master', defaultValue: false, description: 'master branch (ryzen-5950x-1)')

booleanParam(name: 'fmi_v1_26', defaultValue: false, description: 'maintenance/v1.26 branch with FMI (ryzen-5950x-2)')
booleanParam(name: 'fmi_v1_27', defaultValue: false, description: 'maintenance/v1.27 branch with FMI (ryzen-5950x-2)')
booleanParam(name: 'fmi_master', defaultValue: false, description: 'master branch with FMI running OMSimulator (ryzen-5950x-2)')
booleanParam(name: 'fmi_v1_26', defaultValue: false, description: 'maintenance/v1.26 branch with FMI, simulated by OMSimulator, filling v1.26-fmi (ryzen-5950x-2). Ticking it together with fmpy_fmi_v1_26 runs one job that builds every FMU once and simulates it with both.')
booleanParam(name: 'fmi_v1_27', defaultValue: false, description: 'maintenance/v1.27 branch with FMI, simulated by OMSimulator, filling v1.27-fmi (ryzen-5950x-2). Ticking it together with fmpy_fmi_v1_27 runs one job that builds every FMU once and simulates it with both.')
booleanParam(name: 'fmi_master', defaultValue: false, description: 'master branch with FMI, simulated by OMSimulator, filling master-fmi (ryzen-5950x-2). Ticking it together with fmpy_fmi_master runs one job that builds every FMU once and simulates it with both.')

booleanParam(name: 'cs_fmu_cvode_v1_26', defaultValue: false, description: 'maintenance/v1.26 branch with CVODE CS FMUs running OMSimulator (ryzen-5950x-2)')
booleanParam(name: 'cs_fmu_cvode_v1_27', defaultValue: false, description: 'maintenance/v1.27 branch with CVODE CS FMUs running OMSimulator (ryzen-5950x-2)')
booleanParam(name: 'cs_fmu_cvode_master', defaultValue: false, description: 'master branch with CVODE CS FMUs running OMSimulator (ryzen-5950x-2)')

booleanParam(name: 'fmpy_fmi_v1_26', defaultValue: false, description: 'maintenance/v1.26 branch with FMI (ryzen-5950x-2)')
booleanParam(name: 'fmpy_fmi_v1_27', defaultValue: false, description: 'maintenance/v1.27 branch with FMI (ryzen-5950x-2)')
booleanParam(name: 'fmpy_fmi_master', defaultValue: false, description: 'master branch with FMI running FMPy (ryzen-5950x-2)')
booleanParam(name: 'fmpy_fmi_v1_26', defaultValue: false, description: 'maintenance/v1.26 branch with FMI, simulated by FMPy, filling v1.26-fmi-fmpy (ryzen-5950x-2). Ticking it together with fmi_v1_26 runs one job that builds every FMU once and simulates it with both.')
booleanParam(name: 'fmpy_fmi_v1_27', defaultValue: false, description: 'maintenance/v1.27 branch with FMI, simulated by FMPy, filling v1.27-fmi-fmpy (ryzen-5950x-2). Ticking it together with fmi_v1_27 runs one job that builds every FMU once and simulates it with both.')
booleanParam(name: 'fmpy_fmi_master', defaultValue: false, description: 'master branch with FMI, simulated by FMPy, filling master-fmi-fmpy (ryzen-5950x-2). Ticking it together with fmi_master runs one job that builds every FMU once and simulates it with both.')

booleanParam(name: 'newInst_daeMode', defaultValue: false, description: 'master branch, --daeMode with -d=newInst (ryzen-5950x-2)')
booleanParam(name: 'newInst_newBackend', defaultValue: false, description: 'master branch, -d=newInst --newBackend, (ryzen-5950x-1)')
Expand Down Expand Up @@ -129,7 +129,7 @@ pipeline {
}
}

stage('v1.26 FMI with OMSimulator') {
stage('v1.26 FMI') {
agent {
node {
label 'ryzen-5950x-2-1'
Expand All @@ -139,13 +139,13 @@ pipeline {
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmi_v1_26 }
expression { params.fmi_v1_26 || params.fmpy_fmi_v1_26 }
}
steps {
runRegressiontest('maintenance/v1.26', 'v1.26-fmi', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
runRegressiontest('maintenance/v1.26', 'v1.26-fmi', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false, 0, 'configs/conf.json', '', '', fmiSimulators(params.fmi_v1_26, params.fmpy_fmi_v1_26))
}
}
stage('v1.27 FMI with OMSimulator') {
stage('v1.27 FMI') {
agent {
node {
label 'ryzen-5950x-2-1'
Expand All @@ -155,13 +155,13 @@ pipeline {
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmi_v1_27 }
expression { params.fmi_v1_27 || params.fmpy_fmi_v1_27 }
}
steps {
runRegressiontest('maintenance/v1.27', 'v1.27-fmi', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
runRegressiontest('maintenance/v1.27', 'v1.27-fmi', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false, 0, 'configs/conf.json', '', '', fmiSimulators(params.fmi_v1_27, params.fmpy_fmi_v1_27))
}
}
stage('master FMI with OMSimulator') {
stage('master FMI') {
agent {
node {
label 'ryzen-5950x-2-1'
Expand All @@ -171,10 +171,10 @@ pipeline {
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmi_master }
expression { params.fmi_master || params.fmpy_fmi_master }
}
steps {
runRegressiontest('master', 'master-fmi', '', 'origin/master', 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
runRegressiontest('master', 'master-fmi', '', 'origin/master', 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false, 0, 'configs/conf.json', '', '', fmiSimulators(params.fmi_master, params.fmpy_fmi_master))
}
}

Expand Down Expand Up @@ -227,57 +227,6 @@ pipeline {
}
}

stage('v1.26 FMI with FMPy') {
agent {
node {
label 'ryzen-5950x-2-1'
customWorkspace 'ws/OpenModelicaLibraryTestingWork'
}
}
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmpy_fmi_v1_26 }
}
steps {
runRegressiontest('maintenance/v1.26', 'v1.26-fmi-fmpy', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
}
}

stage('v1.27 FMI with FMPy') {
agent {
node {
label 'ryzen-5950x-2-1'
customWorkspace 'ws/OpenModelicaLibraryTestingWork'
}
}
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmpy_fmi_v1_27 }
}
steps {
runRegressiontest('maintenance/v1.27', 'v1.27-fmi-fmpy', '', omsimulatorHash(), 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
}
}

stage('master FMI with FMPy') {
agent {
node {
label 'ryzen-5950x-2-1'
customWorkspace 'ws/OpenModelicaLibraryTestingWork'
}
}
options { skipDefaultCheckout() }
when {
beforeAgent true
expression { params.fmpy_fmi_master }
}
steps {
runRegressiontest('master', 'master-fmi-fmpy', '', 'origin/master', 'ripper2', 'LibraryTestingRipper2DB', false, '', '', false, false)
}
}

stage('newInst-daeMode') {
agent {
node {
Expand Down Expand Up @@ -763,7 +712,17 @@ def sccachePreamble() {
* `.CI/wasm-jit`. If non-empty, the omc build and test.py run in that image
* instead of on the node; everything using the node's own omc stays outside.
*/
def runRegressiontest(branch, name, extraFlags, omsHash, dbPrefix, sshConfig, omcompiler, extrasimflags, testFlags, boolean removePackageOrder, boolean conversionScript, int jobs=0, libs_config_file = 'configs/conf.json', cmakeFlags = '', dockerfile = '') {
/* The FMI simulators a job runs, from the parameters that used to start one job each.
* Both ticked is one job that builds every FMU once and simulates it with both,
* filling <branch>-fmi and <branch>-fmi-fmpy exactly as the two jobs did. */
def fmiSimulators(boolean omsimulator, boolean fmpy) {
def simulators = []
if (omsimulator) simulators << 'OMSimulator'
if (fmpy) simulators << 'fmpy'
return simulators
}

def runRegressiontest(branch, name, extraFlags, omsHash, dbPrefix, sshConfig, omcompiler, extrasimflags, testFlags, boolean removePackageOrder, boolean conversionScript, int jobs=0, libs_config_file = 'configs/conf.json', cmakeFlags = '', dockerfile = '', fmiSimulators = null) {
sh '''
find /tmp -name "*openmodelica.hudson*" -exec rm {} ";" || true
mkdir -p ~/TEST_LIBS_BACKUP
Expand Down Expand Up @@ -814,8 +773,13 @@ def runRegressiontest(branch, name, extraFlags, omsHash, dbPrefix, sshConfig, om
}
}

FMI_TESTING_FLAG=""
if (!name.contains('fmpy') && omsHash) {
// A job that does not say which simulators it wants gets the one its name implies.
def simulators = fmiSimulators
if (simulators == null) {
simulators = name.contains('fmpy') ? ['fmpy'] : (omsHash ? ['OMSimulator'] : [])
}
FMI_TESTING_FLAG = ""
if (simulators.contains('OMSimulator') && omsHash) {
sh """
if ! test -d OMSimulator; then
git clone --recursive https://openmodelica.org/git-readonly/OMSimulator.git || exit 1
Expand Down Expand Up @@ -848,19 +812,23 @@ def runRegressiontest(branch, name, extraFlags, omsHash, dbPrefix, sshConfig, om
echo OMSimulator version:
${env.HOME}/saved_omc/OMSimulator/install/bin/OMSimulator --version
"""
FMI_TESTING_FLAG="--fmi=true --fmisimulator=${env.HOME}/saved_omc/OMSimulator/install/bin/OMSimulator --default=ulimitExe=50"
if (name.contains('cvode')) {
FMI_TESTING_FLAG += " --fmuType=cs"
}
FMI_TESTING_FLAG = " --fmisimulator=${env.HOME}/saved_omc/OMSimulator/install/bin/OMSimulator"
}

if (name.contains('fmpy')) {
if (simulators.contains('fmpy')) {
sh """
# update fmpy
pip install FMPy || true
python3 -m fmpy -h || exit 1
"""
FMI_TESTING_FLAG="--fmi=true --fmisimulator='python3 -m fmpy' --default=ulimitExe=50"
FMI_TESTING_FLAG += " --fmisimulator='python3 -m fmpy'"
}

if (FMI_TESTING_FLAG) {
FMI_TESTING_FLAG = "--fmi=true${FMI_TESTING_FLAG} --default=ulimitExe=50"
if (name.contains('cvode')) {
FMI_TESTING_FLAG += " --fmuType=cs"
}
}

OMCPATH = "${omcompiler ? '../' : './'}OMCompiler"
Expand Down
91 changes: 89 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,12 @@ Options:
- `--ompython_omhome=''`: Path to OpenModelica for OMPython (can be different to
the OM running the tests)
- `--noclean=False`: Clean (most) generated files.
- `--fmisimulator=''`: The default is nothing but you can use the path to
OMSimulator executable or 'fmpy'
- `--fmisimulator=''`: The FMI simulator to run the FMUs with, as `name=command`
or just the command, e.g. the path to the OMSimulator
executable or `'python3 -m fmpy'`. Repeat the option to
simulate every FMU with several tools without building it
more than once, see [Testing FMI with several
simulators](#testing-fmi-with-several-simulators)
- `--ulimitvmem=8388608`: Virtual memory limit (in kB)
- `--default=[]`: Add a default value for some configuration key, such as
`--default=ulimitExe=60`. The equals sign is mandatory
Expand All @@ -168,6 +172,89 @@ Options:
`procCCompile` above for more insight into individual test
parallelization.

### Testing FMI with several simulators

Building an FMU costs far more than simulating it. Measured on the twelve
models of ExternData: 178 seconds building the FMUs, 0.7 simulating them with
OMSimulator and 2.6 with FMPy. Testing the same FMUs with a second tool
therefore used to cost almost twice as much as testing them with one, because
each job built its own copy of them.

Give `--fmisimulator` once per tool and the FMUs are built once and simulated
with each of them:

```bash
./test.py --branch=v1.27-fmi --fmi=true \
--fmisimulator=/path/to/OMSimulator \
--fmisimulator='python3 -m fmpy' \
configs/myConf.json
```

`--branch` names the job; every simulator stores its results in a branch of its
own derived from it, so the run above fills

| simulator | branch | published to |
| --- | --- | --- |
| OMSimulator | `v1.27-fmi` | `branches/v1.27-fmi` |
| FMPy | `v1.27-fmi-fmpy` | `branches/v1.27-fmi-fmpy` |

which is where those results have always been. OMSimulator keeps the plain
`-fmi` branch; every other tool adds its name. The branch a tool fills depends
on the tool and not on the order, so asking for FMPy alone still fills
`v1.27-fmi-fmpy` and leaves `v1.27-fmi` alone.

A branch directory looks the same as it always did, file names included. The
`.err` of a model is written by the build, so every simulator of it publishes
the same one; the `.sim` and the difference files are the ones that simulator
produced.

Only the simulator may differ between the results that share an FMU. Anything
that changes the FMU itself - a different compiler, a different library, a
different `--fmuType` or `--fmiFlags` - is a different job, which is why the
Co-Simulation jobs with CVODE are not merged with the Model Exchange ones.

In Jenkins the parameters keep their meaning: `fmi_v1_27` asks for OMSimulator
and `fmpy_fmi_v1_27` for FMPy. Ticking both runs one job that builds every FMU
once and simulates it with both; ticking one runs that tool alone.

### Adding an FMI simulator

The simulators live in [configs/fmi-simulators.json](configs/fmi-simulators.json).
Adding one is an entry there and no change to any script:

```json
"fmusim": {
"resultExtension": "csv",
"versionArgument": "--version",
"optionalArguments": { "stepSizeArgument": " --output-interval {stepSize:g}" },
"arguments": "--interface-type ModelExchange --output-file {result} --start-time {startTime:g} --stop-time {stopTime:g}{stepSizeArgument} {fmu}"
}
```

- `arguments` is the command line, a template over `simulator`, `fmu`,
`result`, `requestedResult`, `tempDir`, `startTime`, `stopTime`, `tolerance`,
`timeout`, `stepSize` and anything named in `optionalArguments`.
- `optionalArguments` are the flags that have to disappear when there is nothing
to put in them. OMSimulator hangs on `--stepSize=0` rather than ignoring it,
so its step size flag lives here, while FMPy wants `--output-interval 0` all
the same and writes the value straight into its `arguments`.
- `command` is how the tool is invoked, `{simulator}` by default. FMPy needs a
subcommand, `{simulator} simulate`.
- `resultExtension` is what the tool writes, so that the comparison against the
reference file knows what to read.
- `versionArgument` prints the version, which is recorded with the results.
- `branchSuffix` overrides the `-<name>` a tool adds to the branch. Only
OMSimulator needs it, with `""`.
- `untested` marks an entry nobody has run yet; the run then says so instead of
failing every model with a puzzling error. Remove it once it works.

Then run it with `--fmisimulator=fmusim=/path/to/fmusim`, or just
`--fmisimulator=/path/to/fmusim` if the command contains the name.

A tool that is a Python package rather than a command line needs a small driver
script that takes the arguments its entry passes, simulates, writes the result
file and exits non-zero when it fails; the entry then points `command` at it.

### Generate HTML results

```bash
Expand Down
46 changes: 46 additions & 0 deletions configs/fmi-simulators.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
{
"_comment": [
"The FMI simulators the testing knows how to run. Adding one is an entry here and nothing else.",
"",
"arguments the command line, a template over: simulator, fmu, result, tempDir,",
" startTime, stopTime, tolerance, timeout, stepSize, and every name",
" defined in optionalArguments.",
"optionalArguments flags that have to disappear when there is nothing to put in them, as a",
" result is always the file the tool should write; requestedResult is empty",
" when the run wants no result file at all, which only OMSimulator acts on.",
" name and its own template. OMSimulator crashes on --stepSize=0 instead of",
" ignoring it, so its step size flag lives here; FMPy wants --output-interval 0",
" all the same and writes the value straight into arguments.",
"command how the tool is invoked, {simulator} by default; FMPy needs a subcommand.",
"resultExtension what the tool writes, so the verification knows what to compare.",
"versionArgument the flag that prints the version, recorded with the results.",
"branchSuffix appended to --branch to get the table of this simulator; the default is",
" -<name>, so OMSimulator is the only one that needs to say anything here.",
"untested set it while nobody has run the entry yet; the run then says so."
],
"OMSimulator": {
"branchSuffix": "",
"resultExtension": "mat",
"versionArgument": "-v",
"arguments": "{resultArgument} --tempDir={tempDir} --startTime={startTime:g} --stopTime={stopTime:g}{stepSizeArgument} --timeout={timeout:g} --tolerance={tolerance:g} {fmu}",
"optionalArguments": {
"resultArgument": "-r={requestedResult}",
"stepSizeArgument": " --stepSize={stepSize:g}"
}
},
"fmpy": {
"resultExtension": "csv",
"versionArgument": "--version",
"command": "{simulator} simulate",
"arguments": "--output-file {result} --start-time {startTime:g} --stop-time {stopTime:g} --timeout {timeout:g} --relative-tolerance {tolerance:g} --interface-type ModelExchange --solver CVode --output-interval {stepSize:g} {fmu}"
},
"fmusim": {
"untested": true,
"resultExtension": "csv",
"versionArgument": "--version",
"arguments": "--interface-type ModelExchange --output-file {result} --start-time {startTime:g} --stop-time {stopTime:g}{stepSizeArgument} {fmu}",
"optionalArguments": {
"stepSizeArgument": " --output-interval {stepSize:g}"
}
}
}
Loading
Loading