CI: Test sdk deserialization against generated aas-core-testdatagen files - #608
Draft
hpoeche wants to merge 6 commits into
Draft
CI: Test sdk deserialization against generated aas-core-testdatagen files#608hpoeche wants to merge 6 commits into
aas-core-testdatagen files#608hpoeche wants to merge 6 commits into
Conversation
For now only the JSON examples are considered. Errors are sorted into three categories: Errors that occur from deserialization, NotImplementedErrors that are explicit developer choices and Unexpected errors to cover future errors of the sdk.
…n toeliminate code duplication
We add a composite github action `sdk-test-testdatagen` that fetches the example files from the `aas-specs-metmodel` repository in the correct verion, runs the `example_tests.py` script and uploads the output as an artifact to github. For testing this action is temporarily called in the ci, that runs on every pull_request. This should be changed to only run on PRs into main before this PR is approved.
For usage in Github CI, the test script now outputs a markdown file comprehending the test results. For each format (JSON, XML) the number of passed and failed tests are shown, as well as a tabular overview on the amount of failed tests per error message. The summary also contains the sha of the commit of the `basyx-python-sdk` repo that the test ran on and the tag of the `aas-specs-metamodel` repository that provided the example files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Currently, the sdk implementation of the AAS objects and the deserialization is tested using the handwritten unittests that rely on handwritten example json input. With every change of the metamodel that requires changes in our SDK implementation, this test data needs a corresponding adjustment. This process is resource intensive and prone to errors.
Changes
To overcome both of these issues, #109 proposed to utilize test cases which are auto-generated from the metamodel itself. The project mentioned was updated to the name of
aas-core-testdatagenand the generated example files are currently shipped with the metamodel itself in theschemas/.../examplesfolder.In these changes we add a test script
example_tests.pythat test the deserialization of all generated example files. Occurring errors are caught, roughly grouped by their cause and reported to output files, that contain the input which rose the error as well as the complete stacktrace.We use this script to run deserialization testing in our CI pipeline on pull requests to
main. For quick overview of the test result status, a job summary can be viewed in the Github web interface. For more detailed analysis, the output reports are uploaded as artifact.Fixes #109