Feature/osc validation integration #747
Conversation
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
Signed-off-by: Thomas Sedlmayer <[email protected]>
|
Thanks for this interesting initiative. Potentially it can contribute in harmonizing tools. We'll first look into the ID topic. We understand any ID shall be globally unique, across categories of simulation entities. So far esmini have reused any specified ID in the OpenDRIVE objects, e.g. stationary objects. This traceability from OSI to scenario components is important. We'll now look into whether we can utilize ExternalReference to achieve this instead. |
|
@thomassedlmayer We have an experimental branch which aims to address the issue of overlapping ID's in OSI. Would it be possible for you to verify that branch's functionality against the ASAM OSI Quality Checker? It would be great to get your feedback in case there's still some issues. The branch in question: feature/global_osi_ids |
- track internal scenario ID in OSI source reference
|
We have now made a release 2.57.0 which hopefully complies with ASAM OSI Quality Checker. This is a first step for this PR. |
|
Hi @slundel6 and @eknabevcc, Though, you probably noticed that the validation step still fails. There are four remaining issues of which 3 will be fixed by some updates on the side of osc-validation. I will update this PR once this is solved on our side. But there is one remaining issue which I unfortunately overlooked to mention in my initial request: OSI's rules require the Once this is solved, this should allow more focus on actually validating the OpenSCENARIO part of ESMini (which is the actual goal of osc-validation! Thank you again for your cooperation and your help regarding the OSI output of ESMini! |
|
We added the |
This is a draft/proposal to include osc-validation in the CI pipeline of ESMini.
Coming from the European Synergies project, we developed an OpenSCENARIO engine validation tool called
osc-validation(open source). The goal is to detect behavioral differences between scenario engines based on their OSI output but also potential deviations from the OSC standard (for OSC definitions that are unambiguous). Another long-term goal is to identify underspecified or ambiguous parts of the OSC standard.osc-validationis temporarily hosted at https://github.com/PMSFIT/osc-validation. It's soon going to be part of the OpenMSL project.The only integrated test currently verifies that the vehicle trajectories in the OSI output deviate only marginally from the input data, which ESMini is expected to satisfy.
Though, with the current state of the test case, ESMini would fail because of an integrated ASAM OSI Quality Checker (https://github.com/OpenSimulationInterface/qc-osi-trace) which checks the tool output for compliance with OSI 3.7.0 rules which are based on the
/rulessections in the proto definitions. The checker result shows that there are some problems with unique assignment of IDs in ESMini's OSI output (different object types have overlapping ID space).If you are interested in integrating this into ESMini's pipeline, I would ask you to review the CI setup and check out
osc-validation.I had to upgrade from Python 3.8 to 3.10 for compatibility reasons. But this should be fine since Python 3.8 already reached its EOL about a year ago. We could even go higher since 3.10 is already close to its EOL timeline.
Sidenote:
Also, I noticed that there are some issues with the current pipeline setup when running it in a fork (due to missing tags etc.); see the changes in OSMP_FMU/CMakeLists.txt. This should be removed before ever merging this PR. This is just there so that the pipeline runs properly in my fork.
@jdsika