Adding check files for verification integration tests #1641
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR extends the verification test suite to optionally validate the VCs results.
For each
.scalafile, a corresponding.checkfile dictates the expected output.The latter is a CSV with the following columns:
<line>:<column>precondition,assertand so on)valid,invalid,unknown)Note that VCs are allowed to be undistinguishable (i.e. all 5 properties are equal), for instance
BigIntMonoidLaws.checkhas 3 times the entry10:9;append;precondition;validbecause there are 3 corresponding VCs.To opt-in for
.checkfiles test for benchmarks that are expected to fail, the functionVerificationComponentTestSuite#testNegAllWithCheckFilesmay be used.It is currently only used by
VerificationSuiteto testverification/invalid. For now, a warning is issued for benchmarks missing a corresponding.checkfile.To quickly generate
.checkfiles, a utilitycheck-files-utilmodule is provided.It essentially runs Stainless and writes the results.
The utility must be passed a folder or individual files (which must resides in
frontends/benchmarks).It accepts the following options:
--ignore=<comma-separated-without-spaces-files>to ignore specific files--parallelism=<n>sets the number of threads for StainlessOptions can be passed to Stainless by adding a separation
--.For instances, the following command (in SBT shell):
will:
.checkfiles for theverification/invalidbenchmarks exceptBadConcRope.scalaandBadOpaque.scalaas indicated by the--ignoreoptions (note that the files are concatenated with commas without any space!)smt-cvc5as solverstrict-arithmeticoption)Note that the primary motivation for this is to fix missing positions in VCs (which are probably the second most annoying things after timed-out VCs).
If a benchmark has VCs with missing positions, the utility will issue a warning but it won't be able to know from which file the VC comes (since the position is also responsible to contain the filename).