Add 248 negative tests to p4_16_errors
#5369
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds 248 negative tests and their respective output files to
p4_16_errors. These have been generated by fuzzing against a mechanized version of the spec, the details of which @jaehyun1ee shared at the recent August LDWG meeting. The new error programs test violating conditions that are not covered by existing tests.For example,
expr-binary-shift-by-negative.p4tests the below condition in the spec.The program tests when the right operand of a shift is a negative integer, which was not previously covered by any of the existing tests. All 248 added tests cover such conditions.
p4_16_samplesare reduced then mutated, until we find a program that hits a "new violating condition". The source program is specified as a comment at the top of each file. From 1409 generated programs, we manually filtered out conditions covered by other tests underp4_16_errorsand purged duplicates. We also only add programs that p4c successfully rejects, as the rest have been reported as separate issues. The remaining 248 programs have been given descriptive names of their root cause. All tests are target-independent.expr-last-access-on-non-stackhas two versions, one that tries to accesslaston a tuple expression and the other on a boolean.