Initial TrustInSoft CI setup #680
Open
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi, @hawicz!
I contacted you in June about deploying the new tool TrustInSoft CI on JSON-C: #641
At the time you gave me valuable feedback concerning both what you found advantageous and inconvenient about the tool.
In the meantime TIS CI has matured and we have learned more about how to deploy it on open-source projects. I would like to propose you a new TIS CI configuration for JSON-C, which addresses your concerns.
First, you were concerned about compromising the readability of your code just to make some checks in TIS CI pass. We have added new options that allow to ignore certain alarms if you consider them spurious.
Second, we have learned how to make TIS CI configurations more robust and more advantageous for long-term Continuous Integration. The script
regenerate.pycan be used to recreate all the TIS CI configuration files automatically. And a whole batch of tests generated through fuzzing was included to increase the verification coverage and spot regressions.About the files that are part of this PR:
tis.config: The main configuration file that describes all the 1377 tests to analyze (32 from your test suite, the rest from inputs created through fuzzing).trustinsoft/common.config: Analysis options common to all tests.trustinsoft/fuzz_inputs/*.jsonandtrustinsoft/test_fuzz.c: The fuzzing inputs (generated from https://github.com/nst/JSONTestSuite) and the test driver for analyzing them.trustinsoft/include/*.h: Three header files (generated when building JSON-C) that TIS CI requires for analyses.trustinsoft/regenerate.py: A python script which recreates the TIS CI configuration files and which copies the 3 generated header files totrustinsoft/include.README.md: Includes a TIS CI badge that displays the latest analysis status.In this new configuration almost all the tests are green. That's because:
The one exception is the single red test which concerns the
json_tokener_new_exfunction. You explained that this function expects thedepthparameter to be non-negative. So instead of looking for the problem inside the function's body I gave it a preconditiondepth >= 0. This precondition is violated by this test because it callsjson_tokener_new_exwithdepthequal-2. That's why the test is red.Please let me know if this configuration is satisfactory to you or if you have any more concerns.