Dev p3.0/param testcases #879

ChristineZh0u · 2025-06-30T16:51:30Z

Merging param test cases branch into major 3.0

…ramtest

…into p-org-paramtest

* update assumption * fix: add deps back * refactor: constant -> param * fix * clean the code * unify normal/param/assume tests * clean the code * rename * merging safety test parser rules * clean the code * Update github CI action v2 to v3 * iUpdate github CI action v2 to v3 on ubuntu * rename variable "param" into "parameter" to avoid keyword conflicts

* add twise * clean the code

* Solved Concerns from Lewis: + new allow twise number be 1 + unify twise number valid condition + add unit test * add new test case * unit test for ParamAssignment.IterateAssignments

* Update build system and Java compiler, remove dependency JARs * Update CheckerCore logging and state machine components

…ametric test (#875) Co-authored-by: Christine Zhou <[email protected]>

ankushdesai · 2025-07-01T05:27:00Z

.github/workflows/macosci.yml

      run: dotnet build --configuration Release
    - name: Test
-      run: dotnet test --configuration Release
+      run: dotnet test --configuration Release --blame-crash --blame-hang --blame-hang-timeout 300s --logger:"console;verbosity=detailed"


Can you please explain this change?

This change is to get better logs when the GitHub Action fails. Without it, we just see that something crashed, but not what caused it. With these flags, we get more detailed output and can tell which test hung or crashed, which makes debugging a lot easier.

ankushdesai · 2025-07-01T05:27:54Z

Src/PChecker/CheckerCore/SystematicTesting/TestingEngine.cs

                        Console.Out.WriteLine($"{mi.DeclaringType.Name}");
                    }

-                    Environment.Exit(0);


Why this change?

This is needed because, for parametric tests, we have to fetch all the test case names using this function. The problem is that it calls Environment.Exit(0), which ends the entire regression test run early. So we had to handle it differently to avoid stopping the whole suite.

ankushdesai · 2025-07-01T05:28:15Z

Src/PCompiler/CompilerCore/Backend/PChecker/PCheckerCodeGenerator.cs

        public bool HasCompilationStage => true;
-        private static List<Variable> _globalParams = [];
+        [ThreadStatic]
+        private static List<Variable> _globalParams;


why this change?

This was causing issues when I ran regression tests in parallel. Since _globalParams was static, all the tests were sharing the same list, which led to conflicts. Adding [ThreadStatic] makes sure each test thread gets its own copy, so they don’t mess with each other.

Instead, can we just make it non-static?

This is a global variable which can't be made non-static, otherwise it defeats the purpose of global variable.

ankushdesai · 2025-07-01T05:28:37Z

Src/PCompiler/CompilerCore/Backend/PChecker/PCheckerCodeGenerator.cs

            // write the top level declarations
            foreach (var decl in globalScope.AllDecls)
            {
+                // Console.WriteLine("top-level decl: " + decl.Name);


Please make a pass to remove unwanted changes.

ankushdesai · 2025-07-09T23:55:40Z

@lewisbru @aishu-j can I get reviews on this PR; would like to get this merged.

Copilot

Pull Request Overview

This PR merges parametric test cases into major version 3.0, introducing support for parameterized testing functionality within the P language testing framework. The changes implement a comprehensive system for running and validating parametric tests with various parameter combinations.

Adds parametric test infrastructure with test case discovery and validation
Enhances test runners to handle parameter-based test execution
Improves test reliability with retry mechanisms and better error handling

Reviewed Changes

Copilot reviewed 20 out of 21 changed files in this pull request and generated 12 comments.

Show a summary per file

File	Description
Tst/UnitTests/TestAssertions.cs	Added Thread.Sleep retry logic for directory deletion and improved error output
Tst/UnitTests/Runners/PCheckerRunner.cs	Major refactor to support parametric test discovery, validation, and in-process execution
Tst/UnitTests/PCheckerLogGeneratorTests.cs	Updated test to use source-based naming instead of hardcoded "Main"
Multiple .p test files	Added new parametric test cases with parameter definitions and test declarations
Src/PCompiler/CompilerCore/TypeChecker/ParamVisitor.cs	Added validation to prevent equal start/end values in parameter ranges
Src/PChecker/CheckerCore/Checker.cs	Enhanced checker to handle test case listing mode
.github/workflows/macosci.yml	Improved CI configuration with detailed logging and test result uploads

Src/PCompiler/CompilerCore/TypeChecker/ParamVisitor.cs

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

Tst/RegressionTests/Feature1SMLevelDecls/StaticError/ParamTest3/ParamTest3.p

Tst/RegressionTests/Feature1SMLevelDecls/StaticError/ParamTest4/ParamTest4.p

Tst/RegressionTests/Feature1SMLevelDecls/DynamicError/ParamTest2/ParamTest2.p

Tst/UnitTests/TestAssertions.cs

lewisbru

Also, resolve the various copilot recommendations and fix the typos

lewisbru · 2025-07-10T16:24:51Z

Src/PChecker/CheckerCore/Checker.cs

 using PChecker.IO.Logging;
-using PChecker.SystematicTesting;
 using PChecker.Testing;
+using PChecker.SystematicTesting;


Nit: why the reordering? All of the formatting conventions I know of recommend alike imports/usings to be in alphabetical order.

Will revert the reordering change here

lewisbru · 2025-07-10T16:26:21Z

Src/PChecker/CheckerCore/Checker.cs

            {
                case CheckerMode.BugFinding:
-                    TestingProcess.Create(configuration).Run();
+                    var testingEngine = TestingProcess.Create(configuration);


Why create the testingEngine before checking configuration.ListTestCases? TestingProcess.Create() change the configuration?

We still call TestingProcess.Create() even when configuration.ListTestCases is true because the logic to list test cases lives inside the Create() method. That method is responsible for discovering and printing the available test cases, so we need to call it regardless.

The reason for making this change now is that the original implementation used Environment.Exit to terminate after listing test cases. This caused issues when running tests in-process because Environment.Exit would terminate the entire test run, not just the individual test. To fix this, I replaced the exit call with a simple return, which exits the method cleanly without affecting other tests.

lewisbru · 2025-07-10T16:30:47Z

Src/PCompiler/CompilerCore/Backend/PChecker/PCheckerCodeGenerator.cs

        public bool HasCompilationStage => true;
-        private static List<Variable> _globalParams = [];
+        [ThreadStatic]
+        private static List<Variable> _globalParams;


Instead, can we just make it non-static?

Src/PCompiler/CompilerCore/Backend/PChecker/PCheckerCodeGenerator.cs

lewisbru · 2025-07-10T16:35:09Z

Src/PCompiler/CompilerCore/TypeChecker/ParamVisitor.cs

+                if (values.Count == 2 &&
+                    values[0] is IntLiteralExpr first &&
+                    values[1] is IntLiteralExpr second &&
+                    first.Value == second.Value)


Is this exception check too limited? I think it should verify that first.Value < secondValue.

lewisbru · 2025-07-10T16:37:37Z

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

+	start state Init {
+		entry {
+			print format("global varaible g1 = {0}", g1);
+      print format("global varaible g2 = {0}", g2);


lewisbru · 2025-07-10T16:40:59Z

Src/PCompiler/CompilerCore/TypeChecker/ParamVisitor.cs

+                {
+                    throw handler.InternalError(context, new Exception("Invalid range: start and end must not be equal (e.g., [2, 2] is not allowed)"));                }
+
+                if (values.Count == 0)


I know this is copied from the original code, but what does a sequence of 0 values mean?

if it is supposed to be an empty sequence, I don't think PrimitiveType.Int is an appropriate default?

lewisbru · 2025-07-10T16:50:06Z

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

@@ -0,0 +1,33 @@
+param g1 : int;


The param types in the test should include all of the allowed types; if they have to be int, then there should be a negative test with non-int param types.

added a new param (g5) for boolean

lewisbru · 2025-07-10T16:53:25Z

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

+
+test BasicNonParametricTest [main=Main]:{Main};
+
+test param (g1 in [1, 2], g2 in [3, 4], g3 in [5, 6], g4 in [7, 8]) FullCartesianProductParametricTest [main=Main]:{Main};


Especially given the special handling of two int sequences above; there should be more variation of possible values:

single int sequence

two int sequence that is a range

two int sequence that is not a range?

two int sequence

Tst/TestParamTester/1_ClientServer/PTst/TestDriver.p

lewisbru · 2025-07-14T15:38:54Z

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

+						print format("global variable g1 = {0}", g1);
+						print format("global variable g2 = {0}", g2);
+						print format("global variable g3 = {0}", g3);
+						print format("global variable g4 = {0}", g4);


add print for g5 here.

lewisbru · 2025-07-14T15:40:07Z

Tst/RegressionTests/Feature1SMLevelDecls/Correct/ParamTest/ParamTest.p

 test BasicNonParametricTest [main=Main]:{Main};

-test param (g1 in [1, 2], g2 in [3, 4], g3 in [5, 6], g4 in [7, 8]) FullCartesianProductParametricTest [main=Main]:{Main};
+test param (g1 in [-1, 2], g2 in [3, 4], g3 in [5, 6], g4 in [7, 8], g5 in [true, false]) FullCartesianProductParametricTest [main=Main]:{Main};


Somewhere in here include some cases where one of the parameters has more than two values.

good point! I'll add some cases for that

zhezhouzz and others added 28 commits June 11, 2024 13:58

Add global constant varaibles; TODO backend

6be2b01

Updated BuildGlobalScope

82637c6

add global variable

7b55329

fix global variable types

8cfe5c9

Allow multiple tests

d630b9f

Add int list literal syntax

f39c431

add parametric test

246d1d6

Quick-fix

71b519b

fix bad quick-fix

740de07

add rich syntax

5def42a

Merge branch 'master' into paramtest

38dbf6d

Merge branch 'mainp' into paramtest

08ee970

Merge branch 'paramtest' of github.com:p-org/P-TestGeneration into pa…

57d7c4b

…ramtest

Merge branch 'master' into paramtest

1a9754d

Merge branch 'master' into paramtest

50463f5

Merge branch 'paramtest' of https://github.com/p-org/P-TestGeneration …

658c304

…into p-org-paramtest

Merge branch 'p-org-paramtest' into dev_p3.0/param_testcases

924eced

Small merge error

60d3fb4

T-wise combinatorial test (#837)

c7e86ab

* add twise * clean the code

T-wise combinatorial (Solved Concerns from Lewis) (#838)

ef898a0

* Solved Concerns from Lewis: + new allow twise number be 1 + unify twise number valid condition + add unit test * add new test case * unit test for ParamAssignment.IterateAssignments

Dev/remove libhandler (#847)

55a1199

Update build system and Java compiler, remove dependency JARs (#849)

c5c6c72

* Update build system and Java compiler, remove dependency JARs * Update CheckerCore logging and state machine components

Merge branch 'master' into major/P3.0

28833bf

Merge branch 'major/P3.0' into dev_p3.0/param_testcases

5c77847

Modified regression tests to run in process, added test cases for par…

a4a6195

…ametric test (#875) Co-authored-by: Christine Zhou <[email protected]>

resolve merging conflict

e8c605a

Fix pex regression test

5320e40

ChristineZh0u requested a review from ankushdesai June 30, 2025 23:19

ankushdesai requested changes Jul 1, 2025

View reviewed changes

remove unnecessary line

936c47e

ankushdesai requested review from aishu-j, Copilot and lewisbru July 9, 2025 23:54

Copilot AI reviewed Jul 9, 2025

View reviewed changes

ankushdesai assigned lewisbru Jul 9, 2025

lewisbru requested changes Jul 10, 2025

View reviewed changes

p-org deleted a comment from Copilot AI Jul 13, 2025

ChristineZh0u force-pushed the dev_p3.0/param_testcases branch from 7ab51ff to ef6b7b5 Compare July 13, 2025 19:48

lewisbru requested changes Jul 14, 2025

View reviewed changes

Addressing PR comments

0a8c3b8

ChristineZh0u force-pushed the dev_p3.0/param_testcases branch from ef6b7b5 to 0a8c3b8 Compare July 14, 2025 16:35

lewisbru approved these changes Jul 17, 2025

View reviewed changes

ankushdesai merged commit cedacd2 into major/P3.0 Jul 22, 2025
10 checks passed

ankushdesai deleted the dev_p3.0/param_testcases branch July 22, 2025 23:12


		test BasicNonParametricTest [main=Main]:{Main};

		test param (g1 in [1, 2], g2 in [3, 4], g3 in [5, 6], g4 in [7, 8]) FullCartesianProductParametricTest [main=Main]:{Main};

Dev p3.0/param testcases #879

Dev p3.0/param testcases #879

Uh oh!

Conversation

ChristineZh0u commented Jun 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankushdesai commented Jul 9, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lewisbru left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers