Thanks to visit codestin.com
Credit goes to github.com

Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 89 additions & 9 deletions Docs/docs/manual/testcases.md
Original file line number Diff line number Diff line change
@@ -1,23 +1,103 @@
P Test cases are used to define different finite scenarios under which we would like to check the correctness of our system.
P test cases define different finite scenarios under which we would like to check the correctness of our system. Each test case is automatically discharged by the P Checker.

??? note "P Test Cases Grammar"

```
testcase
| test iden [main=iden] : modExpr ; # TestDecl
| test iden [main=iden] : modExpr ; # TestDecl
| test param (paramList) iden [main=iden] : modExpr ; # ParamTestDecl
| test param (paramList) assume (expr) iden [main=iden] : modExpr ; # AssumeTestDecl
| test param (paramList) (num wise) iden [main=iden] : modExpr; # TWiseTestDecl
;

paramList
| iden in [valueList] # SingleParam
| iden in [valueList], paramList # MultiParam
;

valueList
| value # SingleValue
| value, valueList # MultiValue
;

value
| num # NumberValue
| bool # BoolValue
;
```

`modExpr` represent the P module defined using the module expressions described in [P Module System](modulesystem.md)

### Test Case Declaration
### Basic Test Case

A basic test case checks the correctness of a module under a specific scenario.

**Syntax**: `test tName [main=mName] : module_under_test ;`

- `tName` is the name of the test case
- `mName` is the name of the **main** machine where execution starts
- `module_under_test` is the module to be tested

=== "Basic Test"
```
test tcSingleClient [main=TestWithSingleClient]:
assert BankBalanceIsAlwaysCorrect in
(union Client, Bank, { TestWithSingleClient });
```

P allows programmers to write different scenarios under which we would like to check the correctness of the module (or system) under test. More concretely, the system module to be tested is unioned with different environment modules (or test harnesses/drivers) to check its correctness for different inputs scenarios generated by the environment modules. Each test case is automatically discharged by the P Checker.
### Parameterized Test Cases

**Syntax:**: `test tName [main=mName] : module_under_test ;`
Parameterized tests allow systematic exploration of different system configurations. Before using parameters in test cases, they must be declared as global variables with their types.

`tName` is the name of the test case, `mName` is the name of the **main** machine where the execution of the system starts, and `module_under_test` is the module to be tested (manual: [modules in P](../manual/modulesystem.md)).
**Parameter Declaration Syntax**: `param name : type ;`

!!! info "Properties checked for a Test Case"
For example:
```
param nClients : int; // For numeric parameters
param b1 : bool; // For boolean parameters
param g1 : int; // For another numeric parameter
```

=== "Basic Parameter Test"
```
test param (nClients in [2, 3, 4]) tcTest [main=TestWithConfig]:
assert BankBalanceIsAlwaysCorrect in
(union Client, Bank, { TestWithConfig });
```

=== "Multiple Parameters"
```
test param (nClients in [2, 3, 4], g1 in [1, 2], g2 in [4, 5]) tcTest [main=TestWithConfig]:
assert BankBalanceIsAlwaysCorrect in
(union Client, Bank, { TestWithConfig });
```

=== "With Assumption"
```
test param (nClients in [2, 3, 4], g1 in [1, 2], g2 in [4, 5])
assume (nClients + g1 < g2) tcTest [main=TestWithConfig]:
assert BankBalanceIsAlwaysCorrect in
(union Client, Bank, { TestWithConfig });
```

=== "N-wise Testing"
```
test param (nClients in [2, 3, 4], g1 in [1, 2], g2 in [4, 5], b1 in [true, false])
(2 wise) tcTest [main=TestWithConfig]:
assert BankBalanceIsAlwaysCorrect in
(union Client, Bank, { TestWithConfig });
```

For each testcase, the P checker by default asserts that for each execution of the system (i.e., `module_under_test`): (1) there are no `unhandled event` exceptions; (2) all local assertions in the program hold; (3) there are no deadlocks; and finally (4) based on the [specification monitors that are attached](modulesystem.md#assert-monitors-module) to the module, these safety and liveness properties asserted by the monitors always hold.
!!! info "Properties Checked"
For each test case, the P checker asserts:
1. No `unhandled event` exceptions
2. All local assertions hold
3. No deadlocks
4. All [specification monitors](modulesystem.md#assert-monitors-module) properties hold

!!! example "Tutorial Examples"
For complete examples of parameterized testing in action, see:

- [Client Server](../tutorial/clientserver.md#parameterized-tests): Basic example showing parameter declarations and usage
- [Two Phase Commit](../tutorial/twophasecommit.md#test-scenarios): Shows pairwise testing of system configurations with assumptions
- [Failure Detector](../tutorial/failuredetector.md#parameterized-tests): Demonstrates testing different system sizes and monitoring loads
- [Espresso Machine](../tutorial/espressomachine.md#parameterized-tests): Illustrates testing various user interaction scenarios
32 changes: 27 additions & 5 deletions Docs/docs/tutorial/clientserver.md
Original file line number Diff line number Diff line change
Expand Up @@ -86,8 +86,8 @@ The test scenarios folder in P has two parts: TestDrivers and TestScripts. TestD
The test scenarios folder for ClientServer ([PTst](https://github.com/p-org/P/tree/master/Tutorial/1_ClientServer/PTst)) consists of two files [TestDriver.p](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p) and [TestScript.p](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/Testscript.p).

??? tip "[Expand]: Let's walk through TestDriver.p"
- ([L36 - L60](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L36-L60)) &rarr; Function `SetupClientServerSystem` takes as input the number of clients to be created and configures the ClientServer system by creating the `Client` and `BankServer` machines. The [`CreateRandomInitialAccounts`](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L25-L34) function uses the [`choose`](../manual/expressions.md#choose) primitive to randomly initialize the accounts map.
The function also [`announce` the event](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L49-L51) `eSpec_BankBalanceIsAlwaysCorrect_Init` to initialize the monitors with initial balance for all accounts (manual: [annouce statement](../manual/statements.md#announce)).
- ([L49 - L72](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L49-L72)) &rarr; Function `SetupClientServerSystem` takes as input the number of clients to be created and configures the ClientServer system by creating the `Client` and `BankServer` machines. The [`CreateRandomInitialAccounts`](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L34-L47) function uses the [`choose`](../manual/expressions.md#choose) primitive to randomly initialize the accounts map.
The function also [`announce` the event](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L64) `eSpec_BankBalanceIsAlwaysCorrect_Init` to initialize the monitors with initial balance for all accounts (manual: [annouce statement](../manual/statements.md#announce)).
- ([L3 - L22](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L3-L22)) &rarr; Machines `TestWithSingleClient` and `TestWithMultipleClients` are simple test driver machines that configure the system to be checked by the P checker for different scenarios. In this case, test the ClientServer system by first randomly initializing the accounts map and then checking it with either one `Client` or with multiple `Client`s (between 2 and 4)).

??? tip "[Expand]: Let's walk through TestScript.p"
Expand All @@ -96,6 +96,20 @@ The test scenarios folder for ClientServer ([PTst](https://github.com/p-org/P/tr
- ([L4 - L16](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/Testscript.p#L4-L16)) &rarr; Declares three test cases each checking a different scenario and system. The system under test is the `union` of the modules representing each component in the system (manual: [P module system](../manual/modulesystem.md#union-module)). The `assert` module constructor is used to attach monitors or specifications to be checked on the modules (manual: [assert](../manual/modulesystem.md#assert-monitors-module)).
- In the `tcAbstractServer` test case, instead of composing with the Bank module, we use the AbstractBank module. Hence, in the composed system, whenever the creation of a BankServer machine is invoked the binding will instead create an AbstractBankServer machine.

### Parameterized Tests

The ClientServer tutorial demonstrates P's parameterized testing capabilities that allow systematic exploration of different system configurations. These tests enable checking the system with varying numbers of clients to validate scalability and concurrent access patterns.

??? tip "[Expand]: Let's walk through parameterized TestDrivers.p"
- ([L24](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L24)) &rarr; Parameter declaration `param nClients: int;` defines a configurable parameter for the number of clients to create in the test scenarios.
- ([L26 - L32](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/TestDriver.p#L26-L32)) &rarr; `TestWithConfig` machine creates a configurable number of clients based on the `nClients` parameter, enabling systematic testing of different client loads against the bank server.


??? tip "[Expand]: Let's walk through parameterized TestScripts.p"
- ([L18 - L21](https://github.com/p-org/P/blob/master/Tutorial/1_ClientServer/PTst/Testscript.p#L18-L21)) &rarr; Parameterized test case `tcParameterizedMultipleClients` with `param (nClients in [2, 3, 4])` demonstrates how to systematically test the bank system with different numbers of concurrent clients (2, 3, and 4 clients), generating 3 distinct test scenarios.
- This parameterized approach allows verification of the bank's correctness properties under different concurrency levels, ensuring that the `BankBalanceIsAlwaysCorrect` and `GuaranteedWithDrawProgress` specifications hold regardless of client count.
- The test cases generated are: `tcParameterizedMultipleClients___nClients_2`, `tcParameterizedMultipleClients___nClients_3`, and `tcParameterizedMultipleClients___nClients_4`, each testing the system with the corresponding number of clients.

### Compiling ClientServer

Navigate to the [1_ClientServer](https://github.com/p-org/P/tree/master/Tutorial/1_ClientServer) folder and run the following command to compile the ClientServer project:
Expand Down Expand Up @@ -152,22 +166,25 @@ p check

??? note "Expected Output"

```hl_lines="8 9 10"
```hl_lines="8 9 10 11 12 13"
$ p check

.. Searching for a P compiled file locally in the current folder
.. Found a P compiled file: P/Tutorial/1_ClientServer/PGenerated/CSharp/net6.0/ClientServer.dll
.. Checking P/Tutorial/1_ClientServer/PGenerated/CSharp/net6.0/ClientServer.dll
Error: We found '3' test cases. Please provide a more precise name of the test case you wish to check using (--testcase | -tc).
Error: We found '6' test cases. Please provide a more precise name of the test case you wish to check using (--testcase | -tc).
Possible options are:
tcSingleClient
tcMultipleClients
tcAbstractServer
tcParameterizedMultipleClients___nClients_2
tcParameterizedMultipleClients___nClients_3
tcParameterizedMultipleClients___nClients_4

~~ [PTool]: Thanks for using P! ~~
```

There are three test cases defined in the ClientServer project, and you can specify which
There are six test cases defined in the ClientServer project, and you can specify which
test case to run by using the `-tc` parameter along with the `-s` parameter for the number of schedules to explore.

Check the `tcSingleClient` test case for 1000 schedules:
Expand All @@ -182,6 +199,11 @@ Check the `tcMultipleClients` test case for 1000 schedules:
p check -tc tcMultipleClients -s 1000
```

Check the parameterized generated `tcParameterizedMultipleClients` test cases for 1000 schedules each:
```shell
p check -tc tcParameterizedMultipleClients -s 1000
```

Check the `tcAbstractServer` test case for 1000 schedules:

```shell
Expand Down
Loading
Loading