Orthogonal Array Approach for Test Case Optimization

ABSTRACT: In Software Testing, we try to cover all test scenarios and test cases to test an application/product to achieve 100% test coverage. However, most of the times we do not have the leisure of time to test all the test cases [1]. In such cases, w e have to balance between the specified time and quality so that we achieve maximum test coverage. This becomes a very big challenge for the Software Test Managers. Managers adopt multiple strategies to optimize the testing effort and achieve the right balance between Cost, Schedule and Quality. Striking a right balance decides the success or failure of any software solution.In this paper, we have proposed a technique called Orthogonal Array Approach for reducing the number of test cases that needs to be tested for any given software and hence improve the efficiency of testing. With this technique, reduced numbers of test cases are generated automatically. Fewer test cases would reduce time consumption of the testing as a whole and hence the cost of testi ng will also reduce. The details of the technique are described along with two case studies for illustration. The advantages of the method are clearly brought out with the help of analysis graphs.

I. INTRODUCTION Software testing has now become a very complex and challenging task. To achieve this challenging task, we need to build a proper strategy. The strategy provides a road map that describes the steps to be undertaken as part of testing. It also lays down the effort, time, and resources required to achieve the same. Thus, testing strategy consists of test planning, test case design, test case execution and test result data collection and evaluation . Software testing activity is not just the identification and specification of defects. It covers reporting and also offers suggestions and recommendations for appropriate actions to be taken for improving the software product/solution. For any given software application, we have a huge number of test cases. We require to identify only those test cases that would lead us to expose maximum number of undetected errors. Despite the importance of techniques in identifying these test cases, developing the techniques

remains one the most difficult aspects of software testing.

II. WHY OPTIMIZATION? Testing process, being the last stage before the release of the product to the customer, holds the key to success of the product in the market. With the day to day increasing competition in the market, there is a need to reduce the testing lifecycle so that the desired testing can be performed with high quality and less cost. Considering the fact that time is always at a premium, the need for having an optimized testing process is therefore very essential.

This requires the Testing Team and Test Manager to optimize the Testing Process. We have explored the usage of the Orthogonal Array Approach described in the next section to reduce the number of test cases significantly. III. ORTHOGONAL ARRAY APPROACH The following terminologies have been used in this approach 1) Factor (f): Those parameters that the tester intentionally changes during testing to study its effect on the output. Levels (p): The different values of factors used in testing.

e3 = (e2 + e1) mod 3 = (1,3,2)

8.

Let us now construct the 9 rows and 4 columns.

The 9 rows represent the 9 test cases. The 1st column is constructed as follows: Rows 1 2 3 4 1 1 1 1 2 2 2 3 3 3 The 1st p (p=3) rows are constructed as follows: 1 1 1 1 2 2 2 3 3 3 2 1 2 3 3 1 2 3 4 1 2 3 2 3 4

2)

During testing, we basically observe the average change in the response when a factor is changed from one level to another level. Hence to achieve the entire test coverage, we should have (f*p) number of test cases. As mentioned above, all these test cases can be executed in an ideal situation with infinite time and budget available. We are using orthogonal array approach to reduce the number of test cases. The following steps are followed to construct the orthogonal array for testing a program with f factors, each factor having p levels:

5 6 7 8 9

9. Rows 1 2 3 4 5 6 7 8 9

1. 2.

Check if p is a prime number. In case where each factor has different levels, Check whether the highest level is a prime number. If the highest level not a prime, identify the next highest prime number. Check if f <=p+1.If not, check if f is a prime number, else identify the next highest prime number. There exists an OA (Orthogonal Array) with p 2 rows and (p+1) columns. When p=3, we have an OA with 9 rows and 4 columns. We construct p tuples (e1, e2, .ep) as follows: e1 = (0,1,2,.,p-1) = (0,1,2) e2 = (1,2,,p) = (1,2,3)

3. 4.

5. 6. 7.

Now we will make use of the tuples constructed in Step 7. 10. Insert eis in ((i-1)*p) +1 th rows, i = 2, 3..., p In our case, for p =3, e2 in ((2-1)*3) + 1 th row = 4 th row; e2 = (1,2,3) e3 in ((3-1)*3) + 1 th row = 7 th row; e3 = (1,3,2) Rows 1 1 2 1 3 1 4 1

ei=(ei-1 + e1) mod p, for i= 3 to p

2 3 4 5 6 7 8 9

1 1 2 2 2 3 3 3

2 3 1

2 3 2

2 3 3

7 8 9

3 3 3

1 2 3

3 1 2

2 3 1

13. When p=5, we have an OA with 25 rows and 6 columns. Let us construct the OA with p=5.

11. Fill (i*p) + 2 th row as (e(i+1) + 1) mod p, i = 1,2,,p In our case, for p =3, 5th row = (e2 + 1) mod 3; ((2, 3, 4) mod 3) i.e. (2, 3, 1) 8th row = (e3 + 1) mod 3; ((2, 4, 3) mod 3) i.e. (2, 1, 3) Rows 1 2 3 4 5 6 7 8 9 1 1 1 1 2 2 2 3 3 3 1 2 3 1 2 3 2 1 2 3 1 2 3 1 2 3 2 3 4 1 2 3 3 1

14. We construct 5 tuples as follows: e1 = (0,1,2,.,p-1) = (0,1,2,3,4) e2 = (1,2,,p) = (1,2,3,4,5) ei=(ei-1 + e1) mod p, for i= 3 to p e3 = (e2 + e1) mod 5 = (1,3,5,7,9) mod5 = (1,3,5,2,4) e4 = (e3 + e1) mod 5 = (1,4,7,5,8) mod 5 = (1,4,2,5,3) e5 = (e4 + e1) mod 5 = (1,5,4,8,7 ) mod 5 = (1,5,4,3,2) 15. The 1st column is constructed as follows: Rows 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 1 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 2 3 4 5 6

12. Fill (i*p) + 3 rd row as (e (i+1) + 2) mod p, i = 1, 2,.., p In our case, for p =3, 6th row = (e2 + 2) mod 3; ((3, 4, 5) mod 3) i.e. (3, 1, 2) 9 row = (e3 + 2) mod 3; ((3, 5, 4) mod 3) i.e. (3, 2, 1) Rows 1 2 3 4 5 6 1 1 1 1 2 2 2 2 1 2 3 1 2 3 3 1 2 3 2 3 1 4 1 2 3 3 1 2
th

17. Insert eis in ((i-1)*p) +1 th rows, i = 2, 3,.., p In our case, for p =5, e2 in ((2-1)*5) + 1 th row = 6 th row; e2 = (1,2,3,4,5) e3 in ((3-1)*5) + 1 th row = 11 th row; e3 = (1,3,5,2,4) e4 in ((4-1)*5) + 1 th row = 16th row; e4 = (1,4,2,5,3) e5 in ((5-1)*5) + 1 th row = 21st row; e5 = (1,5,4,3,2) Rows 16. THE 1ST P (P=5) ROWS ARE CONSTRUCTED AS FOLLOWS: Rows 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 5 1 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 2 1 2 3 4 5 3 1 2 3 4 5 4 1 2 3 4 5 5 1 2 3 4 5 6 1 2 3 4 5 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 1 5 4 3 2 1 4 2 5 3 1 3 5 2 4 2 1 2 3 4 5 1 3 1 2 3 4 5 2 4 1 2 3 4 5 3 5 1 2 3 4 5 4 6 1 2 3 4 5 5

18. Fill (i*p) + 2 th row as (e(i+1) + 1) mod p, i = 1,2,,p In our case, for p =5,

7th row = (e2 + 1) mod 5; ((2, 3, 4, 5, 6) mod 5) i.e. (2, 3, 4, 5, 1) 12th row = (e3 + 1) mod 5; ((2, 4, 6, 3, 5) mod 5) i.e. (2, 4, 1, 3, 5) 17th row = (e4 + 1) mod 5; ((2, 5, 3, 6,4) mod 5) i.e. ( 2,5,3,1,4) 22nd row = (e5 + 1) mod 5; ((2, 6, 5, 4, 3) mod 5) i.e. (2, 1, 5, 4, 3) Rows 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 1 2 5 1 4 5 3 4 2 3 1 2 4 5 2 3 5 1 3 4 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 1 2 3 4 5 2 3 1 2 3 4 5 3 4 1 2 3 4 5 4 5 1 2 3 4 5 5 1

8th row = (e2 + 2) mod 5; ((3, 4, 5, 6, 7) mod 5) i.e. (3, 4, 5, 1, 2) 13th row = (e3 + 2) mod 5; ((3, 5, 7, 4, 6) mod 5) i.e. (3, 5, 2, 4, 1) 18th row = (e4 + 2) mod 5; ((3, 6, 4, 7, 5) mod 5) i.e. (3, 1, 4, 2, 5) 23rd row = (e5 + 2) mod 5; ((3, 7, 6, 5, 4) mod 5) i.e. (3, 2, 1, 5, 4) Rows 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 1 2 3 5 1 2 4 5 1 3 4 5 2 3 4 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 3 4 5 5 1 2 2 3 4 4 5 1 1 2 3 4 5 1 2 3 1 2 3 4 5 2 3 4 1 2 3 4 5 3 4 5 1 2 3 4 5 4 5 1 1 2 3 4 5 5 1 2

19. Fill (i*p) + 3 rd row as (e (i+1) + 2) mod p, i = 1, 2, .., p In our case, for p =5,

20. Fill (i*p) + 4 th row as (e (i+1) + 3) mod p, i = 1, 2, .., p In our case, for p =5,

9th row = (e2 + 3) mod 5; ((4, 5, 6, 7, 8) mod 5) i.e. (4, 5, 1, 2, 3) 14th row = (e3 + 3) mod 5; ((4, 6, 8, 5, 7) mod 5) i.e. (4, 1, 3, 5, 2) 19th row = (e4 + 3) mod 5; ((4, 7, 5, 8, 6) mod 5) i.e. (4, 2, 5, 3, 1) 24th row = (e5 + 3) mod 5; ((4, 8, 7, 6, 5) mod 5) i.e. (4, 3, 2, 1, 5) Rows 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 4 5 1 2 2 3 4 5 5 1 2 3 3 4 5 1 1 2 3 4 3 4 5 1 5 1 2 3 2 3 4 5 4 5 1 2 1 2 3 4 5 1 2 3 4 1 2 3 4 5 2 3 4 5 1 2 3 4 5 3 4 5 1 1 2 3 4 5 4 5 1 2 1 2 3 4 5 5 1 2 3

10th row = (e2 + 4) mod 5; ((5, 6, 7, 8, 9) mod 5) i.e. (5, 1, 2, 3, 4) 15th row = (e3 + 4) mod 5; ((5, 7, 9, 6, 8) mod 5) i.e. (5, 2, 4, 1, 3) 20th row = (e4 + 4) mod 5; ((5, 8, 6, 9, 7) mod 5) i.e. (5, 3, 1, 4, 2) 25th row = (e5 + 4) mod 5; ((5, 9, 8, 7, 6) mod 5) i.e. (5, 4, 3, 2, 1) Rows 1 2 3 4 5 6 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3 4 4 4 4 4 5 5 5 5 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5 2 3 4 5 1 3 4 5 1 2 4 5 1 2 3 5 1 2 3 4 1 2 3 4 5 3 4 5 1 2 5 1 2 3 4 2 3 4 5 1 4 5 1 2 3 1 2 3 4 5 4 5 1 2 3 2 3 4 5 1 5 1 2 3 4 3 4 5 1 2 1 2 3 4 5 5 1 2 3 4 4 5 1 2 3 3 4 5 1 2 2 3 4 5 1

21. Fill (i*p) + 5 th row as (e (i+1) + 4) mod p, i = 1, 2, .., p In our case, for p =5,

Case Study 1: The above approach was used for a project where Compatibility Testing had to be performed for various Browser-OS-Database combinations. The factors and various levels for each of the factors are listed below in Table 1: Table 1: Factors and Levels listed for the Compatibility Testing Scenario
Factors A. Web Browser B. Web Server OS C. Type of Operation D. Database Level 1 IE 5.0 Sun OS 2.8 Retrieval of Data Oracle 8i Level 2 Netscape 4.7 HP-UX 11 Saving of Data SQL Server 7.0 Level 3 Mozilla 1.3.1 Windows NT Server 4.0 Data Deletion Sybase ASE 12.5

Case Study 2: Customer was delighted with our approach and suggested us to add 1 more browser and 1 more OS to this combination. Hence, the factors and various levels for each of the factors are now as follows in Table 3: Table 3: Factors and Levels listed for the Compatibility Testing Scenario with 1 more browser and 1 more OS added
Factors A. Web Browser B. Web Server OS C. Type of Operation D. Database Level 1 IE 5.0 Sun OS 2.8 Retrieval of Data Oracle 8i Level 2 Netscape 4.7 HP-UX 11 Saving of Data SQL Server 7.0 Level 3 Mozilla 1.3.1 Windows NT Server 4.0 Data Deletion Sybase ASE 12.5 Level 4 Safari Windows Vista

Here the highest level =3, which is a prime number.

Table 2: Orthogonal Array Compatibility Testing Scenario

constructed

for

the

Type of Operation Retrieval of Data Saving of Data Data Deletion Saving of Data Data Deletion Retrieval of Data Data Deletion Retrieval of Data Saving of Data

Database Oracle 8i Sybase ASE 12.5 SQL Server 7.0 SQL Server 7.0 Oracle 8i Sybase ASE 12.5 Sybase ASE 12.5 SQL Server 7.0 Oracle 8i

Here the highest level =4, which is not a prime number. The second highest level =3, which is a prime number. Hence we can construct the Orthogonal Array with 4*3= 12 rows and 4 (3+1) columns. The OA is constructed as follows in Table 4: Orthogonal Array constructed for the Compatibility Testing Scenario with 1 more browser and 1 more OS added
Test Number 1 2 3 4 5 6 7 8 9 Web Browser IE 5.0 IE 5.0 IE 5.0 Netscape 4.7 Netscape 4.7 Netscape 4.7 Mozilla 1.3.1 Mozilla 1.3.1 Mozilla 1.3.1 Web Server OS Sun OS 2.8 HP-UX 11 Windows NT Server 4.0 Windows Vista Sun OS 2.8 HP-UX 11 Windows NT Server 4.0 Windows Vista Sun OS 2.8 Type of Operation Retrieval of Data Saving of Data Data Deletion Saving of Data Data Deletion Retrieval of Data Data Deletion Retrieval of Data Saving of Data Database Oracle 8i Sybase ASE 12.5 SQL Server 7.0 SQL Server 7.0 Oracle 8i Sybase ASE 12.5 Sybase ASE 12.5 SQL Server 7.0 Oracle 8i

Benefit shown to the customer: For 4 Factors, each with 3 levels, the total no. of test cases = 34 = 81. With the Orthogonal Array Approach, we have been able to reduce it to 9 Test Cases. Thus we have been able to reduce the testing effort to (1/9) i.e. 11.11% of the total effort. Hence, Effort Saved = 88.89%.

10 11 12

Safari Safari Safari

HP-UX 11 Windows NT Server 4.0 Windows Vista

Retrieval of Data Saving of Data Data Deletion

SQL Server 7.0 Oracle 8i Sybase ASE 12.5

150 100 50 81

144

Benefit shown to the customer: For 4 Factors, 2 factors with 3 levels and the other 2 factors with 4 levels, the total no. of test cases = 3 2 * 42 = 144. With the Orthogonal Array Approach, we have been able to reduce it to 12 Test Cases. Thus we have been able to reduce the testing effort (1/12) i.e. 8.33% of the total effort. Hence, Effort Saved = 91.67%. V. RESULTS

9 0 Case Study1

12

Case Study2

Total TCs Reduced TCs

Total TCs and Reduced TCs for the above 2 Case Studies

Below Table illustrates the results from the above 2 Case Studies. Case Study 1 shows the reduction of Total Test Cases from 81 to 9 with the use of this algorithm. 88.89% efforts have been saved leading to a Time Saving of 144 hrs and Cost Saving of $3456 (Considering the customer billing rate of $ 24 /hr). Case Study 1 shows the reduction of Total Test Cases from 144 to 12 with the use of this algorithm. 91.67% efforts have been saved leading to a Time Saving of 264 hrs and Cost Saving of $6336 (Considering the customer billing rate of $ 24 /hr). Results for the above 2 Case Studies Parameter All possible Test cases Reduced Test Cases Effort Saving (%) Time Saving(hrs) Cost Saving ($) Case Study1 81 9 88.89 144 3456 Case Study2 144 12 91.67 264 6336

160 140 120 100 80 60 40 20 0

144

81

12

Case Study1

Case Study2

Total TCs Reduced TCs

Conclusion

There are various different factors that determine the effectiveness of the testing. To keep the client happy, the managers have to constantly look for opportunities to decrease the overall cost of testing. Matured organizations are now looking at newer and long term solutions for defining the test effectiveness of their testing functions. The above proposed technique has achieved huge Reduction in the percentage of the test cases. Based on the analysis done, the proposed method can be considered a superior technique. Limitation of the proposed technique lies in the fact that by selecting few optimum test cases, we are introducing a risk factor as only few test cases are executed selectively. Another limitation of this technique is the assumption that each of the factors is independent. This may not be the case always. The future work on the technique would try to address these problems.