Mutation

Mutation Testing • Error seeding – Introduce artificial faults into a program in some suitable random fashion – Test the program with a test suite – Let r the ratio defined by r = n1 / n2 where • n1 = # of the artificial faults found during error seeding • n2 = # of the artificial faults introduced during error seeding 1 Mutation Testing • Use error seeding to estimate the number of faults – Assume that we test a program with a test suite • Let n be # of faults found during testing – How many faults remain in the program' • n1 / r 2 1 Mutation Testing • Mutants int getMax(int x, int y) { int max; if (x > y) max = x; else max = y; return max; } } int getMax(int x, int y) { int max; if (x >= y) max = x; else max = y; return max; } int getMax(int x, int y) { int max; if (x > y) max = x; else max = x; return max; 3 Mutation Testing • Equivalent mutants int getMax(int x, int y) { int max; if (x > y) max = x; else max = y; return max; } } int getMax(int x, int y) { int max; if (x >= y) max = x + 1; else max = y; return max; } int getMax(int x, int y) { int max; if (x > y) max = x * 1; else max = y; return max; 4 2 Mutation Testing • Assumptions – Competent programmer assumption • Programmers are not so stupid • They produce programs that are close to being correct • Faults should be detectable as small deviations from the intended program – Coupling effect assumption • If a test suite kills a mutant, it also can kill mutants of mutants 5 Mutation Testing • Mutation analysis – We want to evaluate the quality of a test suite • Assume that we are given a program with a test suite • Make a set of mutants • Test the program and a mutant with the test suite – If the outputs are different, the mutant is said to be killed by the test suite – Otherwise, the mutant is live » The test suite is inadequate » The mutant is equivalent to the original program 6 3 Mutation Testing • Mutation Adequacy – Evaluate the quality of a test suite for a program • D / (M – E) where – D is # of killed mutants by the test suite – M is # of total mutants – E is # of equivalent mutants 7 Mutation Testing int getMax(int x, int y) { int max; if (x > y) max = x; else max = y; return max; } } int getMax(int x, int y) { int max; if (x >= y) max = x; else max = y; return max; x = 2, y = 2 x = 7, y = 2 x = 2, y = 2 x = 7, y = 2 8 4 Mutation testing For this approach of testing we consider : i) the programmer is competent ii) the “coupling” This testing is based on the program and its mutants. Test take the following instruction of the program; if a > 8 then x := y (initial Program) We consider the following as mutants of the above program: if a < 8 then x := y if a >= 8 then x := y if a > 10 then x := y if a > 8 then x := y +1 if a > 8 then x := x The mutant consists in a simple modification of the initial program. Mutation operators dependent of programming languages should be defined. For technical and practical raisons the number of mutants to consider should be limited (memory, execution time,…). 9 The main objective is to select efficient Test Data which have the error detection power. To criteria for this Test Data is to differentiate the initial program from the mutant. This distinguishability between the initial program and its mutants will based on test results. If the selected Test Data distinguish the initial program from a particular mutant, we say that The Test Data TD kills the mutant. The game in this testing is to “kill” mutants as much as possible. An example: a := b + c; writeln (a); Lets take the following mutants: i) a := b + 1; writeln (a); ii) a := b - c; writeln (a); iii) a := b + c; writeln (b); Test Data TD1 (c #1) TD2 (b and c #0) TD3 (c#0) b 2 4 8 This class of mutants is called Primary Mutants because based on a, b, c values we can distinguish them from the initial program. c 3 2 5 Initial prog. Mutant 5 6 13 3 (i) 2 (ii) 8 (iii) 10 5 Lets take another example: var a, b, c: integer; begin if a < b then c:= a; end Mutants for this program: i) if a a[r] then r:=i; M2) r:= 1; For i:= 2 to 3 do if i > a[r] then r:=i; M3) r:= 1; For i:= 2 to 3 do if a[i] >= a[r] then r:=i; M4) r:= 1; For i:= 2 to 3 do if a[r] > a[r] then r:=i; 12 6 Lets consider the following Test Data a[1] DT1 DT2 DT3 1 1 3 a[1] 2 2 1 a[1] 3 1 2 We apply these Test Data to mutants M1, M2, M3 and M4 P M1 M2 M3 M4 Killed Mutants M4 M2 & M4 none DT1 DT2 DT3 3 2 1 3 2 1 3 3 1 3 2 1 1 1 1 13 We have to look at the effeciency of the proposed test Data. We notice that M1 and M3 are not “killed” yet. We deduce that the test is incomplete. We have to add a new Test Data DT4 = {2,2,1} , P will have r=1 and M3 will have r=2, then this test data permits to kill M3. Now, we have to analyze M1 in order to derive a “killer” test. The difference between P and M1 is the starting point. M1 starts with i=1 and P starts with i=2. This has no impact on the result r. We conclude that M1 is a secondary mutant Conclusions In mutation technique , there is a need for mutants generation, test data selection and execution of all mutants. This consume time and money. Weak mutation technique have introduced to reduce time execution. Weak mutation analyzes the status at mutation point, if different from initial the search is continued. 14 7