Mutation Testing using Mutpy Module in Python

Prerequisite: Mutation Testing

Mutpy is a Mutation testing tool in Python that generated mutants and computes a mutation score. It supports standard unittest module, generates YAML/HTML reports and has colorful output. It applies mutation on AST level.

Installation:

This module does not come built-in with Python. To install it type the below command in the terminal.

pip install mutpy

Here, we will use the Mutpy library to execute test cases written for a simple program.

Now, we perform Mutation Testing for a program that checks if a number is prime or not. 

Code: This code is save as isPrime.py

Now, we need to write test cases for the above program using Pytest or Unittest library. Test cases should be written with an approach to kill all the mutants, i.e. test cases should be effective enough to give a good mutation score. The test cases are written using the Unittest library in the file shown below. 

Test cases are written using self.assertEqual() that is a test assertion generally used to determine if a test case has passed or failed. We have written three test functions below to check the three types of input: 

1. Input is prime
2. Input is non-prime
3. Input is invalid

Note: The function name and the test file name should always start with the word ‘test’.

Code: This code is save as test_isPrime.py.

To execute these test cases, we need to create two separate files isPrime.py and test_isPrime.py in a single folder and run the following command in the command prompt: 

mut.py --target isPrime --unit-test test_isPrime -m --runner pytest

In the above command, we have to specify three things: 

• Target: the target file on which test cases will be run, which in our case is isPrime.py
• Unit-test: the file which has the unit tests that have to be executed, i.e. test_isPrime.py in our case.
• Runner: pytest or unittest

The output will be a set of mutants along with details like mutation score, the number of mutations killed, survived, etc. 

[*] Start mutation process:
   - targets: isPrime
   - tests: test_isPrime
[*] 3 tests passed:
   - test_isPrime [3.07469 s]
[*] Start mutants generation and execution:
   - [#   1] AOR isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num * i == 0:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.45151 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_nonprime
   - [#   2] COI isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if not (num > 1):
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.25723 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_invalid
   - [#   3] COI isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if not (num % i == 0):
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.28817 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
   - [#   4] CRP isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num > 2:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.23510 s] survived
   - [#   5] CRP isPrime:
--------------------------------------------------------------------------------
   2: def isPrime(num):
   3:
   4:     if num > 1:
   5:
-  6:         for i in range(2, num):
+  6:         for i in range(3, num):
   7:             if num % i == 0:
   8:                 return False
   9:         else:
  10:
--------------------------------------------------------------------------------
[1.20360 s] survived
   - [#   6] CRP isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num % i == 1:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.23499 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
   - [#   7] ROR isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num < 1:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.24164 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_invalid
   - [#   8] ROR isPrime:
--------------------------------------------------------------------------------
   1:
   2: def isPrime(num):
   3:
-  4:     if num > 1:
+  4:     if num >= 1:
   5:
   6:         for i in range(2, num):
   7:             if num % i == 0:
   8:                 return False
--------------------------------------------------------------------------------
[1.21934 s] survived
   - [#   9] ROR isPrime:
--------------------------------------------------------------------------------
   3:
   4:     if num > 1:
   5:
   6:         for i in range(2, num):
-  7:             if num % i == 0:
+  7:             if num % i != 0:
   8:                 return False
   9:         else:
  10:
  11:             return True
--------------------------------------------------------------------------------
[1.32597 s] killed by testing program mutpy/test_isPrime.py::CalculatorTest::test_prime
[*] Mutation score [14.91747 s]: 66.7%
   - all: 9
   - killed: 6 (66.7%)
   - survived: 3 (33.3%)
   - incompetent: 0 (0.0%)
   - timeout: 0 (

We can see from the output above, 6 mutants have been killed and only 3 mutant was able to survive. Also, a mutation score of 66.7% was achieved. We can further improve this mutation score by analyzing the mutants that survived the test cases and writing new or modifying test cases to kill the mutants that survived.