JUnit – Ordered Tests

JUnit is a popular framework in the Java ecosystem designed for creating and executing automated tests. Although the framework encourages writing independent tests that can run in any order, there are scenarios where tests need to be executed in a specific order. This article explores how to order tests in JUnit and provides examples on using annotations and techniques to control the test execution order.

Ordered Tests in JUnit

JUnit encourages writing isolated tests that do not rely on execution order. In cases like integration testing, where tests may depend on prior test results, controlling the test execution order becomes essential.

JUnit 5 introduced the @TestMethodOrder annotation, which allows developers to specify the order in which test methods within a class are executed. This annotation supports several predefined ordering strategies, as well as custom approaches for more specific requirements.

1. Default Test Execution in JUnit

By default, JUnit does not guarantee any specific order of test execution. The default behavior ensures that tests are independent and can run in parallel or individually without being affected by other tests.

class DefaultOrderTests {
    @Test
    void test1() {
        System.out.println("Test 1");
    }

    @Test
    void test2() {
        System.out.println("Test 2");
    }

    @Test
    void test3() {
        System.out.println("Test 3");
    }
}

In the above example, JUnit does not guarantee that test1() will run before test2(), and so on. The tests might execute in any order.

2. @TestMethodOrder Annotation

The @TestMethodOrder annotation allows explicit ordering of test methods. The annotation takes a MethodOrderer implementation as its parameter to control the order.

import org.junit.jupiter.api.*;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderedTests {

    @Test
    @Order(1)
    void testA() {
        System.out.println("Test A");
    }

    @Test
    @Order(2)
    void testB() {
        System.out.println("Test B");
    }

    @Test
    @Order(3)
    void testC() {
        System.out.println("Test C");
    }
}

In this case, the tests are executed based on their @Order value.

3. Ordering Strategies in JUnit 5

JUnit offers several built-in ordering strategies through MethodOrderer implementations.

a. Order by Annotation (@Order)

This strategy allows ordering based on the @Order annotation, where lower numbers execute first.

import org.junit.jupiter.api.*;

@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class OrderedByAnnotationTests {

    @Test
    @Order(1)
    void testInitialize() {
        System.out.println("Initializing");
    }

    @Test
    @Order(3)
    void testCleanUp() {
        System.out.println("Cleaning up");
    }

    @Test
    @Order(2)
    void testProcessData() {
        System.out.println("Processing data");
    }
}

Use Case: When tests have a clear dependency or sequence, such as setting up resources, performing operations, and cleaning up.

b. Alphanumeric Ordering

Tests are ordered by method names in lexicographical (alphanumeric) order using MethodOrderer.Alphanumeric.

import org.junit.jupiter.api.*;

@TestMethodOrder(MethodOrderer.Alphanumeric.class)
public class AlphanumericOrderTests {

    @Test
    void testZeta() {
        System.out.println("Test Zeta");
    }

    @Test
    void testAlpha() {
        System.out.println("Test Alpha");
    }

    @Test
    void testBeta() {
        System.out.println("Test Beta");
    }
}

Use Case:

Alphanumeric ordering can be helpful in the cases where test methods follow a logical or natural naming pattern (e.g., testStep1(), testStep2(), etc.) and need to be executed in that order. This can be especially useful when you want to execute tests in the readable order without having to use @Order.

c. Random Ordering

Tests are executed in random order each time the tests are run using MethodOrderer.Random.

import org.junit.jupiter.api.*;

@TestMethodOrder(MethodOrderer.Random.class)
public class RandomOrderTests {

    @Test
    void testX() {
        System.out.println("Test X");
    }

    @Test
    void testY() {
        System.out.println("Test Y");
    }

    @Test
    void testZ() {
        System.out.println("Test Z");
    }
}

Each time the tests are run, they will execute in a different order, promoting test independence.

d. Custom Ordering

JUnit 5 allow the developers to define their own test ordering logic by implementing the MethodOrderer interface. This can be useful when we need more complex or custom logic to determine the order of tests.

import org.junit.jupiter.api.*;
import java.lang.reflect.Method;
import java.util.Comparator;

@TestMethodOrder(CustomOrderer.class)
class CustomOrderTests {

    @Test
    void testShort() {
        System.out.println("Test with short name");
    }

    @Test
    void testLonger() {
        System.out.println("Test with longer name");
    }

    @Test
    void testLongestName() {
        System.out.println("Test with longest name");
    }
}

class CustomOrderer implements MethodOrderer {
    @Override
    public void orderMethods(MethodOrderContext context) {
        // Custom ordering logic: based on the length of method names
        context.getMethodDescriptors().sort(Comparator.comparingInt(
                descriptor -> descriptor.getMethod().getName().length()));
    }
}

Use Case:

Customer ordering can be useful when we need the fine-grained control over the test execution order based on the more complex logic, such as metadata, runtime conditions, or other attributes.

Considerations When Using Ordered Tests

• Test Independence: Unit tests should remain independent to avoid introducing dependencies that make tests fragile.
• Integration Tests: Ordered tests are more suitable for integration or system tests where certain steps must occur in sequence.
• Performance: Ordered tests may limit parallel test execution, affecting performance.

Implementation of Ordered Tests in JUnit

This example project demonstrates how to create ordered tests in JUnit 5. The project will showcase how to set up test classes and control test execution order using the @TestMethodOrder.

Step 1: Create a New Maven Project

Create a new Maven project using IntelliJ IDEA with the following options:

• Name: OrderedTestsDemo
• Build System: Maven
• Click on the Create button.

Project Structure

Once the project is created, set up the file structure as follows:

Step 2: Add the JUnit 5 Dependency

Open the pom.xml file and add the following JUnit 5 dependencies:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.gfg</groupId>
    <artifactId>OrderedTestsDemo</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <!-- JUnit 5 Dependency -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-api</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter-engine</artifactId>
            <version>5.7.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <!-- Maven Surefire Plugin to run JUnit 5 tests -->
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
                <configuration>
                    <includes>
                        <include>**/*Test.java</include>
                    </includes>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

Step 3: Create CalculatorService Class

The CalculatorService class will perform basic arithmetic operations. This class will serve as the target for the tests.

CalculatorService.java:

package com.gfg;

// Service class for basic arithmetic operations
public class CalculatorService {

    public int add(int a, int b) 
    {
        return a + b; // Returns the sum of a and b
    }

    public int subtract(int a, int b) 
    {
        return a - b; // Returns the difference of a and b
    }

    public int multiply(int a, int b) 
    {
        return a * b; // Returns the product of a and b
    }

    public int divide(int a, int b) 
    {
        if (b == 0) 
        {
            throw new IllegalArgumentException("Cannot divide by zero"); // Error handling for division by zero
        }
        return a / b; // Returns the quotient of a and b
    }
}

Step 4: Main Class

The main class demonstrates the use of the CalculatorService.

Main.java:

package com.gfg;

// Main class to execute the CalculatorService
public class Main {
    public static void main(String[] args) {
        CalculatorService calculatorService = new CalculatorService(); // Create a CalculatorService instance
        
        // Performing arithmetic operations and displaying results
        System.out.println("Addition: " + calculatorService.add(3, 4));
        System.out.println("Subtraction: " + calculatorService.subtract(9, 5));
        System.out.println("Multiplication: " + calculatorService.multiply(3, 5));
        System.out.println("Division: " + calculatorService.divide(10, 2));
    }
}

Step 5: Create JUnit Test Class

The test class uses ordered tests to validate the CalculatorService methods.

CalculatorServiceTest.java:

import com.gfg.CalculatorService;
import org.junit.jupiter.api.*;

import static org.junit.jupiter.api.Assertions.*;

// Test class for CalculatorService
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class CalculatorServiceTest {

    private static CalculatorService calculatorService; // Service to be tested

    @BeforeAll
    static void setup() {
        calculatorService = new CalculatorService(); // Initialize the CalculatorService before all tests
    }

    @Test
    @Order(1) // First test to execute
    void testAddition() {
        int result = calculatorService.add(3, 4);
        assertEquals(7, result); // Validate addition result
    }

    @Test
    @Order(2) // Second test to execute
    void testSubtraction() {
        int result = calculatorService.subtract(9, 5);
        assertEquals(4, result); // Validate subtraction result
    }

    @Test
    @Order(3) // Third test to execute
    void testMultiplication() {
        int result = calculatorService.multiply(3, 5);
        assertEquals(15, result); // Validate multiplication result
    }

    @Test
    @Order(4) // Fourth test to execute
    void testDivision() {
        int result = calculatorService.divide(10, 2);
        assertEquals(5, result); // Validate division result
    }

    @Test
    @Order(5) // Additional test to check division by zero
    void testDivisionByZero() {
        Exception exception = assertThrows(IllegalArgumentException.class, () -> {
            calculatorService.divide(10, 0); // Attempt to divide by zero
        });
        assertEquals("Cannot divide by zero", exception.getMessage()); // Validate exception message
    }
}

Step 6: Run the Application

Once completed, the project will run and show the below output:

Step 7: Test the Application

Now we will test the application, and it will show the below result. We can use the below command to run the test cases:

mvn test

Output:

This example project demonstrate how to create the JUnit 5 project with ordered tests. Using the @TestMethodOrder and @Order, we can control the sequence in which the test methods are executed. This can be useful in scenarios where tests depend on the specific execution order, such as the integration tests or system tests that involve multiple stages.