JUnit 4 - Conditional Test Execution and Ignoring Strategies

In software testing, there are scenarios where we might to conditionally run or skip the certain tests based on the specific factors, such as operating system, environment variables, or application conditions. JUnit provides various mechanisms to handle these situations. It allows the tests to be ignored or executed conditionally. This flexibility ensures that the tests run only when applicable by reducing the chance of irrelevant test failures and improving test reliability.

In this article, we will learn different approaches to conditionally run or ignore the tests using JUnit 4, including the @Ignore annotation, Assume class, and custom rules.

Prerequisites:

• Basic understanding of Java and JUnit.
• Maven for building dependency management.
• JDK and IntelliJ IDEA installed in your system

Conditional Test Execution and Ignoring Tests in JUnit 4

1. Ignoring Tests using the @Ignore Annotation

The @Ignore annotation in the JUnit 4 is the simplest way to skip the test. It allows us to mark the test method or the entire test class as ignored. Ignored tests will not executed during the test run, and the test report will indicate that they were skipped.

Example:

import org.junit.Test;
import org.junit.Ignore;

public class SimpleIgnoreTest {

    @Test
@Ignore (This test is temporarily ignored due to bug XYZ")
    public void testIgnored() {
        Some test logic here
    }
}

In this example, the @Ignore annotation prevents the testIgnored() method from being executed. The optional string inside the annotation provides the reason, which can be useful for tracking why the test can be ignored (e.g., due to a known bug or an unimplemented feature).

When to Use:

• We want to temporarily skip the test, such as when it's not yet implemented or if it's currently known to fail due to the bug.
• We want to exclude the specific tests from the current execution run but still keep them in the test suite for future use.

2. Conditionally Ignoring Tests using Assume

In more dynamic scenarios, where we want to conditionally ignore the tests based on the runtime factors (like environment variables, operating systems, or certain conditions), JUnit 4 provides the Assume class. Assumptions are used to skip tests if certain conditions are not met. This makes Assume a more flexible way to control test execution.

When the assumption is false, the test is ignored rather than failed, allowing us to skip the irrelevant tests for the current environment or situation.

Example:

import org.junit.Assume;
import org.junit.Test;

public class AssumeExampleTest {

    @Test
    public void testOnlyOnWindows() {
        Assume.assumeTrue(System.getProperty("os.name").startsWith("Windows"));
        // Test logic that will only run on Windows
        System.out.println("This test runs only on Windows");
    }

    @Test
    public void testOnlyOnLinux() {
        Assume.assumeTrue(System.getProperty("os.name").startsWith("Linux"));
        // Test logic that will only run on Linux
        System.out.println("This test runs only on Linux");
    }
}

In this example:

• The testOnlyOnWindows() method will only run if the operating system starts with "Windows". If the assumption got fail (i.e., the OS is not Windows), then the test will be ignored.
• Similarly, testOnlyOnLinux() will be ignored on the non-Linux environments.

Usage Scenarios:

• Testing the OS-specific functionality (e.g., Windows, Linux, Mac).
• It is skipping the tests based on the environment variables or application properties (e.g., feature flags, specific configurations).
• This is avoiding tests in certain phases of development or deployment (e.g., skipping tests in production).

3. Skipping Tests Based on Environment Variables or System Properties

Another practical use case of the assumption is to skip tests based on the environments variables or system properties. This allows us to configure the tests according to the external factors, such as whether a feature flag is enabled or if we are in particular environment (e.g., dev, test, or prod).

Example:

import org.junit.Assume;
import org.junit.Test;

public class EnvironmentVariableTest {

    @Test
    public void testIfFeatureEnabled() {
        String featureFlag = System.getenv("FEATURE_ENABLED");
        Assume.assumeTrue("true".equals(featureFlag));
        // Test logic that runs only when the FEATURE_ENABLED environment variable is true
        System.out.println("Feature is enabled, running the test");
    }
}

In this example:

• The test checks whether the environment variable FEATURE_ENABLED set to "true". If it is, the test will run, otherwise, it will be skipped.
• This is particularly useful in the CI/CD pipelines, where different tests may be run depending on the environment (development, testing, or production) the pipeline is operating in.

4. Conditionally Ignoring Tests with Custom Annotations

While Assume works well for the simple conditions, sometimes more complex logic is needed to decide whether the test should be run. In these case, we can create the custom annotations and rules to handle the conditional test execution.

Define Custom RunIf Annotation

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(RetentionPolicy.RUNTIME)
public @interface RunIf {
    String value();
}

Create the Custom Rule

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class ConditionalRule implements TestRule {

    @Override
    public Statement apply(Statement base, Description description) {
        RunIf runIf = description.getAnnotation(RunIf.class);
        if (runIf != null) {
            String condition = runIf.value();
            if (!"ENABLE_TEST".equals(System.getenv(condition))) {
                return new Statement() {
                    @Override
                    public void evaluate() {
                        Skip the test
                    }
                };
            }
        }
        return base;
    }
}

Apply the Rule to Test Class:

import org.junit.Rule;
import org.junit.Test;

public class CustomAnnotationTest {

    @Rule
    public ConditionalRule conditionalRule = new ConditionalRule();

    @Test
    @RunIf("RUN_TEST")
    public void testConditionalRun() {
        // This test will only run if the RUN_TEST environment variable is set to ENABLE_TEST
        System.out.println("Test executed based on custom condition");
    }
}

In this example:

• The custom annotation @RunIf is created to specify the condition (e.g., an environment variable).
• A conditional rule is defined to evaluate the condition at runtime and decide whether the test should run.
• This setup allows for the highly flexible and reusable conditional test mechanism across the test suite.

5. Ignoring the Tests Dynamically in Code

Another option is to include the logic inside the test methods to skip or ignore the tests based on certain runtime conditions.

Example:

import org.junit.Test;
import static org.junit.Assume.assumeTrue;

public class DynamicIgnoreTest {

    @Test
    public void testWithDynamicCondition() {
        boolean condition = checkSomeCondition();
        assumeTrue(condition);
        // Test logic will only execute if the condition is true
        System.out.println("Condition is true, running the test.");
    }

    private boolean checkSomeCondition() {
        // Custom logic to check a condition, e.g., checking a database state or a service response
        return false; // For this example, returning false to simulate skipping the test
    }
}

In this example, the test dynamically checks the condition inside the method using the assumeTrue(), if the condition fails, then the test is skipped.

Project Implementation to Conditionally Run Tests in JUnit 4

We will create a simple Maven project that demonstrates how to conditionally run or ignore the tests using the JUnit 4. This project will include example of:

• Using the @Ignore annotation.
• Conditionally skipping tests using the assumption.
• Custom conditional test execution using the custom annotation and rule.

Step 1: Create a Maven Project

Create a new Maven Project using the IntelliJ IDEA. Choose the following options:

• Name: conditional-tests
• Build System: Maven

Click on the Create button.

Project Structure

After the project creation done, then set the project folder structure as shown in the below image:

Step 2: Add the JUnit Dependency into pom.xml

Open the pom.xml file and add the JUnit 4 dependency into the maven project.

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.gfg</groupId>
    <artifactId>conditional-tests</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <dependencies>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.13.2</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>2.22.2</version>
            </plugin>
        </plugins>
    </build>

</project>

Step 3: AssumeExampleTest Class

Create the AssumeExampleTest class and this class demonstrate how to use the assumption to skip tests based on the conditions, such as the operating system.

AssumeExampleTest.java:

import org.junit.Assume;
import org.junit.Test;

public class AssumeExampleTest {

    @Test
    public void testOnlyOnWindows() {
        Assume.assumeTrue(System.getProperty("os.name").startsWith("Windows"));
        // Test logic that runs only on Windows
        System.out.println("This test runs only on Windows.");
    }

    @Test
    public void testOnlyOnLinux() {
        Assume.assumeTrue(System.getProperty("os.name").startsWith("Linux"));
        // Test logic that runs only on Linux
        System.out.println("This test runs only on Linux.");
    }
}

Explanation:

• testOnlyOnWindows() runs only if the operating system is Windows. If the condition is not met, then the test will be skipped.
• testOnlyOnLinux() runs only if the OS is Linux.

Step 4: Define the Custom Annotation

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;

@Retention(RetentionPolicy.RUNTIME)
public @interface RunIf {
    String value();
}

Step 5: Create the Custom Rule

import org.junit.rules.TestRule;
import org.junit.runner.Description;
import org.junit.runners.model.Statement;

public class ConditionalRule implements TestRule {

    @Override
    public Statement apply(Statement base, Description description) {
        RunIf runIf = description.getAnnotation(RunIf.class);
        if (runIf != null) {
            String condition = runIf.value();
            if (!"ENABLE_TEST".equals(System.getenv(condition))) {
                return new Statement() {
                    @Override
                    public void evaluate() {
                        // Skipping the test
                    }
                };
            }
        }
        return base;
    }
}

Step 6: Apply the Rule in the Test Class

import org.junit.Rule;
import org.junit.Test;

public class CustomAnnotationTest {

    @Rule
    public ConditionalRule conditionalRule = new ConditionalRule();

    @Test
    @RunIf("RUN_TEST")
    public void testConditionalRun() {
        // This test will run only if the environment variable "RUN_TEST" is set to "ENABLE_TEST"
        System.out.println("Test executed based on custom condition.");
    }
}

Step 7: DynamicIgnoreTest class

Create the DynamicIgnoreTest class, and this test demonstrates how to dynamically skip the test based on the custom conditions within the test method itself.

DynamicIgnoreTest.java:

import org.junit.Test;
import static org.junit.Assume.assumeTrue;

public class DynamicIgnoreTest {

    @Test
    public void testWithDynamicCondition() {
        boolean condition = checkSomeCondition();
        assumeTrue(condition);
        // Test logic will only execute if the condition is true
        System.out.println("Condition is true, running the test.");
    }

    private boolean checkSomeCondition() {
        // Custom logic to check a condition
        return false; // Returning false to skip the test
    }
}

Step 8: SimpleIgnoreTest Class

Create the SimpleIgnoreTest class and this class demonstrate how to use the @Ignore annotation to skip the test.

SimpleIgnoreTest.java:

import org.junit.Test;
import org.junit.Ignore;

public class SimpleIgnoreTest {

    @Test
    @Ignore("This test is ignored because it is under development.")
    public void testToBeIgnored() {
        // Test logic here
        System.out.println("This test will be ignored.");
    }

    @Test
    public void testThatRuns() {
        // Test logic here
        System.out.println("This test runs normally.");
    }
}

Step 9: Main Runner

import org.junit.runner.JUnitCore;
import org.junit.runner.Result;
import org.junit.runner.notification.Failure;

public class MainTestRunner {

    public static void main(String[] args) {
        // Run specific test classes
        runTest(SimpleIgnoreTest.class);
        runTest(AssumeExampleTest.class);
        runTest(DynamicIgnoreTest.class);
        runTest(CustomAnnotationTest.class);
    }

    public static void runTest(Class<?> testClass) {
        System.out.println("Running tests for: " + testClass.getSimpleName());

        Result result = JUnitCore.runClasses(testClass);

        // Display results
        if (result.getFailureCount() > 0) {
            for (Failure failure : result.getFailures()) {
                System.out.println("Test failed: " + failure.getDescription());
                System.out.println("Error: " + failure.getMessage());
            }
        } else {
            System.out.println("All tests passed for: " + testClass.getSimpleName());
        }

        System.out.println("Total tests run: " + result.getRunCount());
        System.out.println("Tests ignored: " + result.getIgnoreCount());
        System.out.println("---------------------------------------------------\n");
    }
}

Step 10: Running Tests

When we execute MainTestRunner, it is automatically run the tests from the test classes SimpleIgnoreTest, AssumeExampleTest, DynamicIgnoreTest, and CustomAnnotationTest. We can modify this to include any test class we want to run.

Step 11: Run the Test cases

We can use the below command to test the maven project:

mvn test

Output:

In this example project, we demonstrate how to conditionally run or ignore the tests in the JUnit 4 using several approaches that are @Ignore, Assume, and Custom Annotation.

These techniques provides flexibility in controlling the test execution based on the runtime conditions, environment setups, or specific use cases.