16) Spring Batch Lesson

Part 2: Managing Spring Batch Processes

14 min to complete · By Ryan Desmond, Jared Larsen

In the 2nd and final part of this Spring Batch tutorial, you will continue working on the demo project you started in the previous lesson. Pause here and think about what you have learned so far. You know what batch processing is now and how to define a job. But just defining a job is not enough, right?

Next, you learn how to utilize your knowledge in a real-world scenario. Primarily, this lesson will teach you how the JobLauncher is defined and used to start the jobs. You will also learn about restarting/skipping jobs or steps during the batch processing run.

Running a Job

To run a job in batch processing, you need at least two things - a job launcher and a job. The JobLauncher can be called manually in many different ways, or automatically via code. As mentioned in the previous lesson, your demo project will have a web endpoint to launch jobs. To get this started, you'll need a controller:

@RestController
@RequestMapping("/batch")
public class BatchController {

    @Autowired
    private JobLauncher jobLauncher;

    @Autowired
    private Job job;

    @GetMapping(value = "/job")
    public String testJob(@RequestParam(name = "id") String jobId) {
        
        JobParametersBuilder jobParametersBuilder = 
                new JobParametersBuilder();
        
        if (StringUtils.hasLength(jobId)) {
            jobParametersBuilder.addString("jobId", jobId);
        }
        JobExecution jobExecution;
        try {
            jobExecution = 
                    jobLauncher.run(
                            job, 
                            jobParametersBuilder.toJobParameters()
                    );
        } catch (JobExecutionAlreadyRunningException 
                 | JobRestartException 
                 | JobInstanceAlreadyCompleteException 
                 | JobParametersInvalidException e) {
            e.printStackTrace();
            // return exception message
            return e.getMessage();
        }
        // return job execution status
        return jobExecution.getStatus().name();
    }
}

In the code above, you have autowired a JobLauncher and the job bean defined in your BatchConfiguration class. You will launch the job when the /batch/job endpoint is hit. Although you don't need to return anything from this endpoint method, in this example code, the endpoint returns the BatchStatus for demo purposes.

Now with the controller in place, run the project and hit this endpoint to launch the job:

http://localhost:8080/batch/job?id=1000

Handling Bad Data

At this point, you should get the following output in your browser:

FAILED

And in your IntelliJ console you'll get:

Encountered an error executing step designation-step in job employee-loader-job
javax.persistence.EntityNotFoundException: null

ERROR! ERROR! ERROR! Don't worry, this is the expected output, as we have snuck some bad data in your CSV file, oops! The second step fails because the CSV file intentionally contains a wrong designation code - tests instead of test which is defined in your Designation enum.

However, Step 1 was completed; so you’ll see some updates in your JPA database:

MARK 
HARRY 
GEORGE 
Employee Data 1

Now you can run the following queries in the MySQL command line to see the batch tables status. These tables will have all the execution details including your error stacktrace:

SELECT * FROM BATCH_JOB_INSTANCE;
SELECT * FROM BATCH_JOB_SEQ;
SELECT * FROM BATCH_JOB_EXECUTION;
SELECT * FROM BATCH_JOB_EXECUTION_CONTEXT;
SELECT * FROM BATCH_JOB_EXECUTION_PARAMS;
SELECT * FROM BATCH_JOB_EXECUTION_SEQ;
SELECT * FROM BATCH_STEP_EXECUTION;
SELECT * FROM BATCH_STEP_EXECUTION_CONTEXT;
SELECT * FROM BATCH_STEP_EXECUTION_SEQ;

You can run the following query to fix your data so that Step 2 does not fail any longer:

UPDATE `employee` SET `designation` = 'test' WHERE `employee`.`id` = 3

Now that the data is fixed, hit the URL again from the browser:

http://localhost:8080/batch/job?id=1000

This time, you’ll see "COMPLETED". The employee table data will be as expected:

Developer
Designer 
Tester
Employee Data 2

Now, you should also fix the bad data in your CSV file. And the next time your co-worker provides bad data, you know how to diagnose/fix the issue and proceed!

Colorful illustration of a light bulb

Tip: Because this CSV file is stored within your project structure in resources, you'll need to reboot the application after fixing the bad data or the job will continue to fail. This will not be the case when working with files stored outside the project.

Stopping a Job

Now that you have learned how to launch a job, the next natural step would be to learn how to stop a running job. You can stop a running job using the JobOperator as shown below:

Set<Long> executions = 
        jobOperator.getRunningExecutions("employee-loader-job");
jobOperator.stop(executions.iterator().next());

You can also autowire JobOperator similar to how you autowired JobLauncher to run a job. A job will stop immediately if there are no other services related to it. This is because if there is any other related service, Spring Batch will wait for it to be completed and when that service returns a response, Spring Batch will then gracefully update status and stop the job.

Restarting a Job

Now that you can start and stop a job, what would be the next thing? You are right, a combination of both a.k.a. restarting a job. And the best part is that during the course of this demo project, you have already restarted a job once. Did you realize when that happened? Restarting a job means starting a job all over again with the same job parameters as before.

Did you notice that you passed id=1000 as a parameter to run your job? Do you know why that was used? This parameter is your job identifier, which will cause Spring Batch to run the same job with the exact same parameters. You actually restarted the job as you tried the same URL a second time in the browser. When a job is meant to be restarted, Spring Batch will look up the job repository and find the last status and follow the job and step settings.

If the job is configured to not restart via preventRestart(true), the job won’t restart and will throw an exception. If it’s set to false (which is the default setting), the job will restart and follow the step settings.

A completed step won't restart unless allowStartIfComplete() is set to true (not the default behavior), and the limitStart() counter has not reached its limit. Hit this URL again from the browser:

http://localhost:8080/batch/job?id=1000

This time, since it has previously completed successfully, it won't restart and the controller will return the exception message:

A job instance already exists and is complete for parameters={jobId=1000}. 
If you want to run this job again, change the parameters.

When restarting a stopped or failed job, the previous execution remains in place as a record of what happened. This is generally the preferred behavior, but in case you want to manually restart a jobExecution, you can use jobOperator.restart().

Retrying Steps

Retry logic is needed when a step is prone to fail due to network issues, like if you try to launch the job from a web container. In such cases, you can provide retry logic when configuring your steps.

First, you have to call faultTolerant(), and then you will be able to define retryLimit(), retryPolicy(), exception classes etc.

@Bean
public Step designationStep(
        JobRepository jobRepository, 
        ItemReader<Employee> repositoryReader, 
        DesignationProcessor processor, 
        EmployeeWriter writer, 
        PlatformTransactionManager transactionManager) {

    return new StepBuilder("designation-step", jobRepository)
            .<Employee, Employee>chunk(100, transactionManager)
            .reader(repositoryReader)
            .processor(processor)
            .writer(writer)
            .faultTolerant()
            .retryLimit(3)
            .retry(NoSuchEntityException.class)           
            .build();
}

Skipping Steps

If an exception occurs while running a step, and you want to skip that step, you can specify that as well.

After calling faultTolerant(), you can define skipLimit(), skip exception class, noSkip exception class etc. The exceptions that can be skipped will go in the skip() section, but those that should not be skipped will go in the noSkip() section. For example, FileNotFoundException should be passed to noSkip() but FlatFileParseException can be skipped.

@Bean
public Step designationStep(
        JobRepository jobRepository,
        ItemReader<Employee> repositoryReader,
        DesignationProcessor processor,
        EmployeeWriter writer,
        PlatformTransactionManager transactionManager) {

    return new StepBuilder("designation-step", jobRepository)
            .<Employee, Employee>chunk(100, transactionManager)
            .reader(repositoryReader)
            .processor(processor)
            .writer(writer)
            .faultTolerant()
            .skipLimit(10)
            .skip(NoSuchEntityException.class)
            .build();
}

Wrapping Up

Learn by Doing

Package: your.batch.project

  • Take some time to explore the code and database tables further.
  • Demonstrate your ability to start, stop, and restart jobs.
  • Implement some form of retry and skip logic.
  • If you get stuck, and your project is not functioning as expected, you can find a complete working example here.

Please be sure to push your work to GitHub when you're done.

Summary: Spring Batch Tutorial Part 2

In this section, you continued to work on the previous demo example and learned how to start, stop, and restart jobs. You also learned how to manipulate the steps involved in a job, such as skipping or retrying. This should have given you a pretty robust understanding of batch-processing functionality in your code.