Spring Batch: Processing Large Data Reliably

Spring Batch is for jobs that process large amounts of data in the background: importing files, nightly reports, migrations. It gives you chunk processing, transactions per chunk, restart after failure, skip and retry policies and a job history.

Concepts

Job "importPeopleJob"
  └─ Step "importStep"  (chunk oriented)
        ┌─────────────┐   item    ┌──────────────┐   item   ┌────────────┐
        │ ItemReader  │ ────────► │ItemProcessor │ ───────► │ ItemWriter │
        │ (CSV, DB..) │ (one at   │ (validate,   │ (returns │ (DB, file) │
        └─────────────┘  a time)  │  transform)  │ null =   └────────────┘
                                  └──────────────┘ skip)      called ONCE per chunk with the whole list
  Chunk size = 100: read 100 items ─► process each ─► write the 100 ─► COMMIT transaction ─► next chunk

  JobRepository (database tables BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, BATCH_STEP_EXECUTION ...)
  stores status, counts and the position, so a failed job can RESTART from the last committed chunk.

Setup

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-batch</artifactId>
</dependency>

spring.batch.job.enabled=false          # do not run all jobs at startup; we launch them ourselves
spring.batch.jdbc.initialize-schema=always   # create the BATCH_* tables (embedded DB: automatic)

Complete job: CSV file to database (Spring Batch 5)

# src/main/resources/people.csv
firstName,lastName
Ram,Sharma
Sita,Verma
Amit,Kumar

public class Person {
    private String firstName;
    private String lastName;
    public Person() { }
    public Person(String firstName, String lastName) { this.firstName = firstName; this.lastName = lastName; }
    public String getFirstName() { return firstName; }
    public void setFirstName(String v) { this.firstName = v; }
    public String getLastName() { return lastName; }
    public void setLastName(String v) { this.lastName = v; }
}

@Configuration
public class ImportJobConfig {

    @Bean
    public FlatFileItemReader<Person> reader() {
        return new FlatFileItemReaderBuilder<Person>()
                .name("personReader")
                .resource(new ClassPathResource("people.csv"))
                .linesToSkip(1)
                .delimited()
                .names("firstName", "lastName")
                .targetType(Person.class)
                .build();
    }

    @Bean
    public ItemProcessor<Person, Person> processor() {
        return p -> p.getLastName().isBlank() ? null                                  // null = filter this item out
                : new Person(p.getFirstName().toUpperCase(), p.getLastName().toUpperCase());
    }

    @Bean
    public JdbcBatchItemWriter<Person> writer(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<Person>()
                .dataSource(dataSource)
                .sql("insert into people (first_name, last_name) values (:firstName, :lastName)")
                .beanMapped()
                .build();
    }

    @Bean
    public Step importStep(JobRepository repository, PlatformTransactionManager tx,
                           FlatFileItemReader<Person> reader, ItemProcessor<Person, Person> processor,
                           JdbcBatchItemWriter<Person> writer) {
        return new StepBuilder("importStep", repository)
                .<Person, Person>chunk(100, tx)
                .reader(reader)
                .processor(processor)
                .writer(writer)
                .faultTolerant()
                .skipLimit(10).skip(FlatFileParseException.class)                     // bad lines are skipped, up to 10
                .retryLimit(3).retry(DeadlockLoserDataAccessException.class)          // retry temporary DB problems
                .build();
    }

    @Bean
    public Job importJob(JobRepository repository, Step importStep) {
        return new JobBuilder("importPeopleJob", repository)
                .incrementer(new RunIdIncrementer())
                .start(importStep)
                .build();
    }
}

Start the job (REST or scheduler)

@RestController
public class JobController {

    private final JobLauncher launcher;
    private final Job importJob;

    public JobController(JobLauncher launcher, Job importJob) { this.launcher = launcher; this.importJob = importJob; }

    @PostMapping("/jobs/import")
    public String run() throws Exception {
        JobParameters params = new JobParametersBuilder()
                .addLong("startedAt", System.currentTimeMillis())          // a new JobInstance for each run
                .toJobParameters();
        JobExecution execution = launcher.run(importJob, params);
        return execution.getStatus().toString();
    }
}

Tasklet step for simple actions

@Bean
public Step cleanupStep(JobRepository repository, PlatformTransactionManager tx) {
    return new StepBuilder("cleanupStep", repository)
            .tasklet((contribution, chunkContext) -> {
                System.out.println("Deleting temp files ...");
                return RepeatStatus.FINISHED;
            }, tx)
            .build();
}
// job: .start(importStep).next(cleanupStep) or conditional flow with .on("FAILED").to(alertStep)

Restart, scaling and design rules

• Restart: if the job fails at chunk 40, restarting the same JobInstance continues after the last committed chunk (readers must save state, the standard ones do).
• Chunk size: too small = many commits (slow). Too large = big transactions and memory. Typical values are 100 to 1000.
• Scale up with multi-threaded steps or partitioning (split data ranges across workers), or remote partitioning for many machines.
• Make writers idempotent when possible, and log skipped items to a table or file.
• Run jobs on one instance at a time (JobInstance uniqueness plus a scheduler lock), or use Kubernetes CronJob.

Common mistake: Loading all records into memory in one list and saving them in one transaction. That is exactly what chunk processing prevents. Read, process and write in chunks with a transaction per chunk.

Interview questions

What is the difference between a chunk step and a tasklet step?

A chunk step reads, processes and writes items in transactional chunks. A tasklet step runs one piece of code, such as cleaning a directory.

How does Spring Batch restart a failed job?

The JobRepository stores the execution context and the last committed chunk. Restarting the same JobInstance continues from there.

Spring Batch or @Scheduled?

@Scheduled only triggers code on a timer. Spring Batch adds the processing model, transactions, restart, skip and retry, and history. Use @Scheduled to launch a batch job.