What Is a Batch Job? Processing Data in Chunks Explained

A batch job processes a bounded set of data all at once, typically on a schedule, rather than handling records continuously as they arrive.

Batch processing is the oldest and still most common way to crunch data. A batch job gathers up a defined chunk of work - all of yesterday transactions, this hour logs - and processes it in one run. It trades latency for simplicity and throughput.

What a batch job is

A batch job is a process that runs over a finite, bounded dataset and completes. It usually runs on a schedule (nightly, hourly) and processes whatever data accumulated since the last run. Examples include a nightly aggregation, a daily report build, or a weekly model retrain.

Batch versus streaming

Batch processes data in chunks with higher latency but simpler logic and easy reprocessing. Streaming processes events continuously with low latency but more complex state handling. Many platforms use batch for heavy historical work and streaming for fresh data.

Idempotency and retries

Because batch jobs are re-run on failure or for backfills, they should be idempotent: running the same job twice produces the same result, not duplicates. This makes retries safe, which matters when a job processing hours of data hits a transient error near the end.

Batch jobs in CI

A batch job runs naturally as a scheduled CI workflow. CI also tests the job logic on a sample chunk before it ships.

A scheduled batch job in CI

on:
  schedule:
    - cron: "0 2 * * *"   # run nightly at 02:00
jobs:
  nightly:
    steps:
      - run: python batch_job.py --date $(date -I)

Latchkey note

Batch jobs can be long and resource-heavy, processing large datasets in one run. On Latchkey, route them to larger runners, cache the input data and dependencies, and rely on auto-retry so a transient failure late in a long job does not force a full re-run from scratch.

Key takeaways

A batch job processes a bounded chunk of data all at once, usually on a schedule.
It trades latency for simplicity and throughput compared to streaming.
Batch jobs should be idempotent so retries and backfills are safe, and they run well as scheduled CI workflows.