Batch processing: Measurement and limits

Defines measurable batch-job capacity through bounded partitions, leases, checkpoints, idempotent effects, backpressure, resource budgets, and completion objectives instead of a single nominal throughput number.

Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.

Decision

Plan capacity from arrival volume and completion deadline, then process stable partitions with bounded concurrency and resumable checkpoints. Admit work only within database, queue, provider, memory, and cost budgets and degrade by pausing rather than corrupting progress.

Scope

• Work identity, partitioning, admission, concurrency, leases, checkpoints, retries, poison items, resource limits, fairness, completion SLO, and capacity tests.
• Finite or windowed background jobs over many records.
• Operational scaling and measurement.

Outside this block

• Streaming systems with unbounded event-time semantics.
• Assuming average records per second predicts tail completion under skew.

Contract

• Each batch and item has a stable idempotency identity and input snapshot or watermark.
• Partitions are deterministic, balanced enough to measure skew, and do not process one item concurrently without coordination.
• Workers claim bounded leases, heartbeat, checkpoint only durable effects, and permit safe takeover after expiry.
• Concurrency is controlled by the tightest downstream budget, not CPU availability alone.
• Retries are capped and classified; poison items enter a reviewable terminal queue without blocking all progress.
• Completion SLO includes queue wait, processing, retries, and reconciliation and is reported by batch cohort.

Implementation guidance

• Estimate required steady throughput as input count divided by processing window, then add measured headroom for variance and recovery.
• Load-test real distributions of item cost and partition skew with downstream throttling.
• Use adaptive concurrency bounded by explicit minimum and maximum when feedback signals are reliable.
• Separate item success from batch completeness and reconcile counts/amounts before success.

Failure handling and safeguards

• On lease loss, the worker stops committing and a new owner resumes from durable state.
• On downstream throttle, reduce admission and honor bounded retry timing rather than accumulating unbounded in-memory work.
• On budget breach, pause the batch and expose incomplete state; do not mark partial output complete.

Verification and operations

• Measure arrival, queue age, throughput, item duration distribution, partition skew, concurrency, lease loss, retry, poison rate, downstream saturation, cost, and completion SLO.
• Run failure tests at checkpoint boundaries and a recovery-capacity test for backlog after outage.
• Alert from projected deadline miss as well as current error rate.

The executable-looking examples in this package are fixtures and acceptance contracts. Run python tools/validate.py from the collection root to check package structure and metadata; then implement and execute the scenarios in the target repository.

Adoption assumptions

• Names and numeric values in example.yaml are an adoption profile, not universal defaults.
• The adopting team must map actors, data classes, error vocabulary, and ownership to its system.

References

• No external normative source is required; this package defines a project decision.