Uses visibility timeout plus heartbeat, caps attempts, moves poison messages to a dead-letter queue with payload redaction, and requires idempotent handlers.
Package status: reference context ready for human review. The contract and test scenarios are complete, but no claim is made that an adopting implementation has passed them.
Decision
Async workers lease messages with a visibility timeout and extend that lease with heartbeats while work continues. Handlers are idempotent on a stable job key. After a maximum attempt count, the message is poison: it moves to a dead-letter queue with payload fields redacted according to policy. Operators replay from the dead-letter queue only after the handler defect is fixed.
Scope
- Visibility timeout, heartbeat, attempt caps, poison classification, dead-letter redaction, idempotent handling, and replay for async workers.
- Queues fed by an outbox or API enqueue.
- Interaction with notification bounce handling when the job is an email send.
Outside this block
- Exactly-once publication into the queue, which belongs to the outbox block.
- Worker autoscaling policies.
Contract
- A received message is invisible to other workers until timeout; the worker heartbeats before the timeout if processing continues.
- If heartbeat stops, the message becomes visible and may be retried; duplicate delivery must hit the same idempotency key and not double-apply side effects.
- Attempts increment on each lease; at max attempts the message is sent to the dead-letter queue, not deleted silently.
- Dead-letter payloads redact secrets, tokens, and personal data listed in the poison policy; operators still see job type, id, and error class.
- Poison is for handler or payload defects, not for downstream rate limits that should retry with backoff under the cap.
- Replay from dead-letter requires a live handler version that records the replay actor.
Implementation guidance
- Set visibility timeout above the p99 handler time plus heartbeat interval, using the same idea as Amazon SQS visibility timeout.
- Store processed job keys with a TTL longer than the maximum retry horizon.
Failure handling
- Heartbeat failure mid-work: assume another worker may start; the first worker must still be idempotent if it later completes.
- DLQ publish failure: keep the message visible for retry of the poison path; never drop after max attempts without an operator signal.
Verification and operations
- Measure lease expiries, attempt histograms, DLQ depth, redaction misses, and duplicate side effects (must be zero).
- Page when DLQ depth grows for a job type that is not in a known incident.
- Prove with tests that two overlapping leases for one job id apply the side effect once.
The executable-looking examples in this package are fixtures and acceptance contracts. Run
python tools/validate.py from the collection root to check package structure and metadata; then
implement and execute the scenarios in the target repository.
Adoption assumptions
- Names and numeric values in
example.yamlare an adoption profile, not universal defaults. - The adopting team must map actors, data classes, error vocabulary, and ownership to its system.