Long-running import and export work runs as durable jobs with explicit states, content-addressed artifacts, resumable checkpoints, and idempotent completion so clients poll status without guessing partial outcomes or duplicate side effects.
Package status: AI-assisted reference package. Static package validation has passed; implementation scenarios remain not run and human domain review is required before stable adoption.
Decision
Bulk import and export operations never hold an HTTP request open until completion. Each request creates a durable job record with a stable id, initial state queued, an idempotency key scoped to the initiating principal, and a payload fingerprint. Workers acquire a lease with owner and expiry, advance jobs through explicit states, persist checkpoints, and store output artifacts under content-addressed keys. Clients poll job status; this block does not define a subscribe or push event contract. Completion is idempotent so duplicate worker runs or client retries cannot double-apply mutations or publish two different artifacts for the same job id.
Scope
- Job lifecycle, state transitions, artifact storage, checkpointing, lease recovery, and client-visible status APIs accessed by polling.
- Import validation phases that may fail partially with row-level error reports.
- Export generation that may span multiple shards aggregated into a downloadable bundle.
Outside this block
- Real-time streaming export over WebSocket or server-sent events.
- Synchronous CSV upload endpoints for small payloads below the profile size threshold.
- Virus scanning of uploaded import files (see file-storage scanning blocks).
Contract
- Creating a job with the same idempotency key and equivalent payload fingerprint returns the existing job id and does not enqueue a second worker run.
- Submitting the same idempotency key with a different payload fingerprint returns a stable conflict error and does not mutate the existing job.
- Terminal states are only
succeeded,failed, andcancelled;retry_waitand every other state remain incomplete to clients. - While state is
running, a worker holds lease owner and lease expiry; stale leases trigger recovery from last checkpoint perjob-states.yaml. - On
succeeded, exactly one artifact digest is published; subsequent status reads return the same digest unless the job is explicitly re-run as a new revision. - Artifact download is allowed only when state is
succeeded; running, failed, and cancelled jobs do not serve partial artifact bytes. - On
failed, a structured error report artifact is available when row-level validation caused failure; the job does not partially commit undeclared side effects outside the profile transaction boundary. - Cancellation transitions
queued,running, orretry_waitjobs tocancelledwithin the profile cancellation SLA; workers checkpoint before stopping when safe. - Job records retain audit fields: creator principal, created_at, last_transition_at, idempotency key hash, and payload fingerprint.
- Expired job artifacts are deleted after the profile retention window; status remains queryable with
artifact_expired: true.
Implementation guidance
- Store job metadata in a transactional store; store large artifacts in object storage keyed by digest.
- Implement lease renewals while running and reclaim work when lease_expires_at passes with stale_lease_grace_seconds.
- Expose problem-details errors for invalid state transitions and fingerprint conflicts rather than generic 500 responses.
- Separate validation-only dry-run jobs when the product needs preview without commit.
Failure handling and safeguards
- Worker crash mid-run resumes from last checkpoint rather than restarting from zero when checkpoints are enabled.
- Duplicate completion handlers must verify expected prior state before marking
succeeded. - Object storage unavailability checkpoints work, releases the worker lease, and moves the job to
retry_waitwith a bounded next-attempt timestamp. A later worker returns it torunning; the job is never left looking actively leased or falsely marked successful.
Verification and operations
- Integration tests cover idempotent create, fingerprint conflict, lease recovery, object-storage
retry_wait, poll-only status, and artifact download gating. - Metrics track job age by state, checkpoint stall duration, lease reclaim count, and idempotency replay count.
- Alert when running jobs exceed the profile maximum duration without checkpoint progress.
Adoption assumptions
- Names and numeric values in
example.yamlandjob-states.yamlare an adoption profile, not universal defaults. - The adopting team defines payload schemas, validation rules, and which mutations are allowed inside a job transaction.