Production agent runs are durable jobs, not fragile HTTP requests. A queue absorbs bursts, a lease prevents permanent ownership, checkpoints support continuation, idempotency protects side effects, and reconciliation resolves uncertain outcomes. Retries must have one owner, a classified reason, bounded backoff, and enough state to avoid restarting expensive or consequential work blindly.
You will design the job state machine, retry taxonomy, and exactly-once-effect strategy that works over at-least-once delivery.
Workers renew leases. If a worker dies, another can reclaim after expiry using the checkpoint and tool-attempt records.
Avoid retries in HTTP client, adapter, worker, and queue simultaneously. Choose ownership and count total attempts.
Derive a stable key from run, proposal, tool version, and normalized arguments. Persist not_started before the call, then transition to in_progress, succeeded, or unknown. On duplicate delivery, return stored success or reconcile unknown; never issue a second blind mutation.
Failure injection: Kill the worker immediately after an external system commits. The next worker should query by idempotency key or business reference and record the existing result.
Test worker death during model call, before tool call, after commit, while waiting for approval, and during event publication. Also test duplicate queue delivery, lease expiry, cancellation, poison jobs, and dead-letter replay authorization.
Implement a fake queue and tool ledger. Use property tests with random crashes to prove the external counter increments at most once per proposal.