How to Recover Failed AI Music API Jobs with a Dead-Letter Queue

Put an AI music job in a dead-letter queue only after a bounded worker-delivery budget is exhausted or the provider reports a terminal failure. If you already have a task ID, redrive the task read, not the generation request. If the original POST outcome is unknown, quarantine it for reconciliation instead of automatically creating another potentially billable job.

By Sonilo Team · Facts verified August 27, 2026

Disclosure: Sonilo publishes this guide and operates the API used as the concrete example. The `ai-media-dead-letter-v1` classifier and redrive planner were tested with deterministic local fixtures. The tests did not call a production account, generate media, inspect billing, measure queue or provider availability, or validate a customer deployment.

Separate queue failure from provider failure

A dead-letter queue, or DLQ, isolates work that a consumer could not process after a bounded number of deliveries. It is not a list of every provider task that returns `failed`, and it is not permission to repeat the operation that created the task.

Three independent states matter:

The current Sonilo Retrieve Task reference documents `GET /v1/tasks/{task_id}` for asynchronous work and identifies `processing`, `succeeded`, and `failed` as native states. It also tells adapters to treat normalized `completed` and `canceled` states as terminal. That provider state is evidence about one task. Your queue delivery count is evidence about whether one worker message has been handled. Keep the two counters separate.

This distinction prevents the most expensive DLQ mistake: redriving a message by repeating a generation POST when the original job may already exist. The application-level duplicate-job guard owns request identity before submission. The DLQ owns inspection and controlled recovery after normal processing stops.

Use one decision table for every delivery

Classify each delivery before acknowledging, retrying, or quarantining it:

The current Sonilo API error guide distinguishes authentication, balance, permission, validation, rate-limit, and upstream failures. The machine-readable OpenAPI 1.0.0 contract maps `401`, `402`, `403`, `404`, and `429` to different next actions. Preserve the class instead of flattening every response into “retry failed.”

Treat `429` as capacity feedback. Honor `Retry-After` on the task read and stay inside the shared account budget described in the rate-limit admission guide. Treat a repeated upstream outage as dependency health, which belongs to the circuit-breaker workflow. A DLQ should receive the work only after those bounded controls have done their jobs.

Choose the delivery budget from the operation

Queue products expose different controls, but the design question is the same: how many deliveries should one work item receive before normal processing stops?

Amazon SQS dead-letter queue guidance uses `maxReceiveCount` and warns against setting it so low that one ordinary failure sends a message to the DLQ. It also recommends a DLQ retention period longer than the source queue's retention period for standard queues. Google Cloud Pub/Sub dead-letter guidance supports a configured delivery-attempt range of 5 through 100 and describes that maximum as approximate, not an exact application counter.

Do not copy either provider's default into your application without a time budget. Define:

`delivery window = visibility or acknowledgment deadline + retry delay + worker startup margin`

`maximum recovery time = sum of allowed delivery windows`

The delivery budget must fit inside your product deadline and the provider-task poll deadline. The bounded polling guide covers the task-read schedule. The graceful worker shutdown guide covers preserving task ownership when a deployment interrupts a worker.

Ordered workloads need extra care. Amazon's guidance says a DLQ can break FIFO order, using a video edit decision list as an example where order changes meaning. If later work depends on an earlier task, quarantine the entire dependency group or pause that ordering key rather than letting subsequent edits continue against an incomplete timeline.

Store a safe dead-letter envelope

A useful DLQ record explains what to resume without becoming a second secret store. The tested `ai-media-dead-letter-v1` envelope keeps only:

Do not copy the bearer key, signed source URL, prompt, filename, temporary output URL, or an arbitrary provider payload into the queue. The safe logging contract applies the same allowlist principle to telemetry. OWASP's Logging Cheat Sheet recommends excluding access tokens, authentication passwords, and sensitive personal data from logs unless specifically required and protected; a DLQ deserves the same review because operators and support systems often read it.

The local gate constructs this shape only after the classifier chooses `quarantine`:

`{`

` "version": "ai-media-dead-letter-v1",`

` "applicationRequestId": "req-123",`

` "taskId": "task-123",`

` "phase": "task_read",`

` "reason": "task_read_attempts_exhausted",`

` "deliveryAttempt": 5,`

` "firstSeenAt": "2026-08-27T16:00:00.000Z",`

` "quarantinedAt": "2026-08-27T16:10:00.000Z",`

` "providerCode": "upstream_error"`

`}`

The envelope intentionally omits source and output capabilities. Store customer inputs in the system that already owns their access controls and retention policy. Use identifiers in the DLQ to retrieve reviewed context when an operator has permission.

Run the tested Node.js gate

`dead-letter-job-gate.mjs` is a dependency-free classifier and redrive planner. The module accepts one delivery observation and returns one of four actions: acknowledge, retry the known task read, quarantine, or hold for review. Every result fixes `allowGenerationPost` to `false`.

The central rule is short:

`if (phase === "submission_unknown") {`

` return { action: "quarantine", reason: "submission_outcome_unknown", allowGenerationPost: false };`

`}`

`if (taskStatus === "processing" && deliveryAttempt < maxDeliveryAttempts) {`

` return { action: "retry_task_read", reason: "provider_processing", allowGenerationPost: false };`

`}`

Run the module and its tests with `node --test dead-letter-job-gate.test.mjs`. The stable Node.js test runner executed 21 deterministic tests on Node.js v22.23.2 on August 27, 2026. The suite covers processing, success, terminal failure, `400`, `401`, `402`, `403`, `422`, `429`, `503`, exhausted reads, ambiguous submissions, envelope redaction, operator approval, and three redrive paths.

The tests prove the local decision policy against controlled JavaScript objects. They do not prove SQS or Pub/Sub delivery semantics, provider availability, billing behavior, output quality, secret-management controls, or a production operator's judgment.

Redrive by resuming, repairing, or reconciling

“Redrive” should name a reviewed recovery operation, not a bulk replay button.

Resume a known task read

When the DLQ record has a task ID and the reason is exhausted task reads, query the exact task path again. Do not create another job. If the task now succeeds, validate its media and continue the temporary-output durability workflow. If it is still processing, place it into a new bounded poll window rather than an unbounded loop.

Repair a blocking condition

Authentication, account balance, permissions, invalid input, and malformed responses need different owners. Repair the condition first, then decide whether a new business intent exists. The live account usage workflow can provide dated service and usage evidence, while the pre-submit cost estimator can apply a reviewed budget ceiling before any new generation.

Reconcile an unknown submission

A timeout around a generation POST does not prove that the server rejected it. RFC 9110 section 9.2.2 limits automatic retry of a non-idempotent request unless the client knows the semantics are idempotent or knows the original request was not applied. Keep an unknown submission in a reconciliation state until account, task, or application evidence supports a decision.

If no recovery identifier can be found, a person may approve a new generation as a new intent. Give it a new application request ID, revalidate the input, and apply the current budget gate. A DLQ redrive token must never bypass those checks.

Configure the queue without outsourcing the policy

Use the managed queue's DLQ feature for delivery isolation, but keep the media-job classifier in application code.

For Amazon SQS, configure a source queue redrive policy, set `maxReceiveCount` from the recovery-time budget, restrict which source queues may use the DLQ, and keep the DLQ retention longer than the source queue where the queue type's timestamp behavior requires it. The AWS documentation also supports moving messages out through a controlled DLQ redrive operation.

For Google Cloud Pub/Sub, attach the dead-letter topic to the subscription, create a subscription for the dead-letter topic, and grant the documented service-account permissions. Pub/Sub wraps forwarded messages with source-subscription attributes, and its delivery-attempt threshold is best effort. Use the application's durable request record as the authoritative audit trail rather than assuming the broker count is exact.

Pub/Sub's retry policy applies exponential backoff per message and continues delivering other messages unless ordered delivery changes that behavior. Pub/Sub exactly-once delivery can confirm acknowledgments for supported pull subscriptions within its documented regional scope, but it does not merge multiple unique publishes into one business intent. Keep the application request ID even when the broker offers stronger acknowledgment semantics.

Monitor recovery, not just queue depth

Alert on the first newly quarantined message, then group by safe reason and age. Useful measures include:

• New DLQ items by reason and endpoint.
• Oldest quarantined item and time to operator decision.
• Percentage resumed with the same task ID.
• Percentage requiring credential, balance, permission, or input repair.
• Number of unknown submissions reconciled without a second generation.
• Number of explicitly approved new intents after terminal provider failure.

Do not put prompts, signed URLs, output URLs, or authorization values into metric labels. Use the existing structured logging workflow for safe correlation, and test the whole release boundary with the Node.js API contract gate before changing classifier behavior.

Review DLQ retention as part of privacy and incident-response policy. Queue retention is not an archive, and an old message can outlive the source record it references. Delete or tombstone reviewed items according to a documented retention schedule after the operator decision and audit record are durable.

When not to use Sonilo

Do not use Sonilo when your application only needs a generic queue exercise. A local fake provider and synthetic task records are enough to test acknowledgment, delivery attempts, quarantine, redrive permissions, and secret exclusion.

Do not use a media-generation API as the queue itself. Keep application delivery, provider task state, and durable asset state in separate records so each can be recovered without guessing.

Do not automatically regenerate media after a provider failure when customer approval, cost, or creative continuity matters. Repair the blocking condition, review the original intent, and require a new request identity and budget decision.

Do not use a DLQ to conceal an unbounded retry loop. Bound ordinary reads first, instrument their reasons, and treat a rising DLQ as a production incident that needs diagnosis.

Make the recovery path safer than the retry path

The production rule is simple: quarantine enough context to understand one failed intent, but never enough authority to repeat it automatically. Resume a known task read. Repair a known blocker. Reconcile an unknown submission. Create a new generation only as an explicitly reviewed new intent.

Read the current Sonilo API reference before adopting the example. If your platform needs a reviewed failure-recovery design, capacity plan, or commercial integration, talk to Sonilo.