How to Add a Circuit Breaker to an AI Music API

Put a circuit breaker around new generation submissions, not around the whole job lifecycle. Count repeated dependency failures, open the circuit before more work reaches a failing service, allow one controlled half-open probe after a cooldown, and keep replay safety in a separate durable request record.

Put a circuit breaker around new generation submissions, not around the whole job lifecycle. Count repeated dependency failures, open the circuit before more work reaches a failing service, allow one controlled half-open probe after a cooldown, and keep replay safety in a separate durable request record. A failure can justify opening the circuit without making the failed POST safe to send again.

By Sonilo Team · Facts verified August 18, 2026

Disclosure: Sonilo publishes this guide and operates the API used as the concrete example. The `ai-music-circuit-breaker-v1` module was tested with synthetic outcomes and a deterministic clock. It did not send generation requests, measure uptime or latency, verify billing, or prove distributed coordination.

Use a breaker for dependency failure, not every error

A circuit breaker is a stateful guard between your application and a remote dependency. Microsoft describes the pattern as three states: closed calls flow normally, open calls fail fast, and half-open allows limited traffic to test recovery. The Microsoft Circuit Breaker pattern also distinguishes the breaker from retry: retry attempts a transient operation again, while the breaker stops operations that are likely to fail.

That distinction matters for a paid generation API. Repeated failures can consume worker time, network connections, queue capacity, and user patience. The AWS Circuit Breaker pattern explains how repeated calls during a persistent failure can amplify contention and how an open circuit returns immediately until a recovery probe is due.

Do not count every non-success as a dependency incident. Invalid input, authentication, account balance, access, file size, duration, and rate limits have different remedies. Opening one global circuit for those responses can block healthy traffic without helping the caller fix the actual problem.

The current Sonilo error guide documents those request, account, capacity, and upstream-processing classes. The classification above is an application policy built on that contract. It is not a statement that every future server error has the same cause.

Keep four controls separate

A breaker is only one control in a production generation path. Combining all failure handling into one counter creates unsafe shortcuts.

The generation admission queue protects account requests-per-minute and concurrent-generation capacity before a submission. It handles expected load, including a definite `429` response.

The duplicate-job prevention contract protects user intent across duplicate clicks, queue redelivery, and uncertain POST outcomes. It decides whether a particular request may be replayed.

The async polling workflow schedules status reads after the provider has returned a task ID. It does not decide whether new generations should start.

The circuit breaker aggregates recent dependency health. It protects the service and your application by pausing unrelated new submissions when failures cross a threshold.

Use all four only where the workload needs them. A low-volume internal tool may be better served by clear errors and bounded manual recovery. A customer-facing queue with many workers has more to gain from a shared breaker.

Scope the breaker to new submissions

The current Sonilo API reference defines two completion shapes. Music can return an NDJSON stream, while async endpoints return `202` with a `task_id` that the client reads later.

Do not wrap task-status GET requests and generation POST requests in one breaker. A failing generation path and a failing read path have different user impact and recovery evidence. One merged counter can stop useful reconciliation exactly when operators need it most.

Use a submission breaker for generation POSTs. Let already accepted jobs continue through their existing stream or polling lifecycle. The current Retrieve Task reference defines `processing`, `succeeded`, and `failed` task states for async work.

Keep polling known task IDs when that read path remains healthy, even if the submission circuit is open. A circuit breaker should stop new exposure; it should not erase accepted work or pretend that in-flight tasks were cancelled.

Current Sonilo documentation says the API does not send callbacks and has no dedicated task-cancel endpoint. Stopping local polling therefore does not cancel server work. Recheck that interface fact against the current API reference before adopting it as long-lived product behavior.

Classify outcomes before changing state

`ai-music-circuit-breaker-v1` uses a small classifier before its state machine. The classifier keeps health evidence separate from job replay semantics.

Save the policy in a server-only module:

1. `export function classifyGenerationOutcome({status, transportError = false, streamError = false} = {}) {`
2. ` if (transportError) return {`
3. ` kind: "dependency_failure",`
4. ` countsBreaker: true,`
5. ` ambiguousSubmission: true,`
6. ` };`
7. ` if (streamError) return {`
8. ` kind: "dependency_failure",`
9. ` countsBreaker: true,`
10. ` ambiguousSubmission: false,`
11. ` };`
12. ` if (status >= 200 && status < 300) return {`
13. ` kind: "success", countsBreaker: false, ambiguousSubmission: false,`
14. ` };`
15. ` if ([500, 502, 503, 504].includes(status)) return {`
16. ` kind: "dependency_failure", countsBreaker: true, ambiguousSubmission: false,`
17. ` };`
18. ` if (status === 429) return {`
19. ` kind: "capacity_signal", countsBreaker: false, ambiguousSubmission: false,`
20. ` };`
21. ` if ([400, 401, 402, 403, 413, 422].includes(status)) return {`
22. ` kind: "request_or_account_failure", countsBreaker: false, ambiguousSubmission: false,`
23. ` };`
24. ` return {kind: "unknown_failure", countsBreaker: false, ambiguousSubmission: false};`
25. `}`

Only `502` is part of the current documented Sonilo error table. The other listed `5xx` values are conservative application handling for common server-failure classes, not a claim that the current API emits them.

An unknown status stays neutral until the integration owner classifies it from current documentation and observed behavior. Silently treating every unknown response as transient can create an open circuit during a request bug. Treating every unknown response as healthy can hide a real incident. Neutral plus an alert is the safer default for an unreviewed code.

Open after consecutive dependency failures

Choose a threshold and cooldown from observed traffic, not from a copied example. The sample defaults to three consecutive dependency failures and a 30-second cooldown only to make the state transitions concrete.

The core transition rules are:

1. In `closed`, allow submissions and count only classified dependency failures.
2. Reset the consecutive count after a successful submission.
3. Move to `open` when the count reaches the configured threshold.
4. In `open`, reject new submissions locally until the cooldown boundary.
5. After the cooldown, move to `half_open` and allow one controlled probe.
6. Close after a successful probe.
7. Reopen after a failed probe.
8. Reopen after a neutral probe result because dependency health remains unknown.

One subtle rule prevents a retry storm: the first failure opens the circuit at a fixed timestamp, and later rejected local calls do not push that timestamp forward. AWS recommends that concurrent callers not extend the expiration window indefinitely.

The sample exposes `retryAtMs` for an open circuit. That value schedules a local eligibility check; it is not permission to replay the same failed job. The worker must re-read both the circuit and the request record when it wakes.

Make the half-open probe deliberate

A half-open probe is real traffic. For a generation API, it may also be billable work. Do not create synthetic customer media, hide probe cost, or send the same timed-out POST again just to test recovery.

Use one already-authorized queued generation whose request record still permits a first submission. Mark it as the half-open probe before the network call. If multiple workers share the circuit, acquire that probe token atomically.

The probe closes the circuit only after a usable successful submission response. For an async endpoint, a valid `202` and task ID are enough to show that submission acceptance recovered; the job still needs normal polling. For a streaming endpoint, decide whether your breaker measures initial response acceptance or full stream health, and name that scope in metrics.

This article's sample treats a stream-level error as dependency evidence. That is intentionally conservative. If the API exposes stable error codes that separate customer content from service failure, refine the classifier instead of counting all stream errors forever.

A half-open `429` does not prove that the dependency is still failing. It also does not prove recovery. The sample reopens for another cooldown and sends no second concurrent probe. The admission queue should separately reduce capacity pressure.

Do not turn a timeout into an automatic retry

Node.js v22 provides `AbortSignal.timeout()` for bounding an operation. A local timeout can protect worker resources, but it cannot tell you whether a remote POST was accepted before the response was lost.

Record a transport timeout twice:

• As dependency evidence for the breaker.
• As an ambiguous outcome for the exact application request.

The breaker may open because of that evidence. The request record must still freeze automatic replay until reconciliation or a deliberate new user intent. The AWS retry guidance notes that retry safety depends on idempotency; a generation POST without a provider recovery key deserves stricter handling.

Do not attach one short timeout to the entire NDJSON response and call it a submission timeout. A music stream can remain active while valid output is generated. Separate connection or first-response deadlines from the full generation lifecycle, and let the NDJSON stream parser enforce terminal events and file integrity.

Route each failure to the right recovery path

Run the video preflight workflow before the breaker so invalid media never becomes false outage evidence. Keep the bearer key behind the Next.js server boundary, and use the safe logging contract for circuit events.

After success, the breaker has no role in asset durability. Copy and verify generated media through the temporary-output handoff.

The twelve-test contract

The module ran with the official Node.js v22 test runner and no external packages. Twelve deterministic tests passed on August 18, 2026:

1. Start closed and allow a normal submission.
2. Open exactly at the consecutive dependency-failure threshold.
3. Fail fast while open and report the cooldown boundary.
4. Allow only one half-open probe.
5. Close and reset after a successful probe.
6. Reopen after a failed probe.
7. Reopen after a neutral probe because health remains unknown.
8. Count the documented `502` class as dependency failure.
9. Keep `429` capacity signals out of the failure count.
10. Keep request and account failures out of the failure count.
11. Mark transport failure as both breaker evidence and an ambiguous submission.
12. Reject an invalid threshold or cooldown before serving traffic.

These tests prove local state transitions and classification only. They do not prove a provider outage, service-level objective, billing behavior, output quality, or exactly-once execution. They do not exercise real network timing.

The module is in memory. A multi-process deployment needs shared, atomic state for the open timestamp and half-open probe token. Microsoft calls out concurrency, resource differentiation, observability, and manual override as design considerations. Test those properties against the datastore and deployment model you actually run.

Observe the breaker without logging customer data

Emit one structured event for every transition rather than logging every blocked payload. Useful fields include `breaker_name`, `previous_state`, `next_state`, `outcome_kind`, `http_status`, `opened_at`, `retry_at`, `request_id`, and `provider_task_id` when one exists.

Do not log bearer values, prompts, uploaded filenames, source URLs, temporary output URLs, or raw response bodies by default. A circuit dashboard needs health signals, not customer media details.

Alert on sustained open time, repeated half-open failures, and a rising rate of ambiguous submissions. Keep `429` rate separate from dependency-failure rate. Otherwise a busy healthy account can look like a provider outage.

Track the breaker by operation. A submission breaker for video-to-music should not automatically block task retrieval, account-limit reads, or another independently operated endpoint. Merge signals only after production evidence shows that they share a failure domain.

When not to use a circuit breaker

Do not add this pattern when normal bounded error handling is enough. A breaker adds shared state, recovery policy, operator controls, and another way to delay customer work.

Do not use it to hide invalid inputs or expired credentials. Do not use it as a substitute for account admission. Do not use it to cancel accepted tasks. Do not use it as permission to retry ambiguous generation POSTs.

Avoid an in-memory breaker when many workers must make one consistent decision. Either implement a shared atomic contract or keep the feature off until you can test distributed recovery.

If the workload is message-driven and already has bounded retries, delayed delivery, and a dead-letter path, its queue may provide enough isolation. Microsoft notes that circuit breakers can be unnecessary where infrastructure or event systems already manage failure recovery.

Ship the failure contract before the incident

Define which outcomes count, persist request identity, test the three states, and decide who can force the circuit open or closed. During an incident, leave accepted tasks alone, keep reconciliation reads available, and stop only the new submissions protected by that breaker.

Start with the broader Sonilo Video-to-Music API guide, then verify the current endpoint and error shapes in the Sonilo OpenAPI document. The operational rule is compact: open on repeated dependency evidence, probe once, and never confuse service recovery with replay safety.