How to Gracefully Shut Down an AI Music API Worker

On SIGTERM, stop claiming new generation jobs first. Then persist every accepted task ID, let active submissions reach a known state within a fixed deadline, and hand unfinished polling to the next worker. Never treat process shutdown as proof that an interrupted POST was rejected.

On SIGTERM, stop claiming new generation jobs first. Then persist every accepted task ID, let active submissions reach a known state within a fixed deadline, and hand unfinished polling to the next worker. Never treat process shutdown as proof that an interrupted POST was rejected.

By Sonilo Team · Facts verified August 19, 2026

Disclosure: Sonilo publishes this guide and operates the API used as the concrete example. The `graceful-generation-worker-v1` controller was tested with synthetic job states, an event emitter, and a deterministic clock. It did not send generation requests, run inside Kubernetes, measure deployment downtime, or prove queue and database atomicity.

Drain in four stages

Graceful shutdown is a state transition, not a delay before `process.exit()`. A worker needs to know whether each claimed job is still local, currently being submitted, or already accepted by the remote API.

Use this order:

1. Close intake so the worker claims no new jobs.
2. Let each active submission reach a usable response or the application drain deadline.
3. Persist every returned `task_id` before acknowledging the queue job.
4. At the deadline, route unresolved work by its last durable phase and exit.

Node.js emits a signal event when the process receives `SIGTERM` or `SIGINT`. Its Node.js v22 process documentation also warns that installing a listener removes the default exit behavior for those signals on non-Windows platforms. The handler must therefore finish the drain and set an exit path; receiving the signal alone no longer ends the process.

Kubernetes follows a bounded termination flow. The current Pod lifecycle documentation says the default grace period is 30 seconds, the container runtime sends TERM to process 1, and remaining processes receive SIGKILL when that period expires. Your worker's internal deadline must finish before the platform deadline.

Prefer async task IDs for drainable workers

The current Sonilo API reference documents both streaming music responses and asynchronous tasks. Music endpoints can return an NDJSON stream, or a `task_id` when called in async mode; other asynchronous generation endpoints return `202` with a `task_id`.

For a background worker that must survive rolling deploys, a durable task ID creates a cleaner handoff boundary. Once the worker has persisted the accepted task ID, another process can continue the task polling workflow without resubmitting the generation request.

The current Retrieve Task reference lists `processing`, `succeeded`, `failed`, or `canceled`. The current OpenAPI contract also treats `completed` and `canceled` as terminal values, but exposes no client task-cancel operation. Stopping local polling therefore does not itself cancel remote work; shutdown should transfer responsibility for an accepted task and preserve any later terminal result.

Streaming remains useful when one process intentionally owns the connection through completion. If you choose that shape, the tested NDJSON parser needs its own terminal-event, partial-file, and restart policy. Do not claim that a broken local stream proves no remote work occurred.

Persist the phase before changing it

Store the job state in a durable record or queue lease. An in-memory map is useful for tests and local coordination, but it cannot be the only recovery source in a multi-process deployment.

The difficult row is `submitting`. A terminated worker cannot know whether a remote service accepted a POST just because the local connection ended. Use the separate duplicate-job prevention contract to freeze or reconcile that request. Graceful shutdown does not create provider idempotency.

Persist the phase change before its side effect where possible. Mark a request `submitting` before the network call. Store `accepted` and the returned task ID in one durable transaction before acknowledging the queue item. A process crash can occur between any two instructions; write the record so the next worker can choose a conservative action.

Keep shutdown separate from capacity and dependency health

A drain controller answers whether this process should accept work. It does not decide whether the account has capacity or whether the provider is healthy.

The generation admission queue protects live requests-per-minute and concurrent-generation budgets. It should stop assigning work to a draining worker while continuing to route eligible work elsewhere.

The circuit-breaker guide protects new submissions during repeated dependency failures. A planned deploy should not count as a provider failure, and an open circuit should not stop status reads for already accepted task IDs.

Keep these controls independent:

• Admission decides whether account capacity permits a new submission.
• The circuit breaker decides whether dependency health permits a new submission.
• The drain controller decides whether this worker may claim new work.
• The request record decides whether this exact job may be submitted or replayed.
• The polling record decides which accepted task ID needs another status read.

That separation keeps a routine deploy from looking like an outage and prevents a provider incident from corrupting process-lifecycle decisions.

Use a small drain controller

`graceful-generation-worker-v1` tracks only the local facts needed for shutdown. Save it in a server-only module. The production queue and database remain the durable authority.

1. `export function createDrainController({graceMs = 30_000} = {}) {`
2. ` let accepting = true;`
3. ` let drain = null;`
4. ` const jobs = new Map();`
5. ``
6. ` const claim = (jobId) => {`
7. ` if (!accepting) return {ok: false, reason: "draining"};`
8. ` if (jobs.has(jobId)) return {ok: false, reason: "already_in_flight"};`
9. ` jobs.set(jobId, {jobId, phase: "claimed", taskId: null});`
10. ` return {ok: true};`
11. ` };`
12. ``
13. ` const beginSubmission = (jobId) => {`
14. ` const job = jobs.get(jobId);`
15. ` if (!job || job.phase !== "claimed") throw new Error("illegal phase");`
16. ` job.phase = "submitting";`
17. ` };`
18. ``
19. ` const recordAcceptedTask = (jobId, taskId) => {`
20. ` const job = jobs.get(jobId);`
21. ` if (!job || job.phase !== "submitting" || !taskId) throw new Error("illegal phase");`
22. ` job.phase = "accepted";`
23. ` job.taskId = taskId;`
24. ` };`
25. ``
26. ` const beginDrain = ({signal = "SIGTERM", nowMs = Date.now()} = {}) => {`
27. ` if (!drain) {`
28. ` accepting = false;`
29. ` drain = {signal, startedAtMs: nowMs, deadlineMs: nowMs + graceMs};`
30. ` }`
31. ` return snapshot();`
32. ` };`
33. ``
34. ` const evaluate = (nowMs = Date.now()) => {`
35. ` if (!drain) return {readyToExit: false, reason: "not_draining", actions: []};`
36. ` if (jobs.size === 0) return {readyToExit: true, reason: "drained", actions: []};`
37. ` if (nowMs < drain.deadlineMs) return {readyToExit: false, reason: "waiting", actions: []};`
38. ` return {readyToExit: true, reason: "deadline", actions: [...jobs.values()].map(disposition)};`
39. ` };`
40. ``
41. ` return {claim, beginSubmission, recordAcceptedTask, beginDrain, evaluate};`
42. `}`

The complete tested module also validates identifiers and times, exposes snapshots, removes finished jobs, maps deadline dispositions, installs signal handlers, and makes repeated signals idempotent.

The first signal owns the deadline. A second SIGTERM or SIGINT must not restart the grace window. Extending it on every signal can keep a broken worker alive until the platform sends SIGKILL.

Wire signals without exiting inside the handler

Register SIGTERM and SIGINT once during worker startup. The handler should close intake, update readiness, and start one drain promise.

1. `const controller = createDrainController({graceMs: 25_000});`
2. ``
3. `installDrainSignals({`
4. ` controller,`
5. ` onSignal: async (state, signal) => {`
6. ` await queue.pauseLocalClaims();`
7. ` await readiness.markNotReady();`
8. ` const deadline = state.drain.deadlineMs;`
9. ` const result = await drainUntil(deadline);`
10. ` await persistDeadlineActions(result.actions);`
11. ` process.exitCode = result.reason === "drained" ? 0 : 1;`
12. ` },`
13. `});`

Do not call `process.exit()` immediately after pausing the queue. Node's `exit` event cannot perform asynchronous cleanup; the process exit documentation says only synchronous operations can run there. Let the event loop finish after the drain promise and set `process.exitCode` instead.

Do not use `beforeExit` as the shutdown signal. Node documents that `beforeExit` is not emitted for explicit termination conditions such as `process.exit()` or uncaught exceptions. Handle the operating-system signals directly.

Budget the platform deadline explicitly

Choose the application deadline from measured shutdown work, then make the platform grace period longer. A useful configuration rule is:

`terminationGracePeriodSeconds > preStop time + application drain time + safety margin`

The current Kubernetes container lifecycle-hook documentation says the termination grace countdown begins before a `PreStop` hook runs. A long hook therefore consumes time that the application might otherwise use to drain. Keep the hook light, or include its worst-case duration in the budget.

The 25-second controller value above is only an example under Kubernetes' documented 30-second default. It is not a universal recommendation. Measure queue pause latency, database writes, current submission response times, and sidecar shutdown behavior for your deployment.

If generation can legitimately take longer than the process grace period, do not make the worker wait for remote completion. Persist the task ID and hand polling to another worker. The platform deadline should bound local handoff, not the full media-generation duration.

Route every deadline state deliberately

The current Sonilo error guide separates request, authentication, balance, access, media, capacity, and upstream-processing failures. Shutdown should preserve that definite response when one exists; it should not overwrite the failure with a generic worker-terminated label.

After success, use the temporary-output durability workflow to copy and verify the generated file. The drain is complete when ownership of that work is durable, not when the remote media itself has necessarily finished.

Protect source access and credentials during handoff

A replacement worker needs the same trusted server boundary, but it should not receive secrets through queue payloads or logs. Keep the API key behind the server-side credential boundary, and log only safe correlation fields through the structured logging contract.

Useful drain fields include `worker_id`, `signal`, `drain_started_at`, `deadline_at`, `job_id`, `phase`, `task_id`, `handoff_action`, and `exit_reason`. Exclude bearer tokens, prompts, source URLs, temporary output URLs, raw request bodies, and uploaded filenames by default.

If the queued job points to a remote video, verify that its access window still covers the next legitimate submission attempt. The presigned source URL gate owns that check. Do not refresh an input URL for an ambiguous job and submit again until the request record permits replay.

Run the video preflight workflow before the job enters `submitting`. That keeps invalid media out of the ambiguous window and gives redelivered, unsubmitted work a clear compatibility record.

The twelve-test contract

The controller ran with the official Node.js v22 test runner and no external packages. Twelve deterministic tests passed on Node.js v22.23.2 on August 19, 2026:

1. Claim work while the worker accepts jobs.
2. Reject a duplicate in-flight claim.
3. Stop new claims immediately when drain begins.
4. Keep the first deadline after a second signal.
5. Exit cleanly after every local job finishes.
6. Wait for unresolved jobs before the deadline.
7. Release claimed but unsubmitted work at the deadline.
8. Freeze a submission interrupted before a usable response.
9. Hand an accepted job to polling by durable task ID.
10. Reject an accepted transition without a task ID.
11. Install SIGTERM and SIGINT handlers and invoke the drain callback once.
12. Reject invalid grace periods and illegal phase transitions.

These tests prove local phase rules, deadline decisions, and signal-listener behavior. They do not prove that a queue lease and database transaction are atomic, that Kubernetes delivered a signal, that a provider accepted or rejected a real POST, or that another worker resumed polling.

The controller's in-memory map is not a distributed lock. A production worker needs queue-specific lease semantics and a datastore transaction around request phase, task ID, and acknowledgement. Test process crashes before and after each durable write, not only planned SIGTERM handling.

Test the deploy, not only the module

Run a controlled rolling deployment with synthetic queue jobs that cannot create customer cost. Observe at least these cases:

• Signal arrives before a claim.
• Signal arrives after a claim but before submission.
• Signal arrives while the POST is awaiting a response.
• Signal arrives after `202` but before task ID persistence.
• Signal arrives after task ID persistence but before queue acknowledgement.
• Signal arrives while a successor is polling the accepted task.
• A second signal arrives during drain.
• The platform deadline expires and sends SIGKILL.

Confirm that each job has one explainable durable state afterward. Count duplicate submissions, orphaned task IDs, redelivery delay, drain duration, forced exits, and ambiguous outcomes separately. A zero in a synthetic deploy is evidence about that test, not a guarantee for production traffic.

Use the machine-readable Sonilo OpenAPI document and the full agent reference to recheck current request and response shapes when the integration changes.

When not to wait for graceful completion

Do not keep a compromised or corrupted process serving work merely to finish a drain. Node's guidance on uncaught exceptions treats the process as being in an undefined state and recommends synchronous cleanup followed by restart. Planned SIGTERM handling and crash recovery are different paths.

Do not wait for every remote generation to finish when a durable task ID can transfer ownership. A long wait consumes the platform grace period without improving recoverability.

Do not add process-local drain state if the queue already supplies a tested worker shutdown primitive that stops intake and awaits active handlers. Use the queue's contract, then add only the generation-specific phase and task-ID rules it lacks.

Do not claim graceful shutdown makes submissions exactly once. It narrows uncertainty by recording phases. The ambiguous interval still exists unless the provider and application share a recoverable idempotency contract.

Make deploys a handoff, not a replay

Close intake, persist the request phase, save accepted task IDs before acknowledgement, and leave enough platform grace for the local handoff. At the deadline, release work that never started, freeze work whose submission result is unknown, and resume accepted work by task ID.

Start with the broader Video-to-Music API guide, then verify the current async fields in the Video-to-Music endpoint reference. The safe deployment rule is short: stop claiming, preserve what is known, and never turn a shutdown into an unreviewed replay.