Meta's AudioCraft Framework: A Builder's Guide to MusicGen, AudioGen, and EnCodec

Developers researching AI-generated audio almost always encounter Meta's AudioCraft first. The name is broad, the GitHub star count is high, and the demos are compelling. But the framework packages three models that are purpose-built for completely different tasks, and picking the wrong one is the single most common integration mistake we see at Sonilo. This guide breaks down what each model actually does, what it cannot do, and the licensing and infrastructure realities that matter before you write a single line of production code.

What AudioCraft Actually Is

AudioCraft is a single open-source Python codebase released by Meta AI in August 2023. It does not run on Meta's servers on your behalf. There is no hosted endpoint, no managed API, and no usage dashboard. To use it, you clone the GitHub repository, set up a compatible Python environment, provision GPU compute, and run inference yourself.

The framework bundles three separate generative audio models:

• MusicGen - text-to-music generation
• AudioGen - text-to-sound-effect generation
• EnCodec - neural audio compression and reconstruction

Meta's stated goal for AudioCraft was to give researchers and developers a shared foundation for audio-generation experimentation. It was explicitly designed as a research codebase, not as a competitor to production API providers. That distinction has real consequences for teams with shipping deadlines.

Because AudioCraft is self-hosted, every production deployment decision sits with your team: hardware provisioning, model loading latency, inference batching, scaling under load, monitoring, and uptime. None of that is abstracted away the way it would be with a managed audio API.

MusicGen: Text-to-Music

MusicGen is an autoregressive transformer model that generates music from a text description. You supply a prompt such as "upbeat electronic music with a driving bassline" and the model produces a short audio clip matching that description.

Model Sizes

MusicGen is available in three parameter sizes:

• 300M parameters
• 1.5B parameters
• 3.3B parameters

Larger parameter counts generally allow the model to handle more complex prompts and produce more coherent musical structure, though the relationship between size and output quality is task-dependent and may vary by prompt type and evaluation metric. Teams should benchmark each size against their specific use case rather than defaulting to the largest available.

Melody Conditioning

A melody-conditioned variant of MusicGen accepts a short audio reference clip alongside the text prompt. This allows the model to generate music that follows the melodic contour of the reference while still responding to the text description. It is not audio style transfer in the traditional sense; it is conditioning, meaning the reference guides but does not deterministically control the output.

Training Data

MusicGen was trained on music that Meta either owns or licensed specifically for this purpose. It was not trained on public-domain audio or scraped web content. This is an important provenance distinction, but it does not transfer cleared rights to the generated outputs. The output licensing question is governed separately by the weight license (discussed below), not by the training data provenance.

AudioGen: Text-to-Sound-Effects

AudioGen is a text-to-audio model built for environmental sounds and ambient audio, not music. Its intended use cases include foley-style production work: footsteps on gravel, rain on a rooftop, crowd murmur, engine noise, or a dog barking.

Key distinctions that matter for integration:

• AudioGen was trained on publicly sourced sound data, not on licensed music libraries.
• It does not share weights or core architecture decisions with MusicGen; the two models are not interchangeable.
• AudioGen does not generate scored music with melodic or harmonic structure. If you prompt it for a "cinematic orchestral piece," the results will not meet musical expectations.
• Like MusicGen, AudioGen is a research model packaged inside the AudioCraft codebase and requires the same self-hosting setup.

The practical decision rule is straightforward: if your workflow needs game audio assets, podcast sound beds, film foley, or ambient environmental texture, AudioGen is the appropriate model. If it needs composed music, use MusicGen.

EnCodec: Neural Audio Codec

EnCodec is not a generative model. It does not create new audio from prompts or descriptions. It is a high-fidelity neural audio compressor that encodes audio into a compact representation and reconstructs it with fewer perceptual artifacts than earlier codec approaches.

EnCodec serves two roles inside the AudioCraft framework:

1. Internal audio tokenizer. MusicGen and AudioGen use EnCodec internally to convert raw audio waveforms into discrete tokens that the autoregressive models can process and generate. This is an implementation detail, not a user-facing interface.
2. Standalone compression tool. Developers can use EnCodec independently for audio compression tasks entirely separate from any generative application. If your project involves storing or transmitting audio at reduced bitrates with higher fidelity than traditional codecs, EnCodec is worth evaluating on its own merits.

The key point for builders: EnCodec cannot replace MusicGen or AudioGen, and it does not belong in any generative pipeline as a substitute for them. Its role is compression and reconstruction, not generation.

Licensing and Production Realities

This is where many teams discover that AudioCraft requires more legal review than they anticipated.

The Two-License Structure

AudioCraft operates under two separate licenses that apply to different parts of the project:

• The codebase (training scripts, inference code, utilities) is released under the MIT license. MIT permits broad use, including commercial applications of the code itself.
• The model weights for both MusicGen and AudioGen are released under Creative Commons Attribution-NonCommercial 4.0 International (CC-BY-NC 4.0). Under CC-BY-NC 4.0, the licensed material cannot be used for commercial purposes.

This distinction matters enormously. The MIT license on the code does not extend commercial rights to the weights. A team that builds a commercial product using AudioCraft's MusicGen or AudioGen weights is operating outside the terms of the weight license, regardless of how the surrounding codebase is licensed.

What This Means for Product Teams

If your team is building a product that generates music or sound effects for paying customers, internal commercial workflows, advertising, or any revenue-associated use case, you need a licensing arrangement that is not covered by the default CC-BY-NC weight release. Teams in this situation should:

• Consult legal counsel before deploying AudioCraft weights in any commercial context.
• Evaluate third-party APIs and platforms built on separately licensed music-generation stacks, which may offer commercial terms that AudioCraft's default weight license does not.
• Not assume that because MusicGen was trained on licensed music, the outputs inherit cleared commercial rights. Output licensing under CC-BY-NC is a separate question from training data provenance.

Infrastructure Overhead

Beyond licensing, production use of AudioCraft carries meaningful infrastructure overhead. MusicGen at the 3.3B parameter size requires substantial GPU memory. Inference latency for longer clips is significant. There is no built-in queue management, load balancing, or auto-scaling. Teams operating at scale or with latency-sensitive requirements typically find that a wrapper service or managed API layer is more practical than raw self-hosting.

At Sonilo, we help teams evaluate when self-hosting AudioCraft is the right fit for a project and when a managed API is the more pragmatic path to shipping.

Where AudioCraft Fits in a Broader AI Audio Workflow

AudioCraft is a strong starting point for research and prototyping. If your team wants to understand how text-conditioned audio generation works before committing to a production API budget, running AudioCraft locally is a legitimate way to build that intuition.

Here is where it fits well:

• Academic and applied research into generative audio architectures
• Internal proof-of-concept work before selecting a production vendor
• Fine-tuning experiments where you need direct access to model weights
• Non-commercial projects where the CC-BY-NC license is not a constraint

Here is where it does not fit well without additional tooling:

• Video-synchronized music generation. AudioCraft does not natively support video conditioning. It cannot analyze a video clip and generate music that matches scene changes, pacing, or visual mood. Adding that capability requires a separate model or API layer on top of AudioCraft's outputs, or a different tool entirely.
• High-availability production services. Without significant infrastructure investment, AudioCraft is not suitable for serving end users at scale with reliability guarantees.
• Commercial products. As described above, the CC-BY-NC weight license is a hard constraint for most commercial applications.

The audio AI landscape in 2026 includes several managed API options that address these gaps with licensed music stacks, video-conditioning support, and production-grade uptime. At Sonilo, we pair knowledge of foundational frameworks like AudioCraft with practical guidance on production-grade APIs so teams can move from prototype to shipped product without reinventing the infrastructure layer.

Frequently Asked Questions

Can I use MusicGen commercially?

Not under the default weight license. MusicGen's model weights are released under CC-BY-NC 4.0, which prohibits commercial use of the licensed material. The MIT license on AudioCraft's code does not override this restriction. Product teams intending commercial deployment should seek legal guidance and evaluate APIs or platforms with separately negotiated commercial licensing. There is no publicly documented standard commercial licensing arrangement available directly from Meta for AudioCraft weights.

What is the difference between MusicGen and AudioGen?

MusicGen generates music (melodic, harmonic, scored audio) from text prompts, with an optional melody-conditioning variant. It was trained on Meta-owned and specifically licensed music. AudioGen generates environmental sounds and sound effects (rain, footsteps, crowds, engines) from text prompts, and was trained on publicly sourced sound data. The two models do not share weights or architecture, and they are not substitutes for each other. Use MusicGen when the output needs to be music; use AudioGen when the output needs to be ambient or foley-style audio.

Does AudioCraft support video-to-music generation natively?

No. AudioCraft does not include native video-conditioning functionality. None of its three models accept video frames or video features as a conditioning input. Generating music that synchronizes with video content requires either a separate video-analysis and conditioning layer built on top of AudioCraft's outputs, or a different tool designed specifically for video-to-music generation. This is a meaningful capability gap for teams building scoring or synchronization workflows.

The Three-Model Decision Map

AudioCraft is one framework, but it contains three tools with non-overlapping jobs: use MusicGen when you need text-conditioned music, AudioGen when you need text-conditioned sound effects, and EnCodec when you need high-fidelity audio compression. The licensing reality is equally clear: the CC-BY-NC weight license makes commercial deployment a legal question that must be resolved before you ship, regardless of how compelling the demos look.

If you are ready to move beyond research models, explore Sonilo's guides on building production AI audio workflows. We cover how to evaluate managed audio APIs, what to look for in commercial licensing terms, and how to architect audio generation pipelines that can actually go live.

Sources

1. Meta AI. "AudioCraft." Meta AI Resources. https://ai.meta.com/resources/models-and-libraries/audiocraft/
2. Meta AI. "AudioCraft: A Simple One-Stop Shop for Audio Modeling" (August 2023). https://ai.meta.com/blog/audiocraft-musicgen-audiogen-encodec-generative-ai-audio/
3. facebookresearch/audiocraft. GitHub Repository. https://github.com/facebookresearch/audiocraft
4. Copet, J. et al. "Simple and Controllable Music Generation" (2023). arXiv:2306.05284. https://arxiv.org/pdf/2306.05284
5. Creative Commons. "Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)." https://creativecommons.org/licenses/by-nc/4.0/
6. facebookresearch/encodec. GitHub Repository (EnCodec). https://github.com/facebookresearch/encodec