Fugatto

Fugatto, short for Foundational Generative Audio Transformer Opus 1, is NVIDIA's 2.5-billion-parameter generalist for sound. It accepts free-form text instructions, optionally paired with audio, and handles text-to-speech, text-to-audio and audio transformation in a single model: generate a music snippet from a prompt, remove an instrument from a song, or shift the accent and emotion of a voice. Its ComposableART inference technique can combine, interpolate or negate instructions, producing emergent outputs the model was never explicitly trained for, like a saxophone that barks.

The catch is that Fugatto remains a research artifact. NVIDIA announced it in November 2024 with a paper and an impressive demo site, but has released no weights, code or API since, so none of its claims can be independently tested. It belongs in the catalog as a landmark reference model, but anyone wanting to actually use this kind of capability will need to look at the open alternatives.

• Paper (arXiv 2411.04329): reports competitive performance vs specialized models plus emergent compositional abilities (author-reported)
• No public weights, so no independent verification is possible