AudioSep
A foundation model for open-domain sound separation that pulls any sound out of a mixture from a plain-text description.
About
AudioSep reframes source separation as a language query problem: instead of training one model per instrument, it trains a single model that separates whatever you describe in text. A CLAP or CLIP text encoder turns your query into an embedding, and a 30-layer ResUNet separation network uses it to isolate the matching sound — a trumpet solo, a dog bark, a voice in noise — from the rest of the mix.\n\nThe model was trained on large-scale multimodal audio-text data (the authors report 1M training steps on 8 Tesla V100s), and its selling point is zero-shot generalization: it separates sound classes it never explicitly saw, with the MUSIC dataset showing 10.5 dB SDR improvement in the authors' own benchmark. It processes audio at 32 kHz mono, which is fine for analysis but below full music-production fidelity.\n\nIt is genuinely open: MIT-licensed code, released weights, a Hugging Face demo space, and a Replicate endpoint. If your use case is 'isolate this specific thing from a recording' rather than clean vocal/stem splits, this is the most capable free option — just remember the quality ceiling is author-reported and production-grade music separation usually wants a dedicated music model.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
- Author-reported (official repo README): SDR improvement VGGSound 9.14, MUSIC 10.51, ESC-50 10.04, AudioSet 7.74, AudioCaps 8.22, Clotho 6.85 dB\n- Author-reported (v3 paper): 8.16 dB on DCASE 2024 Task 9; 9.21 dB SSNR on Voicebank-Demand\n- No independent replication published yet — all headline numbers are author-reported
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 8/10
Scored 8/10 (TIER verified, metrics-v1). Evidence: peer-reviewed paper (published IEEE/ACM TASLP 2024, arXiv 2308.05037) from a named university team, MIT-licensed code and weights, 1,900+ GitHub stars, and a public Hugging Face demo. Strengths: genuinely novel query-based paradigm, fully reproducible, broad zero-shot coverage. Docked for: (1) all separation numbers are author-reported with no independent replication; (2) 32 kHz mono output limits music-production use. License: MIT.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 7.0/10
- Openness · license + docs
- 10.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · Peer-reviewed paper published in IEEE/ACM TASLP 2024 (arXiv 2308.05037, Aug 2023, v3 revised Dec 2024) from Xubo Liu et al. (University of Surrey / NTU / ByteDance): CLAP/CLIP text encoder plus 30-layer ResUNet separator trained 1M steps on 8 Tesla V100s.
- · MIT-licensed code and weights, 1,900+ GitHub stars, public Hugging Face demo space and Replicate endpoint.
- · Author-reported SDRi: VGGSound 9.14, MUSIC 10.51, ESC-50 10.04, AudioSet 7.74, AudioCaps 8.22, Clotho 6.85; 32 kHz mono output limits music-production use. No independent replication published.