M2UGen
Multi-modal LLM that answers questions about music and generates it from text, images, video, audio, plus music editing.
About
M2UGen (Multi-modal Music Understanding and Generation) wires a LLaMA-2 backbone to several encoders: MERT for music understanding, ViT for images, ViViT for video, with MusicGen and AudioLDM2 acting as the music decoders. That stack lets it do music question answering and generate music conditioned on text, images, video, or audio, along with basic music editing, all inside one model.
The November 2023 paper is light on standard benchmarks, evaluated mostly through demos and task examples, and a successor (MuMu-LLaMA) has since been published by the same group. Code is on GitHub and a public Hugging Face demo space exists, so it is easy to try even though it is firmly a research artifact rather than a product. It is a useful reference for where multimodal music LLMs are headed.
Specs
Context window
Pricing
Free quota
API access
Weights
Released
Benchmarks
No standard published benchmarks found; evaluated via demos and task examples in the paper
Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.
Use cases
Trust score — 5/10
Scored 5/10 (metrics tier, metrics-v1). Evidence: paper (arXiv 2311.11255, Nov 2023) and demo page at crypto-code.github.io/M2UGen-Demo/; public HF demo space confirms real artifact; official repo appears to be a fork-heavy mirror rather than one canonical repo. Strengths: ambitious multimodal coverage (text/image/video/audio-to-music + QA + editing), working public demo. Docked for: (1) no standard benchmarks to verify quality claims; (2) weight/checkpoint license not confirmed in sources checked; (3) largely superseded by the authors' own MuMu-LLaMA. License: not explicitly confirmed — flag for review.
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 5.0/10
- Maintenance · recency, releases, issue load
- 4.0/10
- Openness · license + docs
- 7.0/10
- Manual review · sanity read + red-flag check
- 5.0/10
- · Paper arXiv 2311.11255 (Nov 2023)
- · LLaMA-2 + MERT/ViT/ViViT encoders, MusicGen/AudioLDM2 decoders
- · Public HF demo space (M2UGen/M2UGen-Demo); largely superseded by MuMu-LLaMA