Skip to content
← Models
Open weightsApple/10 trustMetrics-checked

Ferret-UI

First UI-centric multimodal model for referring, grounding, and reasoning on screens.

Visit official page →

About

Ferret-UI is Apple's UI-focused multimodal model, released October 8, 2024 out of the ml-ferret research program with Columbia University. Where most vision models look at photos of the world, Ferret-UI is trained on mobile screens: it can refer to, ground, and reason about UI elements like buttons, menus, and checkout fields, which is the missing piece for on-device AI agents that need to click through apps.

A second generation, Ferret-UI 2, followed in late 2024 with stronger mobile-agent capabilities. Both releases are research-only under Apple's research terms, so they cannot be used commercially. They remain influential reference points for the whole GUI-agent wave, including models like UI-TARS and OS-Atlas.

Specs

Context window

Not disclosed

Pricing

Free; self-hosted, research use only, no official hosted API

Free quota

None (self-hosted)

API access

No official API

Weights

Open weights

Released

Not disclosed

Benchmarks

Authors report strong performance on UI referring, grounding, and reasoning tasks for mobile screens (paper claims); Ferret-UI 2 generation reported to outperform GPT-4o-class baselines on UI referencing (secondary coverage, Dec 2024); no single headline metric extracted here

Provider-reported unless a source says otherwise. Benchmarks are marketing-adjacent by nature — treat them as directional.

Use cases

mobile UI understandingGUI agent researchUI element groundingscreen referring

Trust score — 6/10

/10 trustMetrics-checked

Scored 6/10 (metrics, metrics-v1). Evidence: official apple-aiml-research/ml-ferret repo with dated release notes (2024-10-08), paper lineage from Ferret (ICLR 2024 spotlight). Strengths: pioneering UI-centric MLLM, strong academic pedigree, influential on GUI-agent research. Docked for: (1) research-only license, no commercial use; (2) no hosted access and modest direct maintenance. License: research use only (Apple; weight deltas under CC-BY-NC).

How the trust score is built

Full methodology →
Adoption · stars/downloads, log-scaled vs peers
6.0/10
Maintenance · recency, releases, issue load
5.0/10
Openness · license + docs
6.0/10
Manual review · sanity read + red-flag check
7.0/10
  • · Apple + Columbia, released 2024-10-08; first UI-centric MLLM
  • · Ferret-UI 2 followed late 2024; influential on GUI-agent research

Sources