Skip to content
← Tools / Skills
ToolCrawl4AI (unclecode)/10 trustVerified
Crawl4AI logo

Crawl4AI

Open-source web crawler and scraper for LLMs and AI agents

Visit official site →

About

Crawl4AI turns any website into clean, LLM-ready Markdown for RAG, AI agents, and data pipelines. It is a Python library and CLI that renders pages with a real browser (Chromium/Firefox/WebKit), extracts structured data via CSS/XPath schemas or LLM-powered extraction, and supports deep crawling with BFS/DFS strategies and crash recovery. It can be run entirely locally, as a Docker server with a token-gated REST API and MCP endpoint, or via the hosted Crawl4AI Cloud API launched in 2026. The repository is one of the most-starred open-source web crawlers, with over 85,000 GitHub stars.

What it can do

  • • Clean LLM-ready Markdown output (raw + boilerplate-pruned variants, citation references)
  • • Structured extraction via CSS/XPath/Regex schemas without an LLM, or via LLM into typed JSON schemas
  • • Deep crawling: BFS, DFS, best-first with crash recovery (resume_state)
  • • Adaptive crawling and URL discovery (sitemaps, Common Crawl, 5-10x prefetch)
  • • Browser control: sessions, proxies, cookies, stealth mode, persistent profiles, Chromium/Firefox/WebKit
  • • Screenshots, PDFs, media/link extraction, JS execution hooks, caching, parallel crawls
  • • CLI (`crwl`), Docker server with token-gated REST API + MCP + dashboard/playground
  • • Hosted Cloud API: /scrape, /extract, /search, /serp, /answer endpoints

Setup

  1. Install: pip install -U crawl4ai.
  2. Run the one-time browser setup: crawl4ai-setup (falls back to python -m playwright install --with-deps chromium if it fails).
  3. Verify: crawl4ai-doctor.
  4. Crawl from Python: async with AsyncWebCrawler() as crawler: result = await crawler.arun(url="https://example.com") — Markdown is on result.markdown.
  5. Or use the CLI: crwl https://example.com -o markdown (deep crawl: crwl URL --deep-crawl bfs --max-pages 10).
  6. (Optional) Docker server: export CRAWL4AI_API_TOKEN=<token> then docker run -d -p 11235:11235 -e CRAWL4AI_API_TOKEN=$CRAWL4AI_API_TOKEN --shm-size=1g unclecode/crawl4ai:latest — every endpoint requires the token.
  7. LLM extraction only: set your provider key (e.g. OPENAI_API_KEY) and use LLMExtractionStrategy with a typed schema.

Prerequisites

Python 3.10+; ~1GB disk for the Playwright Chromium browser installed by crawl4ai-setup; Docker for the server deployment; an LLM API key only when using LLM-based extraction.

Use cases

searchdataautomation

Works with

Claude CodeCodexCursorGeneric APIAny agent (.md)

Trust score — 9/10

/10 trustVerified

Scored 9/10 (verified tier, metrics-v1). Evidence: 85,382 GitHub stars (log-scaled vs catalog peers); blended with ~1,275,996 monthly package downloads; last push 0d ago; 14 releases/12mo; 227 open issues. Manual review: ToS gray zone on stealth/anti-bot features; cloud pricing soft-launched. License: Apache-2.0.

★ 85,381 GitHub stars

How the trust score is built

Full methodology →
Adoption · stars/downloads, log-scaled vs peers
9.0/10
Maintenance · recency, releases, issue load
9.9/10
Openness · license + docs
10.0/10
Manual review · sanity read + red-flag check
9.0/10
  • · 85,382 GitHub stars (log-scaled vs catalog peers)
  • · blended with ~1,275,996 monthly package downloads
  • · last push 0d ago; 14 releases/12mo; 227 open issues
  • · manual review: ToS gray zone on stealth/anti-bot features; cloud pricing soft-launched

Sources

Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.