Firecrawl

Firecrawl is a web data API and retrieval platform for AI agents: scrape pages to markdown/HTML/structured JSON, crawl whole sites with limits, map URLs, run agent-optimized search, and use LLM extraction with schemas. The official SDKs are TypeScript (@mendable/firecrawl-js) and Python (firecrawl-py), and there is an official MCP server so agents can drive it as a tool.

The core is open source (AGPL-3.0) and self-hostable via Docker Compose, but self-hosting is heavyweight (API + PostgreSQL + Redis + RabbitMQ + Playwright) and ships with auth off for the first run. The managed cloud adds the advanced anti-bot Fire-engine, screenshots, page actions, and agent/browser endpoints that the self-hosted stack lacks.

1. Cloud (fastest): sign up at https://firecrawl.dev and create an API key (keys start with fc-).
2. Install an SDK: npm install @mendable/firecrawl-js or pip install firecrawl-py.
3. Scrape a page:

```js

import Firecrawl from '@mendable/firecrawl-js';

const app = new Firecrawl({ apiKey: 'fc-YOUR_API_KEY' });

const res = await app.scrape('https://example.com', { formats: ['markdown'] });

```

1. For crawl/extract: use app.crawl(url, { limit: 100 }) and app.extract({ urls, prompt, schema }) — both verified in the official JS SDK README.
2. Self-host (full control): clone the repo, check out a pinned release (docs verified against v2.11.162), set POSTGRES_PASSWORD in .env, run docker compose up -d, then POST http://localhost:3002/v2/scrape.
3. Do not expose the self-hosted stack publicly without adding auth, network controls, and TLS — the docs' baseline has no authentication.