API reference

One key, plain JSON, one fast endpoint. Turn any URL into clean Markdown, search the web, or pull typed JSON out of a page. Every call is authenticated with your API key and returns in one round trip.

Base URL https://api.crawl4ai.com  ·  Auth header Authorization: Bearer sk_live_...  ·  get a key

OSSfrom the library to the cloud

You already know crawl4ai. The cloud runs the same engine behind one key: no browser to install, no proxies to buy, no cookies to babysit. Each library call has one API call.

# one page → Markdown
async with AsyncWebCrawler() as crawler:
    result = await crawler.arun("https://example.com")
    print(result.markdown)

# many pages
results = await crawler.arun_many(urls, config=run_conf)

# typed JSON out of a page
config = CrawlerRunConfig(extraction_strategy=LLMExtractionStrategy(schema=..., instruction=...))
result = await crawler.arun(url, config=config)

# one page → Markdown                      (POST /scrape)
curl https://api.crawl4ai.com/scrape -H "Authorization: Bearer sk_live_..."   -H "Content-Type: application/json" -d '{"url": "https://example.com", "format": "md"}'

# many pages, results stream as they land   (POST /scrape/batch · /scrape/jobs for thousands)
curl -N https://api.crawl4ai.com/scrape/batch -H "Authorization: Bearer sk_live_..."   -H "Content-Type: application/json" -d '{"urls": ["https://a.com", "https://b.com"], "format": "md"}'

# typed JSON out of a page                  (POST /extract)
curl https://api.crawl4ai.com/extract -H "Authorization: Bearer sk_live_..."   -H "Content-Type: application/json" -d '{"url": "https://example.com", "instruction": "the product name and price"}'

# web search, which the library does not have (GET /search)
curl "https://api.crawl4ai.com/search?q=web+scraping" -H "Authorization: Bearer sk_live_..."

import requests
H = {"Authorization": "Bearer sk_live_..."}
BASE = "https://api.crawl4ai.com"

# one page → Markdown
md = requests.post(BASE + "/scrape", json={"url": "https://example.com", "format": "md"}, headers=H).json()["markdown"]

# typed JSON out of a page
data = requests.post(BASE + "/extract", json={"url": url, "instruction": "the product name and price"}, headers=H).json()

# web search
hits = requests.get(BASE + "/search", params={"q": "web scraping"}, headers=H).json()["results"]

POST/scrape

Fetch a page and return clean Markdown (and/or HTML). Handles JS-heavy pages and bot walls automatically — you don't pick an engine.

curl https://api.crawl4ai.com/scrape \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "format": "both",
    "proxy": "residential",
    "country": "us",
    "parse": { "links": true, "media": true, "metadata": true, "tables": true }
  }'

Body

How long it takes

Most pages come back in 1 to 2 seconds. A page that needs a real browser usually takes 10 to 60 seconds, and the slowest up to 90. Set your client's timeout to 120 seconds and wait for the answer. A call you cancel early is not charged, but you lose the page. A 403 with "blocked" means every step got the site's block page; a 502 means the site did not answer; a 503 with "fleet-busy" means no browser was free for a moment, so send the same request again in a few seconds. None of these is charged. For many URLs, use /scrape/batch (up to 50, each result streams back when its page is ready) or /scrape/jobs (up to 10,000, runs in the background while you poll, so no connection waits on a slow page).

GET/search

Web search, browser-free, results ranked and cleaned. Add rich=1 to also get the on-page extras — follow-up questions, related queries, an entity card and more (see below). For a direct answer, use /answer.

curl "https://api.crawl4ai.com/search?q=rust+web+crawler" \
  -H "Authorization: Bearer sk_live_..."

Query

Rich response rich=1

Every key below is optional — each appears only when that block is on the page. For a direct answer to a question, use /answer.

{
  "results": [ … ranked links, same as always … ],
  "rich": {
    "follow_up_questions": [ "Why is the sky blue at sunset?" ],
    "related_queries":    [ "why is the ocean blue" ],
    "entity":  { "title": "Apple Inc", "subtitle": "NASDAQ: AAPL" },
    "videos": [ { "title": "…", "url": "https://…" } ],
    "news":   [ { "title": "…", "url": "https://…" } ],
    "discussions":  [ … ],
    "did_you_mean": "corrected spelling"
  }
}

GET/answer experimental

Ask a question, get a direct answer. Some questions won't have one yet — then answered is false (use /search for links). Add deep=0 for a direct answer only when one is readily available (no page reading). Experimental: the shape may change as we improve it.

curl "https://api.crawl4ai.com/answer?q=why+is+the+sky+blue" \
  -H "Authorization: Bearer sk_live_..."

{
  "answered": true,
  "answer": {
    "kind": "generated",
    "text": "The sky is blue because Earth's atmosphere scatters sunlight…",
    "sources": [ { "title": "NASA", "url": "https://…" } ]
  },
  "experimental": true
}

POST/extract

Pull structured, typed data out of a page with an instruction and/or a JSON schema. Give a URL (we fetch it) or your own content.

curl https://api.crawl4ai.com/extract \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://news.ycombinator.com",
    "instruction": "the top stories on the front page",
    "schema": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "title":  { "type": "string"  },
          "points": { "type": "integer" },
          "url":    { "type": "string"  }
        },
        "required": ["title", "points"]
      }
    },
    "example": [
      { "title": "Show HN: My project", "points": 128, "url": "https://..." }
    ]
  }'

Body

POST/scrape/batch

Scrape many URLs in one call — up to 50 — and stream a result per line as each finishes.

# -N streams each result line as it lands
curl -N https://api.crawl4ai.com/scrape/batch \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.com", "https://b.com"], "format": "md"}'

import json, requests

with requests.post(
    "https://api.crawl4ai.com/scrape/batch",
    headers={"Authorization": "Bearer sk_live_..."},
    json={"urls": ["https://a.com", "https://b.com"], "format": "md"},
    stream=True,
) as r:
    for line in r.iter_lines():
        if line:
            row = json.loads(line)
            print(row["url"], "ok" if row["ok"] else row["error"])

const res = await fetch("https://api.crawl4ai.com/scrape/batch", {
  method: "POST",
  headers: { "Authorization": "Bearer sk_live_...", "Content-Type": "application/json" },
  body: JSON.stringify({ urls: ["https://a.com", "https://b.com"], format: "md" }),
});

// read the NDJSON stream line by line
const reader = res.body.getReader(), dec = new TextDecoder();
let buf = "";
for (;;) {
  const { value, done } = await reader.read();
  if (done) break;
  buf += dec.decode(value, { stream: true });
  let nl;
  while ((nl = buf.indexOf("\n")) >= 0) {
    const line = buf.slice(0, nl); buf = buf.slice(nl + 1);
    if (line.trim()) {
      const row = JSON.parse(line);
      console.log(row.url, row.ok ? "ok" : row.error);
    }
  }
}

Body

POST/scrape/jobs

For big lists — up to 10,000 URLs — or when your code cannot hold a connection open for slow pages. Submit once, get a job id, then poll for results while it drains in the background.

# 1. submit -> job id
JOB=$(curl -s https://api.crawl4ai.com/scrape/jobs \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{"urls": ["https://a.com", "https://b.com"]}' | jq -r .job_id)

# 2. poll until done
until [ "$(curl -s https://api.crawl4ai.com/scrape/jobs/$JOB \
       -H "Authorization: Bearer sk_live_..." | jq -r .status)" = "done" ]; \
  do sleep 2; done

# 3. fetch results (NDJSON, one line per URL)
curl -s https://api.crawl4ai.com/scrape/jobs/$JOB/results \
  -H "Authorization: Bearer sk_live_..."

import time, requests

H = {"Authorization": "Bearer sk_live_..."}
base = "https://api.crawl4ai.com"

# 1. submit
job = requests.post(f"{base}/scrape/jobs", headers=H,
                    json={"urls": ["https://a.com", "https://b.com"]}).json()["job_id"]

# 2. poll until done
while requests.get(f"{base}/scrape/jobs/{job}", headers=H).json()["status"] != "done":
    time.sleep(2)

# 3. page through the results
after = 0
while True:
    lines = requests.get(f"{base}/scrape/jobs/{job}/results?after={after}",
                         headers=H).text.splitlines()
    if not lines: break
    for line in lines: print(json.loads(line)["url"])
    after += len(lines)

const H = { "Authorization": "Bearer sk_live_...", "Content-Type": "application/json" };
const base = "https://api.crawl4ai.com";

// 1. submit
const { job_id } = await (await fetch(`${base}/scrape/jobs`, {
  method: "POST", headers: H,
  body: JSON.stringify({ urls: ["https://a.com", "https://b.com"] }),
})).json();

// 2. poll until done
let status;
do {
  await new Promise(r => setTimeout(r, 2000));
  ({ status } = await (await fetch(`${base}/scrape/jobs/${job_id}`, { headers: H })).json());
} while (status !== "done");

// 3. fetch results (NDJSON)
const out = await (await fetch(`${base}/scrape/jobs/${job_id}/results`, { headers: H })).text();
console.log(out);

Endpoints

POST/recipes/{name}

Ready-made scrapers. Pick a recipe from the catalog, send its inputs, get rows back. Browse and try them at /recipes/.

curl https://api.crawl4ai.com/recipes/hn-hiring \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "thread": 49522897, "keyword": "Rust" }'

Body

Response

{
  "recipe": "hn-hiring", "version": 1,
  "rows": [ { "company": "Senzing", "author": "samlk", "text": "Senzing | Platform Engineer | Remote (USA) ..." } ],
  "usage": {
    "units": 1, "boxes": ["crawl"], "cache": "miss", "engine": "recipe",
    "lines": [ { "call": "scrape", "url": "https://news.ycombinator.com/item?id=49522897", "units": 1, "cache": "archive", "engine": "archive" } ]
  },
  "meta": { "pages_fetched": 1, "from_cache": false, "elapsed_ms": 812, "llm_calls": 0, "warnings": [] }
}

The result cache

Every recipe declares fresh_for_s. A second call with the same inputs inside that window is served from the cache: usage.cache is "hit", usage.units is 0, meta.from_cache is true. Send "bypass_cache": true to run again. Recipes with a secret input are never cached.

Your session

Some recipes read a site as you: they take a session input, your own login cookies for that site as one Cookie header line. The Crawl4AI Session extension copies it in one click. Crawl4AI uses it for this run only and does not store it: no archive, no cache, no log. Your account and the site's terms are your responsibility. The catalog says per input whether it is a secret and whether the session is required or optional; without an optional session the recipe reads what a logged-out visitor sees. A wrong or expired session costs the run's unit: rows is empty, meta.warnings carries bad_session, and meta.blocked names the site.

Staged recipes and health

Every recipe in the catalog carries a stage: published, or staged for a new recipe that runs but is not yet promoted. Each region checks every recipe every six hours and the catalog carries the result as health (ok, fail or unknown, with the last check's rows, time and error). GET /recipes/health returns the same, without a key.

Errors

The catalog GET /recipes

curl https://api.crawl4ai.com/recipes -H "Authorization: Bearer sk_live_..."

Free (no unit). Returns {"recipes": [...]}: for each recipe its name, version, title, stage, health, kind (json, module or script), inputs (type, default, required), output fields, fresh_for_s, tags, a cost hint, run (sync or job) and a description.

POST/monitors

Watch a page and get told when it changes in the way you care about. Give the URL and a plain sentence; every real change that matches comes back as typed facts (what changed, from what, to what) with links to both page versions. Try it in the dashboard's Playground, mode watch.

curl https://api.crawl4ai.com/monitors \
  -H "Authorization: Bearer sk_live_..." \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://competitor.com/pricing",
        "watch": "a plan gets more expensive, or a plan is removed",
        "every": "1h",
        "notify": { "webhook": "https://you.example/hook", "secret": "s3cr3t" } }'

Body

Response

{
  "id": "mon_7f3a20c1e9b04d55", "created": true,
  "url": "https://competitor.com/pricing",
  "watch": "a plan gets more expensive, or a plan is removed",
  "conditions": { "any": [ { "id": "c1", "text": "the price of a plan goes up" },
                           { "id": "c2", "text": "a plan is removed" } ] },
  "every": "1h", "region": "eu", "health": "ok",
  "baseline": { "status": "pending" },
  "cost": { "per_change_credits": 3 }
}

The other calls

A change

{
  "id": 4181, "event": "change", "monitor": "mon_7f3a20c1e9b04d55",
  "url": "https://competitor.com/pricing",
  "matched": ["c1"],
  "summary": "The Pro plan's price went from $49 / month to $59 / month.",
  "facts": [ { "kind": "changed", "entity": "Pro", "field": "price",
               "before": "$49 / month", "after": "$59 / month", "section": "Pricing > Pro" } ],
  "custom": null,
  "first_seen": "2026-09-26T12:05:32Z", "confirmed_at": "2026-09-26T12:35:41Z",
  "evidence": { "before": "https://api.crawl4ai.com/monitors/mon_7f3a…/versions/75af60efc867a0cb",
                "after":  "https://api.crawl4ai.com/monitors/mon_7f3a…/versions/e70482929f228688" },
  "cost_credits": 3,
  "delivery": { "webhook": { "status": 200, "attempts": 1, "at": "2026-09-26T12:35:42Z" }, "done": true }
}

The webhook

One POST per confirmed change, the event above as the body, with the headers x-c4-event: change, x-c4-monitor, x-c4-timestamp (unix seconds) and, when you gave a secret, x-c4-signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>. Answer 2xx. Otherwise we try again after 1, 5, 30 minutes, 2 and 12 hours, then stop; the timeline keeps the last status.

Credits

1 credit per check, once per every (a page checked more often than that inside the window costs no more). 3 credits per confirmed change that matched. A blocked or failed check, a change that matched nothing, and a paused monitor cost nothing. Up to 100 monitors per account.

Errors

MCP/mcp

Use Crawl4AI as native tools inside Claude, Cursor, or any MCP client — no install, just a URL and your key.

# one line in your terminal
claude mcp add --transport http crawl4ai \
  https://api.crawl4ai.com/mcp \
  --header "Authorization: Bearer sk_live_..."

# ~/.codex/config.toml
[mcp_servers.crawl4ai]
url = "https://api.crawl4ai.com/mcp"
http_headers = { Authorization = "Bearer sk_live_..." }

# opencode.json
{ "mcp": { "crawl4ai": {
    "type": "remote",
    "url": "https://api.crawl4ai.com/mcp",
    "headers": { "Authorization": "Bearer sk_live_..." } } } }

# ~/.cursor/mcp.json  (Windsurf: ~/.codeium/windsurf/mcp_config.json)
{ "mcpServers": { "crawl4ai": {
    "url": "https://api.crawl4ai.com/mcp",
    "headers": { "Authorization": "Bearer sk_live_..." } } } }

# ~/.openclaw/openclaw.json, the mcp.servers block
{ mcp: { servers: { crawl4ai: {
    url: "https://api.crawl4ai.com/mcp",
    transport: "streamable-http",
    headers: { Authorization: "Bearer sk_live_..." } } } } }

# Claude Desktop → Settings → Connectors → Add custom connector
Name:  crawl4ai
URL:   https://api.crawl4ai.com/mcp
# then paste your key when it asks for the Authorization header:
Bearer sk_live_...

OPSregions & data

Three regions, one address. Your request goes to the nearest region by DNS; every region runs the whole stack.

CREDITSpricing

One unit, the credit. Every request costs credits, and its receipt says why. You pay for the effort a page needed, never more: a result already in our archive costs half (we keep it fresh), a plain page is the base price, a page that fought back costs more. Money enters once, at a top-up. The numbers below are read live from GET /v1/prices.

The table

The effort levels

The effort a page needed sets its multiplier. The receipt names the level; the engine that did the work is ours.

Free credit and the packs

The signup credit and bought credit never expire; the 7-day pass expires with its key. A paid top-up turns on the Supporter limits (10 at a time, 120 a minute, 10 keys, no monthly cap). Your own monthly spend cap and auto-recharge are on the dashboard's Billing tab.

On every response

x-c4-cost: 1.500        # what this call cost, in credits
x-c4-balance: 14312.000  # your credit after it

Out of credit, the gate answers 402 with the reason, a message you can show as it is, and every way on: {"error":"no_credit","balance":-0.50,"message":"…","actions":{"topup":"","topup_api":"POST …/v1/billing/topup?amount=<a pack> … with a saved card the pack is charged at once, otherwise the answer carries checkout_url to open","mcp":"call the topup tool"}}. Over your own monthly cap: {"error":"spend_cap","cap_left":…,"actions":{"raise":…,"raise_api":"POST /v1/billing/settings {\"spend_cap_mc\":…}","mcp":"call the spend_cap tool"}}. Your key can call both billing routes, so an agent can act on a 402 by itself. A request the gate refused itself (a bad URL, a rate limit) costs nothing; a 5xx costs nothing.

Estimate before a call POST /v1/estimate

The same body as the call, plus the endpoint. The answer is a range: the fixed part is exact, the level and the tokens are a band from real traffic.

curl -X POST "https://api.crawl4ai.com/v1/estimate" \
  -H "Authorization: Bearer sk_live_..." -H "Content-Type: application/json" \
  -d '{"endpoint":"scrape","url":"https://news.ycombinator.com"}'
# → {"min":"1.00","max":"3.00","text":"about 1.00 to 3.00 credits","parts":[…]}

Balance and top-up from a key

curl "https://api.crawl4ai.com/v1/billing/balance" -H "Authorization: Bearer sk_live_..."
curl -X POST "https://api.crawl4ai.com/v1/billing/topup?amount=25&checkout=1" -H "Authorization: Bearer sk_live_..."
# → {"checkout_url": "https://checkout.stripe.com/…", "credits": …, "usd": 25}  open it, pay, the credits land in seconds

Without checkout=1, an account with a card saved from an earlier top-up is charged at once and there is no page to open: the answer is {"paid": true, "usd": 25, "credits": …, "invoice_url": "…", "balance": {…}}. If that card does not pay, the answer is 402: send the request again with checkout=1, or update the card on the dashboard's Billing tab. An account with no saved card always gets the checkout_url.