Skip to Content

mcp server

the official Model Context Protocol  server — @crawlbrulee/mcp. lets mcp-aware agents (Claude Code, Cursor, Codex, Claude Desktop) scrape pages, map sites, and check usage as native tool calls. it wraps @crawlbrulee/sdk, speaks stdio, and is npx-runnable.

install

Claude Code:

claude mcp add crawlbrulee \
  --env CRAWLBRULEE_API_KEY=cwbl_... \
  -- npx -y @crawlbrulee/mcp

Cursor — add to ~/.cursor/mcp.json:

{
  "mcpServers": {
    "crawlbrulee": {
      "command": "npx",
      "args": ["-y", "@crawlbrulee/mcp"],
      "env": { "CRAWLBRULEE_API_KEY": "cwbl_..." }
    }
  }
}

the same pattern works for Codex, Claude Desktop, and any stdio mcp host: set command: npx, args: ["-y", "@crawlbrulee/mcp"], and forward CRAWLBRULEE_API_KEY in the env block.

configuration

env varrequireddescription
CRAWLBRULEE_API_KEYyesapi key sent as Authorization: Bearer ….

the key is read on the first tool call, so a typo surfaces as a clear tool error rather than a startup failure.

tools

tooldescription
scrapefetch a url → markdown, html, links, images, screenshot, metadata.
scrape_asyncsubmit a scrape job asynchronously; returns a job_id immediately.
scrape_statuscheck an async job’s status (pending / running / done / failed).
scrape_resultfetch a completed async job’s result (same shape as scrape).
mapenumerate a site’s urls (sitemap + in-page discovery).
usagecurrent credits, quota, concurrency, reset time. takes no arguments.
whoamiorganization + token identity behind the configured key.

example scrape input — only url is required:

{
  "url": "https://example.com",
  "extract": { "markdown": true, "links": true },
  "require_js": false,
  "proxy": "basic"
}

async workflow

use scrape_async instead of scrape for long-running pages (heavy js rendering, full-page screenshots). it takes the same input as scrape, plus an optional webhook: { url, metadata }. poll scrape_status with the returned job_id until the status is done, then call scrape_result to fetch the page. if a webhook was attached on submit, we deliver a single signed scrape.complete POST to that url when the job finishes (with metadata echoed back under data.metadata), so an agent can skip polling entirely.

{ "job_id": "job_abc123" }

usage object

scrape, scrape_result, and map responses (and scrape_status once a job is done) carry a top-level response_meta.usage block, so an agent can report what was delivered and what the call cost. every operation includes { credits, engine, proxy }; scrape operations also include screenshot_slices. map does not, and narrows engine to http | cache.

tool failures return an mcp error result: [<errorName>] <message>, with (HTTP <status>) appended only when an http status applies. if CRAWLBRULEE_API_KEY is unset on the host, calls fail with a synthetic missing_api_key error (rather than a startup failure) — agents can branch on the error code either way.

cli