# 🍮 crawlbrulee: agent onboarding you're an AI agent or an automated client. this page is the human landing page with the fluff removed: what crawlbrulee is, whether it fits your task, how to authenticate, and how to make your first call. read it top to bottom and act on it. ## what crawlbrulee is a web-scraping api. you send a url, you get back clean structured data in one json response: markdown, cleaned html, raw html, the page's links and images, a screenshot, and page metadata. you can also enumerate the urls on a site (a "link map"). hosted in the EU, built for AI pipelines and agents. - base url: `https://api.crawlbrulee.com` - auth: every request carries `Authorization: Bearer $CRAWLBRULEE_API_KEY` - every response tells you what it charged, so you can track spend per call ## does it fit your task? **reach for crawlbrulee when you need to:** - turn a page into markdown for llm or rag ingestion - pull a page's links, images, or `` metadata as structured json - screenshot a page: viewport, full-page, desktop, or mobile - find out which urls exist on a site before fetching them - render JavaScript content or use the enhanced proxy tier for a higher retrieval success rate **it won't help you with:** - **searching the web.** crawlbrulee fetches urls you already have; it doesn't find pages by query. - **crawling in one call.** there's no crawl endpoint. compose map → scrape and cap the list yourself. - **llm extraction.** you get markdown, html, and metadata, or named values read by css selector (`extract.elements`: ); run your own model on top. - **interactive browsing.** no clicking, form filling, or multi-step flows. `require_js` renders JavaScript and screenshots take a few scripted scroll/wait actions, but there's no session. if you need a page's content as clean text, this is the right tool. if you need to decide *which* page, find it first, then bring the url here. ## step 1: your user gets a key a person has to do this part, not you. you guide them through it, and you never see the key. 1. **get a key.** if your user has no crawlbrulee account, they sign up at . the free plan needs no credit card. then, in the dashboard menu, they open **tokens** and click **create token**. keys start with `cwbl_`. 2. **put it in the environment.** they add `export CRAWLBRULEE_API_KEY="cwbl_..."` to their shell profile (for example `~/.zshrc`), then start you again from a new terminal. never ask them to paste the key into this chat, and never print it or read it back: anything in the chat stays in its history. 3. **check it.** call `whoami` (the mcp tool, `crawlbrulee whoami`, or `GET /api/whoami`). don't print the variable to check it. the cli, the sdks and curl read `CRAWLBRULEE_API_KEY` on their own. add the mcp server with no key in its config, before or after the export: `claude mcp add crawlbrulee -- npx -y @crawlbrulee/mcp`. the server reads the variable each time it starts. where the variable doesn't reach the server on its own: - **Codex** passes it only when it's listed. in `~/.codex/config.toml`, add `env_vars = ["CRAWLBRULEE_API_KEY"]` under `[mcp_servers.crawlbrulee]`. - **Gemini CLI** passes it only when it's listed. in `~/.gemini/settings.json` (or `.gemini/settings.json` in a project), give the `crawlbrulee` entry under `mcpServers` the line `"env": { "CRAWLBRULEE_API_KEY": "$CRAWLBRULEE_API_KEY" }`. - **Cursor and Windsurf:** name it to be sure. in `~/.cursor/mcp.json` (Cursor) or `mcp_config.json` (Windsurf, opened from its mcp settings), give the `crawlbrulee` entry under `mcpServers` the line `"env": { "CRAWLBRULEE_API_KEY": "${env:CRAWLBRULEE_API_KEY}" }`. - **apps opened from the dock or the start menu**, like Claude Desktop, may not read the shell profile. there, your user writes the key into the app's config themselves. for Claude Desktop that's `~/Library/Application Support/Claude/claude_desktop_config.json` on macOS, `%APPDATA%\Claude\claude_desktop_config.json` on Windows. the full entry for each client: . your user can also skip the shell profile. in Claude Code, the plugin asks for the key in a masked prompt: `/plugin marketplace add crawlbrulee/crawlbrulee-skills`, then `/plugin install crawlbrulee@crawlbrulee`. or they write the key into the client's config themselves, in their own terminal or editor. not with Claude Code's `!` prefix: that adds the command and its output to the chat. **if the key is missing**, your user needs to get one (point 1). **if the api says `invalid_credentials`**, the key is expired, revoked or mistyped: your user creates a new one under **tokens** and replaces the old one themselves. if they use the shell profile, they start you again afterwards. **every plan has the same api surface.** the free tier isn't a crippled demo: async jobs, webhooks, screenshots, and both proxy tiers all work on it. paid plans buy you more credits and more concurrency; what else they add is commercial, not api: top-up packs, a usage dashboard, team management, and priority support. current allowances and prices: . ## step 2: pick an interface | you are… | use | setup | | --- | --- | --- | | an agent that speaks mcp | **mcp server** | register `npx -y @crawlbrulee/mcp` as a stdio server, with no key in its config: it reads `CRAWLBRULEE_API_KEY` from step 1. per-client steps: | | running shell commands or cron | **cli** | `npx crawlbrulee scrape url https://example.com` | | writing Node.js / TypeScript | **js/ts sdk** | `npm install @crawlbrulee/sdk` | | writing Python | **python sdk** | `pip install crawlbrulee` | | in any other language | **raw http**, or generate a client from the spec | see below | if you're a coding agent, install the skills bundle. it teaches you every interface and the api contract behind them: ```bash npx skills add crawlbrulee/crawlbrulee-skills ``` ## step 3: make your first call **curl:** ```bash curl -X POST "https://api.crawlbrulee.com/api/scrape" \ -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "extract": { "markdown": true, "links": true } }' ``` **js/ts sdk:** ```ts import { Crawlbrulee } from '@crawlbrulee/sdk' const crawlbrulee = Crawlbrulee.fromEnv() // reads CRAWLBRULEE_API_KEY const page = await crawlbrulee.scrape({ url: 'https://example.com', extract: { markdown: true, links: true }, }) console.log(page.markdown) ``` **python sdk:** ```python from crawlbrulee import Crawlbrulee, ScrapeExtract client = Crawlbrulee.from_env() # reads CRAWLBRULEE_API_KEY page = client.scrape(url="https://example.com", extract=ScrapeExtract(markdown=True, links=True)) print(page.markdown) ``` > a bare scrape returns `metadata` + `cleaned_html`. ask for `markdown`, `links`, `images`, `raw_html`, or `screenshot` explicitly. the cli is the exception: it defaults to markdown + metadata. ## the operations you'll use - **scrape**: `POST /api/scrape` (wait for it) or `POST /api/scrape/async` (background job you poll, or attach a webhook to). returns whatever you named in `extract`. - **map**: `POST /api/map`. enumerate a site's urls before scraping selected pages. - **account**: `GET /api/usage` for remaining credits and your concurrency cap; `GET /api/whoami` to confirm which token you're on. every interface exposes all of them: sdk `scrape`/`map`, cli `scrape url`/`map`, mcp tools `scrape`/`map`. **results don't wait forever.** screenshot links are signed and expire 24 hours after the scrape, and screenshots aren't publicly accessible, so download the image if you need it later; keep the file, not the link. an async job's status and result answer for 24 hours after you submit it, then return `404 not_found`; fetching the result again doesn't restart the clock. ## what a call costs every successful scrape and map carries a `response_meta.usage` block with the price and its parts: ```jsonc "response_meta": { "usage": { "total_credit_cost": 1, "engine_credit_cost": 1, "proxy_multiplier": 1, "zero_data_retention_credit_cost": 0, "engine": "http", "proxy": "basic" } } ``` `total_credit_cost` is the credits charged, and it always equals `engine_credit_cost` × `proxy_multiplier` + `screenshot_slicing_credit_cost` + `zero_data_retention_credit_cost`. `engine` and `proxy` are what delivered the result. scrape, terminal async-scrape, and successful webhook responses also include `screenshot_slicing_credit_cost`: `1` when slices were generated for the request, otherwise `0`, regardless of tile count. map omits that field and uses only `engine: "http"` or `engine: "cache"`. the older `credits` and `screenshot_slices` fields still come back with the same values, but they are deprecated and will be removed in a future version. **read usage rather than predicting it**, then check `GET /api/usage` before a large job. the formula is engine base (http 1, browser 3, screenshot 5) × proxy multiplier (basic ×1, advanced ×5), plus a flat 1 when screenshot slices are generated, plus a flat 1 for zero data retention. `engine: "cache"` has a 0-credit base, and failures cost 0; the current tables live at . **check `page_status_code` before you trust the content.** a scrape result carries the status the site answered with. a `404` or `503` page comes back as a `200` from us, with the page's content: it's data, not an error, so "page not found" text can look like real content if you skip this check. we bill `2xx` and `4xx` pages, except `403`, `407`, `408`, `429` and `451`; `5xx` pages and errors are never billed. if we could not reach the site (for example it timed out, or its certificate was bad), you get `502 target_unreachable`. a site we could not reach can also come back as `500 internal_server_error`, so check the url before you report a 500. details: . **zero data retention.** send `"zero_data_retention": true` on a scrape or map to keep the result out of the shared cache (must be enabled for your organization, +1 credit): . ## generate your own client the api is described by an OpenAPI 3.0 document. generate a typed client in any language: ```bash npx openapi-typescript https://crawlbrulee.com/docs/openapi.json -o crawlbrulee.d.ts # or any language: npx @openapitools/openapi-generator-cli generate \ -i https://crawlbrulee.com/docs/openapi.json -g python -o ./crawlbrulee-client ``` ## reference - docs: - openapi spec: - api reference: - pricing: - agent skills: