API docs by Redocly

CrawlBrulee API (1.0.0)

Download OpenAPI specification:Download

Web scraping API for extracting structured data from websites.

Overview

crawlbrulee turns any public URL into clean, structured data — markdown, cleaned or raw HTML, links, images, screenshots, and page metadata — in a single JSON response. It can also map a whole site's URLs.

  • Base URL: https://api.crawlbrulee.com
  • Authentication: every request needs an Authorization: Bearer <CRAWLBRULEE_API_KEY> header.
  • Hosting: EU-based company and infrastructure.

Cache hits are free and failed scrapes are never billed. Prefer a typed client over raw HTTP? crawlbrulee ships official SDKs, a CLI, and an MCP server.

Quickstart

1. Get a free key. Create an account at crawlbrulee.com — 750 credits/month, no card — then create an API key in the dashboard.

2. Export it.

export CRAWLBRULEE_API_KEY="cwbl_..."

3. Scrape a page.

curl -X POST "https://api.crawlbrulee.com/api/scrape" \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{ "url": "https://example.com", "extract": { "markdown": true, "links": true } }'

A full walkthrough — sync, async, and site mapping — is in the quickstart guide.

Authentication

All endpoints require a Bearer token, scoped to your organization — usage, rate limits, and credits are tracked per-org. Create and manage tokens in the dashboard.

Authorization: Bearer YOUR_API_KEY

Store it as CRAWLBRULEE_API_KEY and never commit it to source control. Verify a token any time with GET /api/whoami. A 401 with invalid_credentials means the token is missing, malformed, expired, or revoked. See the authentication guide.

Error types

every error uses one consistent format — a stable name, a human-readable message, and an optional details object (present for usage_allocation_error and too_many_requests).

{ "name": "invalid_url", "message": "URL failed validation" }
HTTP Common name values
400 invalid_url, url_too_long, unsupported_url_schema, scrape_error
401 invalid_credentials
403 antibot_blocked
404 not_found, access_denied
408 request_timeout
415 unsupported_content
422 unsupported_screenshot_output (screenshot-only request on a content type that can't be screenshotted; never billed), too_many_redirects (the target redirected in a loop; scrape or map), page_too_large (the page's HTML was too large to process; scrape only — terminal, don't retry)
429 too_many_requests (includes retry_after_ms), usage_allocation_error (includes reason)
499 client_closed_request
500 internal_server_error
503 service_unavailable (transient on our side — retry the request unchanged)

Branch on name, not the HTTP status.

A 503 is not an auth problem: token failures that are genuinely about your key — unknown, expired, or revoked — always come back as 401 invalid_credentials. Retry a 503 with the same key rather than rotating it.

Full reference: errors.

Scraping

Scrape a URL synchronously and return the extracte

Scrape a URL synchronously and return the extracted content. The returned url comes back in normalized form. Known tracking parameters are removed. /map applies the same normalization to every link it returns.

Authorizations:
apiToken
Request Body schema: application/json
required
url
required
string

The URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key.

object
Default: {}

Which content formats to extract. Defaults to metadata + cleaned_html.

object

Cache settings for this request

require_js
boolean
Default: false

Use a headless browser to render JavaScript before scraping

object
Default: {}

What is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot where noted. Never applies to raw_html, which is always the page before we removed anything.

proxy
string
Default: "auto"
Enum: "basic" "advanced" "auto"

Proxy tier to use for fetching

object

Optional locale + country emulation for the scrape

Responses

Request samples

Content type
application/json
{
  • "url": "string",
  • "extract": { },
  • "cache": {
    },
  • "require_js": false,
  • "cleanup": { },
  • "proxy": "basic",
  • "location": {
    }
}

Response samples

Content type
application/json
{
  • "url": "string",
  • "requested_url": "string",
  • "content_type": "string",
  • "unsupported_fields": [
    ],
  • "markdown": "string",
  • "cleaned_html": "string",
  • "raw_html": "string",
  • "images": [
    ],
  • "links": [
    ],
  • "screenshot": {
    },
  • "metadata": {
    },
  • "response_meta": {
    },
  • "warnings": [
    ]
}

Build or return a cached site link map using sitem

Build or return a cached site link map using sitemap discovery plus homepage link discovery. By default the map holds up to 5000 URLs (max_urls, maximum 100000) and returns up to 5000 per page (limit, maximum 10000). Sitemap discovery stops as soon as max_urls URLs have been found, so raising it is a bigger crawl and lowering it is a faster one. When discovery stops early, response_meta.truncation.discovery_cap_reason names the limit that stopped it. Returned URLs come back in normalized form, the same normalization /scrape applies to its returned url. Known tracking parameters are removed. Links are ordered so the most useful ones land on page 1.

Authorizations:
apiToken
Request Body schema: application/json
required
url
required
string

The website URL to map. Mapping always targets the site root, so the path, query string and fragment are dropped; known tracking parameters are removed before the page is fetched.

proxy
string
Default: "auto"
Enum: "basic" "advanced" "auto"

Proxy tier to use for fetching

sitemap_only
boolean
Default: false

Only use sitemap.xml — skip homepage link extraction

object

Filter which link types to include

object

Cache settings for this request

max_urls
integer <= 100000
Default: 5000

Maximum number of URLs to discover and store in the map. Default 5000, maximum 100000. Sitemap discovery stops as soon as this many URLs have been found, so a smaller value is a faster and lighter crawl, not just a smaller answer.

page
integer <= 9007199254740991
Default: 1

Page number for paginated results

limit
integer <= 10000
Default: 5000

Number of URLs to return per page. Default 5000, maximum 10000.

object

Optional country emulation for the map

Responses

Request samples

Content type
application/json
{
  • "url": "string",
  • "proxy": "basic",
  • "sitemap_only": false,
  • "types": {
    },
  • "cache": {
    },
  • "max_urls": 5000,
  • "page": 1,
  • "limit": 5000,
  • "location": {
    }
}

Response samples

Content type
application/json
{
  • "links": [
    ],
  • "response_meta": {
    }
}

Submit an asynchronous scrape job

Submit an asynchronous scrape job

Authorizations:
apiToken
Request Body schema: application/json
required
url
required
string

The URL to scrape. Known tracking parameters are removed before the page is fetched, so they are neither sent to the target site nor part of the cache key. Every other query parameter is kept verbatim and is part of the cache key.

object
Default: {}

Which content formats to extract. Defaults to metadata + cleaned_html.

object

Cache settings for this request

require_js
boolean
Default: false

Use a headless browser to render JavaScript before scraping

object
Default: {}

What is removed from the page before any output is built. Applies to markdown, cleaned_html, links and images on every engine, and to the screenshot where noted. Never applies to raw_html, which is always the page before we removed anything.

proxy
string
Default: "auto"
Enum: "basic" "advanced" "auto"

Proxy tier to use for fetching

object

Optional locale + country emulation for the scrape

object

Optional completion webhook for this job. Configure your signing secret in the dashboard.

Responses

Request samples

Content type
application/json
{
  • "url": "string",
  • "extract": { },
  • "cache": {
    },
  • "require_js": false,
  • "cleanup": { },
  • "proxy": "basic",
  • "location": {
    },
  • "webhook": {}
}

Response samples

Content type
application/json
{
  • "job_id": "string"
}

Check the status of an async scrape job

Check the status of an async scrape job

Authorizations:
apiToken
path Parameters
job_id
required
string

The job ID returned when the async scrape was submitted

Responses

Response samples

Content type
application/json
{
  • "job_id": "string",
  • "status": "pending",
  • "created_at": "2019-08-24T14:15:22Z",
  • "error": "string",
  • "response_meta": {
    }
}

Retrieve the result of a completed async scrape jo

Retrieve the result of a completed async scrape job

Authorizations:
apiToken
path Parameters
job_id
required
string

The job ID returned when the async scrape was submitted

Responses

Response samples

Content type
application/json
{
  • "url": "string",
  • "requested_url": "string",
  • "content_type": "string",
  • "unsupported_fields": [
    ],
  • "markdown": "string",
  • "cleaned_html": "string",
  • "raw_html": "string",
  • "images": [
    ],
  • "links": [
    ],
  • "screenshot": {
    },
  • "metadata": {
    },
  • "response_meta": {
    },
  • "warnings": [
    ]
}

Account

Return the current billing cycle usage: total/used

Return the current billing cycle usage: total/used/available credits, used quota percentage, max allowed concurrency, and when the cycle resets.

Authorizations:
apiToken

Responses

Response samples

Content type
application/json
{
  • "total_credits": 9007199254740991,
  • "used_credits": 9007199254740991,
  • "available_credits": 9007199254740991,
  • "used_quota_percent": 0,
  • "max_concurrency": 9007199254740991,
  • "usage_reset": "2019-08-24T14:15:22Z"
}

Return the organization name and identifying detai

Return the organization name and identifying details of the API token used to authenticate this request.

Authorizations:
apiToken

Responses

Response samples

Content type
application/json
{
  • "organization_name": "string",
  • "token_name": "string",
  • "token_preview": "string"
}