scrape

turn any url into structured data. extract markdown, html, links, images, and screenshots from any web page with a single POST request.

crawlbrulee handles the hard parts — JavaScript rendering, managed proxy routing, caching — so you can focus on what to do with the data, not how to get it.

endpoints

the scrape api has four endpoints: one for synchronous scraping, and three for the async workflow.

see scrape (sync) and scrape (async) for endpoint-specific details.

request body

both POST /api/scrape and POST /api/scrape/async accept the same json body, plus an optional webhook object accepted only by the async endpoint (see webhooks). only url is required — everything else is optional.

a few things worth knowing

• extract has defaults. omit it and you get metadata + cleaned_html. metadata and cleaned_html are on by default — set either to false to drop it; markdown, raw_html, links, images, and screenshot are opt-in. request only what you need for faster responses and smaller payloads.
• cache.max_age set to 0 forces a fresh scrape, bypassing any cached result.
• require_js adds latency because the page is fully rendered before extraction. only use it for JavaScript-rendered pages — see javascript rendering below.
• cleanup runs before extraction, so removed elements won’t appear in markdown, cleaned_html, link/image output, or the screenshot. it never applies to raw_html — that is always the page as it arrived. cleanup.exclude_selectors disables caching for the request; cleanup.ads_and_popups does not.
• proxy: "auto" is the default when you omit proxy. it tries basic first and upgrades to advanced when needed. the request reserves its engine’s advanced-proxy ceiling — 15 credits without a screenshot, 25 with one — then bills at the engine and proxy that delivered. see credits & pricing.
• location emulates a locale + country. location.locale sets Accept-Language/navigator.language; location.country sets the proxy exit country, and also accepts the pseudo-values eu (a random eu member state) and europe (a broader European set including the uk and non-eu countries). useful for geo-targeted or localized pages.

javascript rendering

by default, we extract content the fastest way we can. set require_js: true to render the page with JavaScript before extraction:

{
  "url": "https://example.com",
  "require_js": true,
  "extract": { "markdown": true }
}

use it when:

• the page is a single-page application (React, Vue, Angular)
• content is loaded dynamically via JavaScript
• a default scrape returns empty or incomplete results

browser delivery has a 3-credit engine base before the proxy multiplier. leave require_js off unless the page needs it. you do not need it for screenshots: screenshot requests always render and use the 5-credit screenshot base when the image is delivered.

choosing sync vs async

both modes accept the same request body — async additionally accepts an optional webhook object (see webhooks) — and return the same response schema. the difference is how you receive results.

sync (POST /api/scrape) returns the result directly in the http response. use it when you need data immediately and the page loads in under ~30 seconds.

async (POST /api/scrape/async) returns a job_id immediately. you then receive the result in one of two ways: poll for status and fetch it when the job completes, or register a webhook and have us notify your endpoint the moment it’s done — no polling required. use async for batch pipelines, long-running pages, or queue-driven architectures.

quick example

scrape a page and get markdown:

curl -X POST https://api.crawlbrulee.com/api/scrape \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": {
      "markdown": true
    }
  }'

response (condensed):

{
  "url": "https://example.com",
  "content_type": "text/html",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples ...",
  // cleaned_html is enabled by default, so it comes back even though we only asked for markdown
  "cleaned_html": "<h1>Example Domain</h1><p>This domain is for use in illustrative examples...</p>",
  // metadata is enabled by default too
  "metadata": {
    "title": "Example Domain"
    // ...more metadata fields when present on the page
  },
  "response_meta": {
    "usage": {
      "credits": 1,
      "engine": "http",
      "proxy": "basic",
      "screenshot_slices": 0
    }
  }
}

the response is a flat json object — the extracted content plus page metadata and a response_meta object reporting the charged engine, proxy, slices, and credits (see core concepts). fully cached responses have engine: "cache" and a 0-credit base; a newly produced screenshot-slice variant adds one flat credit (see caching).

what’s next

• scrape (sync) — full reference for the synchronous endpoint
• scrape (async) — submit jobs and poll for results
• extraction options — deep dive into extract fields and behavior
• screenshots — viewport and full-page captures
• caching — cache keys, ttls, and cost optimization
• proxies & location — proxy tiers, choosing between them, and location emulation