core concepts

a few concepts show up everywhere in the crawlbrulee api. understanding them helps you write faster, cheaper integrations.

fetching and rendering

we pick the fastest way to get your content. most pages are fetched and parsed directly. when a page needs JavaScript to produce its content — single-page apps, dynamically loaded sections — set require_js: true and the page is rendered in a real browser before extraction. screenshot requests always render the page.

the delivered engine is part of the price. a direct http result has a 1-credit base, a browser result has a 3-credit base, and a delivered screenshot has a 5-credit base. the proxy tier then applies a multiplier.

proxy tiers

every request goes out through a proxy. the tier you choose determines cost and retrieval success rate.

default is auto: it reserves the request’s advanced-proxy ceiling and bills the tier that actually delivered. pin basic to keep the ×1 multiplier, or advanced when you want the higher-success tier.

see proxies & location for guidance on choosing a tier.

caching

scrape and map results are cached server-side. this is the single biggest lever for reducing cost.

• default freshness window: 2 days (172,800 seconds) for scrape, 7 days (604,800 seconds) for map.
• cache hits are free: 0 base credits, except when a cache hit generates a new screenshot-slice variant — that is +1.
• key normalization: known tracking parameters are stripped before the page is fetched. same page with different tracking params = same cache entry. every other query parameter is part of the key.
• control with cache.max_age: set to 0 to force a fresh scrape. set higher to accept older results.

see caching for cache key details and advanced usage.

credits

credits are the billing unit. every request follows a three-step lifecycle:

1. reserved — when your request starts, credits are reserved from your balance.
2. consumed — on success, reserved credits are consumed.
3. released or adjusted — failures release the reservation; a fully cached result costs 0, while a cache hit that produces a new screenshot-slice variant costs +1.

key costs:

see credits & pricing for the full cost table, plan details, and usage tracking.

the response format

every 200 response carries a response_meta object — this is how you observe what a request actually cost, regardless of what you asked for:

{
  "response_meta": {
    "usage": {
      "credits": 1,
      "engine": "http",
      "proxy": "basic",
      "screenshot_slices": 0
    }
  }
}

• usage.credits: credits actually charged — 0 on a fully cached result.
• usage.engine: the engine that delivered the billed result — http, browser, screenshot, or cache.
• usage.proxy: the proxy tier that actually resolved the request (basic or advanced) — the resolved tier, never auto.
• usage.screenshot_slices: the billed slice-variant increment — 1 when newly produced, otherwise 0.

collection endpoints add more: POST /api/map responses also include response_meta.pagination and response_meta.truncation, describing the page you got back and whether results were capped. async status responses include response_meta once the job reaches done.

a response can also carry a top-level warnings array — stable string codes telling you an output came back capped rather than whole (links_truncated, screenshot_truncated) or couldn’t be extracted at all (links_unavailable, metadata_unavailable). switching on the codes is safe, and they’re stored with the result, so cache hits and async result fetches report them too. see limits and truncation for every code and what sits behind it.

sync vs async

two modes for scraping:

• sync (POST /api/scrape): send a request, get the result in the response. simple. best for real-time lookups, single pages, low-latency needs.
• async (POST /api/scrape/async): submit a job, poll for status, fetch the result when done. best for batch pipelines, long-running pages, queue-driven architectures.

both modes accept the same request body and return the same response schema, with one addition: async also accepts an optional webhook object so you can be notified on completion instead of polling — sync rejects it, since the response is your result. see webhooks for the payload and signature verification.

concurrency (simultaneous active scrapes) is separate from the rate limit, which throttles how fast you can submit. rate limits are per-plan and split into two buckets — sync (/api/scrape + /api/map) and async submit — e.g. 50 sync / 100 async requests per minute on free. see rate limits for the full picture.

extraction model

by default, every scrape returns the page metadata and cleaned_html (the main content, with boilerplate stripped). use the extract object to request more formats — or to turn off a default:

{
  "extract": {
    "markdown": true,
    "links": true,
    "images": true,
    "screenshot": { "type": "viewport" }
  }
}

metadata and cleaned_html are on by default — set either to false to drop it. markdown, raw_html, links, images, and screenshot are opt-in. omit extract entirely and you get metadata + cleaned_html. request only what you need — smaller payloads, faster responses.

see extraction options for the full list of fields and their behavior.