core concepts

a few concepts show up everywhere in the crawlbrulee api. understanding them helps you write faster, cheaper integrations.

fetching and rendering

we pick the fastest way to get your content. most pages are fetched and parsed directly. when a page needs JavaScript to produce its content — single-page apps, dynamically loaded sections — set require_js: true and the page is rendered in a real browser before extraction. screenshot requests always render the page.

the delivered engine is part of the price. a direct http result has a 1-credit base, a browser result has a 3-credit base, and a delivered screenshot has a 5-credit base. the proxy tier then applies a multiplier.

proxy tiers

every request goes out through a proxy. the tier you choose determines cost and retrieval success rate.

default is auto: it bills the tier that actually delivered. pin basic to keep the ×1 multiplier, or advanced when you want the higher-success tier.

see proxies & location for guidance on choosing a tier.

caching

scrape and map results are cached server-side. this is the single biggest lever for reducing cost.

• default freshness window: 2 days (172,800 seconds) for scrape, 7 days (604,800 seconds) for map.
• cache hits are free: 0 base credits, except when a cache hit generates a new screenshot-slice variant — that is +1.
• key normalization: known tracking parameters are stripped before the page is fetched. same page with different tracking params = same cache entry. every other query parameter is part of the key.
• control with cache.max_age: set to 0 to force a fresh scrape. set higher to accept older results.
• keep a result out of the cache: set zero_data_retention: true (must be enabled for your organization). see zero data retention.

see caching for cache key details and advanced usage.

credits

credits are the billing unit. you pay for each page we deliver. failures cost 0, a fully cached result costs 0, and a cache hit that produces a new screenshot-slice variant costs +1. a page the site served with a status we don’t bill (403, 407, 408, 429, 451, or any 5xx) also costs 0 — see what counts as a failure.

key costs:

see credits & pricing for the full cost table, plan details, and usage tracking.

the response format

every 200 response carries a response_meta object — this is how you observe what a request actually cost, regardless of what you asked for:

{
  "response_meta": {
    "usage": {
      "total_credit_cost": 1,
      "engine_credit_cost": 1,
      "proxy_multiplier": 1,
      "screenshot_slicing_credit_cost": 0,
      "zero_data_retention_credit_cost": 0,
      "engine": "http",
      "proxy": "basic"
    }
  }
}

• usage.total_credit_cost: credits actually charged — engine_credit_cost × proxy_multiplier + screenshot_slicing_credit_cost + zero_data_retention_credit_cost. 0 on a fully cached result or a page we don’t bill.
• usage.engine_credit_cost: the engine base charged — 1 http, 3 browser, 5 screenshot, 0 cache. 0 when the page isn’t billed.
• usage.proxy_multiplier: 1 for basic, 5 for advanced.
• usage.screenshot_slicing_credit_cost: 1 when screenshot slices were made on this request, otherwise 0. 0 when the page isn’t billed.
• usage.zero_data_retention_credit_cost: 1 when the request used zero data retention and the result was billed and fresh, otherwise 0.
• usage.engine: the engine that delivered the billed result — http, browser, screenshot, or cache.
• usage.proxy: the proxy tier that actually resolved the request (basic or advanced) — the resolved tier, never auto.

scrape results also carry page_status_code at the top level: the status the site answered with. our 200 means we got a page; page_status_code tells you what kind of page it is (a 404 page is data, not an error). see errors.

collection endpoints add more: POST /api/map responses also include response_meta.pagination and response_meta.truncation, describing the page you got back and whether results were capped. async status responses include response_meta once the job reaches done.

a response can also carry a top-level warnings array — stable string codes telling you an output came back capped rather than whole (links_truncated, screenshot_truncated) or couldn’t be extracted at all (links_unavailable, metadata_unavailable). screenshot_unavailable means you asked for a screenshot, but the page came back through the http engine without one. switching on the codes is safe, and they’re stored with the result, so cache hits and async result fetches report them too. see limits and truncation for every code and what sits behind it.

sync vs async

two modes for scraping:

• sync (POST /api/scrape): send a request, get the result in the response. simple. best for real-time lookups, single pages, low-latency needs.
• async (POST /api/scrape/async): submit a job, poll for status, fetch the result when done. best for batch pipelines, long-running pages, queue-driven architectures.

both modes accept the same request body and return the same response schema, with one addition: async also accepts an optional webhook object so you can be notified on completion instead of polling — sync rejects it, since the response is your result. see webhooks for the payload and signature verification.

concurrency (simultaneous active scrapes) is separate from the rate limit, which throttles how fast you can submit. rate limits are per-plan and split into two buckets — sync (/api/scrape + /api/map) and async submit — e.g. 50 sync / 100 async requests per minute on free. see rate limits for the full picture.

extraction model

by default, every scrape returns the page metadata and cleaned_html (the page body, with scripts, styles, ads and cookie banners removed). use the extract object to request more formats — or to turn off a default:

{
  "extract": {
    "markdown": true,
    "links": true,
    "images": true,
    "screenshot": { "type": "viewport" }
  }
}

metadata and cleaned_html are on by default — set either to false to drop it. markdown, raw_html, links, images, and screenshot are opt-in. omit extract entirely and you get metadata + cleaned_html. request only what you need — smaller payloads, faster responses.

see extraction options for the full list of fields and their behavior.