caching

every scrape result is cached server-side. a subsequent request for the same url has a 0-credit engine base when the requested result is fully cached; a newly produced screenshot-slice variant costs +1. understanding how caching works helps you save credits and get faster responses.

how it works

when a scrape request comes in:

1. we normalize the url and check for a cached result.
2. if a result exists and is younger than max_age, it’s returned immediately — 0 credits charged unless the request produces new screenshot slices.
3. if no result exists or it’s too old, a fresh scrape runs.

fresh results are cached and serve future requests for the same url. only pages we bill are cached: a 404 page is cached like any other page, but a page we don’t bill (a 5xx, 403 or 451 page, for example) is not, so the next request tries the site again. see what counts as a failure. a result from a request with zero data retention is not cached either.

cache key normalization

before looking up the cache, the url is cleaned:

• trailing slashes are normalized
• fragment identifiers are removed
• known tracking parameters are stripped

this means a url carrying tracking parameters and the same url without them hit the same cache entry. you don’t need to strip tracking params yourself.

every other query parameter is part of the cache key, so ?lang=en, ?lang=fr, and no query string are three separate entries. if a url carries params that don’t change the page you care about, drop them before you send it — https://example.com/page and https://example.com/page?ref=newsletter share a cache entry only if you request the same one both times.

tracking parameters

we remove known tracking parameters before the page is fetched, so they never reach the target site either.

the same normalization applies to /api/map. the url you send it is reduced to the site root, so its query string goes anyway — but the links it hands back keep their query strings, minus those tracking parameters. see link url form.

what else is in the cache key

besides the url, these request settings decide whether an entry can serve you:

• screenshot configuration. a cached entry only satisfies a request that asks for a compatible screenshot setup — same capture type, viewport, device mode, and cleanup. change any of those and you get a fresh scrape at full cost.
• location.locale. a page fetched as de-DE is a different entry from the same page fetched as en-US, because the site may serve different content per language. location.country is not part of the key.
• cleanup. cleanup.ads_and_popups and cleanup.exclude_selectors change what is removed from the page, so they are part of the key. requests with the same removals share an entry; requests with different ones don’t. see excluding content.

extract is not part of the key

extract only selects what comes back in the response. changing it doesn’t change whether a request can be served from cache.

that includes extract.elements. on a cache hit, we read your elements from the cached page, and the hit still costs 0 credits.

cache ttl

default time-to-live depends on the endpoint:

max_age controls what counts as “fresh” for your request. a result that’s 3 days old won’t be returned by a default /api/scrape request — set a longer max_age and it can be.

cached results expire after 60 days regardless of what max_age you pass — that’s the effective ceiling, not just the default.

controlling cache behavior

pass a cache object in your request body to control caching:

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cache": {
    "max_age": 86400
  }
}

max_age accepts:

• an integer (seconds): 86400 = accept results up to 1 day old
• an iso 8601 datetime string: "2025-01-15T00:00:00Z" = accept results cached after this timestamp
• 0: bypass cache entirely, always run a fresh scrape

max_age is the only cache control. it’s the same knob on /api/scrape, /api/scrape/async, and /api/map. to keep a result out of the cache, see zero data retention.

cache hits are free

if the result is already cached and fresh enough, you pay nothing — the single best way to reduce your credit usage.

every response includes response_meta.usage.engine; the value "cache" means the base result came from cache:

{
  "response_meta": {
    "usage": {
      "total_credit_cost": 0,
      "engine_credit_cost": 0,
      "proxy_multiplier": 1,
      "screenshot_slicing_credit_cost": 0,
      "zero_data_retention_credit_cost": 0,
      "engine": "cache",
      "proxy": "basic"
    }
  }
}

force-refresh

set max_age to 0 to skip the cache and always get a fresh result:

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cache": {
    "max_age": 0
  }
}

what disables caching

one request option makes a request ineligible for cache hits. such a request always runs a fresh scrape and always costs credits:

• extract.screenshot.actions_before with wait or scroll — pre-screenshot actions modify the page state in ways that can’t be reliably reproduced from cache.

require_js and cache

when require_js is set to true, the request only matches cached results that were rendered with JavaScript. a result that was extracted without rendering won’t be returned for it.

zero data retention

the result of a request sent with zero_data_retention: true is not written to the shared cache. reading from the cache is still allowed, so such a request can be served a result that is already cached; add cache.max_age: 0 to stop that too. see zero data retention.

tips for maximizing cache hits

• don’t force-refresh unless you need to. the default max_age covers most use cases.
• normalize your urls before you send them. if a query string is just a session id or a referrer tag, drop it — a shorter url is a wider cache entry.
• send the same cleanup.exclude_selectors each time. requests with different removals don’t share a cache entry, so a list that changes from call to call misses the cache.
• batch similar urls together. if multiple parts of your pipeline scrape the same page, the second request is free.