caching

every scrape result is cached server-side. a subsequent request for the same url has a 0-credit engine base when the requested result is fully cached; a newly produced screenshot-slice variant costs +1. understanding how caching works helps you save credits and get faster responses.

how it works

when a scrape request comes in:

1. we normalize the url and check for a cached result.
2. if a result exists and is younger than max_age, it’s returned immediately — 0 credits charged unless the request produces new screenshot slices.
3. if no result exists or it’s too old, a fresh scrape runs.

fresh results are cached and serve future requests for the same url.

cache key normalization

before looking up the cache, the url is cleaned:

• trailing slashes are normalized
• fragment identifiers are removed
• known tracking parameters are stripped

this means a url carrying tracking parameters and the same url without them hit the same cache entry. you don’t need to strip tracking params yourself.

every other query parameter is part of the cache key, so ?lang=en, ?lang=fr, and no query string are three separate entries. if a url carries params that don’t change the page you care about, drop them before you send it — https://example.com/page and https://example.com/page?ref=newsletter share a cache entry only if you request the same one both times.

tracking parameters

we remove known tracking parameters before the page is fetched, so they never reach the target site either.

the same normalization applies to /api/map. the url you send it is reduced to the site root, so its query string goes anyway — but the links it hands back keep their query strings, minus those tracking parameters. see link url form.

what else is in the cache key

besides the url, two request settings decide whether an entry can serve you:

• screenshot configuration. a cached entry only satisfies a request that asks for a compatible screenshot setup — same capture type, viewport, device mode, and cleanup. change any of those and you get a fresh scrape at full cost.
• location.locale. a page fetched as de-DE is a different entry from the same page fetched as en-US, because the site may serve different content per language. location.country is not part of the key.

extract is not part of the key

extract only selects what comes back in the response. changing it doesn’t change whether a request can be served from cache.

cache ttl

default time-to-live depends on the endpoint:

max_age controls what counts as “fresh” for your request. a result that’s 3 days old won’t be returned by a default /api/scrape request — set a longer max_age and it can be.

cached results expire after 60 days regardless of what max_age you pass — that’s the effective ceiling, not just the default.

controlling cache behavior

pass a cache object in your request body to control caching:

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cache": {
    "max_age": 86400
  }
}

max_age accepts:

• an integer (seconds): 86400 = accept results up to 1 day old
• an iso 8601 datetime string: "2025-01-15T00:00:00Z" = accept results cached after this timestamp
• 0: bypass cache entirely, always run a fresh scrape

max_age is the only cache control. it’s the same knob on /api/scrape, /api/scrape/async, and /api/map.

cache hits are free

if the result is already cached and fresh enough, you pay nothing — the single best way to reduce your credit usage.

every response includes response_meta.usage.engine; the value "cache" means the base result came from cache:

{
  "response_meta": {
    "usage": {
      "credits": 0,
      "engine": "cache",
      "proxy": "basic",
      "screenshot_slices": 0
    }
  }
}

force-refresh

set max_age to 0 to skip the cache and always get a fresh result:

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cache": {
    "max_age": 0
  }
}

what disables caching

some request options make a request ineligible for cache hits. these requests always run a fresh scrape and always cost credits:

• cleanup.exclude_selectors — custom selectors mean the cached result may not match what you need, so the cache is bypassed entirely. cleanup.ads_and_popups is different: it is part of the cache key, so both true and false stay cacheable.
• extract.screenshot.actions_before with wait or scroll — pre-screenshot actions modify the page state in ways that can’t be reliably reproduced from cache.

require_js and cache

when require_js is set to true, the request only matches cached results that were rendered with JavaScript. a result that was extracted without rendering won’t be returned for it.

tips for maximizing cache hits

• don’t force-refresh unless you need to. the default max_age covers most use cases.
• normalize your urls before you send them. if a query string is just a session id or a referrer tag, drop it — a shorter url is a wider cache entry.
• avoid cleanup.exclude_selectors if you can handle the trimming in post-processing. selectors disable the cache entirely; ads_and_popups does not.
• batch similar urls together. if multiple parts of your pipeline scrape the same page, the second request is free.