Skip to Content

credits & caching

credits and caching

credit cost

a fresh map request uses the 1-credit http engine base, multiplied by the delivered proxy tier:

proxy tiercost
basic1 credit
advanced5 credits
auto1–5 credits — billed at whichever tier actually delivered the result

auto is billed at the tier that delivered. zero data retention adds 1 credit to a billed, fresh map. see proxies & location for how the tiers behave and credits & pricing for the full cost model.

every response includes a response_meta.usage object with the exact cost, engine, and resolved proxy tier — see the response_meta.usage section above, or checking your usage for the platform-wide reference.

how caching works

a successful map result is cached server-side. a subsequent request for the same url returns the cached result if it’s younger than max_age — at 0 credits. if no fresh result exists, a fresh map runs and its result is cached for future requests.

sitemap_url and depth are part of what makes two requests the same map: a map of one sitemap file never answers a request for another file, or for a different depth. a cached map answers a later request only when it holds what that request asks for, so a request for more than the cached map found (a higher max_results, for example) runs a fresh map.

a few results are never cached: a successful result with an empty sitemap (no sitemap found), so each such request runs as a fresh attempt; a map from a request with zero data retention; and a map with modified_since, which also never reads from the cache.

what a map costs when the site has problems

map has no page_status_code: one map run reads several things from the site (robots.txt, sitemap files, the homepage), so one status would be misleading. instead:

  • we found urls → 200 with the links, billed as usual — even if the homepage was a 404.
  • we found no urls, and the site gave a real answer (for example, no sitemap and a 404 for sitemap.xml) → 200 with empty links, billed as usual. “nothing here” is an answer.
  • we found no urls, and the site answered only with errors (5xx, 403 and the like) → 200 with empty links, not billed. every *_credit_cost field in response_meta.usage is 0.
  • filters don’t change the bill. billing looks at what we found before your types filter, so a filter that removes every url still leaves a billed, empty links.
  • the site didn’t answer at all, or we could not connect to it → 502 target_unreachable, not billed. the site may be back later, so a later retry can help.

the map cache works the same way as the scrape cache: cache.max_age is the only knob. there’s no require_js — that one is scrape-only.

default ttl

map results are cached for 7 days (604,800 seconds) by default — longer than the scrape default, since a site’s url inventory changes slowly. this matches the /api/map row in the cache ttl table.

controlling cache behavior

pass a cache object to set max_age, which controls what counts as “fresh” for your request:

{ "url": "https://example.com", "cache": { "max_age": 86400 } }

max_age accepts:

  • an integer (seconds): 86400 = accept results up to 1 day old.
  • an iso 8601 datetime string: "2025-01-15T00:00:00Z" = accept results cached after this timestamp.
  • 0: bypass the cache entirely and always run a fresh map.

so to use cached results (the default), omit cache or set max_age to 604800; to force-refresh, set max_age to 0.

cache hits are free

a cache hit costs 0 credits. if your mapping needs are periodic, lean on the default 7-day cache and save your budget for fresh discoveries.

force-refresh

setting max_age to 0 skips the cache and always runs a fresh map, so you always pay the full credit cost. use it only when you genuinely need the latest url set — for example, right after a site relaunch.

{ "url": "https://example.com", "cache": { "max_age": 0 } }

only max_age: 0 and modified_since force a fresh run. unlike scrape, no other request option turns map caching off.