map a site

discover all urls on a website. map combines sitemap.xml discovery with in-page link discovery to build a comprehensive link index — feed it a domain and get back every url it can find.

endpoint

POST /api/map

requires a Bearer token — see authentication.

basic example

curl


curl -X POST https://api.crawlbrulee.com/api/map \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com"
  }'

js/ts


const response = await fetch('https://api.crawlbrulee.com/api/map', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
  }),
})
 
const data = await response.json()
console.log(data)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const data = await crawlbrulee.map({ url: 'https://example.com' })
console.log(data.links)

python


import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/map",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
    },
)
 
data = response.json()
print(data)

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
data = client.map(url="https://example.com")
print(data.links)

response:


{
  "links": [
    { "url": "https://example.com/about" },
    { "url": "https://example.com/blog" },
    { "url": "https://example.com/pricing" }
  ],
  "response_meta": {
    "usage": {
      "credits": 1,
      "engine": "http",
      "proxy": "basic"
    },
    "pagination": {
      "page": 1,
      "limit": 5000,
      "total": 3,
      "total_pages": 1,
      "has_more": false
    },
    "truncation": {
      "storage_capped": false,
      "response_capped": false,
      "total_before_max_urls": 3,
      "total_detected_before_storage_cap": 3,
      "discovery_capped": false,
      "sitemaps_skipped": 0,
      "discovery_cap_reason": null
    }
  }
}

request parameters

all fields except url are optional.

field	type	required	default	description
`url`	`string`	yes	—	the website url to map
`proxy`	`string`	no	`"auto"`	proxy tier: `"basic"`, `"advanced"`, or `"auto"`. map uses the 1-credit http base; `auto` reserves 5 and bills 1 or 5 based on the delivered tier
`sitemap_only`	`boolean`	no	`false`	only use sitemap.xml discovery; skip in-page link discovery
`types.internal`	`boolean`	no	`true`	include internal links (same domain; `www` and the bare domain are equivalent)
`types.internal_subdomains`	`boolean`	no	`true`	include links on subdomains (other than `www`) of the target domain
`types.external`	`boolean`	no	`true`	include external links (different domain)
`cache.max_age`	`integer \| datetime`	no	`604800` (7 days)	max cache age in seconds, or an iso 8601 timestamp
`max_urls`	`integer`	no	`5000`	how many urls to discover, store, and return (cap: 100,000). discovery stops here, so a smaller value is a faster, lighter crawl — not just a shorter answer
`page`	`integer`	no	`1`	page number for paginated results
`limit`	`integer`	no	`5000`	results per page (cap: 10,000)
`location.country`	`string`	no	—	proxy egress country: an iso 3166-1 alpha-2 code (e.g. `US`), or `eu` / `europe` — see proxies & location

url normalization

this is about the url you send. the urls you get back follow a different rule — see link url form.

map always targets the site root derived from url: the query string and hash are removed, the path is dropped, www. is collapsed, and subdomains are preserved. passing https://example.com/blog/post maps example.com, not the specific blog post. the cache is keyed on this normalized root, so requests for any path on the same site share the same cached result.

response

`links`

an array of link objects. each object has a single url field. the site root itself is not listed — you already have it, it’s the url you asked for.


{
  "links": [{ "url": "https://example.com/about" }, { "url": "https://example.com/pricing" }]
}

ordering

links are sorted so the most useful ones land on page 1. four rules, applied in order:

link type — internal first, then internal_subdomains, then external.
where the link was found — links on the site’s home page come before links found only in sitemaps.
path depth — shallower paths first, so /news beats /2019/07/10/a-story.
alphabetical — codepoint order, as a tie-break.

on a big site this keeps section pages near the front instead of burying them behind thousands of dated archive articles. the order is deterministic, so pagination stays stable — page 2 of the same request always holds the same links.

link url form

every returned url is written the same way /scrape writes its returned url: in normalized form, with known tracking parameters stripped.


{
  "links": [
    { "url": "https://www.example.com/de-en/premium" },
    { "url": "https://www.example.com/search?page=2&sort=new" }
  ]
}

map output and scrape output now agree on the same page. feed a map link straight into /api/scrape and you stay on one host and one cache key — no rewriting, no accidental second fetch of the same page under a different name.

this changed: /api/map used to fold www. down to the bare domain and throw the query string away. if you match map output against stored urls, re-check that matching.

`response_meta.usage`

billing and cache info for this request.

field	type	description
`credits`	`integer`	credits actually charged for this request
`engine`	`string`	`http` for a fresh map or `cache` for a cached result
`proxy`	`string`	proxy tier that resolved the request: `basic` or `advanced`, never `auto`

see credits and caching for the full cost model.

`response_meta.pagination`

tells you where you are in the result set.

field	type	description
`page`	`integer`	current page number
`limit`	`integer`	maximum results per page
`total`	`integer`	total number of urls in the result set
`total_pages`	`integer`	total number of pages
`has_more`	`boolean`	whether more pages are available

`response_meta.truncation`

tells you whether the full url set was truncated at any stage.

field	type	description
`discovery_capped`	`boolean`	`true` if sitemap discovery stopped before reading every sitemap file it found — the site has more pages than this map lists
`discovery_cap_reason`	`string \| null`	which limit stopped discovery first, or `null` if nothing did — see the table below
`sitemaps_skipped`	`integer`	how many sitemap files were skipped or only partly read (file too big, could not be fetched, or a discovery limit was reached)
`response_capped`	`boolean`	`true` if more links were eligible than `max_urls`, so the list was trimmed
`total_before_max_urls`	`integer`	how many links were eligible before the `max_urls` trim
`storage_capped`	`boolean`	`true` if the stored map hit the hard 100,000-url storage cap
`total_detected_before_storage_cap`	`integer`	total urls found during discovery before that storage cap

`discovery_cap_reason` values

value	what it means	can you do anything?
`null`	nothing stopped discovery — the map is complete	—
`"max_urls"`	your own `max_urls` was reached	yes — ask again with a higher `max_urls`
`"time"`	discovery ran out of its time budget	no
`"file_budget"`	the site has more sitemap files than one request reads	no
`"depth"`	the site’s sitemap indexes nest too deeply	no
`"file_size"`	a sitemap file was too large to read	no

the field to watch is discovery_cap_reason. because discovery now stops at max_urls, a map that hit your cap comes back with exactly max_urls links and response_capped: false — nothing in the counts tells you the site has more. discovery_cap_reason: "max_urls" is what tells you, and raising max_urls is the fix. every other reason means we stopped for our own limits; more links are not available on this request.

has_more: true in pagination means more results exist on further pages — paginate to get them. storage_capped: true is the hard ceiling: the map exceeded 100,000 urls, and the remainder was never stored and isn’t retrievable.

filtering by link type

by default all link types are returned. use the types object to narrow results.

internal links only (exclude subdomains and external):

curl


curl -X POST https://api.crawlbrulee.com/api/map \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "types": {
      "internal": true,
      "internal_subdomains": false,
      "external": false
    }
  }'

js/ts


const response = await fetch('https://api.crawlbrulee.com/api/map', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    types: {
      internal: true,
      internal_subdomains: false,
      external: false,
    },
  }),
})
 
const data = await response.json()
console.log(data)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const data = await crawlbrulee.map({
  url: 'https://example.com',
  types: { internal: true, internal_subdomains: false, external: false },
})
 
console.log(data.links)

python


import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/map",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "types": {
            "internal": True,
            "internal_subdomains": False,
            "external": False,
        },
    },
)
 
data = response.json()
print(data)

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
data = client.map(
    url="https://example.com",
    types={"internal": True, "internal_subdomains": False, "external": False},
)
 
print(data.links)

including subdomains but excluding external:


{
  "url": "https://example.com",
  "types": {
    "internal": true,
    "internal_subdomains": true,
    "external": false
  }
}

this would return urls from example.com and blog.example.com but not twitter.com.

sitemap-only mode

set sitemap_only: true to skip in-page link discovery and rely solely on sitemap.xml discovery.

curl


curl -X POST https://api.crawlbrulee.com/api/map \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "sitemap_only": true
  }'

js/ts


const response = await fetch('https://api.crawlbrulee.com/api/map', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    sitemap_only: true,
  }),
})
 
const data = await response.json()
console.log(data)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const data = await crawlbrulee.map({
  url: 'https://example.com',
  sitemap_only: true,
})
 
console.log(data.links)

python


import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/map",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "sitemap_only": True,
    },
)
 
data = response.json()
print(data)

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
data = client.map(
    url="https://example.com",
    sitemap_only=True,
)
 
print(data.links)

when to use sitemap-only mode:

the site has a well-maintained sitemap and you want deterministic results.
you don’t want in-page-discovered links mixed in — sitemap_only doesn’t change latency; it only controls whether those links get merged into the sitemap results.
you’d rather not risk mixing in links from an earlier discovery pass — sitemap_only guarantees a clean, sitemap-only result set.

pagination

results are paginated when the total exceeds limit. use page and limit to iterate through the full set.

js/ts


const API_KEY = process.env.CRAWLBRULEE_API_KEY
 
async function mapAllUrls(siteUrl) {
  let page = 1
  const limit = 10000
  const allLinks = []
 
  while (true) {
    const res = await fetch('https://api.crawlbrulee.com/api/map', {
      method: 'POST',
      headers: {
        Authorization: `Bearer ${API_KEY}`,
        'Content-Type': 'application/json',
      },
      body: JSON.stringify({
        url: siteUrl,
        page,
        limit,
      }),
    })
 
    if (!res.ok) {
      throw new Error(`Map request failed: ${res.status}`)
    }
 
    const data = await res.json()
    allLinks.push(...data.links)
 
    if (!data.response_meta.pagination.has_more) break
    page++
  }
 
  console.log(`Found ${allLinks.length} URLs`)
  return allLinks
}
 
const links = await mapAllUrls('https://example.com')

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
async function mapAllUrls(siteUrl) {
  let page = 1
  const limit = 10000
  const allLinks = []
 
  while (true) {
    const data = await crawlbrulee.map({ url: siteUrl, page, limit })
    allLinks.push(...data.links)
 
    if (!data.response_meta.pagination.has_more) break
    page++
  }
 
  console.log(`Found ${allLinks.length} URLs`)
  return allLinks
}
 
const links = await mapAllUrls('https://example.com')

python


import os
import requests
 
API_KEY = os.environ["CRAWLBRULEE_API_KEY"]
 
def map_all_urls(site_url: str) -> list[dict]:
    page = 1
    limit = 10000
    all_links = []
 
    while True:
        res = requests.post(
            "https://api.crawlbrulee.com/api/map",
            headers={
                "Authorization": f"Bearer {API_KEY}",
                "Content-Type": "application/json",
            },
            json={
                "url": site_url,
                "page": page,
                "limit": limit,
            },
        )
        res.raise_for_status()
 
        data = res.json()
        all_links.extend(data["links"])
 
        if not data["response_meta"]["pagination"]["has_more"]:
            break
        page += 1
 
    print(f"Found {len(all_links)} URLs")
    return all_links
 
links = map_all_urls("https://example.com")

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
def map_all_urls(site_url: str) -> list:
    page = 1
    limit = 10000
    all_links = []
 
    while True:
        data = client.map(url=site_url, page=page, limit=limit)
        all_links.extend(data.links)
 
        if not data.response_meta.pagination.has_more:
            break
        page += 1
 
    print(f"Found {len(all_links)} URLs")
    return all_links
 
links = map_all_urls("https://example.com")

credits and caching

credit cost

a fresh map request uses the 1-credit http engine base, multiplied by the delivered proxy tier:

proxy tier	cost
`basic`	1 credit
`advanced`	5 credits
`auto`	1–5 credits — billed at whichever tier actually delivered the result

auto reserves 5 credits upfront and finalizes at the tier that delivered, so you need at least 5 available credits to submit one even if it ends up costing 1. see proxies & location for how the tiers behave and credits & pricing for the full cost model.

every response includes a response_meta.usage object with the exact credits, engine, and resolved proxy tier — see the response_meta.usage section above, or checking your usage for the platform-wide reference.

how caching works

a successful map result is cached server-side. a subsequent request for the same url returns the cached result if it’s younger than max_age — at 0 credits. if no fresh result exists, a fresh map runs and its result is cached for future requests.

one exception: a successful result with an empty sitemap (no sitemap found) is not cached — each such request runs, and is billed, as a fresh attempt.

the map cache works the same way as the scrape cache: cache.max_age is the only knob. there’s no require_js — that one is scrape-only.

default ttl

map results are cached for 7 days (604,800 seconds) by default — longer than the scrape default, since a site’s url inventory changes slowly. this matches the /api/map row in the cache ttl table.

controlling cache behavior

pass a cache object to set max_age, which controls what counts as “fresh” for your request:


{
  "url": "https://example.com",
  "cache": {
    "max_age": 86400
  }
}

max_age accepts:

an integer (seconds): 86400 = accept results up to 1 day old.
an iso 8601 datetime string: "2025-01-15T00:00:00Z" = accept results cached after this timestamp.
0: bypass the cache entirely and always run a fresh map.

so to use cached results (the default), omit cache or set max_age to 604800; to force-refresh, set max_age to 0.

cache hits are free

when a cache hit occurs, the credits reserved at the start of the request are fully released — you pay 0 credits. if your mapping needs are periodic, lean on the default 7-day cache and save your budget for fresh discoveries.

force-refresh

setting max_age to 0 skips the cache and always runs a fresh map, so you always pay the full credit cost. use it only when you genuinely need the latest url set — for example, right after a site relaunch.


{
  "url": "https://example.com",
  "cache": {
    "max_age": 0
  }
}

a map request is otherwise always cache-eligible — unlike scrape, there are no request options that disable map caching. only max_age: 0 forces a fresh run.

error responses

status	name	when
400	`invalid_url`, `url_too_long`, `unsupported_url_schema`, `url_credentials_not_supported`	bad url, url exceeds the size limit, unsupported url scheme, or url has embedded credentials
401	`invalid_credentials`	missing or invalid api token
403	`antibot_blocked`	the target site’s anti-bot protection blocked the map — same as scrape. a higher proxy tier (`advanced`) is the usual fix
408	`request_timeout`	map request exceeded the server-side timeout
422	`too_many_redirects`	the target’s sitemap or robots.txt redirected in a loop, or through more hops than we follow — same as scrape. not a bad request; retrying rarely helps
429	`too_many_requests`, `usage_allocation_error`	rate limit or credit/concurrency limit hit
499	`client_closed_request`	client disconnected before completion
500	`internal_server_error`	something went wrong on our end
503	`service_unavailable`	transient failure on our end — retry the same request; not an auth problem

all errors follow the standard { name, message, details? } format. see errors for the full reference.

map a site

endpoint

basic example

curl

js/ts

js/ts sdk

python

python sdk

request parameters

url normalization

response

links

ordering

link url form

response_meta.usage

response_meta.pagination

response_meta.truncation

discovery_cap_reason values

filtering by link type

curl

js/ts

js/ts sdk

python

python sdk

sitemap-only mode

curl

js/ts

js/ts sdk

python

python sdk

pagination

js/ts

js/ts sdk

python

python sdk

credits and caching

credit cost

how caching works

default ttl

controlling cache behavior

cache hits are free

force-refresh

error responses

`links`

`response_meta.usage`

`response_meta.pagination`

`response_meta.truncation`

`discovery_cap_reason` values