Skip to Content

sitemaps

how a map reads sitemaps

a site’s sitemaps form a tree. a sitemap file either lists pages, or it is a sitemap index: a file that lists more sitemap files and no pages. map walks that tree one level at a time, and depth says how many levels it goes down. say example.com has this tree:

level 0 the list of the site's top sitemap files, from its robots.txt and the usual addresses level 1 https://example.com/sitemap_index.xml an index of 2 files level 2 https://example.com/sitemap-pages.xml lists pages level 2 https://example.com/sitemap-products.xml an index of 3 files level 3 https://example.com/sitemaps/products-1.xml.gz lists pages
  • a file at level N is read only when N is at most depth. depth is 0 to 3, default 3, so a plain map reads this whole tree. a higher value gets a 400 validation_error.
  • with sitemap_url, level 0 is that file, and it is always read. its nested files are level 1, and so on. without sitemap_url, level 0 is only the list of the site’s top files: depth: 0 returns that list without reading any of them.
  • a sitemap file below depth is not read. it is listed in response_meta.truncation.unread_sitemaps, and discovery_cap_reason is "depth". that is the answer to the depth you asked for, not a fault: read those files with sitemap_url in more calls, or ask with a higher depth.
  • entries picks what comes back: "pages" (the default), "sitemaps" (sitemap files) or "all" (both). max_results counts only the kind you ask for.
  • a sitemap file comes back as one of two types. "sitemap_index" is a file this map found to be a sitemap index, whether it read all of the file, part of it, or only its start. one it did not read all of can also be in unread_sitemaps. "sitemap" is any other sitemap file: one that lists pages, or one whose kind this map didn’t learn, because it was below depth or could not be read. so a listed file shows as "sitemap" until a map opens it, even when it is an index itself.

list a site’s top sitemap files

ask for level 0 only, and for sitemap files:

curl -X POST https://api.crawlbrulee.com/api/map \ -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "depth": 0, "entries": "sitemaps" }'

response (condensed, usage left out):

{ "links": [ { "url": "https://example.com/sitemap_index.xml", "type": "sitemap_index", "lastmod": null }, { "url": "https://example.com/sitemap-news.xml", "type": "sitemap", "lastmod": null } ], "response_meta": { "pagination": { "page": 1, "limit": 5000, "total": 2, "total_pages": 1, "has_more": false }, "truncation": { "storage_capped": false, "response_capped": false, "total_before_max_results": 2, "total_detected_before_storage_cap": 2, "discovery_capped": true, "sitemaps_skipped": 2, "unread_sitemaps": [ "https://example.com/sitemap_index.xml", "https://example.com/sitemap-news.xml" ], "discovery_cap_reason": "depth" } } }

these are the files the site names in its robots.txt, and the ones at the usual addresses. top files have no lastmod. depth: 0 lists them without reading them, so they are also in unread_sitemaps, and discovery_cap_reason says "depth": that is what you asked for.

a top file found at one of the usual addresses can still show as "sitemap_index", as sitemap_index.xml does here, because finding it there shows its kind. the others show as "sitemap".

read one sitemap file

put the file in sitemap_url:

{ "url": "https://example.com", "sitemap_url": "https://example.com/sitemap-pages.xml" }

response (condensed):

{ "links": [ { "url": "https://example.com/about", "type": "page", "lastmod": "2026-09-30T00:00:00.000Z" }, { "url": "https://example.com/blog", "type": "page", "lastmod": "2026-11-12T00:00:00.000Z" } ] }
  • sitemap_url is a full http or https address. it must be on the same site as url (the domain or one of its subdomains), or on a host that the site’s robots.txt names in a Sitemap: line, such as a CDN. a file on another host that robots.txt does not name is refused with a 400 validation_error, like a bad url, and costs nothing. to read a sitemap file of another site, map that site: set url to it.
  • the file is level 0, so it is always read. depth (default 3) says how many levels of nested sitemaps below it are read too.
  • url still names the site: links are sorted into internal, subdomain and external against it, and types filters the pages as usual.
  • a map with sitemap_url returns only what the sitemaps hold: it never adds links from the home page.

why a sitemap index can return no pages

a sitemap index holds only sitemap files. read one at a low depth with the default entries: "pages", and you get no pages:

{ "url": "https://example.com", "sitemap_url": "https://example.com/sitemap-products.xml", "depth": 0 }

response (condensed):

{ "links": [], "response_meta": { "pagination": { "page": 1, "limit": 5000, "total": 0, "total_pages": 0, "has_more": false }, "truncation": { "discovery_capped": true, "sitemaps_skipped": 3, "unread_sitemaps": [ "https://example.com/sitemaps/products-1.xml.gz", "https://example.com/sitemaps/products-2.xml.gz", "https://example.com/sitemaps/products-3.xml.gz" ], "discovery_cap_reason": "depth" } } }

depth: 0 reads sitemap-products.xml and nothing below it. that file lists only sitemap files, and entries: "pages" leaves those out, so links is empty. the answer doesn’t say the file was an index; discovery_cap_reason: "depth" and the files in unread_sitemaps are the sign. to fix it:

  • ask with entries: "sitemaps" (or "all") to get the files the index lists, as links of type "sitemap", or
  • raise depth (here to 1) to read those files too and get their pages.

a plain map can do the same on a deep site: when a site’s pages sit more than three levels down, the map returns no sitemap pages, with discovery_cap_reason: "depth" and the deeper files in unread_sitemaps. read those with sitemap_url.

find what changed since a date

modified_since returns only the entries whose lastmod is at or after a date, or a date and time. a plain date such as 2026-11-01 counts from the start of that day in UTC. a date and time needs a zone, such as Z or +01:00:

{ "url": "https://example.com", "sitemap_url": "https://example.com/sitemap-products.xml", "modified_since": "2026-11-01T00:00:00Z" }

response (condensed):

{ "links": [ { "url": "https://example.com/products/blue-lamp", "type": "page", "lastmod": "2026-11-03T00:00:00.000Z" }, { "url": "https://example.com/products/night-garden", "type": "page", "lastmod": "2026-11-12T00:00:00.000Z" } ] }

it works with or without sitemap_url. three things to know:

  1. an entry with no lastmod is left out, so every entry you get back matches. links found on the home page have no lastmod, so a map with modified_since never returns them. a site whose sitemaps give no dates returns nothing.
  2. every sitemap file is read, whatever its own lastmod. each entry that comes back, a page or a sitemap file, is judged by its own lastmod. so a changed page is found even when the site didn’t update the date of the sitemap file that lists it. with entries: "sitemaps", a sitemap file comes back when its own lastmod matches, not when a page in it does: to find the pages that changed, ask with entries: "pages" (the default).
  3. it always runs fresh. a request with modified_since never reads from the cache and never adds to it, so cache.max_age does nothing, and each call is billed as a fresh map, never as a free cache hit. that goes for every page you ask for too, so set a large limit and page as little as you can. as for any fresh map, an empty answer is billed when the site answered (see what a map costs when the site has problems).

map a very large or deep site in several calls

one map call stops at max_results (100,000 at most) and at its own time and file limits, and it goes three levels down at most. a site bigger or deeper than that is mapped in several calls, one sitemap file per call:

  1. list the sitemap files with depth: 1 and entries: "sitemaps". one call returns the site’s top sitemap files and every file they list.
  2. read each link of type "sitemap", plus each "sitemap_index" link that is also in unread_sitemaps (an index this map read only in part). the files of an index read whole are already in your list. read each file with sitemap_url and a high max_results.
  3. read every file in unread_sitemaps of those calls too. those are the files a call knew about and did not read whole: below depth, stopped by a limit, not readable this time, or cut off mid-download. a file that says it is over the size limit is listed too, but no call reads that one.
  4. page through each answer with page, while pagination.has_more is true.

a listed file shows as "sitemap" until a map opens it, even when it is an index itself. reading it with sitemap_url (default depth: 3) reads what it lists too. to go deeper than that, read the files in that call’s unread_sitemaps the same way.

unread_sitemaps holds up to 1,000 addresses. when it is full, ask the same thing again with entries: "sitemaps": you get every sitemap file the call found, as links, in pages like any answer.

const API_KEY = process.env.CRAWLBRULEE_API_KEY async function map(body) { const res = await fetch('https://api.crawlbrulee.com/api/map', { method: 'POST', headers: { Authorization: `Bearer ${API_KEY}`, 'Content-Type': 'application/json', }, body: JSON.stringify(body), }) if (!res.ok) { throw new Error(`Map request failed: ${res.status}`) } return res.json() } // One map, every page of its answer async function mapEveryPage(body) { const links = [] let page = 1 let data while (true) { data = await map({ ...body, page, limit: 10000 }) links.push(...data.links) if (!data.response_meta.pagination.has_more) break page++ } return { links, unreadSitemaps: data.response_meta.truncation.unread_sitemaps } } async function mapLargeSite(siteUrl) { // 1. List the top sitemap files and every file they list const top = await mapEveryPage({ url: siteUrl, depth: 1, entries: 'sitemaps' }) const partlyRead = new Set(top.unreadSitemaps) const toRead = top.links .filter( (link) => link.type === 'sitemap' || (link.type === 'sitemap_index' && partlyRead.has(link.url)) ) .map((link) => link.url) const seen = new Set() const pages = [] // 2. Read each file once, and every file a call did not read while (toRead.length > 0) { const file = toRead.shift() if (seen.has(file)) continue seen.add(file) const result = await mapEveryPage({ url: siteUrl, sitemap_url: file, max_results: 100000, }) pages.push(...result.links) toRead.push(...result.unreadSitemaps) } console.log(`Found ${pages.length} pages`) return pages } const pages = await mapLargeSite('https://example.com')

this sketch sends each file in sitemap_url only once. a file that could not be read whole comes back in unread_sitemaps, often for a reason that passes, so keep those and try them again later. later pages of the same request are normally served from the cache.

when the site rate-limits the map

when the site answers 429 on the file in sitemap_url, on its robots.txt, or on one of its top sitemap files, the map ends with 429 too_many_requests (limited_by: "target_site", with a Retry-After header when the site sent one). it is not billed. wait, then ask again. a 429 on a nested file further down doesn’t end the map: that file is listed in unread_sitemaps.