sitemaps
how a map reads sitemaps
a site’s sitemaps form a tree. a sitemap file either lists pages, or it is a sitemap index:
a file that lists more sitemap files and no pages. map walks that tree one level at a time, and
depth says how many levels it goes down. say example.com has this tree:
level 0 the list of the site's top sitemap files, from its robots.txt and the usual addresses
level 1 https://example.com/sitemap_index.xml an index of 2 files
level 2 https://example.com/sitemap-pages.xml lists pages
level 2 https://example.com/sitemap-products.xml an index of 3 files
level 3 https://example.com/sitemaps/products-1.xml.gz lists pages- a file at level N is read only when N is at most
depth.depthis0to3, default3, so a plain map reads this whole tree. a higher value gets a400validation_error. - with
sitemap_url, level 0 is that file, and it is always read. its nested files are level 1, and so on. withoutsitemap_url, level 0 is only the list of the site’s top files:depth: 0returns that list without reading any of them. - a sitemap file below
depthis not read. it is listed inresponse_meta.truncation.unread_sitemaps, anddiscovery_cap_reasonis"depth". that is the answer to the depth you asked for, not a fault: read those files withsitemap_urlin more calls, or ask with a higherdepth. entriespicks what comes back:"pages"(the default),"sitemaps"(sitemap files) or"all"(both).max_resultscounts only the kind you ask for.- a sitemap file comes back as one of two types.
"sitemap_index"is a file this map found to be a sitemap index, whether it read all of the file, part of it, or only its start. one it did not read all of can also be inunread_sitemaps."sitemap"is any other sitemap file: one that lists pages, or one whose kind this map didn’t learn, because it was belowdepthor could not be read. so a listed file shows as"sitemap"until a map opens it, even when it is an index itself.
list a site’s top sitemap files
ask for level 0 only, and for sitemap files:
curl -X POST https://api.crawlbrulee.com/api/map \
-H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"depth": 0,
"entries": "sitemaps"
}'response (condensed, usage left out):
{
"links": [
{ "url": "https://example.com/sitemap_index.xml", "type": "sitemap_index", "lastmod": null },
{ "url": "https://example.com/sitemap-news.xml", "type": "sitemap", "lastmod": null }
],
"response_meta": {
"pagination": { "page": 1, "limit": 5000, "total": 2, "total_pages": 1, "has_more": false },
"truncation": {
"storage_capped": false,
"response_capped": false,
"total_before_max_results": 2,
"total_detected_before_storage_cap": 2,
"discovery_capped": true,
"sitemaps_skipped": 2,
"unread_sitemaps": [
"https://example.com/sitemap_index.xml",
"https://example.com/sitemap-news.xml"
],
"discovery_cap_reason": "depth"
}
}
}these are the files the site names in its robots.txt, and the ones at the usual addresses. top
files have no lastmod. depth: 0 lists them without reading them, so they are also in
unread_sitemaps, and discovery_cap_reason says "depth": that is what you asked for.
a top file found at one of the usual addresses can still show as "sitemap_index", as
sitemap_index.xml does here, because finding it there shows its kind. the others show as
"sitemap".
read one sitemap file
put the file in sitemap_url:
{
"url": "https://example.com",
"sitemap_url": "https://example.com/sitemap-pages.xml"
}response (condensed):
{
"links": [
{ "url": "https://example.com/about", "type": "page", "lastmod": "2026-09-30T00:00:00.000Z" },
{ "url": "https://example.com/blog", "type": "page", "lastmod": "2026-11-12T00:00:00.000Z" }
]
}sitemap_urlis a fullhttporhttpsaddress. it must be on the same site asurl(the domain or one of its subdomains), or on a host that the site’s robots.txt names in aSitemap:line, such as a CDN. a file on another host that robots.txt does not name is refused with a400validation_error, like a badurl, and costs nothing. to read a sitemap file of another site, map that site: seturlto it.- the file is level 0, so it is always read.
depth(default3) says how many levels of nested sitemaps below it are read too. urlstill names the site: links are sorted into internal, subdomain and external against it, andtypesfilters the pages as usual.- a map with
sitemap_urlreturns only what the sitemaps hold: it never adds links from the home page.
why a sitemap index can return no pages
a sitemap index holds only sitemap files. read one at a low depth with the default
entries: "pages", and you get no pages:
{
"url": "https://example.com",
"sitemap_url": "https://example.com/sitemap-products.xml",
"depth": 0
}response (condensed):
{
"links": [],
"response_meta": {
"pagination": { "page": 1, "limit": 5000, "total": 0, "total_pages": 0, "has_more": false },
"truncation": {
"discovery_capped": true,
"sitemaps_skipped": 3,
"unread_sitemaps": [
"https://example.com/sitemaps/products-1.xml.gz",
"https://example.com/sitemaps/products-2.xml.gz",
"https://example.com/sitemaps/products-3.xml.gz"
],
"discovery_cap_reason": "depth"
}
}
}depth: 0 reads sitemap-products.xml and nothing below it. that file lists only sitemap files,
and entries: "pages" leaves those out, so links is empty. the answer doesn’t say the file was an
index; discovery_cap_reason: "depth" and the files in unread_sitemaps are the sign. to fix it:
- ask with
entries: "sitemaps"(or"all") to get the files the index lists, as links of type"sitemap", or - raise
depth(here to1) to read those files too and get their pages.
a plain map can do the same on a deep site: when a site’s pages sit more than three levels down,
the map returns no sitemap pages, with discovery_cap_reason: "depth" and the deeper files in
unread_sitemaps. read those with sitemap_url.
find what changed since a date
modified_since returns only the entries whose lastmod is at or after a date, or a date and time.
a plain date such as 2026-11-01 counts from the start of that day in UTC. a date and time needs a
zone, such as Z or +01:00:
{
"url": "https://example.com",
"sitemap_url": "https://example.com/sitemap-products.xml",
"modified_since": "2026-11-01T00:00:00Z"
}response (condensed):
{
"links": [
{ "url": "https://example.com/products/blue-lamp", "type": "page", "lastmod": "2026-11-03T00:00:00.000Z" },
{ "url": "https://example.com/products/night-garden", "type": "page", "lastmod": "2026-11-12T00:00:00.000Z" }
]
}it works with or without sitemap_url. three things to know:
- an entry with no
lastmodis left out, so every entry you get back matches. links found on the home page have nolastmod, so a map withmodified_sincenever returns them. a site whose sitemaps give no dates returns nothing. - every sitemap file is read, whatever its own
lastmod. each entry that comes back, a page or a sitemap file, is judged by its ownlastmod. so a changed page is found even when the site didn’t update the date of the sitemap file that lists it. withentries: "sitemaps", a sitemap file comes back when its ownlastmodmatches, not when a page in it does: to find the pages that changed, ask withentries: "pages"(the default). - it always runs fresh. a request with
modified_sincenever reads from the cache and never adds to it, socache.max_agedoes nothing, and each call is billed as a fresh map, never as a free cache hit. that goes for everypageyou ask for too, so set a largelimitand page as little as you can. as for any fresh map, an empty answer is billed when the site answered (see what a map costs when the site has problems).
map a very large or deep site in several calls
one map call stops at max_results (100,000 at most) and at its own time and file limits, and it
goes three levels down at most. a site bigger or deeper than that is mapped in several calls, one
sitemap file per call:
- list the sitemap files with
depth: 1andentries: "sitemaps". one call returns the site’s top sitemap files and every file they list. - read each link of type
"sitemap", plus each"sitemap_index"link that is also inunread_sitemaps(an index this map read only in part). the files of an index read whole are already in your list. read each file withsitemap_urland a highmax_results. - read every file in
unread_sitemapsof those calls too. those are the files a call knew about and did not read whole: belowdepth, stopped by a limit, not readable this time, or cut off mid-download. a file that says it is over the size limit is listed too, but no call reads that one. - page through each answer with
page, whilepagination.has_moreistrue.
a listed file shows as "sitemap" until a map opens it, even when it is an index itself. reading it
with sitemap_url (default depth: 3) reads what it lists too. to go deeper than that, read the
files in that call’s unread_sitemaps the same way.
unread_sitemaps holds up to 1,000 addresses. when it is full, ask the same thing again with
entries: "sitemaps": you get every sitemap file the call found, as links, in pages like any
answer.
js/ts
const API_KEY = process.env.CRAWLBRULEE_API_KEY
async function map(body) {
const res = await fetch('https://api.crawlbrulee.com/api/map', {
method: 'POST',
headers: {
Authorization: `Bearer ${API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify(body),
})
if (!res.ok) {
throw new Error(`Map request failed: ${res.status}`)
}
return res.json()
}
// One map, every page of its answer
async function mapEveryPage(body) {
const links = []
let page = 1
let data
while (true) {
data = await map({ ...body, page, limit: 10000 })
links.push(...data.links)
if (!data.response_meta.pagination.has_more) break
page++
}
return { links, unreadSitemaps: data.response_meta.truncation.unread_sitemaps }
}
async function mapLargeSite(siteUrl) {
// 1. List the top sitemap files and every file they list
const top = await mapEveryPage({ url: siteUrl, depth: 1, entries: 'sitemaps' })
const partlyRead = new Set(top.unreadSitemaps)
const toRead = top.links
.filter(
(link) =>
link.type === 'sitemap' || (link.type === 'sitemap_index' && partlyRead.has(link.url))
)
.map((link) => link.url)
const seen = new Set()
const pages = []
// 2. Read each file once, and every file a call did not read
while (toRead.length > 0) {
const file = toRead.shift()
if (seen.has(file)) continue
seen.add(file)
const result = await mapEveryPage({
url: siteUrl,
sitemap_url: file,
max_results: 100000,
})
pages.push(...result.links)
toRead.push(...result.unreadSitemaps)
}
console.log(`Found ${pages.length} pages`)
return pages
}
const pages = await mapLargeSite('https://example.com')this sketch sends each file in sitemap_url only once. a file that could not be read whole comes back in unread_sitemaps,
often for a reason that passes, so keep those and try them again later. later pages of the same
request are normally served from the cache.
when the site rate-limits the map
when the site answers 429 on the file in sitemap_url, on its robots.txt, or on one of its top
sitemap files, the map ends with 429 too_many_requests (limited_by: "target_site", with a
Retry-After header when the site sent one). it is not billed. wait, then ask again. a 429 on a nested file further
down doesn’t end the map: that file is listed in unread_sitemaps.