filtering & pagination
filtering by link type
by default all link types are returned. use the types object to narrow results. types filters
pages only: sitemap files are never filtered out.
internal links only (exclude subdomains and external):
curl
curl -X POST https://api.crawlbrulee.com/api/map \
-H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"types": {
"internal": true,
"internal_subdomains": false,
"external": false
}
}'including subdomains but excluding external:
{
"url": "https://example.com",
"types": {
"internal": true,
"internal_subdomains": true,
"external": false
}
}this would return urls from example.com and blog.example.com but not twitter.com.
sitemap-only mode
set sitemap_only: true to leave out the links found on the home page and use the sitemaps only.
a map with sitemap_url or modified_since never adds home page links, so it needs no
sitemap_only.
curl
curl -X POST https://api.crawlbrulee.com/api/map \
-H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com",
"sitemap_only": true
}'when to use sitemap-only mode:
- the site has a well-maintained sitemap and you want deterministic results.
- you don’t want in-page-discovered links mixed in —
sitemap_onlydoesn’t change latency; it only controls whether those links get merged into the sitemap results. - you’d rather not risk mixing in links from an earlier discovery pass —
sitemap_onlyguarantees a clean, sitemap-only result set.
pagination
results are paginated when the total exceeds limit. use page and limit to iterate through the full set.
js/ts
const API_KEY = process.env.CRAWLBRULEE_API_KEY
async function mapAllUrls(siteUrl) {
let page = 1
const limit = 10000
const allLinks = []
while (true) {
const res = await fetch('https://api.crawlbrulee.com/api/map', {
method: 'POST',
headers: {
Authorization: `Bearer ${API_KEY}`,
'Content-Type': 'application/json',
},
body: JSON.stringify({
url: siteUrl,
page,
limit,
}),
})
if (!res.ok) {
throw new Error(`Map request failed: ${res.status}`)
}
const data = await res.json()
allLinks.push(...data.links)
if (!data.response_meta.pagination.has_more) break
page++
}
console.log(`Found ${allLinks.length} URLs`)
return allLinks
}
const links = await mapAllUrls('https://example.com')