Skip to Content

filtering & pagination

by default all link types are returned. use the types object to narrow results. types filters pages only: sitemap files are never filtered out.

internal links only (exclude subdomains and external):

curl -X POST https://api.crawlbrulee.com/api/map \ -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "types": { "internal": true, "internal_subdomains": false, "external": false } }'

including subdomains but excluding external:

{ "url": "https://example.com", "types": { "internal": true, "internal_subdomains": true, "external": false } }

this would return urls from example.com and blog.example.com but not twitter.com.

sitemap-only mode

set sitemap_only: true to leave out the links found on the home page and use the sitemaps only. a map with sitemap_url or modified_since never adds home page links, so it needs no sitemap_only.

curl -X POST https://api.crawlbrulee.com/api/map \ -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com", "sitemap_only": true }'

when to use sitemap-only mode:

  • the site has a well-maintained sitemap and you want deterministic results.
  • you don’t want in-page-discovered links mixed in — sitemap_only doesn’t change latency; it only controls whether those links get merged into the sitemap results.
  • you’d rather not risk mixing in links from an earlier discovery pass — sitemap_only guarantees a clean, sitemap-only result set.

pagination

results are paginated when the total exceeds limit. use page and limit to iterate through the full set.

const API_KEY = process.env.CRAWLBRULEE_API_KEY async function mapAllUrls(siteUrl) { let page = 1 const limit = 10000 const allLinks = [] while (true) { const res = await fetch('https://api.crawlbrulee.com/api/map', { method: 'POST', headers: { Authorization: `Bearer ${API_KEY}`, 'Content-Type': 'application/json', }, body: JSON.stringify({ url: siteUrl, page, limit, }), }) if (!res.ok) { throw new Error(`Map request failed: ${res.status}`) } const data = await res.json() allLinks.push(...data.links) if (!data.response_meta.pagination.has_more) break page++ } console.log(`Found ${allLinks.length} URLs`) return allLinks } const links = await mapAllUrls('https://example.com')