js/ts sdk

the official TypeScript / JavaScript sdk — @crawlbrulee/sdk. fully typed, ships esm + CommonJS, zero runtime dependencies (just fetch). works on Node.js 22+, Deno, Bun, and any runtime with fetch.

install


npm install @crawlbrulee/sdk
# pnpm add @crawlbrulee/sdk · yarn add @crawlbrulee/sdk

quickstart


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// pass the key directly, or use fromEnv() to read CRAWLBRULEE_API_KEY
const crawlbrulee = Crawlbrulee.fromEnv()
 
const page = await crawlbrulee.scrape({
  url: 'https://example.com',
  extract: { markdown: true, links: true },
})
 
console.log(page.markdown)
console.log(page.links?.length, 'links found')

usage object

every scrape/map response, terminal async status, and scrape.complete webhook payload carries billing + routing info on response_meta.usage — no separate usage() call is needed to attribute a single request’s cost.


const page = await crawlbrulee.scrape({ url: 'https://example.com' })
console.log(page.response_meta.usage.credits) // 0 on a fully cached result

Usage ({ credits, engine, proxy, screenshot_slices }) covers scrape, async, and webhook responses. MapUsage is the map-specific { credits, engine, proxy } shape; map cannot generate screenshot slices, and its engine is http | cache.

client options

option	default	description
`apiKey`	—	sent as `Authorization: Bearer …`. required, or use `Crawlbrulee.fromEnv()`.
`baseUrl`	production host	override the api host — for local development and staging.
`timeoutMs`	`0` (no timeout)	per-request timeout, covering the whole request. a per-call `timeoutMs` overrides it.

Crawlbrulee.fromEnv(overrides?) reads CRAWLBRULEE_API_KEY and forwards any other option.

methods

method	description
`scrape(request, opts?)`	scrape a url synchronously; resolves with the page.
`scrapeAsync(request, opts?)`	submit a background job; returns `{ job_id }`.
`getScrapeStatus(jobId, opts?)`	`pending` / `running` / `done` / `failed`.
`getScrapeResult(jobId, opts?)`	result of a finished job.
`waitForScrape(jobId, opts?)`	poll until terminal, then return the result.
`map(request, opts?)`	build (or return a cached) link map for a site.
`usage(opts?)`	current billing-cycle credits, quota, concurrency, reset.
`whoami(opts?)`	organization + token identity.
`fetchScrapeResultFromWebhook(body, opts?)`	fetch the result of a verified `scrape.complete` delivery; throws on a failed/cancelled job.
`verifyWebhookSignature({ payload, headers, secret })`	standalone export (no client needed) — async; verifies a delivery and returns a `{ verified, … }` result instead of throwing.

every method takes an optional second argument: { signal, timeoutMs } — except waitForScrape, whose timeoutMs is the overall wait budget across all polls (default 300_000, 0 = wait forever), not a per-request http timeout.

the two webhook helpers in the table — verifyWebhookSignature (a standalone export you await; no client needed) and client.fetchScrapeResultFromWebhook(body) — are covered in detail on webhook verification.

async jobs


const { job_id } = await crawlbrulee.scrapeAsync({ url: 'https://example.com' })
 
const page = await crawlbrulee.waitForScrape(job_id, {
  intervalMs: 2000, // poll every 2s (default)
  timeoutMs: 5 * 60 * 1000, // give up after 5 min (0 = wait forever)
})

errors

every failure extends CrawlbruleeError. typed subclasses are exported for the actionable cases:

class	when
`AuthenticationError`	401 — missing, invalid, or revoked key. Also a 403 whose error name is not recognized.
`AntibotBlockedError`	403 `antibot_blocked` — the target site’s anti-bot protection blocked the request (scrape or map).
`TooManyRedirectsError`	422 `too_many_redirects` — the target site redirected the request in a loop (scrape or map). not a bad request.
`PageTooLargeError`	422 `page_too_large` — the page’s html was too large to process (scrape). terminal; don’t retry it.
`RateLimitError`	429 — exposes `retryAfterMs`, `limitedBy`.
`UsageAllocationError`	plan limit hit — exposes `reason`, `usage`.
`ValidationError`	bad request (`invalid_url`, `url_too_long`, …).
`NotFoundError`	404 (e.g. unknown async job).
`TransportError`	network failure, abort, non-json response.


import { RateLimitError, UsageAllocationError } from '@crawlbrulee/sdk'
 
try {
  await crawlbrulee.scrape({ url: 'https://example.com' })
} catch (err) {
  if (err instanceof RateLimitError) {
    // back off and retry after err.retryAfterMs
  } else if (err instanceof UsageAllocationError) {
    console.error('plan limit:', err.reason)
  } else {
    throw err
  }
}

for exhaustive branching, switch on err.errorName (the exported ApiErrorName union). to narrow an unknown catch value to CrawlbruleeError, use the exported isCrawlbruleeError() type guard:


import { isCrawlbruleeError } from '@crawlbrulee/sdk'
 
if (isCrawlbruleeError(err)) {
  console.error(err.status, err.errorName)
}

see credits & pricing for what each operation costs, and errors for the full http error catalog.