scrape (async)

for batch pipelines and long-running pages. submit a job, get a job_id back immediately, and pick up the result later.

the async endpoint accepts the same request body as sync scrape and returns the same response schema — the only difference is how you receive results. instead of blocking until the page is scraped, you choose how you’re notified when it’s done.

two ways to get the result

once a job is submitted, there are two ways to receive the result — pick whichever fits your architecture:

polling — you call GET /api/scrape/status/{job_id} on an interval until the job is done, then fetch the result. it’s the simplest option and needs no public endpoint of your own, which makes it the natural fit for scripts, notebooks, or any client that can’t receive inbound http. the cli polls for you — crawlbrulee scrape wait <job-id> (or scrape url … --async --wait) — as do the sdks’ waitForScrape helpers.
webhooks — you register a callback url when you submit, and we send a single signed POST to it the moment the job finishes. nothing to poll, no wasted requests. this is the more scalable choice for production systems and high job volumes, at the cost of standing up (and verifying) a receiving endpoint.

both deliver the same result; they’re just different ways to learn a job is done. polling is covered step-by-step below; see completion webhooks for the webhook payload, signatures, and delivery guarantees.

the polling workflow

submit

POST /api/scrape/async with the same body as sync. you get back a job_id.

poll

GET /api/scrape/status/{job_id} until status is done or failed.

fetch

GET /api/scrape/result/{job_id} to get the full scrape result.

step 1: submit a job

send a POST to /api/scrape/async with the same request body you’d use for sync scrape. the server validates the request, queues the job, and returns http 202 with a job_id.

curl


curl -X POST https://api.crawlbrulee.com/api/scrape/async \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": {
      "markdown": true,
      "links": true
    }
  }'

js/ts


const response = await fetch('https://api.crawlbrulee.com/api/scrape/async', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    extract: {
      markdown: true,
      links: true,
    },
  }),
})
 
const { job_id } = await response.json()
console.log('Job submitted:', job_id)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const { job_id } = await crawlbrulee.scrapeAsync({
  url: 'https://example.com',
  extract: { markdown: true, links: true },
})
 
console.log('Job submitted:', job_id)

python


import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/scrape/async",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "extract": {
            "markdown": True,
            "links": True,
        },
    },
)
 
data = response.json()
job_id = data["job_id"]
print(f"Job submitted: {job_id}")

python sdk


from crawlbrulee import Crawlbrulee, ScrapeExtract
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
job = client.scrape_async(
    url="https://example.com",
    extract=ScrapeExtract(markdown=True, links=True),
)
 
print("Job submitted:", job.job_id)

response (http 202):


{
  "job_id": "683a1f2b4c5d6e7f8a9b0c1d"
}

step 2: check status

poll GET /api/scrape/status/{job_id} to see where the job stands. keep polling until status is done or failed.

curl


curl https://api.crawlbrulee.com/api/scrape/status/683a1f2b4c5d6e7f8a9b0c1d \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY"

js/ts


const statusRes = await fetch(
  `https://api.crawlbrulee.com/api/scrape/status/${job_id}`,
  {
    headers: {
      Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    },
  },
)
 
const status = await statusRes.json()
console.log(status)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const status = await crawlbrulee.getScrapeStatus(job_id)
console.log(status)

python


status_response = requests.get(
    f"https://api.crawlbrulee.com/api/scrape/status/{job_id}",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
    },
)
 
status = status_response.json()
print(status)

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
status = client.get_scrape_status(job_id)
print(status)

response:


{
  "job_id": "683a1f2b4c5d6e7f8a9b0c1d",
  "status": "running",
  "created_at": "2025-01-15T10:30:00.000Z"
}

status values

status	meaning
`pending`	job queued, waiting for execution
`running`	scrape in progress
`done`	result ready — fetch it
`failed`	scrape failed — check the `error` field

once status is "done", the response also includes a response_meta.usage object — the same shape as on a sync scrape response — so you can check billing details without fetching the full result:


{
  "job_id": "683a1f2b4c5d6e7f8a9b0c1d",
  "status": "done",
  "created_at": "2025-01-15T10:30:00.000Z",
  "response_meta": {
    "usage": {
      "total_credit_cost": 1,
      "engine_credit_cost": 1,
      "proxy_multiplier": 1,
      "screenshot_slicing_credit_cost": 0,
      "zero_data_retention_credit_cost": 0,
      "engine": "http",
      "proxy": "basic"
    }
  }
}

when status is "failed", the response includes a generic, customer-safe error string. detailed upstream and service errors are not exposed:


{
  "job_id": "683a1f2b4c5d6e7f8a9b0c1d",
  "status": "failed",
  "created_at": "2025-01-15T10:30:00.000Z",
  "error": "Scrape job failed. Please try again or contact support if the problem persists."
}

step 3: get result

once status is done, fetch the full scrape result from GET /api/scrape/result/{job_id}. the response schema is identical to what you’d get from a sync POST /api/scrape call.

curl


curl https://api.crawlbrulee.com/api/scrape/result/683a1f2b4c5d6e7f8a9b0c1d \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY"

js/ts


const resultRes = await fetch(
  `https://api.crawlbrulee.com/api/scrape/result/${job_id}`,
  {
    headers: {
      Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    },
  },
)
 
const result = await resultRes.json()
console.log(result)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const result = await crawlbrulee.getScrapeResult(job_id)
console.log(result)

python


result_response = requests.get(
    f"https://api.crawlbrulee.com/api/scrape/result/{job_id}",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
    },
)
 
result = result_response.json()
print(result)

python sdk


from crawlbrulee import Crawlbrulee
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
result = client.get_scrape_result(job_id)
print(result)

response:


{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples ...",
  "cleaned_html": "<h1>Example Domain</h1><p>This domain is for use in illustrative examples...</p>",
  "links": [
    {
      "href": "https://www.iana.org/domains/example",
      "text": "More information...",
      "internal": false
    }
  ],
  "metadata": {
    "title": "Example Domain"
  },
  "response_meta": {
    "usage": {
      "total_credit_cost": 1,
      "engine_credit_cost": 1,
      "proxy_multiplier": 1,
      "screenshot_slicing_credit_cost": 0,
      "zero_data_retention_credit_cost": 0,
      "engine": "http",
      "proxy": "basic"
    }
  }
}

as on sync, url is the url actually scraped (post-redirect, in normalized form) and requested_url echoes the url you submitted, verbatim — see the sync response reference for the full field list.

page_status_code is the site’s own status for the page. a job that got a 404 or 503 page from the site ends as done, not failed: the page is data, and its result carries the site’s status. check page_status_code before you use the content, and see what counts as a failure for which statuses are billed.

if the job doesn’t exist, the result endpoint returns 404. if it hasn’t finished yet, it returns 400 scrape_error; if it failed, 400 job_failed.

if you asked for extract.elements, the result carries elements too, like a sync response.

24 hours. the status and result of a job are available for 24 hours after you submit it. after that, GET /api/scrape/status/{job_id} and GET /api/scrape/result/{job_id} return the same 404 as an unknown job — re-run the scrape to get a fresh result. screenshot links in the result expire at the same moment, however late you fetch it, so download the images when you get the result.

zero data retention. the result of a job submitted with zero data retention is also deleted after those 24 hours.

full workflow

end-to-end examples that submit a job, poll until completion, and fetch the result.

js/ts


async function asyncScrape(url, extract) {
  const API_BASE = 'https://api.crawlbrulee.com'
  const headers = {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  }
 
  // 1. Submit the job
  const submitRes = await fetch(`${API_BASE}/api/scrape/async`, {
    method: 'POST',
    headers,
    body: JSON.stringify({ url, extract }),
  })
 
  if (!submitRes.ok) {
    throw new Error(`Submit failed: ${submitRes.status} ${await submitRes.text()}`)
  }
 
  const { job_id } = await submitRes.json()
  console.log(`Job submitted: ${job_id}`)
 
  // 2. Poll for status
  let delay = 2000 // start at 2 seconds
  const maxDelay = 30000
 
  while (true) {
    await new Promise((resolve) => setTimeout(resolve, delay))
 
    const statusRes = await fetch(`${API_BASE}/api/scrape/status/${job_id}`, {
      headers,
    })
 
    if (!statusRes.ok) {
      throw new Error(`Status check failed: ${statusRes.status}`)
    }
 
    const { status, error } = await statusRes.json()
    console.log(`Status: ${status}`)
 
    if (status === 'done') break
    if (status === 'failed') throw new Error(`Job failed: ${error}`)
 
    // Exponential backoff, capped at maxDelay
    delay = Math.min(delay * 1.5, maxDelay)
  }
 
  // 3. Fetch the result
  const resultRes = await fetch(`${API_BASE}/api/scrape/result/${job_id}`, {
    headers,
  })
 
  if (!resultRes.ok) {
    throw new Error(`Result fetch failed: ${resultRes.status}`)
  }
 
  return resultRes.json()
}
 
// Usage
const result = await asyncScrape('https://example.com', {
  markdown: true,
  links: true,
})
console.log(result.markdown)

js/ts sdk


import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const { job_id } = await crawlbrulee.scrapeAsync({
  url: 'https://example.com',
  extract: { markdown: true, links: true },
})
 
// waitForScrape handles the polling loop for you
const result = await crawlbrulee.waitForScrape(job_id, {
  intervalMs: 2000,
  timeoutMs: 5 * 60 * 1000,
})
 
console.log(result.markdown)

python


import os
import time
import requests
 
def async_scrape(url: str, extract: dict) -> dict:
    api_base = "https://api.crawlbrulee.com"
    headers = {
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    }
 
    # 1. Submit the job
    submit_res = requests.post(
        f"{api_base}/api/scrape/async",
        headers=headers,
        json={"url": url, "extract": extract},
    )
    submit_res.raise_for_status()
 
    job_id = submit_res.json()["job_id"]
    print(f"Job submitted: {job_id}")
 
    # 2. Poll for status
    delay = 2  # start at 2 seconds
    max_delay = 30
 
    while True:
        time.sleep(delay)
 
        status_res = requests.get(
            f"{api_base}/api/scrape/status/{job_id}",
            headers=headers,
        )
        status_res.raise_for_status()
 
        status_data = status_res.json()
        status = status_data["status"]
        print(f"Status: {status}")
 
        if status == "done":
            break
        if status == "failed":
            raise Exception(f"Job failed: {status_data.get('error', 'Unknown error')}")
 
        # Exponential backoff, capped at max_delay
        delay = min(delay * 1.5, max_delay)
 
    # 3. Fetch the result
    result_res = requests.get(
        f"{api_base}/api/scrape/result/{job_id}",
        headers=headers,
    )
    result_res.raise_for_status()
 
    return result_res.json()
 
# usage
result = async_scrape("https://example.com", {
    "markdown": True,
    "links": True,
})
print(result["markdown"])

python sdk


from crawlbrulee import Crawlbrulee, ScrapeExtract
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
job = client.scrape_async(
    url="https://example.com",
    extract=ScrapeExtract(markdown=True, links=True),
)
 
# wait_for_scrape handles the polling loop for you
result = client.wait_for_scrape(job.job_id)
 
print(result.markdown)

polling best practices

poll at 1-2 second intervals. faster polling wastes your non-scrape rate limit (999/min). use exponential backoff for long-running jobs — start at 2 seconds and increase by 1.5x each iteration, capping at around 30 seconds.

async jobs share your org’s concurrency pool with sync requests. a job may sit in pending if your concurrency limit is reached — it will execute automatically when a slot opens.

receiving results via webhook

the more scalable alternative to polling: pass a webhook object when you submit, and we’ll send a single signed POST to your url the moment the job finishes — success or failed (cancelled is reserved; there is no cancel endpoint today). no polling loop, no wasted status calls — well suited to production pipelines and high job volumes.


{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "webhook": {
    "url": "https://your-app.com/hooks/crawlbrulee",
    "metadata": { "order_id": "A-1234" }
  }
}

see completion webhooks for the payload shape, signature verification, and key rotation.

error handling

submit rejected. POST /api/scrape/async can fail before a job is ever created. a 400 with name: "usage_allocation_error" means the request itself was rejected — details.reason is credit_limit or concurrency_limit. a 429 means you’re being rate-limited (or hit the same usage-allocation limits under load). a 403 with name: "zero_data_retention_not_enabled" means you sent zero_data_retention: true, but it is not enabled for your organization; it costs nothing (see zero data retention). see errors for the full error format and credits & pricing for what each limit means and how to raise it.

job not found. both the status and result endpoints return 404 if the job_id doesn’t match any known job, or if more than 24 hours have passed since you submitted it (see step 3). double-check the id and make sure you’re hitting the same environment (production vs staging) where the job was submitted.

request interrupted. submit, status, and result requests can return 408 request_timeout when the server-side request deadline elapses. a disconnected client is represented as 499 client_closed_request when a response can still be sent.

service temporarily unavailable. submit, status, and result can all return 503 service_unavailable when a dependency we need is briefly unavailable. the request itself was fine — retry it unchanged, backing off between attempts. this is never a signal to rotate your api key; a key that’s genuinely unknown, expired, or revoked comes back as 401 invalid_credentials.

job failed. the status response includes status: "failed" and a generic error field. a job fails only when we could not get a page — for example, the site blocked us or could not be reached — or something broke on our side. a page the site served with an error status is not a failure: the job ends as done with that page. retry transient failures; if a job keeps failing, contact support with the job_id so the underlying failure can be investigated without exposing service details in the public response.

screenshot-only jobs. the result path follows the same screenshot semantics as sync: if the job asked only for a screenshot and none can be delivered, the result endpoint returns 422 unsupported_screenshot_output (content type can’t be screenshotted) or 500 (capture failed) instead of an empty 200 — and you’re not billed. see when no screenshot can be delivered.

credits. when the job is done you’re charged for the engine, proxy, slice-variant increment, and zero data retention credit that apply to the delivered result, unless the page’s page_status_code is one we don’t bill (403, 407, 408, 429, 451, or any 5xx), which costs 0. a failed job costs 0 credits. a cache hit has a 0-credit base; generating a new slice variant adds one flat credit (for live or cached scrapes) no matter how many slices the split produces.

when to use async

processing many urls in a pipeline
scraping pages that take a long time to render
building queue-driven architectures
when you don’t need the result immediately

for single, fast pages — use sync scrape instead.