extraction options

we return the page metadata and cleaned_html by default. use the extract object to add more formats — markdown, raw_html, links, images, screenshot — or to turn off a default. if you don’t pass an extract object, you get metadata + cleaned_html back. to get just a few named values from the page instead, use elements.

every option below is a boolean flag on the extract object, except screenshot and elements, which each have their own page: screenshots and elements.

markdown

set extract.markdown: true to get a cleaned markdown representation of the page content. scripts, styles, ads and cookie banners are removed, then the remaining html is converted to markdown. navigation and footers stay; to drop them, pass their selectors in exclude_selectors (see excluding content).

best for: ai/llm pipelines, content indexing, rag systems.

request

curl -X POST https://api.crawlbrulee.com/api/scrape \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": {
      "markdown": true
    }
  }'

const response = await fetch('https://api.crawlbrulee.com/api/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    extract: {
      markdown: true,
    },
  }),
})
 
const data = await response.json()
console.log(data)

import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const page = await crawlbrulee.scrape({
  url: 'https://example.com',
  extract: { markdown: true },
})
 
console.log(page.markdown)

import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/scrape",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "extract": {
            "markdown": True,
        },
    },
)
 
data = response.json()
print(data)

from crawlbrulee import Crawlbrulee, ScrapeExtract
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
page = client.scrape(
    url="https://example.com",
    extract=ScrapeExtract(markdown=True),
)
 
print(page.markdown)

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
  "metadata": { "title": "Example Domain" }
}

the cleaned markdown is returned in the markdown field. full responses also carry a response_meta object reporting credits, engine, and proxy tier — see the sync response reference.

cleaned html

cleaned html (the cleaned_html field) is returned by default. scripts, styles, iframes and html comments are removed, and your cleanup block is applied before the result is built. to skip it, set extract.cleaned_html: false.

request

{
  "url": "https://example.com",
  "extract": {
    "cleaned_html": true
  }
}

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "cleaned_html": "<h1>Example Domain</h1><p>This domain is for use in illustrative examples...</p>",
  "metadata": { "title": "Example Domain" }
}

the cleaned html is returned in the cleaned_html field.

raw html

set extract.raw_html: true to get the page’s source html without the readability cleaning and markdown conversion applied to cleaned_html and markdown. nothing is stripped for readability — you get the full document as returned by the server or browser.

one thing to know: cleanup never applies to raw_html. neither ads_and_popups nor your own exclude_selectors touch it — raw_html is always the page as it arrived, before anything was removed. that is deliberate: it means you can always get back what we started from, whatever cleanup you asked for. for non-html responses, raw_html is the untouched fetched body.

use this when you need to parse the page yourself or need access to elements that the cleaning step would remove.

request

{
  "url": "https://example.com",
  "extract": {
    "raw_html": true
  }
}

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "raw_html": "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>...</html>",
  "metadata": { "title": "Example Domain" }
}

the raw source is returned in the raw_html field.

links

set extract.links: true to get every link on the page. each link includes:

beyond absolutization, hrefs aren’t cleaned: duplicates and self-links survive, and deduplication is up to you. empty, unparseable, and non-http(s) hrefs (mailto:, tel:, javascript:) are dropped.

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "links": [
    {
      "text": "More information...",
      "href": "https://www.iana.org/domains/example",
      "internal": false
    },
    {
      "text": "About Us",
      "href": "https://example.com/about",
      "internal": true
    }
  ],
  "metadata": { "title": "Example Domain" }
}

images

set extract.images: true to get every image on the page. each image includes:

url is always absolute: document-relative src values are resolved against the full page url (browser-parity, matching link extraction), and any query string on the original src is preserved.

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "page_status_code": 200,
  "content_type": "text/html",
  "images": [
    {
      "url": "https://example.com/logo.png",
      "alt": "Example logo"
    },
    {
      "url": "https://example.com/hero.jpg",
      "alt": null
    }
  ],
  "metadata": { "title": "Example Domain" }
}

metadata

metadata is returned by default on every scrape request. you don’t need to opt into it — it’s always there unless you explicitly disable it with extract.metadata: false.

all fields are optional — availability depends on what the target page provides. fields that aren’t present on the page are omitted from the response entirely, with one exception: favicon_url is a best-effort lookup (checked against <head> icon hints and the manifest, falling back to /favicon.ico) and can be returned as null when no favicon is found, instead of being omitted.

elements

set extract.elements to read named values from the page by css selector: a heading, a price, or one object per product card. the values come back in the top-level elements field, under the names you picked. they cost no extra credits.

{
  "url": "https://books.toscrape.com/",
  "extract": {
    "elements": {
      "heading": "h1",
      "prices": { "selector": ".price_color", "all": true }
    }
  }
}

response (condensed: prices has 20 items, the first 3 are shown)

{
  "elements": {
    "heading": "All products",
    "prices": ["£51.77", "£53.74", "£50.10"]
  }
}

the request forms, fields, the selector rules and the limits are on the elements page.

limits and truncation

extraction is bounded per page, so a single pathological page can’t produce an unbounded response. the caps are generous — real pages sit far below them — and when one is reached we truncate that output and tell you, in the response’s top-level warnings array. nothing is capped silently.

{
  "warnings": ["links_truncated", "raw_html_truncated"]
}

warnings is a flat array of stable string codes — switch on the code, don’t match on message text. truncated html is cut back to a tag boundary, so what you get still parses.

metadata_truncated is retired and no longer sent: metadata has no size limit of its own any more. you can still see it on a result stored before that change.

when a field can’t be extracted

truncation means the field is there but capped. the other half of warnings covers the opposite case: an optional field whose extraction failed outright. rather than failing your whole scrape over one section, we return everything else and name what’s missing.

this is what lets you tell “the page genuinely had no links” apart from “we couldn’t read them” — an empty array alone can’t distinguish the two.

the page body is the exception. if the body can’t be extracted there’s nothing worth returning, so the scrape fails with an error instead of handing you a hollow 200 — and a failed scrape isn’t billed.

in elements, a null value (or [] with all: true) has more than one cause: nothing matched, the match has no such attribute, or the link was unsafe, such as javascript:. a value too big for the size limit is null too, and then warnings carries elements_truncated. on a json, plain text, xml or markdown page, elements is left out and unsupported_fields lists it. see elements for the details.

which warnings come back

warnings is stored alongside the result, so it survives past the original call: cache hits and async result fetches carry the same codes a fresh scrape would.

they’re filtered to the outputs you actually asked for — request markdown only and you won’t hear about the page’s links. raw_html_truncated is the exception and always surfaces, because a truncated body feeds markdown and cleaned_html too.

excluding content

the cleanup block says what comes off the page before anything is built from it. it has two fields.

ads_and_popups is on by default and removes ads, cookie banners, consent dialogs and chat widgets. set it false when you want the page as-is — or when a site refuses to serve content to a client that looks like an ad blocker.

exclude_selectors takes an array of css selectors for anything the default doesn’t catch: your own navigation, footers, sidebars, or site-specific noise. at most 100 selectors, each at most 500 characters.

both shape markdown, cleaned_html, links, images, elements and the screenshot. neither touches raw_html.

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cleanup": {
    "ads_and_popups": true,
    "exclude_selectors": ["nav", "footer", ".cookie-banner", "#sidebar"]
  }
}

disabling the defaults

metadata and cleaned_html are on by default. set either to false inside extract to skip it and reduce response size.

{
  "url": "https://example.com",
  "extract": { "markdown": true, "metadata": false, "cleaned_html": false }
}

this is useful when you only care about one format — here, markdown — and want the leanest possible response.