extraction options

we return the page metadata and cleaned_html by default. use the extract object to add more formats — markdown, raw_html, links, images, screenshot — or to turn off a default. if you don’t pass an extract object, you get metadata + cleaned_html back.

every option below is a boolean flag on the extract object (except screenshot, which is covered on its own page).

markdown

set extract.markdown: true to get a cleaned markdown representation of the page content. scripts, navigation, footers, and other non-content elements are stripped, then the remaining html is converted to markdown.

best for: ai/llm pipelines, content indexing, rag systems.

request

curl -X POST https://api.crawlbrulee.com/api/scrape \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "extract": {
      "markdown": true
    }
  }'

const response = await fetch('https://api.crawlbrulee.com/api/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://example.com',
    extract: {
      markdown: true,
    },
  }),
})
 
const data = await response.json()
console.log(data)

import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const page = await crawlbrulee.scrape({
  url: 'https://example.com',
  extract: { markdown: true },
})
 
console.log(page.markdown)

import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/scrape",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://example.com",
        "extract": {
            "markdown": True,
        },
    },
)
 
data = response.json()
print(data)

from crawlbrulee import Crawlbrulee, ScrapeExtract
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
page = client.scrape(
    url="https://example.com",
    extract=ScrapeExtract(markdown=True),
)
 
print(page.markdown)

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "content_type": "text/html",
  "markdown": "# Example Domain\n\nThis domain is for use in illustrative examples in documents...",
  "metadata": { "title": "Example Domain" }
}

the cleaned markdown is returned in the markdown field. full responses also carry a response_meta object reporting credits, engine, and proxy tier — see the sync response reference.

cleaned html

cleaned html (the cleaned_html field) is returned by default. scripts, styles, and non-content elements are removed, and your cleanup block is applied before the result is built. to skip it, set extract.cleaned_html: false.

request

{
  "url": "https://example.com",
  "extract": {
    "cleaned_html": true
  }
}

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "content_type": "text/html",
  "cleaned_html": "<h1>Example Domain</h1><p>This domain is for use in illustrative examples...</p>",
  "metadata": { "title": "Example Domain" }
}

the cleaned html is returned in the cleaned_html field.

raw html

set extract.raw_html: true to get the page’s source html without the readability cleaning and markdown conversion applied to cleaned_html and markdown. nothing is stripped for readability — you get the full document as returned by the server or browser.

one thing to know: cleanup never applies to raw_html. neither ads_and_popups nor your own exclude_selectors touch it — raw_html is always the page as it arrived, before anything was removed. that is deliberate: it means you can always get back what we started from, whatever cleanup you asked for. for non-html responses, raw_html is the untouched fetched body.

use this when you need to parse the page yourself or need access to elements that the cleaning step would remove.

request

{
  "url": "https://example.com",
  "extract": {
    "raw_html": true
  }
}

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "content_type": "text/html",
  "raw_html": "<!doctype html>\n<html>\n<head>\n    <title>Example Domain</title>...</html>",
  "metadata": { "title": "Example Domain" }
}

the raw source is returned in the raw_html field.

links

set extract.links: true to get every link on the page. each link includes:

beyond absolutization, hrefs aren’t cleaned: duplicates and self-links survive, and deduplication is up to you. empty, unparseable, and non-http(s) hrefs (mailto:, tel:, javascript:) are dropped.

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "content_type": "text/html",
  "links": [
    {
      "text": "More information...",
      "href": "https://www.iana.org/domains/example",
      "internal": false
    },
    {
      "text": "About Us",
      "href": "https://example.com/about",
      "internal": true
    }
  ],
  "metadata": { "title": "Example Domain" }
}

images

set extract.images: true to get every image on the page. each image includes:

url is always absolute: document-relative src values are resolved against the full page url (browser-parity, matching link extraction), and any query string on the original src is preserved.

response (condensed)

{
  "url": "https://example.com",
  "requested_url": "https://example.com",
  "content_type": "text/html",
  "images": [
    {
      "url": "https://example.com/logo.png",
      "alt": "Example logo"
    },
    {
      "url": "https://example.com/hero.jpg",
      "alt": null
    }
  ],
  "metadata": { "title": "Example Domain" }
}

metadata

metadata is returned by default on every scrape request. you don’t need to opt into it — it’s always there unless you explicitly disable it with extract.metadata: false.

all fields are optional — availability depends on what the target page provides. fields that aren’t present on the page are omitted from the response entirely, with one exception: favicon_url is a best-effort lookup (checked against <head> icon hints and the manifest, falling back to /favicon.ico) and can be returned as null when no favicon is found, instead of being omitted.

limits and truncation

extraction is bounded per page, so a single pathological page can’t produce an unbounded response. the caps are generous — real pages sit far below them — and when one is reached we truncate that output and tell you, in the response’s top-level warnings array. nothing is capped silently.

{
  "warnings": ["links_truncated", "raw_html_truncated"]
}

warnings is a flat array of stable string codes — switch on the code, don’t match on message text. truncated html is cut back to a tag boundary, so what you get still parses.

when a field can’t be extracted

truncation means the field is there but capped. the other half of warnings covers the opposite case: an optional field whose extraction failed outright. rather than failing your whole scrape over one section, we return everything else and name what’s missing.

this is what lets you tell “the page genuinely had no links” apart from “we couldn’t read them” — an empty array alone can’t distinguish the two.

the page body is the exception. if the body can’t be extracted there’s nothing worth returning, so the scrape fails with an error instead of handing you a hollow 200 — and a failed scrape isn’t billed.

which warnings come back

warnings is stored alongside the result, so it survives past the original call: cache hits and async result fetches carry the same codes a fresh scrape would.

they’re filtered to the outputs you actually asked for — request markdown only and you won’t hear about the page’s links. raw_html_truncated is the exception and always surfaces, because a truncated body feeds markdown and cleaned_html too.

excluding content

the cleanup block says what comes off the page before anything is built from it. it has two fields.

ads_and_popups is on by default and removes ads, cookie banners, consent dialogs and chat widgets. set it false when you want the page as-is — or when a site refuses to serve content to a client that looks like an ad blocker.

exclude_selectors takes an array of css selectors for anything the default doesn’t catch: your own navigation, footers, sidebars, or site-specific noise. at most 100 selectors, each at most 500 characters.

both shape markdown, cleaned_html, links, images and the screenshot. neither touches raw_html.

{
  "url": "https://example.com",
  "extract": { "markdown": true },
  "cleanup": {
    "ads_and_popups": true,
    "exclude_selectors": ["nav", "footer", ".cookie-banner", "#sidebar"]
  }
}

disabling the defaults

metadata and cleaned_html are on by default. set either to false inside extract to skip it and reduce response size.

{
  "url": "https://example.com",
  "extract": { "markdown": true, "metadata": false, "cleaned_html": false }
}

this is useful when you only care about one format — here, markdown — and want the leanest possible response.