elements

use extract.elements when you know which parts of a page you want. you give each value a name and a css selector. we send back just those values, under the same names, in the top-level elements field of the result. no need to parse the whole page yourself.

elements cost no extra credits, also when the page comes from cache. see cost.

a first example

this request reads the page heading, every book on the page (with its title, price and link), and the link to the next page:

curl -X POST https://api.crawlbrulee.com/api/scrape \
  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://books.toscrape.com/",
    "extract": {
      "elements": {
        "heading": "h1",
        "books": {
          "selector": "article.product_pod",
          "all": true,
          "fields": {
            "title": { "selector": "h3 a", "output": "attribute", "attribute": "title" },
            "price": ".price_color",
            "url": { "selector": "h3 a", "output": "attribute", "attribute": "href" }
          }
        },
        "next_page": { "selector": "li.next a", "output": "attribute", "attribute": "href" }
      }
    }
  }'

const response = await fetch('https://api.crawlbrulee.com/api/scrape', {
  method: 'POST',
  headers: {
    Authorization: `Bearer ${process.env.CRAWLBRULEE_API_KEY}`,
    'Content-Type': 'application/json',
  },
  body: JSON.stringify({
    url: 'https://books.toscrape.com/',
    extract: {
      elements: {
        heading: 'h1',
        books: {
          selector: 'article.product_pod',
          all: true,
          fields: {
            title: { selector: 'h3 a', output: 'attribute', attribute: 'title' },
            price: '.price_color',
            url: { selector: 'h3 a', output: 'attribute', attribute: 'href' },
          },
        },
        next_page: { selector: 'li.next a', output: 'attribute', attribute: 'href' },
      },
    },
  }),
})
 
const data = await response.json()
console.log(data.elements.books)

import { Crawlbrulee } from '@crawlbrulee/sdk'
 
// reads CRAWLBRULEE_API_KEY from the environment
const crawlbrulee = Crawlbrulee.fromEnv()
 
const page = await crawlbrulee.scrape({
  url: 'https://books.toscrape.com/',
  extract: {
    elements: {
      heading: 'h1',
      books: {
        selector: 'article.product_pod',
        all: true,
        fields: {
          title: { selector: 'h3 a', output: 'attribute', attribute: 'title' },
          price: '.price_color',
          url: { selector: 'h3 a', output: 'attribute', attribute: 'href' },
        },
      },
      next_page: { selector: 'li.next a', output: 'attribute', attribute: 'href' },
    },
  },
})
 
console.log(page.elements?.books)

import os
import requests
 
response = requests.post(
    "https://api.crawlbrulee.com/api/scrape",
    headers={
        "Authorization": f"Bearer {os.environ['CRAWLBRULEE_API_KEY']}",
        "Content-Type": "application/json",
    },
    json={
        "url": "https://books.toscrape.com/",
        "extract": {
            "elements": {
                "heading": "h1",
                "books": {
                    "selector": "article.product_pod",
                    "all": True,
                    "fields": {
                        "title": {"selector": "h3 a", "output": "attribute", "attribute": "title"},
                        "price": ".price_color",
                        "url": {"selector": "h3 a", "output": "attribute", "attribute": "href"},
                    },
                },
                "next_page": {"selector": "li.next a", "output": "attribute", "attribute": "href"},
            },
        },
    },
)
 
data = response.json()
print(data["elements"]["books"])

from crawlbrulee import Crawlbrulee, ScrapeElementSpec, ScrapeExtract
 
# reads CRAWLBRULEE_API_KEY from the environment
client = Crawlbrulee.from_env()
 
page = client.scrape(
    url="https://books.toscrape.com/",
    extract=ScrapeExtract(
        elements={
            "heading": "h1",
            "books": ScrapeElementSpec(
                selector="article.product_pod",
                all=True,
                fields={
                    "title": ScrapeElementSpec(selector="h3 a", output="attribute", attribute="title"),
                    "price": ".price_color",
                    "url": ScrapeElementSpec(selector="h3 a", output="attribute", attribute="href"),
                },
            ),
            "next_page": ScrapeElementSpec(selector="li.next a", output="attribute", attribute="href"),
        },
    ),
)
 
books = page.elements["books"] if page.elements is not None else []
if isinstance(books, list):  # `all=True` gives a list
    for book in books:
        if isinstance(book, dict):  # `fields` gives an object per match
            print(book["title"], book["price"])

response (condensed: books has 20 items, the first 3 are shown)

{
  "url": "https://books.toscrape.com/",
  "requested_url": "https://books.toscrape.com/",
  "page_status_code": 200,
  "content_type": "text/html",
  "elements": {
    "heading": "All products",
    "books": [
      {
        "title": "A Light in the Attic",
        "price": "£51.77",
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html"
      },
      {
        "title": "Tipping the Velvet",
        "price": "£53.74",
        "url": "https://books.toscrape.com/catalogue/tipping-the-velvet_999/index.html"
      },
      {
        "title": "Soumission",
        "price": "£50.10",
        "url": "https://books.toscrape.com/catalogue/soumission_998/index.html"
      }
    ],
    "next_page": "https://books.toscrape.com/catalogue/page-2.html"
  }
}

the page links to each book with a short relative path. url and next_page come back as full urls, because href and src values always do.

elements comes on top of the other outputs. metadata and cleaned_html stay on by default, so the full response carries them too. if you only want the values, turn them off:

{
  "url": "https://books.toscrape.com/",
  "extract": {
    "metadata": false,
    "cleaned_html": false,
    "elements": { "heading": "h1" }
  }
}

how to ask for a value

extract.elements is an object. each key is a name you pick, and it comes back as a key of elements. each value is a selector string or an object.

a selector string

the short form. you get the text of the first match.

{ "heading": "h1", "price": ".price_color" }

an object

the long form, when you want more than the text of the first match:

{
  "next_page": { "selector": "li.next a", "output": "attribute", "attribute": "href" },
  "all_prices": { "selector": ".price_color", "all": true },
  "first_book_html": { "selector": "article.product_pod", "output": "html" }
}

fields: one object per match

when a value has parts, like a product card with a title, a price and a link, use fields. each field is a name and a selector (string or object), read inside that match only. so every card gives you one object, and its title, price and link always belong together.

a field can’t read the match itself. if you match <article> and a field asks for an attribute of that <article>, it comes back null. read it with output: "attribute" on the outer spec instead, or match one level higher.

with all: true you get a list of objects, one per match. without it, you get one object from the first match.

fields can have fields of their own, up to 3 levels deep. the innermost level has no fields. here a category page lists sections, each with products, each with tags:

{
  "sections": {
    "selector": "section.category",
    "all": true,
    "fields": {
      "name": "h2",
      "products": {
        "selector": ".product",
        "all": true,
        "fields": {
          "title": "h3",
          "tags": { "selector": ".tag", "all": true }
        }
      }
    }
  }
}

a spec with fields can’t also set output or attribute, and fields needs at least one field. either mistake is a 400.

what comes back

the values come back in the top-level elements field, under the names you sent:

every name you asked for is always there, at every level of fields. a name with nothing to read is null (or [] with all: true), never missing. so you can read book["price"] without checking first that the key exists.

with output: "attribute", a match that doesn’t have the attribute gives null, and is left out of an all: true list.

selector rules

• standard css selectors. tags, classes, ids, attribute selectors, combinators, and pseudo-classes such as :nth-child(), :not() and :has() all work.
• pseudo-elements and invalid selectors are refused. a selector like p::before, or one with a typo, is a 400 before we fetch the page, so it costs nothing (see when a request is refused).
• values are read after cleanup. cleanup.ads_and_popups (on by default) and your own cleanup.exclude_selectors remove parts of the page first, so a selector that points into a removed part matches nothing. see excluding content. elements are read from the same page as links and images.
• <script> and <style> can’t be selected. they are not part of the page we read, so JSON-LD (script[type="application/ld+json"]) and data inside inline scripts come back null. use extract.raw_html when you need those.
• attribute names match in any case. data-sku and DATA-SKU read the same attribute.
• href and src come back as full urls, resolved against the page, like links and images. a link to a #fragment on the same page stays as written, such as #reviews.
• unsafe links are dropped. with output: "attribute", a link such as javascript:… gives null, and is left out of an all: true list. so two all: true lists from the same page can have different lengths, and pairing them by position can match the wrong values. read values that belong together as fields of one match instead.
• tables are read the way a browser reads them. a browser adds <tbody> to a table when the page’s html leaves it out. so table > tbody > tr finds the rows either way, whether or not the page was rendered with JavaScript.
• fields look only inside the match. a field selector that points outside the match finds nothing. that includes one that starts with + or ~, because a sibling is outside the match, and the match itself: an attribute of the matched element comes back null from a field (see fields).

limits

more than 50 selectors, or a selector over 500 characters, is a 400.

the other limits never fail the request. when a value hits one, the response’s warnings array carries elements_truncated:

{
  "warnings": ["elements_truncated"]
}

it means at least one of these happened:

• a list stopped at 1,000 matches.
• the request reached 10,000 matches, so later matches were left out.
• a value didn’t fit in the 1 MB or so of values a response can carry. that value is null: a value is never cut short.

output: "html" on a large match is the usual way to reach the size limit. ask for the parts you need with fields instead.

pages that aren’t html

elements need an html page. for json, plain text, markdown or xml, the response has no elements field and unsupported_fields includes "elements". the rest of the scrape comes back as usual.

{
  "content_type": "application/json",
  "unsupported_fields": ["elements"]
}

cost

elements cost no extra credits. a scrape with elements costs what the same scrape costs without them (see credits & pricing).

that holds for a cache hit too. extract is not part of the cache key, so a cached page can answer a request with any elements. we read your values from the cached page, and the hit still costs 0. to read the live page instead, send cache.max_age: 0.

async jobs

POST /api/scrape/async takes the same extract.elements. the result from GET /api/scrape/result/{job_id} carries elements, like the rest of the result, for the same 24 hours (see scrape (async)).

the scrape.complete webhook doesn’t carry results, elements included. fetch the result when the webhook arrives.

when a request is refused

a request that breaks the rules above is refused with a 400 validation_error before we fetch the page, so it costs nothing. that covers things like a selector that isn’t valid css, more than 50 selectors, or fields together with output. the message says what’s wrong and where. for example, a typo in a pseudo-class:

{
  "name": "validation_error",
  "message": "Request validation failed: extract.elements.books.fields.price: invalid CSS selector: Unknown pseudo-class :frist-child"
}

switch on name, not on the message text. see errors for the error format.

in the sdks, cli and mcp server

• js/ts sdk: extract: { elements: { … } }, and the values on page.elements. see js/ts sdk.
• python sdk: ScrapeExtract(elements={…}). each value is a selector string, a ScrapeElementSpec, or the same spec as a plain dict (ScrapeElementSpecDict). the values come back on page.elements. see python sdk.
• cli: --element name=selector (repeatable), or --elements with the full object as json. on their own, they return only the elements (plus metadata unless --no-metadata); add -m or another content flag to get the page too. see cli.
• mcp server: the scrape and scrape_async tools take extract.elements. see mcp server.