how to scrape ecommerce product data

welcome to crawl academy 🎓. in this series we take one real website per lesson and turn its pages into clean data you can use. every lesson comes with working TypeScript code in our public crawl-academy repo. this post shows the important pieces, trimmed for reading; clone the repo for the full code.

lesson one is the most common job in web scraping: one product page in an online shop. we picked saturn.de, one of Germany's biggest electronics shops, and a product most people know: Apple's AirPods 4.

what you'll get

one run of the script gives you the product as json:

1{
2  "name": "APPLE AirPods 4 mit Aktiver Geräuschunterdrückung, In-ear Kopfhörer Bluetooth Weiß",
3  "article_number": "2954282",
4  "price": 159,
5  "currency": "EUR",
6  "rating": 4.8,
7  "review_count": 2172,
8  "variants": [
9    { "name": "Airpods 4 mit ANC", "current": true, "url": "https://www.saturn.de/de/product/…-2954282.html" },
10    { "name": "Airpods 4", "current": false, "url": "https://www.saturn.de/de/product/…-2954343.html" },
11    { "name": "Airpods Pro 3", "current": false, "url": "https://www.saturn.de/de/product/…-3013574.html" }
12  ],
13  "offers": [
14    { "name": "Neues Produkt", "price": 159 },
15    { "name": "Refurbished Produkte", "price": 130.08 }
16  ],
17  "images": ["https://assets.mmsrg.com/isr/166325/c1/-/ASSET_MMS_144665910", "…"]
18}

and, from the same request, a clean screenshot of the page, with no cookie banner in the way. here is the product, cropped from our screenshot:

before you start

• Node.js 22.18 or later. it runs .ts files directly, so there is no build step.
• pnpm. if you don't have it yet: npm install -g pnpm.
• a crawlbrulee api key. the free plan has 750 credits a month and needs no credit card.

1git clone https://github.com/crawlbrulee/crawl-academy.git
2cd crawl-academy/01-saturn-product-page
3pnpm install
4export CRAWLBRULEE_API_KEY=cwbl_…
5pnpm scrape

step 1: name the values you want

the usual way to scrape a shop is to download the html, load it into a parser and walk the tree yourself. with crawlbrulee you skip that part. you list the values you want, each with a css selector, and they come back by name:

1import { Crawlbrulee, type ScrapeElements } from '@crawlbrulee/sdk'
2
3const elements: ScrapeElements = {
4  name: 'h1',
5  price: "[data-test='mms-product-price'] [data-test='cofr-price mms-branded-price']",
6  rating: { selector: "a[href='#reviews'] [aria-label*='Basierend auf']", output: 'attribute', attribute: 'aria-label' },
7  variants: { selector: "a[data-sctrack='variant-tile-link']", all: true },
8  images: { selector: "[data-test='mms-pdp-gallery'] img", output: 'attribute', attribute: 'src', all: true },
9}
10
11const crawlbrulee = Crawlbrulee.fromEnv()
12const page = await crawlbrulee.scrape({ url: productUrl, extract: { elements } })
13
14console.log(page.elements.name)
15// "APPLE AirPods 4 mit Aktiver Geräuschunterdrückung, In-ear Kopfhörer Bluetooth Weiß"

three things to know:

• a plain selector string returns the text of the first match.
• output: 'attribute' reads an attribute, like an image src. links and image urls come back as full urls.
• all: true returns every match as a list. that's how you get all variants and all gallery images.

how to find the selectors: open the page in your browser, right-click the price, and pick "inspect". saturn.de marks its parts with data-test attributes, which change less often than class names. that makes them good selectors. in the repo, the selectors sit in their own file, so when the shop changes its page, you update one file.

step 2: leave the proxy on auto

the request in step 1 has no proxy setting, and that's on purpose. leave it on auto, the default, and we pick the proxy for you: we start with the basic proxy and move up to the advanced proxy when the basic one doesn't get the page. for saturn.de most requests end up on the advanced proxy, but some still go through via the basic proxy. you pay for the tier that got the page, and we tell you in the response which one it was.

two more settings go into the request:

1const page = await crawlbrulee.scrape({
2  url: productUrl,
3  location: { country: 'de', locale: 'de-DE' },
4  cache: { max_age: 0 },
5  extract: { elements },
6})

• location asks for the page the way a shopper in Germany sees it, in German, with prices in euro.
• cache: { max_age: 0 } asks for a fresh copy. prices change, so you want a fresh copy, not an older one from our cache.

step 3: turn the shop's text into numbers

the values come back the way the shop shows them, so src/parse.ts cleans them up and builds the Product from src/types.ts:

• the price box says 159, – € 159,00€ inkl. MwSt. the script takes the full price, 159,00, and turns the German number format into 159:

1// we use regex to convert a value from "159, – € 159,00€ inkl. MwSt." to
2// a number we can use, e.g. 159
3const [, amount] = text.match(/(\d{1,3}(?:\.\d{3})*,\d{2})\s*€/)
4const price = Number(amount.replace(/\./g, '').replace(',', '.'))

• the rating label says 4.8 von 5 Sternen, Basierend auf 2172 Bewertungen. one regular expression gives you the average and the review count.
• the gallery can list a picture twice, large and as a thumbnail, with resize settings in the url. the script keeps one link per picture, without the settings. that link opens the full-size image.

step 4: add a screenshot to the same request

the data is ready. now for a picture of the page. a screenshot needs a real browser, but not a second request: add screenshot to the same extract block, and we load the page in a browser, take the picture and read your elements from that same page.

1const page = await crawlbrulee.scrape({
2  url: productUrl,
3  location: { country: 'de', locale: 'de-DE' },
4  cache: { max_age: 0 },
5  extract: {
6    elements,
7    screenshot: { type: 'viewport', viewport: { width: 1440, height: 1000 } },
8  },
9})
10
11// the link expires 24 hours after the scrape, so keep the file
12const image = await fetch(page.screenshot.url)
13await writeFile('screenshot.webp', Buffer.from(await image.arrayBuffer()))

what it costs

• product data and a screenshot: the screenshot engine (5 credits) on the advanced proxy (×5), so 25 credits.

one command, one request: 25 credits per product. the values come with the screenshot at no extra cost, and the free plan's 750 credits cover 30 product pages a month. don't need the picture? leave out screenshot, and the same request runs without a browser for 5 credits. a request that fails costs nothing.

scrape with care

before we wrote the script, we checked the site:

• robots.txt allows all visitors on product pages. it closes search and some list pages, and the script doesn't visit them.
• the terms of use don't forbid reading public product pages.
• personal data. reviews are written by people, so the script reads only the average rating and the review count. it keeps no names and no review texts.
• images belong to the shop and the brands. the script keeps links, not copies.
• load. one page, one request per run.

do the same checks for the pages you scrape. this is not legal advice.

make it yours

• another product: pass its url, pnpm scrape "https://www.saturn.de/de/product/_….html".
• a price watch: run it once a day and compare price with yesterday's value.
• when a value comes back null: the shop changed its page. inspect the element in your browser and update its selector in src/selectors.ts. nothing else changes.

next in crawl academy: how to scrape blog posts into markdown, with plain fetch and no sdk at all.

try it on your own pages

crawlbrulee is EU-native web scraping for AI agents and developers. the free plan has 750 credits a month, no credit card needed.

create your free account and run this lesson in five minutes.