web scraping with or without a browser

there are two ways to scrape a web page. you can send one plain request for the page and read the html the server sends back, with no browser involved. or you can open the page in a real browser, let it run its javascript, and read what ends up on the screen. both work, but they break in different places, and one costs a lot more to run than the other. if you use AI agents, this matters for them too: when an agent reads a page for you, it usually takes the first way, the plain request, with the same limits.

this is the third lesson of crawl academy. lessons one and two each took one real site and turned it into clean data. this one steps back to the question that comes before every scraper you write: does this page need a browser?

by the end of this post you'll know how each one works, where each one breaks, how to tell in a minute which one a page needs, and why some pages look empty to your AI agent, even though they look perfectly fine to you when you open them in your browser. along the way, short code samples show each idea. they use a made-up shop at shop.example.com.

how a page reaches your screen

first, let's look at what the server actually sends. when you open a page, your browser asks the site's server for that page's html. the server answers with html, but what's in that html depends on how the site is built.

some sites build the full page on the server and send it back complete, with the content already in it. this is called server-side rendering, or SSR. our made-up shop builds its blue mug page this way, so the html from the server already holds what you see on the screen:

1<html>
2<head>...</head>
3<body>
4  <!-- other content -->
5  ...
6
7  <!-- the data you want -->
8  <div id="product">
9    <h1>blue mug</h1>
10    <span class="price">€12.90</span>
11  </div>
12
13  <!-- more content -->
14  ...
15</body>
16</html>

at the other end, some sites send back html with almost no content, just a script. your browser loads the script and runs it, and the script then gets the data from the site's server and puts it on the page. so the first version of the page the browser gets isn't very useful yet: the data isn't there. this is called client-side rendering, or CSR, and it's quite common. our shop builds its red mug page this way:

1<html>
2<head>...</head>
3<body>
4  <!-- no content yet. the div is empty -->
5  <div id="product"></div>
6
7  <!-- your browser runs this script, and the script fills in the div above -->
8  <script src="/app.js"></script>
9</body>
10</html>

once the script has run, the content is on the page:

1<html>
2<head>...</head>
3<body>
4  <!-- the data you want, put here by the script -->
5  <div id="product">
6    <h1>red mug</h1>
7    <span class="price">€14.90</span>
8  </div>
9
10  <!-- the script stays, even after it has loaded the data -->
11  <script src="/app.js"></script>
12</body>
13</html>

these two pages are the two ends. most sites sit somewhere in between. they build the parts that rarely change on the server, like the menu, the product name and the description, and load the rest with javascript, like the price, the stock, the reviews or the "you may also like" row. others build most pages on the server and only a few pages, or a few products, in the browser. so one site can need both ways of scraping.

on your screen, both mug pages look the same. to a scraper, they are very different:

• the blue mug page has its content in the html the scraper downloads.
• the red mug page only gets its content after the javascript runs. if nothing runs the javascript, the scraper finds nothing useful on the page.

this difference is what the rest of this post is about. by the end of it you'll know what it means for your scraper, and which kind to use, plain http or a browser, for the way a page is built.

step 1: scrape with plain http

let's start with the simple way: one GET request to download the page's html, and a parser to read it. here we use fetch, which is built into Node.js, to send the request, and cheerio, a small library that finds things in html with css selectors:

1import * as cheerio from 'cheerio'
2
3// one GET request: the server sends back the page's html as text
4const res = await fetch('https://shop.example.com/products/blue-mug')
5const html = await res.text()
6
7// cheerio reads the html so we can search it with css selectors.
8// it doesn't run the page's javascript
9const $ = cheerio.load(html)
10
11console.log($('h1').text())     // "blue mug"
12console.log($('.price').text()) // "€12.90"

that's the whole scraper. it makes one request and gets back the page's html. no javascript runs at all, and that's not a setting you forgot: running a page's javascript is a browser's job. fetch only downloads the html as text, and cheerio only reads that text. on a page built on the server, like the blue mug page, that's all you need. it's fast, cheap and hard to beat.

plain http has limits too, and they all come from the same place: there's no browser.

• it only reads. it can't click a button, scroll, fill in a form or take a screenshot. it sees the html the server sent, and nothing more.
• it's easier to spot as a bot. many sites send a small javascript check that a real browser runs and plain http doesn't. and the way a plain http client opens a secure connection often doesn't look like a real browser either.

step 2: when the html comes back empty

now point the same code at the red mug page:

1const res = await fetch('https://shop.example.com/products/red-mug')
2const $ = cheerio.load(await res.text())
3
4console.log($('h1').text())     // ""
5console.log($('.price').text()) // ""

it still runs. the request still answers 200, but both lines print an empty string.

this is the tricky part. technically, nothing failed, so nothing tells you something is wrong. the data simply isn't in the html, because the browser was supposed to fill it in later. a scraper that crashes is easy to fix. a scraper that quietly returns empty fields can run for weeks before anyone notices.

this is how many AI agents see the web

the empty red mug page is also what many AI agents see. when you ask an agent to read a page, it usually calls a built-in web fetch tool. many of these tools work like step 1: they download the html and turn it into text, and no javascript runs. on the blue mug page, the agent gets the name and the price. on the red mug page, it gets an empty page, or a "please turn on javascript" line, and it can't tell that it missed anything. it also has the other limits of plain http: it's easy to spot as a bot, so some sites block it.

some agents can drive a real browser, through a browser extension or computer use, but that's a lot slower and heavier just to read one page, so it's usually not what happens when an agent "looks something up". so when your agent says a page has no price, the page may just be built in the browser.

step 3: scrape with a browser

when the data really isn't in the html, you need something that runs the javascript: a browser. not the one you're reading this in, but a real browser, like Chrome, that your code controls. here we use Playwright, a library that lets you control a browser like Chrome or Firefox from javascript code:

1import { chromium } from 'playwright'
2
3// start Chrome and open a new tab
4const browser = await chromium.launch()
5const page = await browser.newPage()
6
7// load the page. Chrome downloads everything the page asks for and runs its javascript
8await page.goto('https://shop.example.com/products/red-mug')
9
10// wait until the script has put the price on the page
11await page.waitForSelector('.price')
12
13console.log(await page.textContent('h1'))     // "red mug"
14console.log(await page.textContent('.price')) // "€14.90"
15
16await browser.close()

once the javascript has run, the content is on the page. the same selectors that came back empty in step 2 now find the red mug's name and price.

compare this with step 1. there, one GET request downloaded the page's html and nothing else. here, the browser downloads the html, then every script, style sheet, font and image the page asks for. and it runs the page's javascript by itself as the page loads, the same way it does for you, without us asking it to.

a browser also gets past more anti-bot checks. it runs the javascript checks that plain http skips, and with careful setup, Playwright can make it look like a person using Chrome on their own computer. set up well, most anti-bot systems let it through. getting that setup right is not simple, though, and we come back to it below.

step 4: find out which one a page needs

you've now seen both ways, and you know one of them is a lot more work to run. so before you pick one, check the page. it takes a minute:

1. turn javascript off. open the page in your browser, open the developer tools, open the command menu (Ctrl+Shift+P, or Cmd+Shift+P on a Mac), run "disable javascript" and reload. if the data you want is still on the screen, plain http will do. if it's gone, you need a browser.
2. check it in code. download the page with plain http and look for a value you can see on the screen:

1const html = await (await fetch(url)).text()
2const needsBrowser = !html.includes('€14.90')

one thing to watch for: if the html you get back is a "checking your browser" page or a captcha, the page doesn't need javascript. the site is blocking you. that's a different problem, and we get to it below.

the two ways side by side

now that you've seen both, here's how they compare.

plain http

• fast. one request for the page's html, usually done in well under a second.
• light. a small server can run many requests at the same time.
• cheap to run. no browser to start, keep up to date or restart when it crashes.
• misses what javascript builds. on pages built in the browser, you get the empty page.
• only reads. no clicking, no scrolling, no forms, no screenshots.
• easier to spot as a bot. it doesn't run the javascript checks some sites send, and the way it opens a secure connection often doesn't match a real browser.

browser

• sees what a person sees. if it's on the screen, you can read it.
• can do more than read. click, scroll, wait for something to appear, take a screenshot.
• passes checks that need javascript. and with careful setup, it looks like a person using Chrome on their own computer.
• slower. it loads every script, style sheet and image, so a page takes a few seconds instead of a fraction of one.
• heavy. every open page needs its own share of memory and cpu, so the same server runs far fewer at once.
• can still leave some traces. a browser controlled by code can leave traces that anti-bot scripts look for, especially one that isn't set up well.

the rule of thumb: start with plain http, and use a browser only for the pages that need one, or that block plain http. most scrapers that run at scale do exactly that: http first, and a browser for the pages that come back empty or blocked.

blocks are a third problem

there's one problem neither way solves on its own, and you saw it at the end of step 4. many sites run anti-bot systems that look at more than your code. they check where a request comes from (a data center or a home connection), how fast requests arrive, and whether the visitor behaves like a person.

a well set up browser helps with some of these checks, but not with the address your requests come from. that takes proxies: other connections your requests go out through, including home connections in the country you need. without them, even a perfectly set up browser often gets flagged once you run it on a server, because anti-bot systems can see it's a browser running in a data center. add retries and some patience on top, and it's a project of its own.

not sure which one your pages need?

by now you've seen the full list: a parser, a browser, a way to tell them apart, proxies and retries. that's a lot to build and keep running when what you wanted was the data.

with crawlbrulee, it's one request. we start with plain http, the fast and cheap way, and move up to a real browser by ourselves when a site blocks it, so you don't have to build both:

1import { Crawlbrulee } from '@crawlbrulee/sdk'
2
3const crawlbrulee = Crawlbrulee.fromEnv() // reads CRAWLBRULEE_API_KEY
4
5const page = await crawlbrulee.scrape({
6  url: 'https://shop.example.com/products/red-mug',
7  extract: {
8    markdown: true,
9    // ...and any other formats you need
10  },
11})
12
13console.log(page.markdown) // the red mug's name, price and description, as markdown

if you know a page is built by javascript, like the red mug page, add require_js: true and we go straight to the browser.

• we run the browsers. Chrome runs on our servers, so there's nothing for you to install, update or restart.
• blocks are on us. if a site blocks the light way, we retry through a more advanced proxy setup and a real browser, by ourselves.
• you pay for what worked. a block or a timeout costs nothing, and every response tells you what it cost.
• clean output. ads, cookie banners and popups are removed before we build the markdown, so it's ready for your code or your AI agent.

try it on your own pages

crawlbrulee is EU-native web scraping for AI agents and developers. the free plan has 750 credits a month, no credit card needed.

create your free account and find out which of your pages need a browser.