yes, your agent can scrape. give it a scraping provider anyway

last week, a founder shared something on LinkedIn that stuck with us. they're building a small project where AI agents do most of the legwork. one of the sites they need blocks agents. the other 95% let them through.

the surprising part was what happened next: nothing. the agent finished the job with the other sites, and the one that blocked it quietly dropped out of the results. going back to do it by hand wasn't worth it.

that's the real cost of a blocked agent. it doesn't crash. it moves on, and you get an answer with a hole in it that nobody sees.

so yes, your agent can scrape the web, and most of the time it does fine. this post is about the rest of the time: why agents miss pages, why that won't go away soon, and the small change that gets your agent more of them.

why your agent misses pages

when an agent reads a page, it usually calls a built-in web fetch tool. the tool asks the server for the page, turns the html into text and hands it to the model. it's quick, and on most pages it works. it fails in two ways.

the page needs JavaScript

many built-in fetch tools don't run the page's JavaScript. that's fine on a page built on the server. but many shops, dashboards and web apps send an almost empty page and build the content in your browser. the fetch tool gets the empty shell: no price, no listings, no text.

the agent can't tell an empty shell from a page with nothing on it. so it often decides the information isn't there and moves on. we explain the two kinds of pages, and how to tell them apart, in web scraping with or without a browser.

some agents can drive a full browser too. it's slow and costly per page, so most of them reach for the quick fetch tool first.

the site blocks it

many sites sit behind an anti-bot system. it looks at where a request comes from, how it opens the connection and whether it behaves like a person. an agent's fetch tool doesn't run the checks the site sends, doesn't open the connection the way a real browser does, and often names itself in its user agent. in chat apps, the fetch usually runs on the AI provider's servers too, so the request comes from a data center. to an anti-bot system, that looks like a bot, because it is one.

so the agent gets a 403, a captcha, or a "checking your browser" page instead of the content. some agents say so. many just carry on without that site.

sites and anti-bot vendors are still working this out

it's easy to blame the sites for this, or the anti-bot vendors they pay to keep bots out. but both have a hard job right now.

for the first time, the right answer to a lot of bot traffic is to treat it like a person. an agent that compares prices or reads the docs for someone is doing what that person would do in a browser. anti-bot vendors still have to stop mass scraping that overloads a site or copies it whole. now they also have to let the normal agents in.

if they get it wrong, the site owner loses both ways. block the agents, and the site drops out of the answers people get from their AI. let everything in, and the mass scrapers come in too.

and the traffic is already here. Vercel's founder, Guillermo Rauch, shared that 58% of the traffic across Vercel's network came from bots over the last 30 days, up from 32% in January 2024. the web is turning agent-first, and everyone (sites, anti-bot vendors, people who build agents) is still adjusting to what that means.

the rules will change, the anti-bot vendors will adjust, and some of these blocks will go away. but even when the vendors agree that agents aren't bots in the old sense, each site owner still has to agree too. until then, and maybe even after, agents will sometimes get blocked.

what to do today: give your agent a scraping provider

until the sites and the vendors work it out, the fix is on your side: let your agent read a page the way your own browser would.

you don't have to build that yourself. a scraping provider already runs real browsers, sends requests through different kinds of proxies, including home connections, and retries the ones that get blocked. add one to your agent as an mcp server or a cli, and the agent gets a second, stronger way to read a page.

the site's rules still apply, though. its terms and its robots.txt matter as much as before, whichever way your agent reads the page. is web scraping legal? walks through the rules in the EU.

once set up, here's how it works: your agent keeps using its own fetch tool, which is free and fast. when a page comes back blocked or empty, it reads the page again through the provider. most agents work this out alone once the tool is there. to make sure, tell yours once, in its instructions file (AGENTS.md, CLAUDE.md or your agent's rules file):

1when a web page comes back blocked or empty, or asks you to prove you're human,
2read it again with the crawlbrulee scrape tool. set require_js to true when the
3page builds its content with JavaScript.

will it get every page? no. a few sites are hard to read even with the most advanced browser setup. they're a small minority, though, and you'll get many of the pages that used to drop out quietly, and the ones that still fail come back as a clear error, not as a gap nobody sees.

set it up in a minute with crawlbrulee

we built crawlbrulee for exactly this kind of job, and setting it up takes one sentence. tell your agent:

the agent finds the setup steps on our site (we publish a guide written for agents), asks you for an api key, and that's it. to get a key, create a free account: 750 credits a month, no credit card needed.

prefer a terminal? the cli works the same way for agents that run shell commands: npx crawlbrulee scrape url https://example.com.

what your agent gets:

• JavaScript pages. with require_js, we render the page in Chrome on our servers before we turn it into markdown or extract data from it.
• retries when a site blocks it. on auto, the default proxy setting, we try the light route first and retry through our stronger proxy tier by ourselves.
• you pay only for pages that came back. agents retry a lot, so this matters: a block, a timeout or a server error from the site costs 0 credits.
• clean markdown. ads, cookie banners and popups are removed before we build the markdown, so your model reads the page, not the clutter around it.
• EU servers. the fetch, the render, the cache and the result all run in the EU.