this is general information, not legal advice. the EU regulators' guidelines on web scraping are still a draft. we will update this post when the final text is out.
TL;DR: yes, it is legal
no EU law bans web scraping, and most business projects fit within the rules without much effort. which rules apply depends mostly on what is on the pages you collect, and on how you collect them.
here are the questions we hear most, each with a one-line answer. the rest of the post explains them.
can I scrape prices, stock or product details? → yes. it is the most common kind of project. if the pages carry no personal data, the GDPR doesn't apply.
is a public page free to take? → no, not by itself. a page anyone can open can still carry a copyrighted article or photo, a database someone paid to build, or personal data. each brings its own rule.
is ignoring robots.txt illegal? → no law says so. regulators and courts can ask whether you read it.
can I scrape pages that name people? → yes, with care. drop the names you don't need; the GDPR covers the ones you keep.
is breaking a site's terms a crime? → no, it is a civil matter. if you accepted the terms, for example when you opened an account, the site can sue you for breach of contract and ask for damages or a court order to stop.
can I use scraped data to train AI models? → yes, with extra rules on copyright and personal data.
can I scrape behind a login? → it depends. only with the account owner's permission, and only if the site's terms allow it.
is scraping the same as hacking? → no. scraping is reading pages a site shows to anyone. hacking is getting past a lock, like a password, and that is a crime of its own.
first, find your project's risk level
how much legal work a project takes depends mostly on one thing: personal data. that is anything that points to a person: a name, an email address, a phone number, a photo, a home address, even an IP address. so before anything else, ask how much of it your project touches.
we sort projects into three levels, from the most common to the most demanding.
low risk: no personal data
most projects are here. typical examples:
price and stock tracking.
brands checking how retailers show their products.
changes on competitor pages: pricing, features, terms.
legal and regulatory updates from official sites.
travel rates, fuel prices, energy tariffs.
seo audits, and testing your own site in other countries.
the GDPR doesn't apply here. copyright, the database right and the site's terms still do; we cover them further down.
moderate risk: personal details you don't need
the pages carry some personal data, but it isn't what you're after: a seller's name next to a price, a byline, a reviewer's name, a contact person on a tender notice, a sole trader's business details.
the simplest fix is to drop those details as you collect. what you never keep, you never have to protect.
ask for just the values you need with extract.elements, or remove parts of the page with cleanup.exclude_selectors, so the names never reach your systems. see /scrape.
needs attention: the people are the point
here the personal data is the reason for the project: lead lists, contact details, profiles of people, or anything about health, religion, politics or children.
this is the highest level, for a simple reason: the people in your data never chose to be in it, and some of them, like children or patients, are easy to harm. such a project can still be lawful, but you prepare before you collect:
a written reason for the project, weighed against the interests of the people in it.
often a formal risk assessment, called a DPIA.
usually a message to each person, saying that you hold their data.
their consent first, if you plan to email consumers for marketing.
plan it with your DPO. the next section explains each of these steps.
personal data under the GDPR
once a project keeps personal data, whether a few names or a whole lead list, the GDPR applies. the steps are the same at both levels. a project that needs attention goes through all of them in more depth, and writes each one down.
a lawful reason
for a business this is usually "legitimate interest". it has three parts: a real business reason, scraping that is needed for it, and a balance that doesn't hurt the people in the data. in 2024 the Court of Justice of the European Union (CJEU), the EU's top court, confirmed that a commercial interest can qualify. in April 2025 even the Dutch regulator, one of the strictest in the EU, withdrew its earlier view that a purely commercial interest could never count.
write your reasoning down: that note is your answer if anyone asks. consent rarely works here, because you can't ask everyone on every page before you collect.
only the fields you need, and a date to delete them
decide which fields the project needs, collect only those, and set how long you keep them. data you have deleted can't leak.
tell people
when you collect personal data from somewhere other than the person, you usually have to tell them. there is an exception when that is impossible or takes a disproportionate effort, but it is narrow. read Article 14 of the GDPR for the full scope.
where a public notice is enough, it is a short section on your website, for example in the privacy policy linked from your footer: what you collect, why, and from which sources.
sensitive data stays out
don't target data about health, religion, political views or ethnic origin, and delete any that slips in. this can still be legal to acquire but the rules are proportionally more strict.
a way to ask for removal
give people a contact address, and act when they write.
the EU regulators' draft guidelines on scraping (July 2026) say the same for scraping in general. their extra advice is about training AI models, which we cover below.
copyright, databases and site terms
personal data is only one side. even a project with no names in it, like price tracking, has to meet three other rules:
who owns the content;
who built the database;
and what the site's terms say
the good news first: facts are free!
copyright
facts like prices and stock levels aren't protected. texts, photos and articles however can be. EU copyright law still lets companies analyze them automatically ("text and data mining"), unless the owner has said no in a way a machine can read and act on.
a site can say that no in several places: its robots.txt file, a tdm-reservation tag in the page's metadata or a tdmrep.json file (the W3C standard made for exactly this), an ai.txt file, or its terms of use.
how clear does the no have to be? a German appeal court answered in a December 2025 ruling: it counts when an automated process can read it and act on it, and the bar moves with the technology of the day. a plain-text ban in a site's terms didn't pass for a 2021 crawl, because it never named data mining and needed a human to interpret it. a crawl made today, with language models at hand, may well be judged differently.
so check all of these places before you start, note what you found, and respect a clear no. logins, CAPTCHAs and other blocks aren't copyright opt-outs, but they tell you the site doesn't want automated visitors, and EU regulators take them into account in the event of an investigation, when they weigh what the people on that site expect.
the database right
the EU also protects the work that goes into building a database. you can't copy all or a large part of someone's catalog or directory and publish it as your own, and many small copies that add up count too.
this usually applies to large datasets only, and is typically enforced if you try to monetize a dataset scraped from someone else.
if your use case is a legitimate business interest and its main purpose is not to take over someone else's dataset as your own, this usually does not apply.
monitoring the prices of the products you follow, even a few hundred from one online shop, doesn't usually infringe the database right, though it can break that website's terms.
the site's terms
many sites forbid scraping in their terms. whether those terms bind you depends on how you accepted them, and on the country. lawyers sort them into two main kinds:
clickwrap: you ticked a box or clicked "I agree", for example when you opened an account. these usually bind you as a contract.
browsewrap: the terms sit behind a link in the footer, and nobody asked you to accept them. these often don't bind a visitor who never saw them, but courts in different countries judge this differently.
so take most care where you did accept the terms, such as inside a logged-in account. we explain the kinds of website terms, and when they bind, in do a website's terms bind a scraper?.
logins, paywalls & pacing
the rules so far are about what you collect. the next two are about how you collect it.
first: stay on public pages. breaking through a password or another security measure can be a crime, whatever the data behind it. if you need data behind a login, ask the site for access, an api or a data license.
our acceptable use policy, which every customer accepts, forbids getting past logins, paywalls or other access controls unless the account is yours or its owner agreed.
second: be a polite visitor → don't overload the site. send requests at a pace the site can handle, and don't fetch the same page again without a reason. heavy traffic is a common reason sites block scrapers and complain, and it is the easiest one to avoid.
if a recent copy of a page is in our cache, we send you that copy and the site isn't contacted again. a cache hit costs 0 credits. see /docs/caching.
training AI: the AI Act
if you scrape to train AI models, two more sets of rules join in.
1 - the AI Act. it doesn't ban scraping. since August 2025, companies that build general-purpose AI models, the large models behind chat assistants, must have a copyright policy that respects machine-readable opt-outs, and must publish a summary of the content they trained on. the EU's code of practice for these companies (July 2025) spells it out: their crawlers read and follow robots.txt. one use has been banned outright since February 2025: scraping faces from the internet to build face recognition databases.
2 - the EU regulators' draft guidelines (July 2026). they are written for training AI models. they ask you to leave out sites that clearly object to scraping, and to treat logins, robots.txt and CAPTCHAs as signs of what the people on a site expect.
if you only use AI models, or scrape for price tracking or monitoring, these training duties aren't yours.
we don't train on your results or send them to AI providers. and before you train, you can read any site's robots.txt with /scrape.
outside the EU
your project may be in the EU, but the sites you scrape often aren't. here is how the rules differ in the three countries we get asked about most.
🇬🇧 United Kingdom
the UK GDPR keeps almost the same rules, and the UK has its own database right. one difference: the UK's copyright exception for automated analysis covers non-commercial research only. in March 2026 the government decided not to change that for now, so commercial text and data mining needs permission or a license.
🇺🇸 United States
there is no federal privacy law like the GDPR. California's privacy law, like several other state laws, doesn't count publicly available information as personal data, including what a person shared without limiting who could see it. courts have held that scraping public pages isn't hacking under the federal anti-hacking law. contracts still count: a company that had accepted a site's terms through user accounts lost on breach of contract. the US has no database right.
🇨🇦 Canada
collecting personal data needs consent, and the "publicly available" exception is narrow: directories, public registries, court records and publications. in 2021 the privacy regulators found that social media profiles don't count.
🌎 rest of the world
other countries have their own rules on public data. everywhere, logins and paywalls, copyrighted content and the site's terms raise the same questions as in the EU.
who fetches the page matters too
so far, the rules were about your project. the last one is about the tools you use for it. in October 2025, a forum site sued an AI search company in the US and named three scraping vendors as defendants too.
so the question is no longer only: "what do you collect?". it is now also: "who collects it for you, and how?". expect your DPO to ask about the vendor: how long does it keep your results, which uses does it ban, where does it run, and does a written agreement make it your processor? choose a vendor that answers all of that in writing. the questions to ask are in 7 data privacy reasons to switch your web scraping provider.
we built crawlbrulee so the careful way is also the easy one. each rule above has a matching feature:
keep only what you need:extract.elements returns just the values you name; cleanup.exclude_selectors removes parts of a page.
don't keep data longer than needed: every copy deleted after 90 days at most, backups included; with zero data retention we keep no copy for later, and your url stays out of our logs; we keep only the site's domain name.
let people ask for removal: send us the urls and we remove our cached copies.
a vendor that works for you: we're your processor, under a public data processing agreement that applies from signup.
know where your data goes: servers in France; requests go out from an EU country unless you pick another; screenshots are delivered through a CDN with edges worldwide.
don't overload sites: repeat requests are answered from our cache when a recent copy exists.
check for a copyright opt-out: a site's robots.txt is a page like any other: send it to /scrape and read it.
stay out of logins without permission: our acceptable use policy forbids it for every customer.
no AI training on your data: we don't train on your results or send them to AI providers.
you stay responsible for what you collect and what you do with it. it is all written down at personal data at crawlbrulee.
a short checklist
before you start a project, run through these six questions:
what is the project's risk level: low, moderate, or does it need attention?
if it keeps personal data: is the reason written down, are you keeping only the fields you need, and is there a delete date?
is a notice ready, or a message to each person if you hold their contact details?
did you check robots.txt, the terms and the page metadata?
are you staying on public pages, at a pace the site can handle?
can your scraping vendor answer your DPO in writing?
questions people ask
the questions we hear most, with short answers.
is it legal to scrape competitor prices in the EU?
yes. prices and product details aren't personal data and facts aren't protected by copyright. take the products you follow, not a copy of the whole catalog, and stay on public pages.
do I need people's consent to scrape their data?
usually not. most businesses rely on legitimate interest instead, with the reasoning written down. you still have to tell people, keep only what you need, and delete it on time.
what if I skip robots.txt?
nothing in the law punishes that by itself. but robots.txt can carry a copyright opt-out, and regulators read it as a sign of what the site expects. checking it takes a minute.
what about hiQ v LinkedIn and the other well-known cases?
hiQ v LinkedIn (US, 2022): scraping public pages isn't hacking. hiQ still lost in the end, because it had opened LinkedIn accounts and broken the terms that came with them. Meta v Bright Data (US, 2024): a court found Meta's terms didn't cover scraping public pages while logged out, and Meta dropped the case. Ryanair v PR Aviation (CJEU, 2015): a site can limit the use of its data by contract, within national law.
does crawlbrulee make my project GDPR compliant?
no vendor can do that for you, because you decide what to collect and why. what we can do is make our part easy to check: a public agreement, EU servers, short retention and a public list of who processes data for us.
