new in crawlbrulee: zero data retention 🎉

zero data retention is now live in crawlbrulee 🎉

add one field to a request in order to:

• not have your results saved to our shared cache - no other customer ever gets them
• keep scraped urls out of our logs; we keep only domain names, for abuse prevention
• zero data saved on our side from /map and sync /scrape requests without screenshots

why it matters

when you hand a scraping job to a vendor, two questions come up fast: who can access my data? and do you keep my urls?

for a normal request, our answer is already short: every copy is deleted after 90 days at most, backups included. however, some projects need stricter rules than that. a client contract may say "no copies". a DPIA may ask for the shortest retention you can get. or the pages are simply ones you'd rather nobody else ever sees. zero data retention is for those projects.

it fits with the rest of how we work: processing and long-term storage run on EU servers.

what changes when you turn it on

so what does "no copies" look like in practice? turn it on for a request, and here's what happens to your data:

• out of the shared cache. the result is not added to the shared cache, so it is never served to another customer.
• out of our logs. your url stays out of our logs; we keep only the domain name, for abuse prevention.
• • your page content is never logged, even if zero data retention is off
• out of storage. a sync scrape is answered straight from memory. without a screenshot, we store nothing at all.
• 24 hours, only when we have to. two things can't be delivered without storing them for a while: an async result, which you fetch later, and a screenshot, which you download from a link. those are deleted 24 hours after the request, instead of being kept for up to 90 days. don't want us to store anything? use sync scrapes and skip screenshots.
• in writing. the same promise sits in section 6 of our terms and annex 2 of our DPA, so your DPO has something to point to.

what we still keep: usage and billing records (which organization made the request, when, and what it cost), and the domain name in our logs for up to 90 days.

one detail worth knowing: reading from the cache is still allowed, so a request can be answered with a result that is already there. to get a fresh result every time, also set cache.max_age: 0.

turn it on with one field

add zero_data_retention: true at the top level of the request body. it works on scrape, async scrape and map:

1curl -X POST https://api.crawlbrulee.com/api/scrape \
2  -H "Authorization: Bearer $CRAWLBRULEE_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{
5    "url": "https://example.com",
6    "extract": { "markdown": true },
7    "zero_data_retention": true            // <--- here
8  }'

the same option is in our js/ts and python sdks, the cli (--zero-data-retention), the mcp server and our no-code apps.

need it on every request? we can force it to be always on for your whole organization, so a request that forgets the field is still covered.

availability and cost

zero data retention comes with our enterprise plan. on certain occasions, we may consider enabling it for an organization on another plan: write to sales@crawlbrulee.com and tell us about your project.

it adds 1 credit to each billed result that did not come from the cache. a cache hit adds nothing. for example, an http request on the advanced proxy costs 5 + 1 = 6 credits.

keep your data out, starting today

every detail is in the zero data retention docs: the field, what stays out, what we still keep, the 24-hour window and the error you get before it is turned on. read them with your DPO, then write to sales@crawlbrulee.com to switch it on for your organization.

not on the enterprise plan yet? start on the free plan (750 credits a month, no credit card required!), and build your project with sync scrapes. when you're ready for zero data retention, your code needs one more field.

read the zero data retention docs →