how to scrape blog posts into markdown

in lesson one we scraped a product page with our js/ts sdk. this time we use no sdk at all: just fetch and two http endpoints. the code is in our public crawl-academy repo. this post shows the important pieces, trimmed for reading; clone the repo for the full code.

the job: find the posts on a blog and turn them into markdown files, one per post. we picked the blog you're reading right now, so yes, this post ends up in the output too. point the script at your own blog next, and its posts land in your folder as clean markdown, ready to read, search or hand to your AI agent.

what you'll get

1output/
2├── posts.json
3└── posts/
4    ├── crawlbrulee-for-your-dpo.md
5    ├── data-privacy-reasons-to-switch-web-scraping-provider.md
6    ├── do-website-terms-bind-a-scraper.md
7    ├── is-web-scraping-legal.md
8    ├── scrape-blog-posts-to-markdown.md
9    ├── scrape-ecommerce-product-data.md
10    └── zero-data-retention.md

each file starts with the post's facts, then the post itself:

1---
2title: "new in crawlbrulee: zero data retention"
3description: "with zero data retention, your results stay out of our shared cache, your urls stay out of our logs, and a sync scrape stores nothing at all"
4published: "2026-10-07"
5url: "https://crawlbrulee.com/blog/zero-data-retention"
6---
7
8# new in crawlbrulee: zero data retention 🎉
9
10zero data retention is now live in …

posts.json lists the same posts with the same facts and the name of each file.

before you start

• Node.js 22.18 or later. it runs .ts files directly, so there is no build step.
• pnpm. if you don't have it yet: npm install -g pnpm.
• a crawlbrulee api key. the free plan has 750 credits a month and needs no credit card.

1git clone https://github.com/crawlbrulee/crawl-academy.git
2cd crawl-academy/02-crawlbrulee-blog-markdown
3export CRAWLBRULEE_API_KEY=cwbl_…
4pnpm scrape

no install step: the script has no dependencies. one command reads the sitemap and up to 10 posts in it.

step 1: call the api with fetch

every call is a POST with your api key as a Bearer token and a json body. this is the whole client:

1async function post(path: string, body: object) {
2  const res = await fetch(`https://api.crawlbrulee.com${path}`, {
3    method: 'POST',
4    headers: { authorization: `Bearer ${API_KEY}`, 'content-type': 'application/json' },
5    body: JSON.stringify(body),
6  })
7  if (!res.ok) throw new Error(`${path} failed with ${res.status}: ${await res.text()}`)
8  return res.json()
9}

a failed call stops the script with the status and the api's own message, like 401 and unauthorized. in the repo, two small functions wrap post, so the calls below read crawlbrulee.map(…) and crawlbrulee.scrape(…).

step 2: find the posts with a map

you could click through the blog page by page. there's a faster way. like most sites, ours lists every page in a sitemap, for search engines. one call to /api/map reads it:

1const map = await crawlbrulee.map({
2  url: 'https://crawlbrulee.com',
3  sitemap_only: true,
4  types: { internal: true, internal_subdomains: false, external: false },
5  cache: { max_age: 0 },
6})
7// map.links → [{ url: 'https://crawlbrulee.com/docs' }, …, { url: 'https://crawlbrulee.com/blog/zero-data-retention' }, …]

• sitemap_only: true reads only the sitemaps and skips the home page.
• types keeps links on our own site.
• cache: { max_age: 0 } reads the sitemaps fresh, so the list matches the blog right now.

the map gives you every page on the site: the docs, the legal pages and the blog. the script keeps the urls under /blog/, and takes the first 10 the sitemap lists. that keeps a run small, however big the blog gets:

1const postUrls = map.links
2  .map((link) => link.url)
3  .filter((url) => url.includes('/blog/'))
4  .slice(0, 10)

step 3: read each post as markdown

now the script has a list of posts, and it needs their text. ask for markdown and you get the page as clean text, with its headings, lists, links and code blocks.

cleanup.exclude_selectors leaves out the parts of the page that aren't the post. more on that in step 4:

1for (const url of postUrls) {
2  const page = await crawlbrulee.scrape({
3    url,
4    extract: { markdown: true, metadata: true },
5    cleanup: { exclude_selectors: NOT_THE_POST },
6  })
7  // page.markdown                → "# new in crawlbrulee: zero data retention 🎉\n\n…"
8  // page.metadata.date_published → "2026-10-07T09:00:00.000Z"
9}

our blog pages are plain html, so the request needs no browser: 1 credit a post. the plain for loop reads the posts one at a time, so the site never gets many requests at once.

step 4: leave out what isn't the post

a post page has more on it than the post: the menu, a table of contents, the author box and a sign-up box at the end. under the title sits a byline, the line with the author's name, the date and the reading time. you don't want any of that in every file.

two of these parts are more than clutter: the byline and the author box name a person. a name is personal data, and in the EU, collecting it falls under the GDPR. you need a lawful reason to collect it, and rules apply to how long you keep it and who you tell. the GDPR also asks you to collect only the personal data you need, and a markdown copy of a post doesn't need its author's name. so we leave the names out before the markdown is written, and they never reach your files. the easiest personal data to protect is the data you never collect. is web scraping legal in the EU? explains the rules in more detail.

cleanup.exclude_selectors takes a list of css selectors. we remove every element they match, in order, before we write the markdown:

1const NOT_THE_POST = [
2  'header',           // the menu at the top
3  'footer',
4  'aside',            // the table of contents on wide screens
5  'section',          // the author box and the sign-up box at the end
6  'details:has(nav)', // the table of contents on phones
7  'div:has(+ h1)',    // the label right before the title
8  'h1 ~ div',         // the byline: the author's name, the date, the reading time
9]

the list uses three kinds of selectors:

• a tag name. our blog marks these parts with header, footer, aside and section tags, so the tag name is enough. pages built this way are the easy case.
• a tag that holds something. on phones, the table of contents opens from a fold-out details box. the FAQ answers at the end of some posts use the same box, and we want to keep those. details:has(nav) matches only the box with a nav inside, which is the table of contents.
• a position. the byline has no class or id of its own, only styling classes that can change with the next redesign. so we pick it by where it sits. h1 ~ div means "every div after the h1, in the same block". div:has(+ h1) is the div right before the h1: the post's label, like "product updates".

the result: each file starts with the post's title, then the post itself, with the FAQ answers as headings. the files name no author, and the date is not lost: it is in each file's facts (step 5).

to find selectors on another site, open a page in your browser, right-click the part you don't want, and pick "inspect". then try your selector in the browser console before you add it: document.querySelectorAll('h1 ~ div') shows what it matches.

step 5: save one file per post

the markdown has the post, and the metadata has its facts: the title, the description and the date it was published. the script writes the facts at the top of each file, as a small yaml block, with the markdown below:

1const slug = url.split('/blog/')[1]
2const title = page.metadata.og_title.replace(' · crawlbrulee', '')
3
4const facts = [
5  '---',
6  `title: ${JSON.stringify(title)}`,
7  `description: ${JSON.stringify(page.metadata.description)}`,
8  `published: ${JSON.stringify(page.metadata.date_published.slice(0, 10))}`,
9  `url: ${JSON.stringify(url)}`,
10  '---',
11].join('\n')
12
13await writeFile(`output/posts/${slug}.md`, `${facts}\n\n${page.markdown}`)

JSON.stringify puts each value in quotes and escapes what needs escaping, and a json string is valid yaml. the repo also writes a posts.json with the same facts for every post.

most markdown tools read this block, and so do AI agents. when your agent quotes a post, it has the title and the url to link back to.

what it costs

• the map: 1 credit per run.
• each post: 1 credit, for up to 10 posts.

a run costs 11 credits at most. when we wrote this, the blog had 7 posts, this one included, so a run cost 8 credits. the free plan's 750 credits a month go a long way.

make it yours

• your own blog. in the repo's src/scrape-blog.ts, change SITE and POST_PATH to the blog's address and the path its posts start with. then open one post in your browser, inspect the parts around it, and put their selectors in NOT_THE_POST. on your own blog, you can raise the 10-post limit, MAX_POSTS. pointing it at someone else's blog? read its robots.txt and terms of use first. most blog posts are copyrighted: unless the owner says yes, keep the title, the date and a link, not the full text. this is not legal advice.
• an AI agent's reading list. point your agent, or the tool that indexes your documents, at output/posts/. each file carries its own title, date and url.

try it on your own pages

crawlbrulee is EU-native web scraping for AI agents and developers. the free plan has 750 credits a month, no credit card needed.

create your free account and turn your first blog into markdown today.