cli

the official command-line interface — crawlbrulee on npm. scrape pages, map sites, and inspect your account from the terminal. it wraps @crawlbrulee/sdk and is npx-runnable with zero install.

install

# one-off — recommended
npx crawlbrulee scrape url https://example.com
 
# or install globally
npm install -g crawlbrulee

first run

crawlbrulee login                 # paste your key (or pass --api-key)
crawlbrulee view-config           # check resolved config (key masked)
crawlbrulee scrape url https://example.com

you can skip login and authenticate per-call instead:

export CRAWLBRULEE_API_KEY="cwbl_..."
crawlbrulee scrape url https://example.com

commands

scrape url

crawlbrulee scrape url https://example.com            # markdown to stdout
crawlbrulee scrape url https://example.com --all      # every extract field
crawlbrulee scrape url https://example.com -ss full   # full-page screenshot
crawlbrulee scrape url https://example.com --proxy advanced --require-js
crawlbrulee scrape url https://example.com -l | jq .links

extract toggles — -m/--markdown, -c/--cleaned-html, -r/--raw-html, -l/--links, -i/--images, -ss/--screenshot, --all, --no-metadata. if any toggle is given it replaces the default. other flags: --proxy <basic|advanced|auto>, --require-js, --exclude-selectors, --cache-max-age, --locale, --country, -o/--output.

-ss/--screenshot takes an optional positional value: -ss <mode>[,width,height[,device[,slice-height]]] (mode is viewport | full | full_page; full is shorthand for full_page, the default).

async scrape

--async submits the scrape as a background job and prints its job_id immediately instead of holding the connection open — useful for heavy js rendering or long-page screenshots. retrieve the result by polling (scrape status / scrape wait) or by registering a completion webhook with --webhook-url.

crawlbrulee scrape url https://example.com --async
# job_id: job_abc123
 
# submit and wait for the result in one command
crawlbrulee scrape url https://example.com --async --wait
 
# or get notified via webhook instead of polling
crawlbrulee scrape url https://example.com --async \
  --webhook-url https://hooks.example.com/crawlbrulee \
  --webhook-metadata '{"order":"abc","attempt":2}'

act on a job_id you already have with the three job subcommands:

crawlbrulee scrape status job_abc123   # pending | running | done | failed
crawlbrulee scrape result job_abc123   # the finished result (errors if not done yet)
crawlbrulee scrape wait   job_abc123   # poll until done, then print the result

scrape wait (and --async --wait) accept --interval <seconds> (default 2) and --timeout <seconds> (default 300; 0 = wait forever); --wait requires --async. --webhook-url (requires --async) is called with a signed scrape.complete POST when the job finishes; --webhook-metadata (requires --webhook-url) must be a json object and is echoed back verbatim in the webhook payload.

map

crawlbrulee map https://example.com --limit 500 --page 2
crawlbrulee map https://example.com --sitemap-only
crawlbrulee map https://example.com --internal-only --no-subdomains
crawlbrulee map https://example.com --external-only --country DE

flags: --limit, --page, --sitemap-only, --internal-only, --external-only, --no-subdomains, --proxy <basic|advanced|auto>, --cache-max-age, --country, -o/--output. --internal-only and --external-only are mutually exclusive.

auth resolution

api key and base url each resolve independently, flag → env → config file → default:

• api key: --api-key flag → CRAWLBRULEE_API_KEY env → config file written by crawlbrulee login.
• base url: --api-url flag → CRAWLBRULEE_API_URL env → config file → default https://api.crawlbrulee.com.

the config file lives under the OS config dir (e.g. ~/Library/Application Support/crawlbrulee/config.json on macOS), chmod 0600 on posix.

output

stdout is tty-aware: human-readable text in a terminal, json when piped or written to a file. force with --json / --text; --compact for one-line json. errors go to stderr as error: <name> — <message>; exit code is 1 on failure.

every scrape/map response carries a response_meta.usage object. both include { credits, engine, proxy }; scrape also includes screenshot_slices. --json returns the endpoint-specific shape as-is, and text mode prints it as a trailing comment:

crawlbrulee scrape url https://example.com
# usage: 1 credits · engine http · proxy basic · slices 0

the map footer stops after the proxy because map cannot generate screenshot slices.