Skip to Content

Make app

the crawlbrulee app for Make is listed in Make’s app directory: crawlbrulee on Make .

install

in a scenario, add a module and search for “crawlbrulee”. there is nothing else to install.

connection

create an api key in the dashboard (see get a key). the first module you add asks for a crawlbrulee connection:

fieldwhat to put in it
API keyyour key, starting with cwbl_. required.

saving the connection checks the key, at no credit cost.

modules

modulewhat it does
Scrapefetches one url and returns the content you asked for
Scrape (async)starts a background job and returns its job id
Get scrape statustells you whether a job is pending, running, done or failed
Get scrape resultreturns the content of a finished job
Map sitediscovers a site’s urls from its sitemap and its homepage links
Get usagecredits used and left in this billing cycle, plus your concurrency limit
Whoamithe organization and token behind the api key
Make an API callsends a request of your own to the api, for endpoints no module covers
Scrape completedan instant trigger that fires when a background scrape finishes

every field has help text in Make. a field you leave untouched is not sent, so the api’s own default applies. see scrape and map for what each option does.

the page’s own status

Scrape and Get scrape result put the status the site answered with in Page status code. a page the site served is a normal result, whatever its status: a 404 page comes back with its content and the scenario goes on. a Make error handler does not fire for it.

when the status matters to your scenario, add a filter or a router on Page status code, for example 400 or higher.

screenshots

a screenshot comes back as a link, and the link expires 24 hours after the scrape. when you need the image itself, add Make’s HTTP > Get a file module after the scrape and give it the link.

trigger

Scrape completed starts a scenario when a background scrape finishes, so you don’t have to poll for it:

  1. add a Scrape completed trigger to your scenario and copy the webhook address Make shows.
  2. paste that address into the Webhook URL field of a Scrape (async) module.
  3. when the job finishes, we send one scrape.complete event to that address and the trigger fires.

the event tells you which job finished. it does not carry the page content. pass its Job ID to Get scrape result to fetch the content.