Make app
the crawlbrulee app for Make is listed in Make’s app directory: crawlbrulee on Make .
install
in a scenario, add a module and search for “crawlbrulee”. there is nothing else to install.
connection
create an api key in the dashboard (see get a key). the first module you add asks for a crawlbrulee connection:
| field | what to put in it |
|---|---|
| API key | your key, starting with cwbl_. required. |
saving the connection checks the key, at no credit cost.
modules
| module | what it does |
|---|---|
| Scrape | fetches one url and returns the content you asked for |
| Scrape (async) | starts a background job and returns its job id |
| Get scrape status | tells you whether a job is pending, running, done or failed |
| Get scrape result | returns the content of a finished job |
| Map site | discovers a site’s urls from its sitemap and its homepage links |
| Get usage | credits used and left in this billing cycle, plus your concurrency limit |
| Whoami | the organization and token behind the api key |
| Make an API call | sends a request of your own to the api, for endpoints no module covers |
| Scrape completed | an instant trigger that fires when a background scrape finishes |
every field has help text in Make. a field you leave untouched is not sent, so the api’s own default applies. see scrape and map for what each option does.
the page’s own status
Scrape and Get scrape result put the status the site answered with in Page status code. a page the site served is a normal result, whatever its status: a 404 page comes back with its content and the scenario goes on. a Make error handler does not fire for it.
when the status matters to your scenario, add a filter or a router on Page status code, for example 400 or higher.
screenshots
a screenshot comes back as a link, and the link expires 24 hours after the scrape. when you need the image itself, add Make’s HTTP > Get a file module after the scrape and give it the link.
trigger
Scrape completed starts a scenario when a background scrape finishes, so you don’t have to poll for it:
- add a Scrape completed trigger to your scenario and copy the webhook address Make shows.
- paste that address into the Webhook URL field of a Scrape (async) module.
- when the job finishes, we send one
scrape.completeevent to that address and the trigger fires.
the event tells you which job finished. it does not carry the page content. pass its Job ID to Get scrape result to fetch the content.