Web Scraping in Node.js

node.jSTL

8/11/2016

Example: GET + RegEx

Easy way to check for presence of text.
Just the standard libraries
Probably easy to outgrow
Consider using a library to make the requests easier such as got

got('localhost:8080', {
  headers: {
    'User-Agent': 'nodejstl-bot',
  }})
  .then(response => { ... });

Example: Build a DOM With Cheerio

Libraries exist to build a static DOM from the response body
Cheerio creates a DOM, selectors, and a jQuery stlye API

// assuming you have a valid response
let $ = cheerio.load(response.body);
let address = $('address');

More on Cheerio

Built as a lightweight and fast JSDOM replacement
Aims for API compatibility with jQuery
Can manipulate the DOM and render back to HTML

let streetAddress = $('[itemprop="name"]');
$('.team tbody tr').each((index, element) => { ... });

Nightmare

Browser automation library
Based on Electron
API to interact and extract data from the page
Pass data between Node and the browser

const nightmare = require('nightmare')();
nightmare.goto('http://localhost:8080/products')

Web Scraping in Node.js node.jSTL 8/11/2016