Code Responsible

Skip navigation

Scraping the Web with Node.js

There are a lot of great APIs available online to get structured data for your applications. Unfortunately not every website or application provides an API. There is a perfect solution for retrieving the data from these websites and creating your own API using Node.js. There’s a Node.js package called Cheerio that helps create a framework for scraping the data from any website. Once you have the data you create a JSON endpoint to use in your application.

In this example, we’ll use Cheerio and Express to create our own API for WordThink.com, which is a website that provides a daily word and definition.

Getting started

Make sure you have Node.js installed.

  1. Install Cheerio and Request

    Run the following commands in your terminal:

    npm install cheerio
    npm install request
  2. Cheerio script

    Create a new file named ‘app.js’ and paste the following code:

    Test the following script. Run the following command:

    node app.js

    The script should then return a word and description from WordThink.com.

    A terminal console example of word of the day

  3. Install Express

    Run the following command in your terminal:

    npm install express
  4. Final working example

    Update the ‘app.js’ file with the following code:

    Run the app.js script again:

    node app.js

    Visit ‘localhost:3000’ in the browser and you’ll then see a JSON response for the word of the day.

    Example JSON response in the browser

We’ve now created our own API server using Express, Cheerio and Request. It’s that simple. You can now use the endpoint in your next application to display a daily word and definition.

41 Posts

Ryan Burgess

Ryan Burgess is a Manager, UI Engineering at Netflix in San Francisco, California. Previously at Evernote. Listen to Ryan on the Front End Happy Hour podcast. He's also the creator of SensesLost.com, an online-based arts and culture magazine that focuses on graffiti and tattoo art.

7 Responses to “Scraping the Web with Node.js”

Leave a Reply

Comment Form