Scraping the Web with Node.js
There are a lot of great APIs available online to get structured data for your applications. Unfortunately not every website or application provides an API. There is a perfect solution for retrieving the data from these websites and creating your own API using Node.js. There’s a Node.js package called Cheerio that helps create a framework for scraping the data from any website. Once you have the data you create a JSON endpoint to use in your application.
Make sure you have Node.js installed.
Install Cheerio and Request
Run the following commands in your terminal:npm install cheerionpm install request
Create a new file named ‘app.js’ and paste the following code:
Test the following script. Run the following command:node app.js
The script should then return a word and description from WordThink.com.
Run the following command in your terminal:npm install express
Final working example
Update the ‘app.js’ file with the following code:
Run the app.js script again:node app.js
Visit ‘localhost:3000’ in the browser and you’ll then see a JSON response for the word of the day.
We’ve now created our own API server using Express, Cheerio and Request. It’s that simple. You can now use the endpoint in your next application to display a daily word and definition.