Easy Way To Scrape Google Search Results And Auto-update The Data

How to scrape Google Search results
Reading Time: 5 minutes

In this article, we’ll learn how to scrape Google search results and automatically pull the data to Google Sheets. We can scrape data such as titles, links, and snippets among other things.

Scraping the Google Search results page can quickly and easily be done using a simple Google Sheets extension called Amigo. With Amigo we can scrape Google Search results without a single line of coding and set it to retrieve the search engine results page periodically. We only need to specify the search keyword and Amigo will fetch us the latest results for that query without any additional manual input.

Scraping Google Search results with Amigo is a cakewalk. It can be done in a few seconds. Below are the steps.

Steps to scrape Google Search results:

  1. Install the Amigo extension
  2. Get the API endpoint URL
  3. Import the data to Google Sheets

How to scrape Google Search results and pull the data to Google Sheets

Install the Amigo Data extension in Google Sheets

Install the Amigo Data Google Sheets add-on from the Google Workspace Marketplace.

Install Amigo to scrape Google Search results

Click here to install.

Alternatively, you can get it from Google Sheets.

  • Open a new sheet
  • Go to Extensions→Add-ons→Get add-ons
  • Search for Amigo and install

Get the API endpoint URL

Sign up or log in to RapidAPI.

Search for Google Search and click the most relevant one from the results. Or click this link to go directly to the Google Search API documentation page.

Google Search API in RapidAPI Hub

Click on the Subscribe to Test button and subscribe to a plan of your choice, post which you’ll be able to test the API and scrape Google Search results and import the results to Google Sheets.

Subscribe to the API to make calls to the API

Click Endpoints to go back to the API Playground. On the left is the list of the type of data that can be retrieved.

Available endpoints

We can scrape either web search results or images. The method is the same for both. We’ll scrape search results. Click the Web Search endpoint.

Under Required Parameters in the query field, enter the keyword that you want to scrape Google Search results–eg. metaverse. You can also use a combination of words as a query.

Entering the search keyword to scrape Google Search results

Then fill in the optional parameters to filter the results. These parameters are however, as the name suggests, not required. In the example below, the region is set to the US, and will return the top 20 results in English sorted by relevance.

Optional parameters

After filling in the parameters, click the Test Endpoint to run the query and verify whether the API is working. If it is, then we should see the results returned in JSON format.

In the Code Snippets section, we’ll find the URL along with the parameters and the headers.

Change the language to Shell→cURL. The query parameters are appended in the URL so that we do not have to enter the parameter manually to import the data to Google Sheets.

API endpoint URL and headers

Copy the URL and also note the headers. These will be used to scrape Google Search results and import the data to Google Sheets.

Import the data to Google Sheets

Open Google Sheets and launch the Amigo add-on from the sheet into which you want to fill the data. To connect to Amigo, click Extensions→Amigo→Connect.

Launch Amigo to scrape Google Search results

We’ll be presented with a list of sources to connect and import data. Select Custom API.

Select Custom API as the source to connect

Paste the endpoint URL in the API URL field excluding the apostrophes.

Entering the API URL

Then expand the Details tab and enter the headers. The headers can be found in the Code Snippets. The characters to the left of the colon are the header names and those to the right are the header values. Enter these as separate headers. Do not include the apostrophes.

Name: X-RapidAPI-Host, Value: google-search72.p.rapidapi.com

Name: X-RapidAPI-Key, Value: YOUR_API_KEY

Entering the headers
Preview of the data

The data will be imported into Google Sheets in a moment. Below is a snippet of the Google Search results imported into Google Sheets.

Google Search results imported into Google Sheets

Some recommended articles:

How To Scrape And Import Google News Data To Google Sheets

Import LinkedIn Profile Data To Google Sheets

Connect Hubspot To Google Sheets

X
Hire expert data analysts on-demand. Get 30 Days Free Trial