In this article, we’ll learn how to scrape Google search results and automatically pull the data to Google Sheets. We can scrape data such as titles, links, and snippets among other things.
Scraping the Google Search results page can quickly and easily be done using a simple Google Sheets extension called Amigo. With Amigo we can scrape Google Search results without a single line of coding and set it to retrieve the search engine results page periodically. We only need to specify the search keyword and Amigo will fetch us the latest results for that query without any additional manual input.
Scraping Google Search results with Amigo is a cakewalk. It can be done in a few seconds. Below are the steps.
Steps to scrape Google Search results:
How to scrape Google Search results and pull the data to Google Sheets
Install the Amigo Data extension in Google Sheets
Install the Amigo Data Google Sheets add-on from the Google Workspace Marketplace.
Click here to install.
Alternatively, you can get it from Google Sheets.
- Open a new sheet
- Go to Extensions→Add-ons→Get add-ons
- Search for Amigo and install
Get the API endpoint URL
Sign up or log in to RapidAPI.
Search for Google Search and click the most relevant one from the results. Or click this link to go directly to the Google Search API documentation page.
Click on the Subscribe to Test button and subscribe to a plan of your choice, post which you’ll be able to test the API and scrape Google Search results and import the results to Google Sheets.
Click Endpoints to go back to the API Playground. On the left is the list of the type of data that can be retrieved.
We can scrape either web search results or images. The method is the same for both. We’ll scrape search results. Click the Web Search endpoint.
Under Required Parameters in the query field, enter the keyword that you want to scrape Google Search results–eg. metaverse. You can also use a combination of words as a query.
Then fill in the optional parameters to filter the results. These parameters are however, as the name suggests, not required. In the example below, the region is set to the US, and will return the top 20 results in English sorted by relevance.
After filling in the parameters, click the Test Endpoint to run the query and verify whether the API is working. If it is, then we should see the results returned in JSON format.
In the Code Snippets section, we’ll find the URL along with the parameters and the headers.
Change the language to Shell→cURL. The query parameters are appended in the URL so that we do not have to enter the parameter manually to import the data to Google Sheets.
Copy the URL and also note the headers. These will be used to scrape Google Search results and import the data to Google Sheets.
Import the data to Google Sheets
Open Google Sheets and launch the Amigo add-on from the sheet into which you want to fill the data. To connect to Amigo, click Extensions→Amigo→Connect.
We’ll be presented with a list of sources to connect and import data. Select Custom API.
Paste the endpoint URL in the API URL field excluding the apostrophes.
Then expand the Details tab and enter the headers. The headers can be found in the Code Snippets. The characters to the left of the colon are the header names and those to the right are the header values. Enter these as separate headers. Do not include the apostrophes.
Name: X-RapidAPI-Host, Value: google-search72.p.rapidapi.com
Name: X-RapidAPI-Key, Value: YOUR_API_KEY
Click the PREVIEW button. Flatten and expand the nested datasets and delete the columns that are not required. Then click IMPORT.
The data will be imported into Google Sheets in a moment. Below is a snippet of the Google Search results imported into Google Sheets.
Once the Google Search results have been imported into Google Sheets, they can be updated automatically at regular intervals–hourly, daily, weekly or on custom days. And these data will be sent to you via email and or Slack so that you stay updated on any issue or topics of your interest. Click here to learn how to set a refresh schedule and set up alerts in Amigo.
With Amigo you can easily extract data from various sources and import them to Google Sheets in a few clicks. Visit our blog to discover more.
Some recommended articles:
How To Scrape And Import Google News Data To Google Sheets