In this article, we’ll learn how to scrape Google Search results and automatically pull the data to Google Sheets. We can scrape data such as titles, links, and snippets among other things–all without a single line of coding.
In the snippet below, you can see the page title, link and snippets for the keyword that we’ve scraped from Google Search and imported into Google Sheets.
Scraping the Google Search results page can quickly and easily be done using a simple Google Sheets extension called Amigo. With Amigo we can scrape Google Search results without a single line of coding and set it to retrieve the Google Search results periodically. We only need to specify the search keyword and Amigo will fetch us the latest results for that query without any additional manual input.
Scraping Google Search results with Amigo is a cakewalk–and, may we say, a cake feast because it’s fun! So without further ado, let’s begin.
Steps to scrape Google Search results:
How to scrape Google Search results
Install the Amigo Data extension in Google Sheets
Install the Amigo Data Google Sheets add-on from the Google Workspace Marketplace by clicking the link below to scrape Google Search results.
Alternatively, you can get it from Google Sheets.
- Open Google Sheets
- Go to Extensions→Add-ons→Get add-ons
- Search for Amigo and install it.
Get the Google Search API endpoint URL
We’ll use a Google Search scraper API on RapidAPI–an API marketplace–to scrape Google Search results and pull the search results to Google Sheets.
Search for Google Search API and click the most relevant one from the results. or visit the Google Search API documentation page.
Click on the Subscribe to Test button and subscribe to a plan of your choice, post which you’ll be able to use the Google Search API to scrape Google Search results and import the search results to Google Sheets.
Click Endpoints to go back to the Google Search API documentation. On the left is the list of the type of search results that we can scrape–web and image.
We can scrape either web search results or images. The method is the same for both. We’ll scrape Google Search results for web search. Click the Web Search endpoint.
Under Required Parameters in the query field, enter the keyword that you want to scrape Google Search results–eg. metaverse. You can also use a combination of words as a query.
Then fill in the optional parameters to filter the search results. These parameters are however, as the name suggests, not required. In the example below, the region is set to US, and will return the top 20 results in English sorted by relevance.
After filling in the parameters, click the Test Endpoint to run the query and verify whether the API is working. If it is, then we should see the results returned in JSON format.
In the Code Snippets section, we’ll find the Google Search API endpoint URL along with the parameters and the headers.
Change the language to Shell→cURL from (Node.js) Axios. The query parameters are appended in the URL so that we do not have to enter the parameter manually to pull the Google Search results to Google Sheets.
Copy the URL and also note the headers. These will be used to scrape Google Search results and import the search results to Google Sheets.
Import the Google Search results to Google Sheets
Open Google Sheets and launch Amigo to retrieve the Google Search results and pull the data to Google Sheets. To connect to Amigo, click Extensions→Amigo→Connect.
We’ll be presented with a list of sources to connect and import data. Select Custom API.
Enter the API endpoint URL in the API URL field. Make sure that the apostrophes are not included while copying the URL; otherwise the API won’t work.
Then expand the Details tab and enter the headers mentioned in the Code Snippets. The characters to the left of the colon are the header names and those to the right are the header values. Enter these as separate headers. Do not include the apostrophes.
Name: X-RapidAPI-Host, Value: google-search72.p.rapidapi.com
Name: X-RapidAPI-Key, Value: YOUR_API_KEY
Click the PREVIEW button to retrieve the Google Search results and preview them. Flatten and expand the nested datasets and delete the columns that are not required. Then click IMPORT.
The Google Search results will be imported into Google Sheets in a moment. Below is a snippet of the Google Search results imported into Google Sheets.
Automatically fetching Google Search results to Google Sheets
Once we’ve scraped the Google Search results and imported the search results to Google Sheets, we can let Amigo do the heavy lifting of automatically updating the search results.
After successfully importing the Google Search results to Google Sheets, set a refresh schedule to update the search results periodically–hourly, daily, weekly, or on custom days. And, if you so wish, these data will be sent to you and your team via email and or Slack so that you stay updated on any issue or topics of your interest. Learn how to set a refresh schedule and set up alerts in Amigo.
With Amigo you can easily extract data from various sources and import them to Google Sheets in a few clicks. Visit our blog to discover more.
FAQs
How can I scrape Google search results for a specific location or language?
To scrape Google Search results for a specific location or language, specify the region using the two-letter country code in the “gl” parameter. And enter the language code in the “lr” field. For example, to scrape Google Search results in Spanish in the US, we’d use “US” as the region code and “ES” as the language code.
How can I handle pagination when scraping Google search results?
This Google Search API allows you to scrape a maximum of 20 results at a time. If you want to scrape the next 20 Google Search results, then enter the number of results to fetch in the “num” parameter, and mention “20” in the “start” parameter to retrieve the next batch of results. Also ensure that the “sort” parameter is kept the same as the previous import; otherwise there may be repetition or omission of results.
How do I automatically scrape Google Search results?
To automatically scrape Google Search results, simply set a refresh schedule and Amigo will retrieve the Google Search results and pull the data into Google Sheets according to the time and frequency you’ve set.
Some recommended articles:
How To Scrape And Import Google News Data To Google Sheets