Google Big Query Integration

Both Microsoft Excel and Google Sheets are excellent spreadsheet packages and are extensively used on a daily basis by the SEO community at large.

The trouble is there’s only so much data wrangling you can do in either software package before you start to run into issues with data limits.

An excel spreadsheet can handle 1,048,576 rows and 16,384 columns at a time.  Google sheets has a cell limit of 10 million cells or 18,278 columns.  Big Query is a great alternative if you’ve started to reach these limits as it is designed to handle big data and is much more versatile – it’s also unlimited!

If you are an enterprise SEO team tracking hundreds of thousands (or even millions) of keywords daily then you need more firepower to store, manage, manipulate, analyse and visualise your SEO data.  

Google Big Query is the perfect solution and it’s relatively cost-effective.

Ask your SEO tool or platform provider for details of their Big Query integration.  

You’re quite likely to get a blank stare after this request, as, at the time of writing, Authoritas is the only major SEO platform with a comprehensive SEO platform to Google Big Query integration that allows for synchronisation and streaming of data to Big Query.

So, your fallback position is to ask for an API key, so that you can programmatically access and export your data to your Big Query tables.

This obviously does create a technical overhead for you, as this is a production system that will need to be created, monitored, maintained and updated in the future.

If you would like to explore alternative cheaper methods of getting and maintaining your marketing data into Big Query then please speak to the Authoritas team.

Well, this happens more often than you think and it does require a slightly more advanced level of SQL know-how.  

Big Query allows you to create materialized views of your data, which are like virtual versions of your table, transposed, filtered or oriented in a way that makes sense for the view you need.  You can achieve this by writing specific SQL queries and saving them in Big Query.

Fortunately for you, if you are using the Authoritas Big Query integration for Google Data Studio then this complex SQL has already been written by our technical team and there are multiple useful views of your data ready to be visualised waiting for you in your Big Query dataset.

Well, this is often the hard bit, especially if you’ve manually created the tables using .csv uploads that you’ve exported from other SEO tools.  

This process is fine once, but soon becomes a painful and tiresome exercise if you have to repeat this daily, weekly or just monthly.  You have to automate or suffer surrendering the majority of the benefits you will derive from using Big Query in the first place.

Using Big Query necessitates that you create a data pipeline that needs to be managed and monitored.  This requires a mid-level software engineer.  It’s not a big job but one that will take a few days to a few weeks depending upon how many data sources you are connecting to and the state of their APIs.

You need to consider how often each data source is updated.  For example, if you only pull in backlink data from Majestic once a month then that’s fine, but if you are checking new links across all your competitors daily then you need a better solution that uploads data daily.

You also should consider what happens if you change keyword tags or have other content that is updated multiple times a day; again, you will need the services of an engineer, as you need to stream the data to Big Query.

It's probably a lot faster and cheaper to switch SEO tool suppliers and let SEO platforms like Authoritas automate the whole ETL (Extract Transform Load) and maintenance process for you.

Get a Google API key, download the client libraries and find an engineer to setup a process to query and store the data returned for you on a schedule.  

Or, simply authorise the Authoritas platform to do this on your behalf! No engineer or costs required!

As far as we know, only Authoritas has a comprehensive Big Query connection that syncs and importantly streams data to your Big Query instance.  

This is included in Business Advanced and Business Expert level packages and above.