Our crawler is designed to help you make informed and actionable decisions about your and your competitors’ websites and specific pages.
It allows you to crawl domains with a view to comparing competitor sites for structure, content, while identifying and fixing technical SEO and content errors.
For hardcore technical SEO audits, Authoritas’ powerful site crawler can help you identify broken links and duplicate content. Find what’s broken, and fix it fast.
Our crawler can answer many questions about your own website and your competitors’ sites.
Use the in-built filters to find site-wide issues quickly:
By default, we will crawl up to 20,000 pages per domain. If you need more pages ask about our Total Site Crawl service which can crawl potentially million-page websites.
You can display and easily export data for each URL such as:
Tip: Use the quick “Show me” Filters help speed up the insights you can act on;
You can create a crawl schedule for your site and even crawl your competitors’ sites, crawling up to a maximum of 20,000 pages per crawl.
Advanced crawling options include:
Our crawler is geared up to help you analyse and audit your content, learn more about what our crawler can do for you.
If you need to crawl hundreds of thousands or even millions of pages then ask your Client Success Manager about the Total Site Crawl service.
“Authoritas offers a complete solution for creating effective SEO strategies. Their enterprise-level “big data” market analysis, combined with many tools for on-the-job SEO management mean they are the platform of choice for us. Their account management and responsiveness is also first rate.”
Managing Director, SEO Works
Indexing and crawling are two different but related concepts that are important to understand for any website owner and for anyone interested in search engine optimisation. When Google sends its spider (Googlebot Smartphone) to crawl your website it starts to visit pages to download and analyse your content and website code. Google then decides how much of your site content is worthy to get indexed. Google will come back and crawl your site at the time interval that it determines.
This is a fairly large topic. A good starting point is creating a Google Webmaster Tools account and reading Google’s guidelines for webmasters.
The Authoritas spider follows similar methodology to Googlebot, to crawl all the pages on your website. The amount of data and depth of data shown by Google in Google Webmaster Tools will differ from the Authoritas crawler.
But the Authoritas crawler will also merge the website crawl data with other useful metrics from Google or Adobe analytics, backlink data from Majestic and Authoritas ranking data to give you far more actionable advice.
Oh, and of course you cannot crawl competitors’ websites using Googlebot!
Crawling refers to the actions undertaken by a software programme commonly referred to as a web spider which visits your website and downloads content from your pages.
Indexing refers to how the crawled and downloaded data is stored and presented to users.
Crawling and indexing is something that all search engines do.
This will vary according to the size of your website and how authoritative it is in Google’s eyes. A well-established site with millions of users and rapidly changing content, like the BBC home page for example, will get crawled extremely frequently (multiple times per hour) but a newly indexed websites with a few static pages and no links may only get crawled every few days.
To see how often you are getting crawled you can see a chart in your Google Search Console account.
To see which search engine bots are crawling your site the best place to start is to check your server logs. Server logs can be difficult to get at, depending on your technology stack and technical skills and can be very large. We have a new product in beta that makes this much, much easier – feel free to contact us for a free trial.
When SEOs talk about crawling, they are normally referring to one of two things;
If you are interested in trying these SEO tools for your sites, then please book a brief call with one of our platform experts to explore whether we have what you need.
