Crawling your website and finding technical and content issues which are holding you back is an integral part of good SEO. And should you wish, we’ll crawl your website on a weekly basis with a customized depth of crawl, checking potential issues such as HTTP status, canonical tags, page speed and unindexed pages. All of this data is also available via our XML-RPC API.

About the crawl health module

The Authoritas Crawl Health module will give you access to the detail behind the Site Audit module. It reflects all the issues we have encountered when crawling your site. The chart that is displayed will give you an impression of the growth or decline of these issues over time.

Data that can be downloaded for future inspection

Page – the URL we have spidered
Status – the HTTP status of the page (also indicates whether pages are being blocked by meta tags)
Unfriendly URL – if the URL of a page is considered unfriendly to search engines, the reason will be shown in this column
Speed – this will give the download speed for the HTML of each respective URL, i.e. the time taken from requesting a URL to being served that URL (it does not include all external files which may also require downloading)
Canonicals – this will show the canonical value for any URLs that have a rel=canonical tag (these can help to deal with potential issues with duplicate content)
Internal Links In – number of links pointing to this URL from other pages on the same domain
Internal Links Out – number of external links from this URL to other pages on the same domain
External Links In – number of links from other referring domains pointing to this particular URL
External Links Out – number of links from this URL pointing to URLs on other domains

Stipulating where we crawl

Note: Our site spider will only crawl within the specified sub-directory you specify if you set up a campaign that way (e.g. rather than just, but keyword rankings, inbound links, and sitemap checks all operate on the domain-level.

You’ll find our bot listed in your server logs as ‘Curious George –’.

If you’d like to find out more about our crawling, why don’t you arrange a demo to discuss your needs?

<< Technical Auditing Content >>

Find out how our platform can help your agency or enterprise