Big discrepancy in number of known pages in Google Search Console and ScreamingFrog

Question

In a technical SEO audit I used both Google Search Console and ScreamingFrog crawl to review the status errors, however there is big discrepancy in the number of pages that Google knows about, and the ones that ScreamingFrog finds (without any limitations or change in settings of the Spider).

I have 350 overall pages in ScreamingFrog and 7.2K pages in GSC. Same for the different status errors, and there is mismatch on which tool finds what. For example, GSC finds 16 404s but they are totally different pages from the 404s ScreamigFrog finds (just one overlapped).

Did you experience the same?

Answer 1

I put together an FAQ on common reasons why a crawl and GSC (or Google site: index queries) might not match -

https://www.screamingfrog.co.uk/seo-spider/faq/#why-does-the-number-of-urls-crawled-not-match-the-number-of-results-indexed-in-google-or-errors-reported-within-google-webmaster-tools

Also worth mentioning, the comment above is outdated - The SEO Spider can render pages (and see executed JavaScript) in the same way as search engines.

Hope that helps!

Btw - I think I am meant to disclose I am the founder of Screaming Frog. This is an FAQ, and answering direct questions, and not meant to be promotional.

Answer 2

The reason why GSC and SF are probably finding different 404 urls is likely due to server response failure. If one of the web crawlers tries to load a page but it doesn't respond, it will report a 404. This can happen if your webhost's bandwidth or server is overworked.

To check the status of your webpages you can use a header status checker such as: http://tools.seobook.com/server-header-checker/

As long as the page loads from your server it will display the true header status.

Answer 3

For example, GSC finds 16 404s but they are totally different pages from the 404s ScreamigFrog finds (just one overlapped)

The 404s reported by these different tools will always differ, because of the differences in how these tools discover URLs.

Screaming Frog (SF) spiders your site from a specific root URL. So, the reported 404s are only going to consist of internal links that are linking to non-existent pages.

Whereas GSC is also going to report 404s where any inbound link from an external/third party site is linking to a non-existent page on your site. This could be huge.

If you remove a page on your site (together with all internal links to that page) then you wouldn't expect SF to ever report this as a 404 (since there are no links to it). However, if GSC has previously crawled this page then you would now expect it to report it as a 404 (regardless of whether there are any internal links to it) - which is correct.

You might expect GSC to contain the superset of 404s (including all those found by SF), however, this depends on how recently and how thoroughly Google has crawled your internal link structure.

Answer 4

After more than two hours playing with the ScreamingFrog Spider settings the best I could do was to increase the crawled pages to 700 (out of 7.5K on the website). I extracted All known pages from GSC and uploaded them in List mode in ScreamingFrog for crawl.

The downside was that for some page types GSC only gave me the first 1000 pages, but other than that it worked - I managed to crawl 6.5K out of 7.5K which is feasible.

Thanks everyone for jumping in to help, especially to Dan Sharp. Much appreciated.