Many "429" crawl errors in Google Webmaster Tools with GoDaddy Managed Wordpress Hosting and Cloudflare

Question

Following setup:

• GoDaddy Managed Wordpress Hosting + SSL
• Cloudflare as CDN

Google Webmaster Tools shows all pages with 429 crawl error. I tested manually and with tools like monitor.us and I couldn't find any issues loading any of the pages.

It used to work without any flaws before I added Cloudflare and SSL. What is the more likely cause for this issue? How do I best debug?

Answer 1

429 errors are:

429 Too Many Requests

The 429 status code indicates that the user has sent too many requests in a given amount of time ("rate limiting").

The response representations SHOULD include details explaining the condition, and MAY include a Retry-After header indicating how long to wait before making a new request.

Most web hosts don't limit the rate of crawling with this type of error. Search engines need to be able to crawl all your pages to be able to index your website. If Googlebot can't fetch your pages due to this error when it crawls, you are going to have SEO problems.

Here is a thread where somebody has a similar issue, also with GoDaddy hosting. One poster was able to crawl the problematic site with the Screaming Frog crawler and reproduce the issue. When they crawled faster, they got more 429 errors.

The only ways to get this problem fixed would be to:

• Contact GoDaddy and have them disable this error, or allow Googlebot to crawl faster
• Find a new hosting company and move your website to it
• Get Googlebot to crawl slower by adjusting the crawl rate in Google Webmaster Tools (but that only takes effect for 90 days at which point you would have to set it again)

Answer 2

[Disclosure: I run Product Management for Hosting at GoDaddy and am the acting PM on our Managed WordPress product]

Based on your question and feedback we have made a few changes to our settings:

1. Bot rate-limiting has been raised from 3 requests / second to 15 requests / second.
2. Bot rate-limiting now excludes static files. Only dynamic, un-cached content will be rate limited which should be the biggest win given how much of the content can be cached on the Managed WordPress product

Side note: I will have to go test the behavior with CloudFlare to confirm how the Googlebot works with that but we've opened it wider with these changes so you should be good to go.