Google desktop crawler showing unreachable - mobile crawler is working

Question

I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this and it is absolutely fine.

When I went to the old console and ran some fetch as Google (desktop) I got Unreachable for all the URLs I tried on that domain (including /). However, when I tried the mobile crawler all were fine, including rendering.

A bit of background. I host several websites on a shared IP address. I recently moved server / IP address. This was about 4 weeks ago and I haven't had any issues until a few days back, and only on this site.

I do run application firewalls (Fail2Ban and mod_security), but again these shouldn't have changed suddenly. Plus I have now disabled them.

I have checked the site from a few IP addresses to check it is working properly and it is.

Any ideas - or is it it just a wait and see...

Answer 1

OK, should have mentioned to start with this was a Drupal site, so obviously an extra five layers of complexity....

After a bit of rooting around I did find dome 500 errors. It seems like there was a problem with the one of the caching settings which was throwing an error. However, for some reason the cache was working fine for the IP addresses I tried, but apparently not for Google. Don't think this was a case of browser caching as I looked at some fairly obscure pages.

Anyway all seems OK again, panic over!