I noticed a sharp drop in Adsense earnings on one of my sites this week and after a bit of digging in the Search Console found errors suggesting there was a problem with my robots.txt. I checked this and it is absolutely fine.

When I went to the old console and ran some fetch as Google (desktop) I got Unreachable for all the URLs I tried on that domain (including /). However, when I tried the mobile crawler all were fine, including rendering.

A bit of background. I host several websites on a shared IP address. I recently moved server / IP address. This was about 4 weeks ago and I haven't had any issues until a few days back, and only on this site.

I do run application firewalls (Fail2Ban and mod_security), but again these shouldn't have changed suddenly. Plus I have now disabled them.

I have checked the site from a few IP addresses to check it is working properly and it is.

Any ideas - or is it it just a wait and see...

1 Answer 1


OK, should have mentioned to start with this was a Drupal site, so obviously an extra five layers of complexity....

After a bit of rooting around I did find dome 500 errors. It seems like there was a problem with the one of the caching settings which was throwing an error. However, for some reason the cache was working fine for the IP addresses I tried, but apparently not for Google. Don't think this was a case of browser caching as I looked at some fairly obscure pages.

Anyway all seems OK again, panic over!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.