webmaster tools for my site displays that

robots.txt unreachable

and for all links in sitemap it says

network unreachable. sitemap.xml unreachable.

These appear in crawl stats page.

I discussed with the support team of my hosting and they said...


I have verified apache logs, i cannot see any issues on your website/webserver/

Possible issues.

  1. There may the routing issue from the googles server to our server.

  2. When a google bots hits goes high the IP will be automatically blacklisted by our firewall to avoid server loads & downtimes.

As we donot have access to their services, We cannot able to give details of their details/logs etc.

The sitemaps link shows an exclamation mark which means the file was not reachable.

What could be the problem and how to solve it?

  • 1
    Can you reach your robots.txt or sitemap through your browser?
    – John Conde
    Commented Feb 8, 2011 at 21:57
  • Yes. i can read using my browser. Commented Feb 10, 2011 at 7:01

1 Answer 1


Try crawl your site using a fake user agent ID matching Googlebot. (e.g. A1 Website Analyzer allows you change user agent ID)

Some website/webserver "protection" systems also test on if e.g. robots.txt get requested, simultaneous connections etc. You can try run some tests to see how agressive your (if any) protection system its.

However, if your server admin himself states they have an anti bot system then... likely that is the reason.

for future you could try add crawl-delay to robots.txt

  • Ok. i will test and i will delay the crawl rate. and will get back here. Commented Feb 10, 2011 at 7:02
  • Yes, the hosting has blocked ip which ping very often. i contacted them and they said this. also they requested if it is possible to get the ip of google so that they can unblock Commented Feb 11, 2011 at 6:19
  • 1
    Your webhost could lookup IPs to see if Googlebot: googlewebmastercentral.blogspot.com/2006/09/…
    – Tom
    Commented Feb 12, 2011 at 20:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.