Google webmaster tools reports "Network Unreachable" for robots.txt, sitemap.xml, and all pages in the sitemap

Question

webmaster tools for my site displays that

robots.txt unreachable

and for all links in sitemap it says

network unreachable. sitemap.xml unreachable.

These appear in crawl stats page.

I discussed with the support team of my hosting and they said...

Hi,

I have verified apache logs, i cannot see any issues on your website/webserver/

Possible issues.

1. There may the routing issue from the googles server to our server.

2. When a google bots hits goes high the IP will be automatically blacklisted by our firewall to avoid server loads & downtimes.

As we donot have access to their services, We cannot able to give details of their details/logs etc.

The sitemaps link shows an exclamation mark which means the file was not reachable.

What could be the problem and how to solve it?

Answer 1

Try crawl your site using a fake user agent ID matching Googlebot. (e.g. A1 Website Analyzer allows you change user agent ID)

Some website/webserver "protection" systems also test on if e.g. robots.txt get requested, simultaneous connections etc. You can try run some tests to see how agressive your (if any) protection system its.

However, if your server admin himself states they have an anti bot system then... likely that is the reason.

for future you could try add crawl-delay to robots.txt