Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

webmaster tools for my site displays that

robots.txt unreachable and for all links in sitemap it says network unreachable. sitemap.xml unreachable.

These appear in crawl stats page.

I discussed with the support team of my hosting and they said...

Hi,

I have verified apache logs, i cannot see any issues on your website/webserver/

Possible issues.

  1. There may the routing issue from the googles server to our server.

  2. When a google bots hits goes high the IP will be automatically blacklisted by our firewall to avoid server loads & downtimes.

As we donot have access to their services, We cannot able to give details of their details/logs etc.

The sitemaps link shows an exclamation mark which means the file was not reachable.

What could be the problem and how to solve it?

share|improve this question
1  
Can you reach your robots.txt or sitemap through your browser? –  John Conde Feb 8 '11 at 21:57
    
Yes. i can read using my browser. –  Jayapal Chandran Feb 10 '11 at 7:01
add comment

1 Answer

up vote 1 down vote accepted

Try crawl your site using a fake user agent ID matching Googlebot. (e.g. A1 Website Analyzer allows you change user agent ID)

Some website/webserver "protection" systems also test on if e.g. robots.txt get requested, simultaneous connections etc. You can try run some tests to see how agressive your (if any) protection system its.

However, if your server admin himself states they have an anti bot system then... likely that is the reason.

for future you could try add crawl-delay to robots.txt

share|improve this answer
    
Ok. i will test and i will delay the crawl rate. and will get back here. –  Jayapal Chandran Feb 10 '11 at 7:02
    
Yes, the hosting has blocked ip which ping very often. i contacted them and they said this. also they requested if it is possible to get the ip of google so that they can unblock –  Jayapal Chandran Feb 11 '11 at 6:19
1  
Your webhost could lookup IPs to see if Googlebot: googlewebmastercentral.blogspot.com/2006/09/… –  Tom Feb 12 '11 at 20:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.