Hot answers tagged web-crawlers
Are these bots that index my site(s) or are they malicious by nature and should be banned using an IP deny filter? You tell us. If they're interacting with your site as any normal user would there is no need to block them. If they're using the site more than visitors from other regions then your site may appeal to them then users from other regions. ...
This happens when Google or Bing discovers your site and has not been told not to index the site. This happens when there is a link or redirect to the site and the robots.txt restricts the search engine from the site. However, this is not the same as telling a search engine not to index the site. Put <meta name="robots" content="noindex"> in the ...
Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.
Only top voted, non community-wiki answers of a minimum length are eligible