Hot answers tagged web-crawlers
Stephen Ostermiller’s answer explains how to configure it in Google Webmaster Tools (which answers your question). If you care about other search engines, too, or in addition, you could use the Canonical link type. You could specify the canonical link in a HTTP header, or you could specify it in the HTML: On ...
Google Webmaster Tools does have the ability to tell Googlebot to ignore parameters. You can tell Googlebot to ignore the parameter "MobileOptOut" anywhere it appears on your site. To do so: Navigate in Webmaster Tools to Crawl -> URL Parameters. Click the Configure URL parameters » link. Click the Add Parameter button. Put in the name: "MobileOptOut". ...
Pages that are not frequently updated or are not important in the eyes of a search engine, after time, will be crawled (visited) by search engines less and could eventually be dropped from Google's index altogether eventually. This doesn't mean there is an issue at all and is quite common for large sites with many pages.
This happens when Google or Bing discovers your site and has not been told not to index the site. This happens when there is a link or redirect to the site and the robots.txt restricts the search engine from the site. However, this is not the same as telling a search engine not to index the site. Put <meta name="robots" content="noindex"> in the ...
Apparently Google will not crawl them and will report them as being blocked in Google Webmasters Tools. If that error message, which does not affect your rankings, is annoying to you then you will need to remove those pages from your sitemap.
Only top voted, non community-wiki answers of a minimum length are eligible