New answers tagged

1

Is there a way to quiet down MJ12Bot ambitions The MJ12Bot reportedly obeys robots.txt and the (non-standard) Crawl-Delay directive: How can I slow down MJ12bot? You can easily slow down bot by adding the following to your robots.txt file: User-Agent: MJ12bot Crawl-Delay: 5 Crawl-Delay should be an integer number and it signifies number ...


0

Apparently, everyone was sick and tired with receiving useless Googlebot-Mobile traffic, and must have had it blocked, such that Google has decided to ignore the relevant robots.txt entries, giving us less choice of what to allow and not allow. http://support.google.com/webmasters/answer/1061943 ...


2

Robots.txt Google on occasions has been known to ignore robots.txt, it should never be treated as a guarantee. Sensitive pages should always some of form of authentication, or you could opt to block those user agents by returning a 403 forbidden status. However, this is not the issue. Google Probing Googlebot is known to probe sites with different user ...


4

You are right, rel=nofollow is only for links. <img> tag can't have rel=nofollow attribute. What do you mean by disallow google to crawl out of my website through images? Google bot doesn't crawl out when it encounters an external image. It just gives a positive (or sometimes negative) recognition to it and may index it - Google can't crawl an ...


1

In your situation, Don't include "www" because web crawlers that do read the sitemap directive from your robots.txt will attempt to access the "www" version of your sitemap and then be redirected to the non-www version of the URL. Not only will you be wasting a couple extra bytes by including "www", but the crawlers that parse the sitemap directive ...



Top 50 recent answers are included