New answers tagged robots.txt
1
Nowadays, don't worry about it. Although we don't set the x-robot http header noindex for static file css, js, Googlebot know what file should be indexed or not. Except image, because the images can grab organic traffic for our site.
Btw, So many people still don't understand what is robots.txt file and noindex tag. Robots text file what should to crawl and ...
1
From Yandex Webmastertools Support
From February 22, 2018, Yandex doesn't take into account the Crawl-delay directive.
So what you need to do is to login to Yandex.Webmasters and set speed:
To set the speed at which robots downloads the site pages, use the Site crawl rate in Yandex.Webmaster.
1
As an observation the images on blogspot/blogger.com are hosted on http://1.bp.blogspot.com/....file.jpg instead of my own subdomain (http://domain.blogspot.com) and I wonder if this is the cause of the issue?
That is indeed the cause of the issue. A robots.txt file only affects a single subdomain, so it won't affect anything on a different subdomain.
I ...
3
Google allows the robots.txt 301 redirection you're talking about:
Google follows at least five redirect hops as defined by RFC 1945 for HTTP/1.0 and then stops and treats it as a 404.
https://developers.google.com/search/reference/robots_txt
I couldn't find any information about Bing's crawler.
It's my suspicion that smaller, poorly-written, non-...
Top 50 recent answers are included
