New answers tagged robots.txt
2
inhales
While X-Robots-Tag and meta robots are equivalent, robots.txt is different. The former is about indexing, while the latter is about crawling/visiting.
Tell bots not to visit a URL by using robots.txt.
Use only one of the three for each URL. Using both X-Robots-Tag and meta robots on a URL is redundant because they are equivalent, and using both ...
1
Google expanded the documentation for robots.txt for user agents that support the Allow directive. The rule that Googlebot uses (and what Google is trying to make standard) is that the longest matching rule wins.
So when you have:
Disallow: /norobots/
Disallow: /nobot/
Allow: /*.html$
Allow: /****.gif$
/norobots/index.html is blocked because it ...
2
I would not recommend trying to set up your site to disallow everything except certain directories.
The home page of your site will be blocked from crawling. Because most sites get many links to their home page, your site will be throwing away SEO value from a lot of incoming links. You will need to have direct external links to your deep crawlable ...
2
What you have would seem to be "about" correct, assuming you have the appropriate User-agent directive that precedes this?
Disallow: /
Disallow: /*
However, you don't need to repeat the same directive, one with a trailing * and one without. robots.txt is prefix matching. The trailing * is superfluous and these match the same URLs.
User-agent: *
...
1
For each domain, Google will follow the robots.txt at the root of the domain, nothing more and nothing less. Google will be unaware of any reverse proxies.
So if sith.edu adds disallow /rogueone to their robots.txt, Google will not crawl any of the URLs under sith.edu/rogueone (though it could crawl the same content at different URLs under the real rogueone ...
Top 50 recent answers are included
