Tag Info

New answers tagged

2

Good catch! If you're blocking a page with robots.txt then crawlers will not able able to read the noindex meta tag. In these cases you should send the x-robots-tag HTTP header either via server-side code or .htaccess. .htaccess example: X-Robots-Tag: noindex Sample PHP code: header('X-Robots-Tag: noindex');


4

Yes. Whatever the web root for the subdomain is where you would put a robots.txt for that subdomain's contents. It will not affect the root domain and the root domain's robots.txt will not affect the subdomain.


1

Newer apple versions request these files (in this order) /apple-touch-icon-120x120-precomposed.png /apple-touch-icon-120x120.png /apple-touch-icon-precomposed.png /apple-touch-icon.png


0

You should be able to use a $ sign which indicates the end of the string. For example Allow: /$ If you go to the bottom of Google's robots.txt help, you can see a similar example. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt



Top 50 recent answers are included