New answers tagged robots.txt
Good catch! If you're blocking a page with robots.txt then crawlers will not able able to read the noindex meta tag. In these cases you should send the x-robots-tag HTTP header either via server-side code or .htaccess. .htaccess example: X-Robots-Tag: noindex Sample PHP code: header('X-Robots-Tag: noindex');
Yes. Whatever the web root for the subdomain is where you would put a robots.txt for that subdomain's contents. It will not affect the root domain and the root domain's robots.txt will not affect the subdomain.
Newer apple versions request these files (in this order) /apple-touch-icon-120x120-precomposed.png /apple-touch-icon-120x120.png /apple-touch-icon-precomposed.png /apple-touch-icon.png
You should be able to use a $ sign which indicates the end of the string. For example Allow: /$ If you go to the bottom of Google's robots.txt help, you can see a similar example. https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt
Top 50 recent answers are included