New answers tagged noindex
0
If you want to use htaccess, you can use the http header X-Robots-Tag: noindex to accomplish what you need for the specific files you need, or with a pattern.
You can also provide other instructions, such as noarchive, nofollow, or even be specific for a particular crawler, for example Googlebot:
X-Robots-Tag: googlebot: nofollow
X-Robots-Tag: otherbot: ...
-1
You never know, how certain page will get visitors after creation. Thats why your strategy should be like
Set all new urls on index, follow,
After three months after publishing set all URLs with less then 10 organic visits à day to noindex, follow,
After they are away from index, set them with meta robots to noindex, nofollow and links to them to rel="...
Top 50 recent answers are included
