New answers tagged robots.txt
4
You can add a "Noarchive" Directive to prevent Google from caching specific web pages.
<meta name="robots" content="noarchive">
or for just Google:
<meta name="googlebot" content="noarchive">
There is no penalty for doing this.
Here are some typical scenarios to use Noarchive:
Advertising you ...
7
The robots.txt standard controls whether the bot can view foo.html under any circumstances. Just because the bot sees a link to it from a different site doesn't give the bot permission to sneak a peek at it!
Suppose I have a URL /foo.html for which crawling is blocked using robots.txt. Then the bot will not crawl the page, but it might still be indexed if ...
2
Based on official guidelines, don't use the robots.txt file for canonicalization purposes.
You can blocks sessionid URLs using Parameter Handling
If you block using robots.txt, you may lose the search ranking signal from the link to your URL with sessionid/params.
2
Disallow: /*;jsessionid
A more precise syntax for what you want. A trailing * does nothing.
Top 50 recent answers are included