Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

After setting up a blog at blogspot.com I added the site to my webmasters tools of google.

I noticed an error: "Restricted by robots.txt" and looked a little into the matter. I found in the robots.txt of blogspot that Google prevents the /search directory by default in order to avoid duplicate entries in their search engine. As tags can be found under /search/label/SOME_TAG, these are not indexed too.

I run different sites, especially one is an important e-commerce site for me. For each product, we use tags. Each tags leads to a separate site like /tags/tag1/ and lists all products that are linked to this tag.

And this leads me to my question:

Should I also block search and/or tag pages on my sites by using robots.txt?

I feel google might punish my pagerank/results for using this "low-quality" content. However I think they are quite useful. We provide a short description of each tag and how the listed products can be used for the problem the tag describes.

Moreover users landing on tags-pages have very high bounce rates (>90%), which is far above the average bounce rate.

So, what is the best practice?

share|improve this question
up vote 1 down vote accepted

I find a lot of pages in Google's search results from the various StackExchange sites that are the tag pages, most popular questions, etc. So it doesn't look like Google considers them to be low quality. I would leave them unblocked unless you see some sort of problem such as a Panda update having a very negative effect on your rankings. Otherwise these pages are a source of traffic, and although the bounce rate is high, you are receiving visitors who are going further into your site thanks to these pages being in the search results.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.