Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I need to disallow some URLs on my site but I am not sure how to do that. I have a site that has products and reviews. When someone makes a review, the site generates a URL automatically like this:

mysite.com/addreview_1.htm
mysite.com/addreview_2.htm
....
mysite.com/addreview_9999.htm

I need some way to disallow all the URLs which will appear in the future.

share|improve this question
    
Adding commands in robots.txt file will not stop URLs showing in a sitemap.xml file. –  Max May 29 '13 at 13:20
    
I'm having some difficulty understanding what you want. Do want to allow robots to crawl the URLs for the reviews that have already been created, but prevent them from crawling the URLs for reviews that have yet to be created? –  Stephen Ostermiller May 29 '13 at 13:52
    
The nagging query would seem to be why you would want to disallow the crawling of all the review pages? –  w3d May 29 '13 at 14:51
add comment

2 Answers 2

You can add a wildcard entry to the robots.txt like:

Disallow: /addreview*

Google and other big players will honor the wildcards, but as this is a more recent addition to the robots.txt specification, there are probably still crawlers that ignore it.

This will also only work if the URLs you want to disallow have a common element that is not found in URLs you want crawled.

share|improve this answer
    
I added Disallow:/addreview_* and hope this will help. Thanks for reply! –  miller55 May 29 '13 at 13:21
2  
This will work, but you don't actually need the wild card. Simply Disallow: /addreview will suffice: developers.google.com/webmasters/control-crawl-index/docs/… –  Max May 29 '13 at 13:22
add comment

The original robots.txt specification has no concept of "full" URL. Whatever you specify as value for Disallow is always the start of the URL paths you want to block.

For example, see this robots.txt:

# robots.txt for example.com
User-agent: *
Disallow: /foobar.html

This will obviously block example.com/foobar.html. But it will also block:

  • example.com/foobar.html?foo=bar
  • example.com/foobar.html.zip
  • example.com/foobar.html.for.example
  • example.com/foobar.html/foo/bar

So, in your case you just need:

User-agent: *
Disallow: /addreview

It will block all URLs that begin with the string addreview:

  • example.com/addreview
  • example.com/addreview.html
  • example.com/addreview_1.htm
  • example.com/addreview_9999.htm

But it will also block an URL like (let’s assume it exists) example.com/addreviewer, of course. Which may or not what you want (depends on all your URLs you use).

So you need to find a part of a starting URL paths that matches to all the URLs you want to have blocked and doesn’t include any others.

share|improve this answer
    
Some spiders (notably baidu, the chinese search engine) only match full directories and documents. Disallow: /foo would match example.com/foo and example.com/foo/bar.html, but not example.com/foobar –  Stephen Ostermiller Jun 1 '13 at 2:26
    
@StephenOstermiller: Yeah, but they are not following the original robots.txt specification then. I guess there are several other bots that behave differently and even more bots that simply ignore any robots.txt. –  unor Jun 1 '13 at 10:44
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.