1

I need to disallow some URLs on my site but I am not sure how to do that. I have a site that has products and reviews. When someone makes a review, the site generates a URL automatically like this:

mysite.com/addreview_1.htm
mysite.com/addreview_2.htm
....
mysite.com/addreview_9999.htm

I need some way to disallow all the URLs which will appear in the future.

3
  • Adding commands in robots.txt file will not stop URLs showing in a sitemap.xml file.
    – Max
    May 29 '13 at 13:20
  • I'm having some difficulty understanding what you want. Do want to allow robots to crawl the URLs for the reviews that have already been created, but prevent them from crawling the URLs for reviews that have yet to be created? May 29 '13 at 13:52
  • The nagging query would seem to be why you would want to disallow the crawling of all the review pages?
    – MrWhite
    May 29 '13 at 14:51
1

You can add a wildcard entry to the robots.txt like:

Disallow: /addreview*

Google and other big players will honor the wildcards, but as this is a more recent addition to the robots.txt specification, there are probably still crawlers that ignore it.

This will also only work if the URLs you want to disallow have a common element that is not found in URLs you want crawled.

2
0

The original robots.txt specification has no concept of "full" URL. Whatever you specify as value for Disallow is always the start of the URL paths you want to block.

For example, see this robots.txt:

# robots.txt for example.com
User-agent: *
Disallow: /foobar.html

This will obviously block example.com/foobar.html. But it will also block:

  • example.com/foobar.html?foo=bar
  • example.com/foobar.html.zip
  • example.com/foobar.html.for.example
  • example.com/foobar.html/foo/bar

So, in your case you just need:

User-agent: *
Disallow: /addreview

It will block all URLs that begin with the string addreview:

  • example.com/addreview
  • example.com/addreview.html
  • example.com/addreview_1.htm
  • example.com/addreview_9999.htm

But it will also block an URL like (let’s assume it exists) example.com/addreviewer, of course. Which may or not what you want (depends on all your URLs you use).

So you need to find a part of a starting URL paths that matches to all the URLs you want to have blocked and doesn’t include any others.

2
  • Some spiders (notably baidu, the chinese search engine) only match full directories and documents. Disallow: /foo would match example.com/foo and example.com/foo/bar.html, but not example.com/foobar Jun 1 '13 at 2:26
  • @StephenOstermiller: Yeah, but they are not following the original robots.txt specification then. I guess there are several other bots that behave differently and even more bots that simply ignore any robots.txt.
    – unor
    Jun 1 '13 at 10:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.