19

I have two sites I am currently working on which is a shopping cart and another shopper. Both are under the same domain name so for example

http://example.com/first_url 
http://example.com/second_url 

Both these URLs have tons of pages below them

http://example.com/first_url/product1
http://example.com/first_url/product2 
http://example.com/first_url/product3

http://example.com/second_url/product1
http://example.com/second_url/product2 
http://example.com/second_url/product3

How do I disallow the main URL /first_url and /second_url too and all the subpages and sub directories? I was thinking something like this but I don't know if this works or not and whether it was the correct syntax

Disallow: /first_url/*/
Disallow: /second_url/*/

migrated from stackoverflow.com Jul 15 '11 at 19:49

This question came from our site for professional and enthusiast programmers.

  • 1
    I think your syntax is right but not sure about the extra / – Alex Jul 15 '11 at 16:03
26

You don't need the trailing slash to block those directories. This will work:

Disallow: /first_url/*
Disallow: /second_url/*

This Google help page covers this quite well.

-1

You can use this as your default robots.txt:

User-agent: *
Disallow: first_url/*
Disallow: second_url/*

The star will enable everything under that category to be disallowed. You can even disallow one specific thing under that category by typing that specific url.

  • 4
    This is not different than the accepted answer other than you are missing the starting slash. Disallow rules in robots.txt should all start with a slash. That is how all the examples are in the documentation – Stephen Ostermiller Dec 21 '17 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy