Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have two sites I am currently working on which is a shopping cart and another shopper...Both are under the same domain name so for example

http://someurl.com/first_url 
http://someurl.com/second_url 

Both these urls have tons of pages below them

http://someurl.com/first_url/product1
http://someurl.com/first_url/product2 
http://someurl.com/first_url/product3

http://someurl.com/second_url/product1
http://someurl.com/second_url/product2 
http://someurl.com/second_url/product3

How do I disallow the main url /first_url and /second_url to and all the subpages and sub directories ...I was thinking something like this but I don't know if this works or not and whether it was the correct syntax

Disallow: /first_url/*/
Disallow: /second_url/*/
share|improve this question

migrated from stackoverflow.com Jul 15 '11 at 19:49

This question came from our site for professional and enthusiast programmers.

    
I think your syntax is right but not sure about the extra / –  Alex Jul 15 '11 at 16:03

1 Answer 1

up vote 3 down vote accepted

You don't need the trailing slash to block those directories. This will work:

Disallow: /first_url/*
Disallow: /second_url/*

This Google help page covers this quite well.

share|improve this answer
    
You can just do "Disallow: /first_url". The trailing wildcard is ignored. According to developers.google.com/webmasters/control-crawl-index/docs/… –  Evgeny Jun 10 at 23:44

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.