Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error.

The URL which been crawled incorrectly is:

www.abc.com/forum/index.php?option=com_forum

However, The correct page is:

www.abc.com/index.php?option=com_forum

Is robots.txt a correct way of excluding this? I'm thinking about using:

Disallow: /forum/

Will that not block legitimate content in the /forum/ folder of my site?

share|improve this question
    
If /forum/index.php?option=com_forum isn't the correct URL, what are you serving when a crawler hits that page? The correct thing would be to serve a 404 not found error, or a redirect to where the page actually exists. If you do that, you will have no problems with duplicate pages. –  Stephen Ostermiller Nov 25 '13 at 19:41

1 Answer 1

If www.abc.com/forum/index.php?option=com_forum is the only link that's been mis-read then you can just add Disallow: /forum/index.php to your robots.txt file. Adding Disallow: /forum/ will tell robots to ignore everything in that directory, which doesn't sound like what you want.

You could also add a 301 redirect from www.abc.com/forum/index.php?option=com_forum to www.abc.com/index.php?option=com_forum to tell robots what the URL should be. This will also help any users who accidentally land on the wrong URL.

share|improve this answer
    
Thanks, Actually there are close to 100's of urls been crawled like this. Forum was an example. It has been crawled on with some say toys also . Now toys folder do not exist in site, but still google has crawled the site with toys. Will a command like Disallow: /forum/ or Disallow: /toys/ is apt to make Google stop crawling up urls –  Saahil Sinha Jun 29 '13 at 3:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.