How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error.

The URL which been crawled incorrectly is:


However, The correct page is:


Is robots.txt a correct way of excluding this? I'm thinking about using:

Disallow: /forum/

Will that not block legitimate content in the /forum/ folder of my site?

  • If /forum/index.php?option=com_forum isn't the correct URL, what are you serving when a crawler hits that page? The correct thing would be to serve a 404 not found error, or a redirect to where the page actually exists. If you do that, you will have no problems with duplicate pages. Nov 25, 2013 at 19:41

1 Answer 1


If www.abc.com/forum/index.php?option=com_forum is the only link that's been mis-read then you can just add Disallow: /forum/index.php to your robots.txt file. Adding Disallow: /forum/ will tell robots to ignore everything in that directory, which doesn't sound like what you want.

You could also add a 301 redirect from www.abc.com/forum/index.php?option=com_forum to www.abc.com/index.php?option=com_forum to tell robots what the URL should be. This will also help any users who accidentally land on the wrong URL.

  • Thanks, Actually there are close to 100's of urls been crawled like this. Forum was an example. It has been crawled on with some say toys also . Now toys folder do not exist in site, but still google has crawled the site with toys. Will a command like Disallow: /forum/ or Disallow: /toys/ is apt to make Google stop crawling up urls Jun 29, 2013 at 3:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.