Webmasters Stack Exchange is a question and answer site for webmasters. It only takes a minute to sign up.
Sign up to join this community
How do I use robots.txt to disallow folders which are being crawled due to a bad URL structure? They're currently causing a duplicate page error.
The URL which been crawled incorrectly is:
www.abc.com/forum/index.php?option=com_forum
However, The correct page is:
www.abc.com/index.php?option=com_forum
Is robots.txt a correct way of excluding this? I'm thinking about using:
Disallow: /forum/
Will that not block legitimate content in the /forum/ folder of my site?
If www.abc.com/forum/index.php?option=com_forum is the only link that's been mis-read then you can just add Disallow: /forum/index.php to your robots.txt file. Adding Disallow: /forum/ will tell robots to ignore everything in that directory, which doesn't sound like what you want.
You could also add a 301 redirect from www.abc.com/forum/index.php?option=com_forum to www.abc.com/index.php?option=com_forum to tell robots what the URL should be. This will also help any users who accidentally land on the wrong URL.
/forum/index.php?option=com_forumisn't the correct URL, what are you serving when a crawler hits that page? The correct thing would be to serve a 404 not found error, or a redirect to where the page actually exists. If you do that, you will have no problems with duplicate pages.