I'm sure it is my mistake in mod_rewrite rules, but I can't seem to figure it out. I've set up the rules for my new site and all works perfect, unless - I try to create a sitemap. The URLs get duplicated/triplicated/etc... pretty much ten URLs which open the same page.

Here are the rewrite rules:

RewriteRule sport/training/article/(.*)$ ./article.php?q=$1
RewriteRule sport/eating/article/(.*)$ ./article.php?q=$1
RewriteRule sport/track/article/(.*)$ ./article.php?q=$1

RewriteRule sport/training/(.*)$ ./list.php?l=1p=$1
RewriteRule sport/eating/(.*)$ ./list.php?l=2&p=$1
RewriteRule sport/track/(.*)$ ./list.php?l=3&p=$1

When my site gets crawled, the result is something like this:


Sometimes it is even like this:


All of the above open the correct page BTW.

Internal generated link structure design seems to be ok. What am I missing here?

  • I assume you are using a tool to crawl your site to create your sitemap?? – closetnoc Oct 26 '15 at 2:14
  • Yes, an online tool. Is that bad? I tried to scan it with Xenu and it shows all that duplicate content either. – El_Mariachi Oct 26 '15 at 2:20
  • No. Normally, that would be fine. Online tools are somewhat limited in that they will spider your site and return what it sees, but with little intelligence. Some download tools are the same way. If you are using a CMS, then a plug-in could help. If you can write code, then it may be simpler depending. I wrote mine in 20 minutes but it gets it's information from a database. If your site is traditional HTML web pages then it may be best to clean-up any links you have to remove duplicates and try again. Otherwise, you can edit the file, but that could be a chore too. – closetnoc Oct 26 '15 at 2:33
  • It's a CMS that I wrote myself and articles get fetched from DB too. The links are "nonexistent" pretty/SEO friendly links, based on mod rewrite patterns (if it's "/article/" it forwards the friendly url to article.php which then fetches the data). – El_Mariachi Oct 26 '15 at 2:35
  • 1
    I just figured it out. It actually turns out to be a mistake in URL generation. If not the idea about "crawler sees what it sees" I would've gone crazy, because I was concentrating on my rewrite rules. Thank you guys! – El_Mariachi Oct 26 '15 at 16:56

I just figured it out. It actually turns out to be a mistake in URL generation. It was an idea about "crawler sees what it sees" that helped me to solve this.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.