I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a robots.txt file, but I'm wondering if I need one for the mobile site too. Anyone have any thoughts?

  • are the domains different? or you redirect to a sub-folder if a mobile device is detected? Nov 20, 2011 at 0:54
  • The domains are not different. It's www.somehostname.com and m.somehostname.com
    – Wigley
    Nov 20, 2011 at 3:27

2 Answers 2


If both versions of the site share the same content then you will run into duplicate content issues regardless of where the files are located on the server. So from that perspective, canonical URLs would be the best solution for you. A robots.txt file wouldn't really accomplish anything unless there is content on the mobile version of the site that is not present on the regular site that you don't want crawled. Then you would need to use a robots.txt to block those pages.


I agree with the canonical url suggestion. So you could, for example, insert the rel="canonical" meta tag in the pages of the mobile version with the canonical link pointing to the original main website page.

You could also consider using the meta tags "robots" content="no-index, follow" to stop the search bots from indexing the duplicate content but to preserve the flow of the links throughout your site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.