Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a robots.txt file, but I'm wondering if I need one for the mobile site too. Anyone have any thoughts?

share|improve this question

migrated from stackoverflow.com Nov 20 '11 at 4:07

This question came from our site for professional and enthusiast programmers.

are the domains different? or you redirect to a sub-folder if a mobile device is detected? – Book Of Zeus Nov 20 '11 at 0:54
The domains are not different. It's www.somehostname.com and m.somehostname.com – Wigley Nov 20 '11 at 3:27

If both versions of the site share the same content then you will run into duplicate content issues regardless of where the files are located on the server. So from that perspective, canonical URLs would be the best solution for you. A robots.txt file wouldn't really accomplish anything unless there is content on the mobile version of the site that is not present on the regular site that you don't want crawled. Then you would need to use a robots.txt to block those pages.

share|improve this answer

I agree with the canonical url suggestion. So you could, for example, insert the rel="canonical" meta tag in the pages of the mobile version with the canonical link pointing to the original main website page.

You could also consider using the meta tags "robots" content="no-index, follow" to stop the search bots from indexing the duplicate content but to preserve the flow of the links throughout your site.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.