I have a mobile site that is distinct from the corresponding desktop site. The sites share the same content, but the mobile site files are in a completely different directory. The main site had a robots.txt file, but I'm wondering if I need one for the mobile site too. Anyone have any thoughts?
2 Answers
If both versions of the site share the same content then you will run into duplicate content issues regardless of where the files are located on the server. So from that perspective, canonical URLs would be the best solution for you. A robots.txt file wouldn't really accomplish anything unless there is content on the mobile version of the site that is not present on the regular site that you don't want crawled. Then you would need to use a robots.txt to block those pages.
answered Nov 20, 2011 at 5:51
I agree with the canonical url suggestion. So you could, for example, insert the rel="canonical" meta tag in the pages of the mobile version with the canonical link pointing to the original main website page.
You could also consider using the meta tags "robots" content="no-index, follow" to stop the search bots from indexing the duplicate content but to preserve the flow of the links throughout your site.
