I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.
The mobile content is in /mobile/ The regular site will redirect mobile users to the /mobile/ site Regular desktop browsers are allowed into the /mobile/ site and will see the mobile version (used for easy testing of mobile content from desktop).
Can someone help with a sane robots.txt for this site that would work with Google and Bing. I want both crawlers to index the hole site, but only the mobile crawler should index the mobile site.
I have found user-agent: GoogleBot-Mobile for google, but is there something like BingBot-Mobile or does it crawl only with bingbot? Also, is msnbot gone?
Is this ok?
User-agent: * Allow: / User-agent: Googlebot,bingbot Disallow: /mobile/ Allow: / User-agent: Googlebot-Mobile,bingbot-mobile Disallow: / Allow: /mobile/
I guess it would be better if desktop users got redirected away from the /mobile/ site to the regular content, but it's a bit outside scope of question.
Thank you for your help!