Prevent crawling of mobile site

Question

I have an issue with bots crawling the mobile site and returning hits on the mobile site for desktop users.

The mobile content is in /mobile/ The regular site will redirect mobile users to the /mobile/ site Regular desktop browsers are allowed into the /mobile/ site and will see the mobile version (used for easy testing of mobile content from desktop).

Can someone help with a sane robots.txt for this site that would work with Google and Bing. I want both crawlers to index the hole site, but only the mobile crawler should index the mobile site.

I have found user-agent: GoogleBot-Mobile for google, but is there something like BingBot-Mobile or does it crawl only with bingbot? Also, is msnbot gone?

Is this ok?

User-agent: *
Allow: /

User-agent: Googlebot,bingbot
Disallow: /mobile/
Allow: /

User-agent: Googlebot-Mobile,bingbot-mobile
Disallow: /
Allow: /mobile/

I guess it would be better if desktop users got redirected away from the /mobile/ site to the regular content, but it's a bit outside scope of question.

Thank you for your help!

Answer 1

I think you should handle this issue differently. You want google to understand it's an alternate page version that is for different screen sizes. He will make sure a desktop user will be directed to the right place (as well as mobile users).

I would recommend the following:

In all /mobile/ pages use:

  <link rel="canonical" href="/non-mobile-url"/>

And in the non mobile url put this

  <link rel="alternate" media="only screen and (max-width: 640px)" href="/mobile/..." />

Answer 2

Each bot requires its own line.

Your robots.txt should look like this

User-agent: *
Allow: /

User-agent: Googlebot
Disallow: /mobile/
Allow: /

User-agent: bingbot
Disallow: /mobile/
Allow: /

User-agent: Googlebot-Mobile
Disallow: /
Allow: /mobile/

User-agent: bingbot-mobile
Disallow: /
Allow: /mobile/

Additional its always best to use Noindex and Robots.txt

You should consider using both robots and noindex... Simply use the following metas.

On the mobile site and not the normal site

<meta name="Googlebot" content="nofollow, follow">
<meta name="bingbot" content="noindex, follow">

On all the normal pages and not the mobile site use:

<meta name="Googlebot-Mobile" content="nofollow, follow">
<meta name="bingbot-mobile" content="noindex, follow">