The meta data for the homepage on mobile is stating that the robots.txt file is blocking the page from being crawled.

This is not the case for desktop.

Desktop and mobile have the same domain, we have not changed the robots.txt file and this is a recent issue.

I know that there is a separate index for mobile and desktop but was wondering if there were any other reasons this could occur?

1 Answer 1


Open Search Console and go to Crawl -> 'robots.txt Tester'

At the dropdown for the latest version seen, it will list any times at which it had issues getting the robots.txt or times it was recently updated. Double check that Google isn't having problems reading it or has pulled an old copy. Then enter URL for the mobile homepage into the test area and hit 'TEST'.

It's also worth double checking that the page HTML doesn't contain a 'noindex' meta tag.

If all of that is fine, it's likely a bug you'll want to raise with Google. Homepages tend to be high priority and re-scraped often. Use the 'Help' dropdown top right to check.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.