1

I run a site built on wordpress and my question kind of sums up what I am looking for. There has been an update on google according to which "mobile-friendliness" of a site is now considered as a ranking factor. Since my site traffic has taken a significant dive I took a mobile-friendly-test. Although my site resizes pretty well in mobile devices, the test result still says my site is NOT mobile-friendly.

On a closer look, the result also says - This page uses 20 resources which are blocked by robots.txt. And on further reading of this page I found that blocked resources are a part of the Mobile-SEO. It seems like googlebot needs access to css and js files of a site in order to determine the mobile-friendliness of a site.

On the other hand I read long back that for a wordpress site "wp-content" folder should be blocked. So currently I have my robots.txt as given below. I am wondering on what would be the correct contents in a robots.txt if it is a wp-site, so my site could be read by the mobile-friendly tool as a complying site. Also is it okay to allow access to entire wp-content folder ? This is what my robots.txt looks like.

User-agent: *

Disallow: /cgi-bin/
Disallow: /jwp/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /feed/
Disallow: /comments/
Disallow: /xmlrpc.php
Disallow: /*?s
Disallow: /search/
Allow: /wp-content/themes/mysite/style.css

Sitemap: http://mywebsite.com/sitemap_index.xml
  • "googlebot needs access to css and js files of a site in order to determine the mobile-friendliness of a site" - not just for mobile-friendliness. If CSS and/or JS influences the content in any way then googlebot should be able to access it. – MrWhite May 2 '15 at 11:41
  • Note that there must not be an empty line between User-agent and Disallow. – unor May 2 '15 at 22:26
  • @Max: That is not the robots.txt specification (and neither does it mention that blank lines would be allowed, and all examples on this page follow this), robotstxt.org/orig.html is. And there it says that records are separated by blank line(s), so a record can’t contain a blank line. – unor May 4 '15 at 8:48
  • I actually read your original comment wrong, I thought you said there 'must' be an empty line! Sorry. – Max May 4 '15 at 9:36
1

I would pare that down to:

Disallow: /search/
Sitemap: http://example.com/sitemap_index.xml

You should certainly remove wp-content from that list. That has your themes, plugins, and images. For mobile friendly you want all these crawled.

If you want more info on what each of the other pieces do, here is a guide to the WordPress directory structure.

| improve this answer | |
-1

The robots.txt doesn't define mobile friendliness, it is supposed to define what robots are not allowed to access certain resources. In your case, you're preventing every robot from accessing a set of folders except for one CSS file.

When google checks a page, it tries to access all assets in that page. Go on that page in a browser and choose to view the source code and look for all tags that begin with anything similar to:

<link href
<script src

Then you can see all the URL's to the resources your page uses to make it completely loaded. Then if you need to leave your robots.txt file intact, then specifically add allow lines in your robots.txt file to include the resources you find in the HTML.

To make a site mobile friendly, you need to send users with some user agents to one page, and other users to another page, or modify your page so it becomes responsive so it works on all devices. If you choose the first option, you can get source code for your favorite language that you can put in at the beginning before the rest of the site is executed. go to:

http://detectmobilebrowsers.com/

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.