1

I have a situation when I check my Drupal 7 website on Google Mobile Friendly Test, it says that my website is mobile friendly, but beneath there is a sentence saying: "This page uses 56 resources which are blocked by robots.txt". These are all images.

Server company says it is coding and coder says it is server:

The images are found here but they are all blocked:

public_html/sites/all/themes/THEME/images/

but this is in my robots.txt

# JS/CSS
Allow: /core/*.css$
Allow: /core/*.js$
Allow: /misc/*.js
Allow: /misc/*.css
Allow: /modules/*.js
Allow: /modules/*.css
Allow: /profiles/*.js
Allow: /profiles/*.css
Allow: /themes/*.js
Allow: /themes/*.css
Allow: /sites/default/files/advagg_css/*.css$
Allow: /sites/default/files/advagg_js/*.js$
Allow: /sites/all/themes/THEME/img/*.png$
Allow: /sites/all/themes/THEME/img/*.jpg$
Allow: /sites/all/themes/THEME/images/*.gif$
Allow: /sites/all/themes/THEME/images/*.png$
Allow: /sites/all/themes/THEME/images/*.jpg$
Allow: /sites/all/themes/THEME/img/*.jpg$
Allow: /sites/all/themes/THEME/img/*.png$

Cannot work out why it is still blocked


My robots.txt can be found here - is there something I am missing.

http://jpst.it/zZy6

1

I would try adding the Allows before the Disallows. Google actually state the order is not important, only the specificity of the rule based on the length of the entry.

Order of precedence for group-member records

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

However as you can see, they state with wildcards it is undefined, so i would try switching around the order of Allows and Disallows.

And/or I would try simply adding the entries below, allowing all image files without defining their directors,

Allow: /*.jpeg
Allow: /*.jpg
Allow: /*.JPEG
Allow: /*.JPG
Allow: /*.GIF
Allow: /*.gif
Allow: /*.PNG
Allow: /*.png

You can also use the robots.txt tester in Search Console, it may help with the troubleshooting.

  • 1
    Thanks Max your amazing but that did not work :-( I think the server is taking too long to render the images as it says "temporary unreachable" so it must have a max amount that it crawls maybe, the puzzling thing is that when I try to render just the logo (or images individually) it says Error. Google does not give any clue to what that error is. They just say "An error prevented Google from completing the fetch. If this error happens again, we ask that you contact Search Console product support." – FloJo Jul 6 '15 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.