2

To allow the robots to crawl all CSS and JS files we have used the following code:

Allow: /*.css$
Allow: /*.js$

But, some CSS and JS are still being blocked as stated when using the mobile friendly test from Google.

Does the location of the line of code in the robots.txt file matter?

Are there instances were the line of code will be ignored?

Improve this question
3
  • Valid code but impossible to answer without futher information. We know nothing of the error message reported by Google Console (was it a fetch test or was it a index warning). What exact file and location was it complaining about? and what other rules do you have in your robots file, since these could be relevant.
    – Simon Hayter
    Commented Jan 20, 2017 at 16:28
  • 1
    Note that you don't ordinarily need to Allow anything, since that is the default action. The fact that you have Allow directives to begin with suggests that you have other Disallow directives. We would need to see your entire robots.txt file. It's quite possible that you have more specific Disallow directives that are taking priority.
    – MrWhite
    Commented Jan 20, 2017 at 16:43
  • As stated, I used the mobile friendly test from Google, this stated that files were blocked, all the files ended in css and js, some were 3rd party, some were not. From the below answers, I have found the issue is because of the length of the rules.
    – GrapeSoda
    Commented Jan 23, 2017 at 10:43

2 Answers 2

Reset to default
3

Robots.txt uses the longest rule that matches. So when your rules are

Disallow: /directory
Allow: /*.js

Then /directory/file.js would be blocked. Both rules match, and the longer one (the disallow rule) takes precedence. See this article which explains it.

One work around might be to use lots of stars in your wild card rules to make them longer than your other rules:

Allow: /********************.js

Here are some screenshots from Google's robots.txt testing tool showing the technique working:

Improve this answer
3
  • 1
    "One work around might be to use lots of stars in your wild card rules to make them longer" - However, Google states... "The order of precedence for rules with wildcards is undefined." - which could mean that current behaviour might change in the future?
    – MrWhite
    Commented Jan 20, 2017 at 19:11
  • 1
    Yes, Google appears to leave the door open to changing the behavior. Other crawlers might also implement it differently so it would be good to test with tools from Bing, Yandex, and Baidu to see if they do things the same way.
    – Stephen Ostermiller ♦
    Commented Jan 20, 2017 at 19:18
  • This was my issue, I had rules that were longer so took priority. Thank you
    – GrapeSoda
    Commented Jan 23, 2017 at 10:45
3

Regex is not supported in robots.txt and this was asked awhile back on Pro Webmasters. However the code that you have provided is valid which will match certain paths and files. According to Google's bot it defines:

Using /*.js$ matches:

  • /jquery.js
  • /scripts/jquery.js

Using /*.js$ would NOT match:

  • /jquery.js?ver3.0
  • /jquery.js/
  • /jquery.JS.MIN
  • /jquery.JS

If you are using versioning or any other format that is not supported then you can be less specific by using simply /*.js without the $ on the end.

Since your code looks correct the issue you have is likely something else or a rule that is conflicting with your current ones you have. You should be as specific as possible and look at any rules that could be conflicting with them.

See:

Order of precedence for group-member records

At a group-member level, in particular for allow and disallow directives, the most specific rule based on the length of the [path] entry will trump the less specific (shorter) rule. The order of precedence for rules with wildcards is undefined.

Improve this answer
1
  • 1
    The linked document simply refers to these special meta characters as "wildcards".
    – MrWhite
    Commented Jan 20, 2017 at 16:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.