3

For some reason Google Webmaster Tools insists that pretty much all js and css resources are blocked on every page of my website and when I try to render my pages in GWT they will only partially render. However, I have placed all of the following in my robots.txt file:

User-agent: Googlebot
Allow: /*.css$
Allow: /*.js$
Allow: /js
Allow: /css
Allow: .js
Allow: .css
Allow: /js/wow
Allow: /js/stellar

Can anyone tell me why Google is apparently ignoring commands in my robots.txt and how I can correct it?

  • 1
    Just remove the robots.txt if you're not blocking anything, it is completely unnecessary. Set rules in robots.txt only to block something. Checkout my blog post about proper way to use robots.txt – Goyllo Oct 15 '16 at 5:27
  • Thanks, Goyllo, but removing my robots.txt is not a feasible option, as I have other important directives in there. I only added the specific Allow: / directives I mentioned, because Google Webmaster Tools insisted that those file types were being blocked even though they weren't. Can anyone tell me how to resolve this? – M-n-I SEO Guy Oct 15 '16 at 19:13
  • "...were being blocked" - Is GSC (formerly GWT) specifically stating that those resources are being blocked by robots.txt? If you have other directives in robots.txt then maybe it's those that are the problem? – MrWhite Oct 15 '16 at 22:44
  • 1
    No, GWT doesn't state that resources are being blocked specifically in my robots.txt file. It is now my belief that this is being caused by above the fold, render blocking, external CSS references in my <head>. What do you think? – M-n-I SEO Guy Oct 16 '16 at 19:14
  • I have been trying to figure out the easiest way to load my CSS asynchronously, but i'm not having much luck. I have been trying to use something called "loadCSS" by filamentgroup, but due to their poor documentation, I can't get it to work. I think i'm going to create a new thread here at Stack Exchange and see if anyone can help me with that and go from there... – M-n-I SEO Guy Oct 16 '16 at 19:21
1

You may have directories in folders that are blocked. That being the possible case, as you are not mentioning what is blocked, I would use something like this. This way allows you to block subfolders and allow crawling of the folders rooted in the blocked directories.

User-agent: *
Disallow: /admin/
Disallow: /blocked-folder/

User-agent: Googlebot
Allow: /blocked-folder/css/
Allow: /blocked-folder/js/

Updated: Try this based on your code.


<script> 
<script async src="js/jquery.js">
<script async src="js/jquery-migrate-1.2.1.js">
<script async src="js/owl.carousel.min.js">
<script async src="js/jquery.fancybox.pack.js">

<!--[if (gt IE 9)|!(IE)]><!--> 
<script async src="js/wow/wow.js">

$(document).ready(function () { if ($('html').hasClass('desktop')) { new WOW().init(); } }); 
</script>
  • Thanks Johnny. I have already tried allowing specific directories like that and it didn't work. I'm really thinking it probably has to do with render blocking, above the fold external CSS and js references. I am loading js asynchronously, though, so I don't know why js would be being blocked if that's the case... I'm sorta stumped... – M-n-I SEO Guy Oct 17 '16 at 2:18
  • Can you post the code you are using to load the css and js asynchronously – norcal johnny Oct 17 '16 at 2:59
  • So far, I haven't been able to get loadCSS to work properly in order to get my CSS to load asynchronously. loadCSS seems to be what everyone recommends, but as I said, due to its poor documentation I haven't been able to get it to work. No matter how I try to implement it, it breaks my page. Also, I have tried combining my CSS to cut down on rountrips and no matter how I do it, that breaks my pages as well... – M-n-I SEO Guy Oct 17 '16 at 17:31
  • <script async src="js/jquery.js"></script> <script async src="js/jquery-migrate-1.2.1.js"></script> <script async src="js/owl.carousel.min.js"></script> <script async src="js/jquery.fancybox.pack.js"></script> <!--[if (gt IE 9)|!(IE)]><!--> <script async src="js/wow/wow.js"></script> <script> $(document).ready(function () { if ($('html').hasClass('desktop')) { new WOW().init(); } }); </script> – M-n-I SEO Guy Oct 17 '16 at 17:34
  • Above is how I am referencing my js to load it asynchronously. But, according to GWT, those resources are being blocked. It tests perfectly at Google Pagespeed Insights, though... – M-n-I SEO Guy Oct 17 '16 at 17:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.