1

I received an alert from Google Webmaster Tools a couple weeks ago that stated:

"Google systems have recently detected an issue with your homepage that affects how well our algorithms render and index your content. Specifically, Googlebot cannot access your JavaScript and/or CSS files because of restrictions in your robots.txt file. These files help Google understand that your website works properly so blocking access to these assets can result in suboptimal rankings."

I found my robots.txt file to possible remove some blocks (it's a WordPress site if that matters).

User-agent: *
Allow: /
Disallow: /wp-admin/
Host: http://onelittledesigner.com
Sitemap: http://onelittledesigner.com/sitemap.xml

However, despite that change I'm still finding almost 400 blocked resources including many of those that are needed for Google to properly see my page (i.e. stylesheets, scripts, images, etc.).

enter image description here

I'm not sure what could be blocking the resources or how to unblock them. The only instructions Google gave was to fix my robots.txt file, but it appears that the files are being blocked in some other way. Any ideas how to fix this?

migrated from serverfault.com Aug 14 '15 at 21:33

This question came from our site for system and network administrators.

  • 1
    It will take a while... a couple of weeks for a search engine is not enough time- even for Google. – closetnoc Aug 14 '15 at 23:07
  • I'm guessing then that I do just need to wait longer, but there is something really odd here still. Notice the very consistent amount of blocked resources and then the slight drop? I updated the robots.txt on July 28 and then the dip happened when they crawled on August 11 (the previous crawl was on July 25). I'm not sure the dip and my change are related, but I thought I'd mention this just in case. – jhawes Aug 15 '15 at 16:14
  • You will find the mind of Google to be a devious and fiendishly clever one... or not at all clever and just plan lucky sometimes... I am sure it is a coincidence. I get tripped up by things I see from Google from time to time only to figure out, there's nothing to see here... move on!! – closetnoc Aug 15 '15 at 21:19
  • Thanks for the insight. I'll just cross my fingers for a bit then and check on the Webmaster Tools stuff from time to time. – jhawes Aug 15 '15 at 22:59
  • Just want to check you have re-crawled a major page and all it's links. Submitting a sitemap can help too. – Mousey Aug 17 '15 at 1:42
1

When this sort of issue arises it is important to note that search engines don't crawl your website in real time. Depending on the size and what the search engine classes as the importance of your site your site may be re-indexed anywhere from every few hours (for extremely large sites and very high value sites such as Stack Exchange) all the way to the other extreme of every 4-6 weeks (for very small websites). You can help the process along by re-submitting an updated sitemap to Google or using the Fetch as Google tool which will help add the amended page to the queue for re-indexing however as a general rule there is not much you are able to do to force the re-index before it's turn in the queue. In these instances all you can really do is be patient and wait.

I noted that you mentioned the re-index and reduction in errors after the change to your robots.txt file however Google rarely re-indexes a whole site at the same time, rather it does it in bits and pieces over a period of time. The errors you are still seeing will be historical errors which have not yet been resolved as the whole site was not yet completely re-indexed.

Allow 4-6 weeks and you should see everything clear up then.

(Thanks to @closetnoc for the starting point)

0

You can re-index your website anytime with Google Search Console. Just simply navigate to Crawl > Fetch as Google and you should see an input field to put a URL. Then just hit Fetch or Fetch and Render depending on your needs. This will send Google Crawl Bots to your website to recrawl.

I typically do this for all new articles after posting and also when I update publish dates or change on page content for a page or post.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.