3

All wp-content and wp-include files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders.

GSC insists that these folders are blocked by robots.txt, but the robots.txt checker says the "blocked" URLs are allowed, and they're not listed in the robots.txt file. Any ideas what could be wrong?

User-Agent: *    
Disallow: /cgi-bin    
Disallow: /wp-login.php    
Disallow: /wp-admin/    
Disallow: /?author=*    
Disallow: /testimonial/*    
Disallow: /story-page/*    
Disallow: /news-post/*    
Disallow: /wp-post/*    
Disallow: /product-features/*    
Allow: /
| improve this question | |
  • 2
    You should remove the Allow: / line because it serves no purpose. By default everything is allowed and you use Disallow: lines to limit access. You should also remove the * from the end of all your Disallow rules. They serve no purpose either. Normal Disallow rules are "starts with" rules. Wildcards are only needed at the beginning or in the middle of rules, and even then most robots don't understand them. – Stephen Ostermiller♦ Mar 13 '18 at 10:51
  • 3
    Has the robots.txt file been updated in the past couple months? If so, it could just be taking a while for Search Console to re-crawl the URLs and re-test them against the new robots.txt. – Stephen Ostermiller♦ Mar 13 '18 at 10:52
  • Google does have wildcards at the end of URLs in its own robots.txt file: google.com/robots.txt - I'm not sure what the correct markup is for wildcards though – Michael d Mar 13 '18 at 16:59
  • Google only uses a wildcard at the end if they have another wildcard in the middle. When you include a wildcard in the middle, the Disallow rules are no longer "starts with" rules. – Stephen Ostermiller♦ Mar 13 '18 at 22:56
  • @StephenOstermiller the Allow directive overrides Disallow. However, since it’s set to allow everything, it’s the Disallow lines that serve no purpose here. The whole lot can be removed. – DisgruntledGoat Apr 10 '19 at 14:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.