All wp-content
and wp-include
files are blocked according to Google Search Console, and Googlebot can't crawl any JavaScript contained in those folders.
GSC insists that these folders are blocked by robots.txt, but the robots.txt checker says the "blocked" URLs are allowed, and they're not listed in the robots.txt file. Any ideas what could be wrong?
User-Agent: *
Disallow: /cgi-bin
Disallow: /wp-login.php
Disallow: /wp-admin/
Disallow: /?author=*
Disallow: /testimonial/*
Disallow: /story-page/*
Disallow: /news-post/*
Disallow: /wp-post/*
Disallow: /product-features/*
Allow: /
Allow: /
line because it serves no purpose. By default everything is allowed and you useDisallow:
lines to limit access. You should also remove the*
from the end of all yourDisallow
rules. They serve no purpose either. NormalDisallow
rules are "starts with" rules. Wildcards are only needed at the beginning or in the middle of rules, and even then most robots don't understand them. – Stephen Ostermiller♦ Mar 13 '18 at 10:51Disallow
rules are no longer "starts with" rules. – Stephen Ostermiller♦ Mar 13 '18 at 22:56