In Google Webmaster Blocked Resources tab it is showing 500+ blocked resource (like .js and .css), but when I tested those URL in robots.txt Tester it is showing Allowed, then why not Google bot is updating that tab?

Sample resource URL(s)


and this is my robots.txt under (https://www.example.com/robots.txt)

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/themes/
Allow: /wp-includes/js/
Allow: /wp-admin/admin-ajax.php
sitemap: https://www.example.com/sitemap.xml

I know bot takes time to reflect real data, but issue is their form last 2-3 months and the Last detected date: 3/23/17, So I thought that I'm doing something wrong.

  • Allow: isn't a standard robots.txt directive. Rather than use that, just remove the Disallow: /wp-includes/ line. – Stephen Ostermiller May 8 '17 at 19:48
  • thanks @StephenOstermiller for responding, I didn't knew that. I just have 2 query, /wp-includes/ has WordPress lib files which include PHP, js, css, etc. so I don't want google to crawl PHP file and index that, and last question is when I'm testing specific assets URL in Google Webmaster robots.txt Tester it is showing all good. – Raunak Gupta May 9 '17 at 2:56
  • Googlebot should understand allow, but it makes it complicated. It then uses the longest matching rule rather then using the order in the file. – Stephen Ostermiller May 9 '17 at 9:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.