My robots.txt looks as follows:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Google Webmaster Tool recently raised a warning saying:

Indexed, though blocked by robots.txt

I guess the Disallow at the folder level is overwriting the file level Allow. What is the way to get around this?

  • I don’t think so. Google recognized it was blocked, but proceeded anyway. That’s a giant WTF! – CareyB Jun 13 '19 at 11:33
  • Google can index pages based on links to them rather than by crawling them. Just because Google indexed the URL, it doesn't mean that it processed it by crawling it. – Stephen Ostermiller Jun 13 '19 at 12:24
  • Have you tried using Google's robots.txt testing tool? – Stephen Ostermiller Jun 13 '19 at 12:25
  • Hi @StephenOstermiller .. i have and the tool highlights the Disallow line stating that /wp-admin/admin-ajax.php cannot be accessed by GoogleBot – MovieMe Jun 13 '19 at 13:34
  • Have you tried reordering the directives? I thought that googlbot used a longest matching line heuristic but maybe order matters. Or maybe they stopped supporting the nonstandard allow rules. – Stephen Ostermiller Jun 13 '19 at 14:30