My robots.txt looks as follows:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Google Webmaster Tool recently raised a warning saying:

Indexed, though blocked by robots.txt

I guess the Disallow at the folder level is overwriting the file level Allow. What is the way to get around this?

closed as off-topic by Stephen Ostermiller Jun 14 at 20:48

  • This question does not appear to be about webmastering within the scope defined in the help center.
If this question can be reworded to fit the rules in the help center, please edit the question.

  • I don’t think so. Google recognized it was blocked, but proceeded anyway. That’s a giant WTF! – CareyB Jun 13 at 11:33
  • Google can index pages based on links to them rather than by crawling them. Just because Google indexed the URL, it doesn't mean that it processed it by crawling it. – Stephen Ostermiller Jun 13 at 12:24
  • Have you tried using Google's robots.txt testing tool? – Stephen Ostermiller Jun 13 at 12:25
  • Hi @StephenOstermiller .. i have and the tool highlights the Disallow line stating that /wp-admin/admin-ajax.php cannot be accessed by GoogleBot – MovieMe Jun 13 at 13:34
  • Have you tried reordering the directives? I thought that googlbot used a longest matching line heuristic but maybe order matters. Or maybe they stopped supporting the nonstandard allow rules. – Stephen Ostermiller Jun 13 at 14:30