6

My WordPress site in cPanel is serving the admin-ajax.php file which is wasting CPU resources. How can I solve this problem? Should I use this in my robots.txt file or not?

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
3
  • What is the reason your are considering crawlers to access admin-ajax? If that is only used by your site administrators, then you would disallow robot access to it. Even if you did that though, I don't see how it would help with CPU resources. Presumably it only does anything for a logged-in administrator. – Stephen Ostermiller Oct 18 '17 at 17:06
  • If it is being used by something other than an admin, it is possible that hackers are targeting it. If that is the case robots.txt isn't going to help at all. Robots.txt is like a sign that says "please", but it isn't enforced at all and hackers will just ignore it. – Stephen Ostermiller Oct 18 '17 at 17:20
  • 1
    @StephenOstermiller The reason for including admin-ajax.php is because that is how Wordpress' framework derived AJAX system works, even for the public side. So, if you code a theme or plugin using the Wordpress framework that uses AJAX, access to admin-ajax.php must be public. I think it's a stupid situation personally, but that's how it is. Also, I'm not a fan of outing what folders to disallow, especially since there would never be any links to that folder. – Steve Dec 20 '18 at 18:40
6

This file is often used front-end by themes and plugins. It was changed in WordPress robots.txt file a couple of years ago to allow crawling for this reason, after an issue with google being unable to crawl it.

The WP Core ticket thread can be found here and offers further information https://core.trac.wordpress.org/ticket/33156

0

If by “allow” you mean to allow Google to crawl it, yes. Google wants to see how your site is constructed including CSS, JS, and Ajax files.

1
  • I wouldn't think that /wp-admin/ URLs would be used for CSS, JS, or AJAX. The admin files are usually used by the website owner to write new content, approve contents, and such. It sounds like this URL might be an exception to that. If that is the case, it would probably be worth explaining that in your answer. – Stephen Ostermiller Mar 1 '20 at 11:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.