I have a page; home.php, it does an ajax call to /ajax/latestreviews.php.

However Google/Bing are attempting to index latestreviews.php, how can I stop this?

An obvious answer is to include /ajax in the robots.txt or include X-Robots-Tag "noindex" in the header of latestreviews.php. But from my reading Google suggest important resource files (to the page) should not be excluded through robots. Also that included the robots-tag can negatively affect the calling page.

So I was wondering how people are achieving this?

  • What format does /ajax/latestreviews.php return? HTML, or something machine-readable like XML or JSON? May 2, 2020 at 23:03
  • It returns html to be included in the calling page.
    – JAMMac
    May 3, 2020 at 19:55

1 Answer 1


First you should add the X-Robots-Tag "noindex" in the header of the ajax script. But if you to block it definitely, you should prevent loading the javascript in case of google bot. other way more classic is to redirect the ajax call to the referer url.

Remember even if google index Some content added by the ajax script, you should confirm it with the cache: command on your home. It may attribute the content to the ajax script itself.

Also, Google bot javascript renderer doesn't come a lot on website mostly 1 time a month so the freshness of this kind of indexation may not be really efficient. If you really wanna take advantage of those content, you should add them directly in the first dom rendering

  • This is a tough one, because if he adds noindex to latestreviews.php, then googlebot's rendering of home.php won't include any of the latest reviews, and it's more likely to be treated as thinner content. May 3, 2020 at 20:06
  • I just edited the answer with more detail to help you understand the Google bot behavior May 3, 2020 at 20:22
  • @MaximillianLaumeister Yes, that is the reason I didn't add the x-robots tag to latestreviews.php because I had read in a number of posts it would have a negative effect on home.php.
    – JAMMac
    May 5, 2020 at 20:27
  • I'm very sorry @PatrickValibus, I don't understand your answer. I don't want to include the noindex flag for the reason above.
    – JAMMac
    May 5, 2020 at 20:30
  • If I correctly understand, you want to keep the benefits of the fresh content bring by the inclusion of the latest reviews.php content. The content returned by this file is what you want to keep and it's a good thing. But, googlebot, when it crawl the web access let's of page everyday but only to parse the instant html and it doesn't execute javascript (javascript may be executed once a month or less) so trying to keep the call to the ajax file is not the right way. So you should include the content in the initial php and not in a second ajax one. May 5, 2020 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.