1

I have a page; home.php, it does an ajax call to /ajax/latestreviews.php.

However Google/Bing are attempting to index latestreviews.php, how can I stop this?

An obvious answer is to include /ajax in the robots.txt or include X-Robots-Tag "noindex" in the header of latestreviews.php. But from my reading Google suggest important resource files (to the page) should not be excluded through robots. Also that included the robots-tag can negatively affect the calling page.

So I was wondering how people are achieving this?

  • What format does /ajax/latestreviews.php return? HTML, or something machine-readable like XML or JSON? – Maximillian Laumeister May 2 at 23:03
  • It returns html to be included in the calling page. – JAMMac May 3 at 19:55
0

First you should add the X-Robots-Tag "noindex" in the header of the ajax script. But if you to block it definitely, you should prevent loading the javascript in case of google bot. other way more classic is to redirect the ajax call to the referer url.

Remember even if google index Some content added by the ajax script, you should confirm it with the cache: command on your home. It may attribute the content to the ajax script itself.

Also, Google bot javascript renderer doesn't come a lot on website mostly 1 time a month so the freshness of this kind of indexation may not be really efficient. If you really wanna take advantage of those content, you should add them directly in the first dom rendering

| improve this answer | |
  • This is a tough one, because if he adds noindex to latestreviews.php, then googlebot's rendering of home.php won't include any of the latest reviews, and it's more likely to be treated as thinner content. – Maximillian Laumeister May 3 at 20:06
  • I just edited the answer with more detail to help you understand the Google bot behavior – Patrick Valibus May 3 at 20:22
  • @MaximillianLaumeister Yes, that is the reason I didn't add the x-robots tag to latestreviews.php because I had read in a number of posts it would have a negative effect on home.php. – JAMMac May 5 at 20:27
  • I'm very sorry @PatrickValibus, I don't understand your answer. I don't want to include the noindex flag for the reason above. – JAMMac May 5 at 20:30
  • If I correctly understand, you want to keep the benefits of the fresh content bring by the inclusion of the latest reviews.php content. The content returned by this file is what you want to keep and it's a good thing. But, googlebot, when it crawl the web access let's of page everyday but only to parse the instant html and it doesn't execute javascript (javascript may be executed once a month or less) so trying to keep the call to the ajax file is not the right way. So you should include the content in the initial php and not in a second ajax one. – Patrick Valibus May 5 at 20:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.