6

We are getting a number of odd results come up on Google EG:

http://www.somedomain.com/somepage/AjaxFunction.aspx?stuff=XXX&other=XXX 

When I looked at "somepage" the ajax function is not mentioned at all within the HTML, which makes me assume Google is spidering out to the external javascript files and finding this AjaxFunction.aspx call.

My question is: a) Is this possible? b) If so, how can I stop it?

4

I would put the Ajax function into robots.txt:

Disallow: /somepage/AjaxFunction.aspx

That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only index them if they are linked prominently, especially numerous external links. Even if Google does index the URL, it won't index the content of the URL. Google will show "this page was blocked by robots.txt" in the search results.

In many cases it is actually desirable to allow Google to crawl Ajax URLs. Those URLs may provide content that you want Google to index after JavaScript writes it into another page. In that case, robots.txt is not appropriate. You just don't want Google including the Ajax URL itself in the search results. You can use a header directive for that:

X-Robots-Tag: noindex

On an Apache server you can add that header with .htaccess code like:

<Files "AjaxFunction.aspx">
    Header append X-Robots-Tag "noindex"
</Files>

In aspx code you can set it like this:

<% Response.AddHeader "X-Robots-Tag", "noindex" %> 
| improve this answer | |
2

Do not add X-Robots-Tag "noindex" in the AJAX function. That may block the main html page.

We thought it would be a good idea, we did it in a project and what happened is that the AJAX portion of the page affected to the "mother" HTML page and Google considers that we are giving a "noindex" header to the mother page.

This is the result in our GSC. Of course, we have removed the "noindex" header and requested a validation. Now we are looking for a better solution.

GSC image showing how "noindex" on ajax portions affect to the main HTML page

| improve this answer | |
  • Thanks for sharing this Gari, I was about to try this exact solution but wasn't sure if it would affect the parent page. You've just confirmed it for me and saved me a lot of time and headache testing this solution. :) – A Biron Apr 27 at 19:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.