Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
We are getting a number of odd results come up on Google EG:
http://www.somedomain.com/somepage/AjaxFunction.aspx?stuff=XXX&other=XXX
When I looked at "somepage" the ajax function is not mentioned at all within the HTML, which makes me assume Google is spidering out to the external javascript files and finding this AjaxFunction.aspx call.
My question is: a) Is this possible? b) If so, how can I stop it?
I would put the Ajax function into robots.txt:
Disallow: /somepage/AjaxFunction.aspx
That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only index them if they are linked prominently, especially numerous external links. Even if Google does index the URL, it won't index the content of the URL. Google will show "this page was blocked by robots.txt" in the search results.
In many cases it is actually desirable to allow Google to crawl Ajax URLs. Those URLs may provide content that you want Google to index after JavaScript writes it into another page. In that case, robots.txt is not appropriate. You just don't want Google including the Ajax URL itself in the search results. You can use a header directive for that:
X-Robots-Tag: noindex
On an Apache server you can add that header with .htaccess code like:
<Files "AjaxFunction.aspx">
Header append X-Robots-Tag "noindex"
</Files>
In aspx code you can set it like this:
<% Response.AddHeader "X-Robots-Tag", "noindex" %>
Do not add X-Robots-Tag "noindex" in the AJAX function. That may block the main html page.
We thought it would be a good idea, we did it in a project and what happened is that the AJAX portion of the page affected to the "mother" HTML page and Google considers that we are giving a "noindex" header to the mother page.
This is the result in our GSC. Of course, we have removed the "noindex" header and requested a validation. Now we are looking for a better solution.
