We are getting a number of odd results come up on Google EG:
http://www.somedomain.com/somepage/AjaxFunction.aspx?stuff=XXX&other=XXX
When I looked at "somepage" the ajax function is not mentioned at all within the HTML, which makes me assume Google is spidering out to the external javascript files and finding this AjaxFunction.aspx call.
My question is: a) Is this possible? b) If so, how can I stop it?
I would put the Ajax function into robots.txt:
Disallow: /somepage/AjaxFunction.aspx
That will prevent Google from crawling it. Google doesn't typically index URLs it can't crawl. It will only index them if they are linked prominently, especially numerous external links. Even if Google does index the URL, it won't index the content of the URL. Google will show "this page was blocked by robots.txt" in the search results.
In many cases it is actually desirable to allow Google to crawl Ajax URLs. Those URLs may provide content that you want Google to index after JavaScript writes it into another page. In that case, robots.txt is not appropriate. You just don't want Google including the Ajax URL itself in the search results. You can use a header directive for that:
X-Robots-Tag: noindex
On an Apache server you can add that header with .htaccess code like:
<Files "AjaxFunction.aspx">
Header append X-Robots-Tag "noindex"
</Files>
In aspx code you can set it like this:
<% Response.AddHeader "X-Robots-Tag", "noindex" %>
