Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

With the new method of doing things that I've put together, my page content is not actually on index.html or any of the other pages. Instead, it is in index.txt and merely fetched by AJAX.

Is my content, which is in index.txt, going to be invisible to the search engine spiders? Or will it load the AJAX-fetched material and then analyze the whole thing once the material is fetched?

share|improve this question
up vote 6 down vote accepted

In general any content produced dynamically through Ajax will not be crawlable s search engines do not interpret or run JavaScript. The only exception is Google which can handle dynamic JavaScript if youfollow their proposal for crawlable Ajax.

A well built website will make all of its dynamic content available even if JavaScript is not available. It's called progressive enhancement/graceful degradation and really is an accessibility must under most circumstances.

share|improve this answer

It will not be searchable by the spiders. The spiders do not follow Javascript. You'll have to tell the spiders directly to index that content, but they will try to directly link to it then instead of dynamically load it from the previous page.

share|improve this answer
How would I do that? <LINK title="content" type="text/html" rel="alternate" href="index.txt">? – Michael Jan 28 '11 at 21:51
You wouldn't really. The idea if what you want to have the content readable on the website first (this also helps for accessibility reasons) then you would use the AJAX methods to override the basic functionality to show the content in the fancy way. John's answer above contains a great link to explain the process. – XOPJ Feb 14 '11 at 14:11

the only way I can think of exposing the index.txt file is using a non-standard Allow directive in robots.txt file

however, you should always build your site so it serves the content using ordinary links (anchors) and only then adding AJAX calls to improve usability

for example, you can intercept and bypass clicks to menu links on your website (by using onclick="return false"), then load content from pages behind those links via AJAX in the background and extract just the part you need to change on the page

share|improve this answer
The problem with this method is the content lacks the proper context of the page it will be on and thus have no real semantic meaning. Basically without a page to associate it with it's useless to the search engines and to users. – John Conde Jan 28 '11 at 21:41
I believe you are referring to robots.txt? if not, I have most probably missed your logic here – Zathrus Writer Jan 28 '11 at 22:00

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.