With the new method of doing things that I've put together, my page content is not actually on index.html or any of the other pages. Instead, it is in index.txt and merely fetched by AJAX.

Is my content, which is in index.txt, going to be invisible to the search engine spiders? Or will it load the AJAX-fetched material and then analyze the whole thing once the material is fetched?

3 Answers 3


In general any content produced dynamically through Ajax will not be crawlable s search engines do not interpret or run JavaScript. The only exception is Google which can handle dynamic JavaScript if youfollow their proposal for crawlable Ajax.

A well built website will make all of its dynamic content available even if JavaScript is not available. It's called progressive enhancement/graceful degradation and really is an accessibility must under most circumstances.


It will not be searchable by the spiders. The spiders do not follow Javascript. You'll have to tell the spiders directly to index that content, but they will try to directly link to it then instead of dynamically load it from the previous page.

  • How would I do that? <LINK title="content" type="text/html" rel="alternate" href="index.txt">?
    – Michael
    Jan 28, 2011 at 21:51
  • You wouldn't really. The idea if what you want to have the content readable on the website first (this also helps for accessibility reasons) then you would use the AJAX methods to override the basic functionality to show the content in the fancy way. John's answer above contains a great link to explain the process.
    – XOPJ
    Feb 14, 2011 at 14:11

the only way I can think of exposing the index.txt file is using a non-standard Allow directive in robots.txt file

however, you should always build your site so it serves the content using ordinary links (anchors) and only then adding AJAX calls to improve usability

for example, you can intercept and bypass clicks to menu links on your website (by using onclick="return false"), then load content from pages behind those links via AJAX in the background and extract just the part you need to change on the page

  • The problem with this method is the content lacks the proper context of the page it will be on and thus have no real semantic meaning. Basically without a page to associate it with it's useless to the search engines and to users.
    – John Conde
    Jan 28, 2011 at 21:41
  • I believe you are referring to robots.txt? if not, I have most probably missed your logic here Jan 28, 2011 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.