Webmasters Stack Exchange is a question and answer site for pro webmasters. It only takes a minute to sign up.
Sign up to join this community
With the new method of doing things that I've put together, my page content is not actually on index.html or any of the other pages. Instead, it is in index.txt and merely fetched by AJAX.
Is my content, which is in index.txt, going to be invisible to the search engine spiders? Or will it load the AJAX-fetched material and then analyze the whole thing once the material is fetched?
In general any content produced dynamically through Ajax will not be crawlable s search engines do not interpret or run JavaScript. The only exception is Google which can handle dynamic JavaScript if youfollow their proposal for crawlable Ajax.
A well built website will make all of its dynamic content available even if JavaScript is not available. It's called progressive enhancement/graceful degradation and really is an accessibility must under most circumstances.
It will not be searchable by the spiders. The spiders do not follow Javascript. You'll have to tell the spiders directly to index that content, but they will try to directly link to it then instead of dynamically load it from the previous page.
the only way I can think of exposing the index.txt file is using a non-standard Allow directive in robots.txt file
however, you should always build your site so it serves the content using ordinary links (anchors) and only then adding AJAX calls to improve usability
for example, you can intercept and bypass clicks to menu links on your website (by using onclick="return false"), then load content from pages behind those links via AJAX in the background and extract just the part you need to change on the page
