I have an ajax-driven search page that will allow users to search through a large collection of records. Each search result points to index.php?id=xyz (where xyz is the id of the record). The initial view does not have any records listed, and there is no interface that allows you to browse through all records. You can only conduct a search.

How do I build the page so that spiders can crawl each record? Or is there another way (outside of this specific search page) that will allow me to point spiders to a list of all records.

FYI, the collection is rather large, so dumping links to every record in a single request is not a workable solution. Outputting the records must be done in multiple requests.

Each record can be viewed via a single page (eg "record.php?id=xyz"). I would like all the records indexed without anything indexed from the sitemap that shows where the records exist, for example:

<a href="/result.php?id=record1">Record 1</a>
<a href="/result.php?id=record2">Record 2</a>
<a href="/result.php?id=record3">Record 3</a>

<a href="/seo.php?page=2">next</a>

Assuming this is the correct approach, I have these questions:

  1. How would the search engines find the crawl page?

  2. Is it possible to prevent the search engines from indexing the words "Record 1", etc. and "next"? Can I output only the links? Or maybe something like:  

  • 1
    this doesn't seem to be very clear, can you explain better? "Each record can be viewed via a single page (eg "record.php?id=xyz"). I would like all the records indexed without anything indexed from the sitemap that shows where the records exist, for example:"
    – PatomaS
    Oct 22, 2012 at 6:21
  • Does the comment I left below (for @PatomaS) helpful?
    – Redtopia
    Oct 22, 2012 at 7:36

2 Answers 2


You may use mod_rewrite to haver better url's, that would masquerade what you want.

But the main problem here is that you say that you don't have links or a way to discover your content except for the search box.

Even assuming that your users know exactly what to look for, that is still a poor way to provide content, there may be a lot of things that may be useful or interesting but that nobody will ever discover because there is no way to get there.

A crawler is like a user, it will get to the first page it finds, whichever it is, and will follow links around. Obviously they have algorithms to check for alternatives, plus they may get links from other sites that point to specific records in your collection and then index also those, but still, is going to be hard for anybody, human or machine.

One thing you can do, is provide lists or indexes, may be not for all the content, but grouped by categories. There has to be ways to group your content and you can post those categories/groups. You can also provide related links in those categories or in any result, that will also help to index properly all your site/records.

UPDATE Considering your last comment, having related posts in each page will be the best way to go. You will have to work an algorithm that really links to useful subjects, but if they are really useful, then not just the bots, but the users will follow them, and that will provide external links and references back to your site, improving each page relevance and making it more visited. If your related system also considers some rotation of those related links, it will be even better.

An example will be like so, page A talks about a subject, and your related column shows ten results, when I visit that page today, I get results 1 - 10, but if I visit the page tomorrow, I get results 1,2,3 and 11-17. The idea is that some results may be closer than others, so the closest ones, should always be there, but the less related, can be rotated with other of same relevance, that helps the visitors and help you because when the bots comes back, it will get different links to follow.

  • For this data set, there's no reason to allow the user to browse. To narrow the discussion, let's keep the functionality out of the equation. The user can only search, and will be given results matching their search. What I want, is for google to index only the records. I don't want any matches on the search engines to take the user to a list of results... only to the page showing the record data.
    – Redtopia
    Oct 22, 2012 at 7:35
  • 1
    Well, if there is no way to get to the content except by searching for it, and you don't want to make a list of those links for a sitemap or similar, then as far as I know, Google may find something just by sheer luck and index it, but I wouldn't expect the bot to find all the records, also, the more complex the url/query, the hardest it's going to be
    – PatomaS
    Oct 22, 2012 at 8:11
  • I do want to make a list... as I indicated in my question. I'm just wondering how to do it correctly. For example, maybe I can place a link to the list inside the search page... that the user can't see. Let's take a look at how google indexes Stack Exchange sites... why is it that I never see search results that take me to the home page with all the questions?
    – Redtopia
    Oct 22, 2012 at 14:51
  • sites like this accumulate lots of visits and that help, but they don't get almost any direct visitor at the beginning; however each page has a very specific content and the question itself is in the url, plus you have the related posts on the right column, all that combined helps to link to lots of posts all the time. So apart of indexes, I'd say that related posts will be crucial in your system.
    – PatomaS
    Oct 25, 2012 at 0:35
  • about the lists, I think this "I would like all the records indexed without anything indexed from the sitemap that shows where the records exist" got me confused.
    – PatomaS
    Oct 25, 2012 at 0:37

My current strategy is:

  1. The initial view is a paged list of all items, starting with page 1.

  2. The links to each page re-load the entire page, specifying a page number.

  3. Using javascript, I tweak the page list so that clicking on a page triggers an ajax load instead of a full page refresh.

Items 1 and 2 will allow search engines to crawl the entire collection of records, and item 3 will only work for people who have javascript enabled (who doesn't anymore?!)

But to do this right, I should be changing the document hash so that users can copy the URL and link directly to any page.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.