We've got a web application with the main purpose of searching through a set of records search mechanism itself is completely AJAX based. A typical search URL would be something like this:
/search#arg0=v0&arg1=v1
(it can go up to 20 args)
Currently there are about 1000 records added to the database daily. Each 30 to 50 records out of those 1000 may have similar page titles (but not exactly the same). I would like to get those results indexed.
My solution is to add links of some specified search queries to the sitemap. We don't know how frequently crawling happens, or how many records should be display as search results (which makes some records to be missed). We can check the user agent string to get rid of AJAX driven stuff. and send the snapshot of the response to search engines. To solve the missed records issue I thought maybe store the time of last crawl somewhere and show the results according to that value. but that imposes a security issue. if someone changes the user agent value and send a request, that crawl time value can be easily changed.
Any ideas? is there a better, more straight-forward way than this?