4

I have a couple of questions about Ajax crawling of a website, which is kind of search engine itself. The base article explains the mechanism of making AJAX applications crawlable. All this stuff with HTML-snapshots is clear and easy to implement, but I can't understand where the Google Bot "finds pretty AJAX URLs"( ie www.example.com/ajax.html#key=value) to index them.

First thing, that came to mind are breadcrumbs. In a sitemap we are able to specify pages with breadcrumbs/links on them, so the search engine bots may crawl these pages and get HTML-snapshots from there. But I'm sure, there are other ways to give bots these "pretty AJAX URLs".

In our case, we have a simple search site, where users enter keywords, press on "Find", js executes the Ajax request, receives a JSON responce and fill page with results (without any refresh of course). In this case - how can we make sure that the Google bot crawls all the search results in addition to adding a sitemap with links to all search results?

Is there some example of solution, described in article above?

Improve this question
2
  • In the example of a simple site search, as you have here, I'm not sure I can see how this would work, unless you are going to place several seperate AJAX url's on the page for google to crawl initially. Apologies if I've not understood perfectly though. – BizNuge Apr 18 '12 at 11:53
  • @BizNuge if so, how can we change the site, for Google(and other crawlers) can easy crawl our site? – vacuum Apr 18 '12 at 13:15
4
+50

Just create an XML sitemap of your actual content pages (the job vacancies), submit it to Google and use a script to keep it up to date. That's all you need to do.

Generally, trying to get Google to index site search result pages is pointless. Google is a search engine, and is perfectly capable of indexing your content pages directly. Letting one search engine (Google) index the result pages of another search engine (your site search) is just silly and adds a needless layer of indirection. At worst, it could even lead to significant problems:

  • If your site search results were crawlable, Googlebot might decide to spend all its time crawling the nearly infinite space of possible search results, leaving it very little time to index your actual content pages.

  • Also, since the bot would have so many pages to crawl, it would recrawl each of them only rarely. If your site wasn't completely static, this would mean that most pages from your site in Google's index would be stale. This could lead to a lousy user experience and a drop in conversions, as Google would be sending users to pages on your site that no longer have the content they searched for.

    (You can see this happening with some badly designed sites that e.g. let Google crawl their "most recent posts about X" lists. You'll see the page in Google's results, the snippet shows the keywords you were looking for, but when you actually click through, the content you saw in the snippet is no longer anywhere to be seen.)

For these reasons, it is actually often recommended that you deliberately forbid external search engines like Google from crawling your site search results using robots.txt, even if they could technically do so. For example, Google's Webmaster Tools Help pages say:

"Consider using a robots.txt file to block Googlebot's access to problematic URLs. Typically, you should consider blocking dynamic URLs, such as URLs that generate search results, or URLs that can create infinite spaces, such as calendars."

The one exception is where you have a relatively small set of keywords (or "categories" or "tags", like these ones) that associate related pages into lists that users might want to browse. (Here, "relatively small" means a finite number, preferably less than the actual number of content pages on your site, as opposed to "anything a user might think to search for".) In that case, you may want to include those list pages in your sitemap (perhaps with a lower priority than your content pages) so that Google will find and index them too.

You should also consider adding non-AJAX links to those pages from your front page (and from any other page where they might be useful) so that even users with JavaScript disabled (or not supported by their browser) can find them. If the user does have JS enabled, you can use it to replace the links with the corresponding AJAX interface.

In fact, more generally, it's a good idea to design your site so that it works (as far as technically possible) even without JavaScript. This not only makes your site friendly to users with odd browsers (like blind users using an audio browser or lynx with a Braille terminal) or who surf with JS turned off by default for security or performance reasons, but it also automatically makes it accessible to search engines, which see your site much like a user with a JS-less text-only browser would.

Improve this answer
3
  • Currently, we have autogenerated sitemap with all conent pages(about 200k pages).AFAIK spiders will index those pages(that are in sitemap), but if there are no links to these pages on site, the rank of pages will be 0. – vacuum Apr 20 '12 at 8:05
  • That's a good point. So make sure that every page is reachable from your main page (and vice versa) through non-AJAX links. As I noted at the end, you can replace those links with the corresponding AJAX navigation for browsers that support it, but you should still make sure your site is at least minimally navigable even with JS turned off. – Ilmari Karonen Apr 20 '12 at 10:49
  • ok, thx. Now its clear. Thank you for good answer. – vacuum Apr 20 '12 at 10:50
1

Well...

this is a site search you're talking about indexing... and as such would require human interaction to generate/input search terms...

Upon google spidering your site, whether ajax or normal html, it wouldn't hit your site with search queries. It would only hit url's that you specify in anchors/markup.

To this end, the only way I can see that would allow you to get google to index these pages, would be for you to somehow capture users queries, and then render these into your markup somewhere for eventual spidering by bots.

I've put this in an answer rather than a comment, because I do feel this answers the question, although not in a very positive way I guess.

Good Luck...!

Improve this answer
7
  • So you suggest to create separate page with link on default searches? In this approach is there is a possibility to make it with ajax, or this content(search results) must be genereted on the server-side? – vacuum Apr 18 '12 at 14:35
  • erm... to be honest, the point I was trying to illustrate (although perhaps quite vaguely) is that, realistically, since a site search is really a combination of user input and your own search template/page, then google won't necessarily have anything to spider, so to speak. If you could perhaps capture user queries in a db, and later retrieve perhaps the top ten searches, you could then maybe put these into markup on a page of your website somewhere for googlebot to spider. Hope this clarifies my thoughts. – BizNuge Apr 18 '12 at 15:52
  • Why? Lets take some related site - link Spider can crawl all the vacancies by going throu pages in the search. We can do it similarly, but this means that all content will be generated on server. – vacuum Apr 18 '12 at 17:05
  • Yes. but job vacancies are finite, and could be retrieved from a db. search terms (to all intents and purposes) would seem to be infinite, so how would you possibly feed enough terms to google to satisfy all searches, without resorting to cherry-picking the top few most popular searches. perhaps we are talking at crossed purposes here. What EXACTLY is your use-case, and do you maybe have a url/markup to serve as an example...? – BizNuge Apr 18 '12 at 20:43
  • In my case we have finite number of pages with vacancies, that need to be idexed by Google and other spiders. What is the best practice here? – vacuum Apr 18 '12 at 22:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.