1

I'm building a website which is basically a simple 1 page with a search box, and AJAX'd search results displayed in a table.

I'm having trouble finding out the correct information about crawlability and indexing with this.

An article on SearchEngineLand (March 5, 2015) notes that:

Google May Discontinue Its AJAX Crawlable Guidelines

With a suggestion that it no longer needs to offer this as it is capable of crawling an AJAX'd site.

What I'm having difficulty with is:

  1. Whether I should try and make my AJAX search results crawlable, and if so whether I need to do anything to make my site and the asynchronous search results table crawlable
  2. Or whether I should just disallow all indexing of search results, and have individual pages for each search result item instead, that can be indexed separately and included in a sitemap - as explained here.

Some real examples of option 1 and 2:

  1. Build a (somehow) crawlable AJAX search result table so a page with a search for "Foo Bar" displaying a results table of items relating to "Foo Bar" will be indexed.

  2. Or a crawl blocked search result table with results for "Foo Bar", each item in the results table will be linked to an individual page for that item e.g. foo-bar-a.html, foo-bar-d.html etc. These pages are crawlable and submitted in sitemap.

Option 1 I beleive could be limiting in the future, and I have the trouble of finding a way to make the AJAX search function and results crawlable. I think it also throws up a duplicate content issue.

Option 2 gives room for expanding content on these separate pages and hopefully ranking better for each result, but of course involves a lot more work. I can imagine also a problem with duplicate or thin content on these individual pages as there would be very little data to begin with.

What's people's opinion on the best option here?

I hope that all makes sense.

Improve this question

2 Answers 2

Reset to default
1

Site search and search results shoud never be crawlable, regardless of whether they are AJAX or not. See: Matt Cutts: search results in search results. Google penalizes sites that try to get their search results indexed.

Googlebot is now starting to execute JavaScript and index any text that gets written into the page. I have a random password generator website and I found some of the JavaScript generated passwords indexed by Google recently. Google is getting better at indexing the page as the user sees it rather than just what it can find in the initial page source.

Google has also said that using AJAX with push state is a great way to make a site fully crawlable but still powered by AJAX. Moz.com has a full example tutorial for doing it:

  • The initial page load loads the full page including all content inline
  • Clicks load new content into just the content area and use push state to change the URL
  • The back button uses pop state
Improve this answer
7
  • Thanks! I'm curious about the Moz example, how would the results URLs get indexed if Google does not (perhaps, currently) submit queries. I'm guessing I would need to submit the dynamic URLs via sitemap? That said, I'm still under the impression I may be better off having a page for each search result. Currently, a search for "Foo Bar Widget" would return results "Foo Bar Widget A", "Foor Bar Widget B" etc, and each would link to it's own page foo-bar-widget-a.html, foo-bar-widget-b.html etc. They would be very thin pages though, is my only concern, and duplicate content.
    – user2565123
    Commented Apr 8, 2015 at 16:31
  • 1
    As I said, you risk getting a penalty if Google can index pages that look like search results. Google's Panda algorithm also penalizes sites with thin pages. Create pages only of topics where you have enough content and make sure you don't create two pages that are substantially the same.
    – Stephen Ostermiller ♦
    Commented Apr 8, 2015 at 17:06
  • I'm a little confused. If I was to crawl enable my search function using the Moz method, I would be under fire from Google because the crawled results would be considered search results (and probably duplicate content issues). Alternatively, if I disallow search results, but have an individual page for each result, I'd be under fire for thin (and probably duplicate) content pages. So how is it possible to have the content crawled without being penalised in some way?
    – user2565123
    Commented Apr 8, 2015 at 17:19
  • You need to create an XML sitemap of the pages you want to get indexed. For best results, you also need to link to those pages. For indepth information about that part, see The Sitemap Paradox
    – Stephen Ostermiller ♦
    Commented Apr 8, 2015 at 17:20
  • It isn't possible to have content based on search results crawled and indexed without being penalized in some way. You need to create pages that are good landing pages for users with plenty of content other than just links to other pages on your site.
    – Stephen Ostermiller ♦
    Commented Apr 8, 2015 at 17:22
0

Apart from providing your results via the ajax method, you could also interpret the search query from the URL parameters or URL structure (via apache aliasmatch ?).

e.g example.com/zoo.php?q=unicorns or example.com/zoo/jellytiger

and then return a page with the data already in the html, as if a user would have just typed unicorns or jellytiger in the search box.

This has the advantage that you or another website could directly link to a specific content, but also that you can make a sitemap that you can submit to the search engines.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.