Search engine misunderstanding

Question

I have this, probably a misunderstanding, about search engines. How do they search for pages? If i have a page which loads contents from database and / or changes content thought jquery will a search engine grab those content loaded by database and jquery???

help much appreciated!

Answer 1

From the answer to this question about "Ajax generated content, crawling and black listing" I found this document about the way Google crawls AJAX requests which is part of a collection of documents about Making AJAX Applications Crawlable.

In short, it means you need to use <a href="#!data">...</a> rather than <a href="#data">...</a> and then supply a real server-side answer to the URL path/to/path?_escaped_fragment_=data.

Also consider a <link/> tag to supply crawlers with a hint to SEO-friendly content. <link rel="canonical"/>, which this article explains a bit, is a good candidate

Answer 2

Search engines using crawlers to fetch pages from the internet and store it at the server/database. So it depends on the crawler. If you are interested in google, you should try to finde some information about it's crawler. If the crawler execute java script it will cache the content loaded from the DB - then it is searchable.

Here is a relates question that answers your question:
https://stackoverflow.com/questions/2061844/does-googles-crawlers-have-javascript-what-if-i-load-a-page-through-ajax

GoogleBot doesn't interpret Javascript.

Answer 3

In most situations for websites it is preferred to make a website work statically first. Then add the Javascript options for faster loading, ajax actions etcetera. So first only use normal links to normal pages and then continue with adding the jQuery stuff.

That way you get exactly the same result but it also allows you to support older browsers, crawlers etc.

With MVC this is quite easy to realize with simple views with different extensions. For example let's say we want to load a product listing.

We will have 2 urls which the server will respond to:

The first one will generate a classic full page (so with navigation etc):

/pages/products

It would be smart to load the view of products/index in this template so your products table is only coded once off course.

The other one will just generate the table with the products:

/products/index

Offcourse the urls are flexible, you can also load for example: products/index.html and product/index.xml for example. It doesn't matter for this example. As long as you have 2 separate urls.

This will return a normal full page, there is a div which might be filled with ajax but otherwise is just not visible:

<a href="/pages/products" id="productslink">Products overview</a>
<div id="products"></div>

Then you add the jQuery add a onClick handler to the link:

$('#productlink').click(function(e) {
    e.preventDefault(); //make sure the link doesn't work anymore
    $('#products').load('/products/index');
});

Now you get the same result for both a browser without JavaScript and a browser with JavaScript. The nice thing is that it doesn't matter. It will give the same result.

You can do the same with forms, just make a working HTML form. The attach the actions with jQuery afterwards and it will always work.