Crawling an ajax based page with both a hash fragment and a meta tag

Question

According to google's documentation on crawling ajax based web pages, if a url contains a hash fragment, or something at the end of an url that looks like #helloworld, and if there is an ! after the #, as in #!helloworld, google will then request the url url?_escaped_fragment_=helloworld.

I currently have an ajax based webpage that I want google to be able to crawl. Sometimes, the page uses hash fragments, and for those situations I set up the server so it will return an html snapshot for that page using _escaped_fragment_.

However, that webpage often does not load a hash fragment, and when that happens the webpage still loads content using ajax.

I couldn't find a good solution to enable ajax crawling for pages that sometimes have a hash fragment and sometimes don't. How can I tell google to use _escaped_fragment_ when there is a hash fragment, and to use something else to get an html snapshot of a page when there isn't a hash fragment?

Answer 1

Section 3 of Google's documentation on crawling Ajax content declares that you use a META tag on those pages without a hash fragment:

In order to make pages without hash fragments crawlable, you include a special meta tag in the head of the HTML of your page.

So, if pages sometimes have them and sometimes don't, it sounds like you want to determine that server-side and serve up the appropriate META tag when the hash fragments are not being exposed.

Answer 2

As Google decided the protocol, you can't really tell it to use something different. If you ajaxy page doesn't have #! then google will try to crawl it as as normal page. You can add a special metatag (usually just for homepage) to instruct Google to crawl a page that doesn't have #! then Google will do the same replacement with escaped fragment and expects your server to return a result.

So if you add this metatag to index.php:

<meta name="fragment" content="!">

even though that index.php doesn't have a hash fragment, Google will still try to get a page from your server called index.php?_escaped_fragment_= and you can make your server serve the snapshot the same way it does with other pages.

Other than that, I don't really see why you can't technically always use hash fragments.

Depending on the project, I found it more useful and pragmatic to depend on techniques from Progressive Enhancement, i.e. make sure that my pages serve content without javascript so that Google - and other search engines - can crawl them then add my javascript afterwards. This technique worked perfectly on a highly ajaxified knockout-based project I worked on recently and we just ignored Google protocol and we went back to the basics (especially that we were interested in other search engines crawling our site as well).

Answer 3

@Christofian you should be able to determine if a hash tag is being used since they turn up after an event such as a click on a link. That usually triggers some javascript location.hash or jQuery method to add the hash to the URL client side. Google's clear if you have a hash tag use ! if you don't have a hash tag in the URL and your page has AJAX use the meta tag. One or the other none override the other. You should be looking for the function that's adding the hash tag, before it fires check if a hash is being added if not add the meta tag.

Sample

var projectSlug = location.hash.replace("\#","");   
if(projectSlug != "index"){
    processProject(projectSlug);
}