Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

According to google's documentation on crawling ajax based web pages, if a url contains a hash fragment, or something at the end of an url that looks like #helloworld, and if there is an ! after the #, as in #!helloworld, google will then request the url url?_escaped_fragment_=helloworld.

I currently have an ajax based webpage that I want google to be able to crawl. Sometimes, the page uses hash fragments, and for those situations I set up the server so it will return an html snapshot for that page using _escaped_fragment_.

However, that webpage often does not load a hash fragment, and when that happens the webpage still loads content using ajax.

I couldn't find a good solution to enable ajax crawling for pages that sometimes have a hash fragment and sometimes don't. How can I tell google to use _escaped_fragment_ when there is a hash fragment, and to use something else to get an html snapshot of a page when there isn't a hash fragment?

share|improve this question
1  
You should change the application to always have hash fragments on links. –  eduardocereto Oct 19 '12 at 6:44
    
@eduardocereto I can't: google will always try to access index.php, and without javascript, I can't add a hash fragment to that page. –  Christofian Oct 19 '12 at 21:37
add comment

3 Answers 3

Section 3 of Google's documentation on crawling Ajax content declares that you use a META tag on those pages without a hash fragment:

In order to make pages without hash fragments crawlable, you include a special meta tag in the head of the HTML of your page.

So, if pages sometimes have them and sometimes don't, it sounds like you want to determine that server-side and serve up the appropriate META tag when the hash fragments are not being exposed.

share|improve this answer
    
It is technically impossible for me to determine whether a hash fragment is used server side. Does google use the meta tag if there is no hash fragment, or does the meta tag override the hash fragment? –  Christofian Oct 7 '12 at 21:31
    
I took a look at the full specification, and it's very contradictory and confusing. Google says that you can use both in one part of the spec, and that you can only use one in the other. –  Christofian Oct 10 '12 at 23:14
add comment

As Google decided the protocol, you can't really tell it to use something different. If you ajaxy page doesn't have #! then google will try to crawl it as as normal page. You can add a special metatag (usually just for homepage) to instruct Google to crawl a page that doesn't have #! then Google will do the same replacement with escaped fragment and expects your server to return a result.

So if you add this metatag to index.php:

<meta name="fragment" content="!">

even though that index.php doesn't have a hash fragment, Google will still try to get a page from your server called index.php?_escaped_fragment_= and you can make your server serve the snapshot the same way it does with other pages.

Other than that, I don't really see why you can't technically always use hash fragments.

Depending on the project, I found it more useful and pragmatic to depend on techniques from Progressive Enhancement, i.e. make sure that my pages serve content without javascript so that Google - and other search engines - can crawl them then add my javascript afterwards. This technique worked perfectly on a highly ajaxified knockout-based project I worked on recently and we just ignored Google protocol and we went back to the basics (especially that we were interested in other search engines crawling our site as well).

share|improve this answer
add comment

@Christofian you should be able to determine if a hash tag is being used since they turn up after an event such as a click on a link. That usually triggers some javascript location.hash or jQuery method to add the hash to the URL client side. Google's clear if you have a hash tag use ! if you don't have a hash tag in the URL and your page has AJAX use the meta tag. One or the other none override the other. You should be looking for the function that's adding the hash tag, before it fires check if a hash is being added if not add the meta tag.

Sample

var projectSlug = location.hash.replace("\#","");   
if(projectSlug != "index"){
    processProject(projectSlug);
}
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.