2

I'm working on an AJAX-based application and thinking of how to make it crawlable by search engines. I've read the Google's AJAX crawling specification but I couldn't get my head around one thing: if my application uses exclusively pushState (i.e. no hashbangs in URLs at all) wouldn't it be easier just to return a static version of the request URL if the request's user agent was a crawler? Would that be considered cloaking? I mean, I have no problem following the AJAX crawling specification and use <meta name="fragment" content="!"> along with _escaped_fragment_ urls, but it seems a lot easier just to check the user agent in such a case.

I'm aware that returning information based on the request's user agent is often associated with black hat SEO, but in this case, unless I'm missing something, checking for a specific URL ( _escaped_fragment_) seems to be the same thing as checking for a specific user agent. In both cases I'd have to generate and return a static version of the requested resource and I could do bad things if I had bad intentions. AFAIK, no option is safer than the other.

Should I, by all means, stick to the specification or would it be okay to use the user agent info? Or is there a better way to make crawlable a web application that uses pushState only?

Any clarification on this matter will be much appreciated.

| improve this question | |
  • Looks like this question isn't receiving enough attention - are you still interested in getting some feedback, or did you resolve this on your own? – dan♦ Sep 25 '13 at 2:50
  • 1
    Hi dan. Yeah, I'm still interested in hearing what other people think about it. Thanks for asking! – Michael Benford Sep 25 '13 at 3:10
1

I'm aware that returning information based on the request's user agent is often associated with black hat SEO, but in this case, unless I'm missing something, checking for a specific URL ( _escaped_fragment_) seems to be the same thing as checking for a specific user agent.

Cloaking was targeted in Google's Penguin update and is considered a violation of Google’s Webmaster Guidelines as can be read about here.

Cloaking is:

done by delivering content based on the IP addresses or the User-Agent HTTP header of the user requesting the page.

As indicated by Matt Cutts, an important heuristic used by Google to detect cloaking is if your web server is checking for Googlebot's user-agent - see this (at 3:00). Checking for _escaped_fragment_ is not a heuristic, as covered by the specification Google adopted.

Should I, by all means, stick to the specification or would it be okay to use the user agent info?

You're risking your site being penalized if you check the user-agent. So if you can stick to the specification, that would be highly advisable, and would likely require much less effort than trying to recover from a penalty or sandboxing.

| improve this answer | |
  • Thanks for the response dan. I think cloaking isn't simply delivering content based in IP or user agent, but delivering different content based on those parameters, right? Anyway, I watched the Matt Cutts' video you've posted and he says that if you don't treat Googlebot differently than a regular user you will be just fine. And that's precisely what I'm planning to do. I've got no intentions at all of tricking Googlebot or any other web crawlers. – Michael Benford Sep 25 '13 at 4:35
  • One of the heuristics is if the hash of the pages being served match or not. As indicated in that video, you could use the Fetch as Google tool to check that. Googlebot surely can detect if you're checking for its user-agents, and since the inner-workings of its algorithm are unknown, it's a potential risk that might be worth avoiding, since it can take months to recover from penalties... So the safe advice would be to stick to the specifications if possible. – dan♦ Sep 25 '13 at 4:50
  • Matt said himself that that's not a hard and fast rule. In fact, there are several situations where the page you serve to Googlebot won't be exactly the same as the one you serve to a regular user. I thought of checking the user agent in the first place because it would be easier than checking a query string, since I don't use hashbangs in my URLs. I can indeed use <meta name="fragment" content="!"> (as the specs say), but that will make Googlebot hit my server twice for every page, and I thought I could optimize that . I guess I'll think a little more about it. – Michael Benford Sep 25 '13 at 5:08
  • For what it's worth, even if one sticks firmly to the AJAX crawling specification, he/she could easily return a different content to Googlebot when asked for an _escaped_fragment_ url. From the crawler's point of view, it's no safer than any other response in terms of cloaking. – Michael Benford Sep 25 '13 at 5:15
  • Different content, but not based on the user-agent. I try to err on the side of caution when answering questions here so as not to jeopardize someone else's site/livelihood, for others as well... Using the meta sounds like a safer choice. – dan♦ Sep 25 '13 at 5:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.