11

Our company has developed a Single Page Application using AngularJS and its routing. Google indexed our site decently with JavaScript but it did not index some pages very well so we have developed an HTML only version.

We have followed the Ajax Crawling Specification posted here and have a <meta name='fragment' content='!'> tag and canonical urls. We expect http://www.example.com/foo/bar to be fetched from http://www.example.com/?_escaped_fragment_=/foo/bar.

However, we have found out that when we rolled the AJAX specification we now have all pages indexed twice, once with the JavaScript version as http://www.example.com/foo/bar and once with the new version as http://www.example.com/#!/foo/bar. This is harmful to us since it's duplicate content and also mis-representing out site.

I have tried looking for similar questions here and in the Google product forum but could not come up with anything.

  • Have you raised this issue with Google? On their forum? – Jérôme Verstrynge Aug 23 '14 at 12:15
  • @JVerstry yes - I have done that immediately after I posted this question: productforums.google.com/forum/… However my first assumptions is that we did not understand the protocol correctly rather than something is wrong with their product so my chances of getting help here are fairly decent in addition to this question being relevant to a large target audience here (Everyone with a single page application that doesn't use #! in the URLS and want Google to index them). Thanks for the suggestion by the way. – Benjamin Gruenbaum Aug 23 '14 at 12:18
  • 1
    Did you define canonical URLs through rel="canonical"? – fifi finance Aug 23 '14 at 14:14
  • 3
    redirecting the #! urls and cloaking the ugly urls does not seem like you are following the specification. – Tony McCreath Aug 26 '14 at 23:19
  • 1
    Just wondering, if you aren't using #! URLs, why are you using _escaped_fragment_ and the Google AJAX spec? – MrWhite Aug 29 '14 at 20:16
2

When in doubt, use rel="canonical". This could be as simple as putting <link rel="canonical" href="http://blog.example.com/dresses/green-dresses-are-awesome" /> in your <head>. See Google for more info.

This won't keep bots from crawling both "versions", but it will tell Google (and other SERPs) to only index the canonical document.

  • In that same vein, can he use a robots.txt file?? – eyoung100 Aug 29 '14 at 19:50
  • We have those, they did not help in this case. – Benjamin Gruenbaum Aug 29 '14 at 20:18
1

I don't really understand about javascript.
In Google indexing, us the canonical url in <head> like @philtune says. But if you can't expect http://www.example.com/foo/bar to be fetched from http://www.example.com/?_escaped_fragment_=/foo/bar .

Google Fetch just fetched the canonical url.

In this case, the url is http://www.example.com/foo/bar, google will index http://www.example.com/foo/bar too.

If you want Google index you're page, add this script in <head>:

<meta content='index, follow' name='googlebot'/>

and then fetch again using the Google Fetch.

Note:
google will crwal and follow and crawl the website and follow all links on the page. If in you're page have a link like http://www.example.com/?_escaped_fragment_=/foo/bar and redirect to http://www.example.com/foo/bar, google will index http://www.example.com/foo/bar because http://www.example.com/?_escaped_fragment_=/foo/bar are redirect to http://www.example.com/foo/bar.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.