4

My old site used to write the URLs without !.

Eg: www.example.com/#/products

But it has been more than 6 months since I changed it to include ! in the URL like so:

www.example.com/#!/products

But unfortunately Google search results still show my old URL schema.

There is no way I could enter in the Google Removal Tool (via their Webmaster Tools) to remove www.example.com/#/products since Google will complain about an invalid URL.

All I want is to remove www.example.com/#/products and Google to show the one with /#!/products in the results instead.

migrated from serverfault.com Nov 15 '14 at 7:54

This question came from our site for system and network administrators.

  • 1
    Be aware: Neither of those URL styles are search engine friendly, nor are they friendly to users. – Michael Hampton Nov 15 '14 at 7:54
  • Hi @MichaelHampton I understand, thanks for your concern. I actually have snapshops of those pages via ?_escaped_fragment= but right now i need to focus on getting rid of www.example.com/#/products from Google's search result – David Nov 16 '14 at 16:26
  • Presumably www.example.com/#/products now simply returns your homepage, not the "products" page? Or are you doing a (client-side) redirect to the #! URL? – MrWhite Nov 17 '14 at 8:32
  • 2
    @David: How do you redirect users? If I go to site.com/#/example, do you 301-header me? If not, that's your problem (just look into 301-headers) – Martijn Nov 17 '14 at 8:53
  • 2
    Hi everyone, thanks for helping. @Martijn angular is doing the redirect so yeah it does client redirection to #!/products. You can't do 301 redirects for hashed urls the server/nginx/rails won't see that part/url. Also - can you explain what is the relationship of getting rid of www.example.com/#/products from Google's search result here? – David Nov 17 '14 at 10:05
2
+25

It's been a while since i have been in google webmaster's tools but I do remember that in there is a tool for removing indexed pages for any website..

**You will need to link the website with the particular webmaster's tool account to do so...

https://www.google.com/webmasters/tools/dashboard

login or create new account (link the website as mentioned above)

click on google index & select remove urls

on the opening page insert all urls you want removed.....**

  • 1
    Hi Norman thanks for your help, but as I mentioned in my post, i already went through the URL removal tool and its not going to work – David Nov 24 '14 at 7:58
2

First, you don't have to go through the trouble of removing the old URLs with the hash. If your content is getting crawled and indexed appropriately under the new hash-bang URL, it'll soon get replaced.

It's not entirely clear what the problem is without troubleshooting against the live site, but hopefully these steps will help you discover a solution.

You'll want to start by eliminating common problems.

Check if Googlebot is Accessing Your HTML Snapshots

The whole idea of AJAX crawlability is to get your product images, descriptions, and specifications indexed. I'm assuming you've followed the appropriate instructions from Google's AJAX crawlability specifications, so you'll want to make sure that Googlebot is able to see your HTML snapshots. If Google isn't seeing the correct HTML snapshots, this is likely the reason why Google hasn't updated it's index to reflect the new hashbang URLs you created.

Using Google Webmaster Tools (GWT), go to the "Crawl » Fetch as Google" section. Then have Goolgebot fetch and render the full _escaped_fragment_ URL => www.example.com/?_escaped_fragment_=products. If all is well, you should see the status as "Complete". You should also be able to click on the status, to see the HTML snapshot of the particular product.

If you see a "Not Found" for the status or "Blocked", that's likely the reason Google hasn't updated the URLs in the index.

If it's "Blocked", that means you have a rule in your robots.txt file that is preventing Googlebot from accessing the HTML snapshot.

If it's "Not Found", that means you probably haven't configured nginx to appropriate route requests with ?_escaped_fragment_= to the server side rendering of the HTML snapshot.

If you see a "Partial", then you'll need to click through on the status, and see the various resources which are blocked from Googlebot. But basically, if you don't see the appropriate content in the thumbnail image from the Google's render of the page, it's likely that you're blocking some resource that Google needs to render out the content.

Check if Angular is Rendering the HashBang URLs in the HREFs

If the rendered thumbnail of the page is correct, we've established that Googlebot is able to see the server side rendered HTML and the routing is correct for the _escaped_fragment_ requests. This then points to the front end since Googlebot doesn't seem to be discovering the new hashbang URLs.

A quick check of your access logs should help you confirm this issue. If you have Splunk, LogStash, or good ole grep handy, you'll want to look for requests that have the following criteria:

  • user agent contains "googlebot"
  • request URL contains "?_escaped_fragment_=products"
  • requesting IP starts with "66.249"

Basically, you want to look for all verified Googlebot requests. Googlebot mostly comes from 66.249.*.* these days.

If you're not seeing many requests compared to the # of Pages Crawled Per Day listed under "Crawl » Crawl Stats" in GWT, then it's likely that Googlebot isn't able to see the hashbang URLs in the href attributes of the anchor tags.

If you navigate to www.example.com/#!/products and Inspect Element in Chrome or Firefox, you'll want to make sure that the rendered HTML has clear anchor tags with hrefs that include the hashbang versions of the URLs. If not, then you'll want to reconfigure Angular to do so.

If you do see them there, you'll want to then want to fetch and render www.example.com/#!/products or another page like your homepage that would have anchor tags with href attributes linking to hashbang URLs.

When you click on the status to see the "rendered" version of the page, you can click on the "Fetching" tab which is next to the default "Rendering" tab, to see the HTTP response headers and resulting HTML.

If you don't find the hashbang references in the appropriate spots from the "Fetching" tab, you'll have to step through your Angular stack to see where the disconnect is happening. Likely you're doing a javascript redirect in an asset file that is blocked by Googlebot, and it's probably not seeing the updated references.

When All Else Fails

If it's none of the things above, well then it's more likely a bug with Google itself. This actually happens from time to time, and although Googlers are smart folks, they do get it wrong sometimes.

You'll want to jump into the Google Webmaster Forums: Crawling, index, and ranking, and hopefully one of the many Googlers who frequent the forums will investigate it for you.

An Alternate Solution

So after all that's been said and done, I strongly recommend against using the _escaped_fragment_ standard. Although it's something that Google put out there and it's been adopted by Bing, you rarely see sites implement it correctly, and you rarely see the hashbang URLs appear in the top search results for indexed pages.

My suggestion, if you have the engineering time & resources to do it, is to move to using pushState(). With pushState you'll need to add more logic to your routes to handle both client and server side requests, but you can then take advantage of using canonical tags and you don't have to mess with the ?_escaped_fragment_ overhead.

I know Angular + pushState isn't the most well documented thing out there, but there are some good posts out there depending on the rest of your stack.

Lastly, I don't want to be overly self promotional, but as these things go, the conversation eventually leads to "pre-render" vs "server side" rendering. I did a talk at SMX Advanced covering some common AJAX crawlability issues. If you jump to slide 34, you can see some of the pros and cons. Also on slide 49 and 50 there are references to a few other good Angular + SEO posts.

0

If you have changed your URLs you need to 301 redirect OLD urls to new one, Once you do this Google will automatically update its indexing status within a month or so depending on the crawling rate of your website.

Here is a complete guide to 301 redirection

http://moz.com/learn/seo/redirection

  • Hash URLs cannot be redirected because the portion after the has is not sent to the server. It is only handled client side by JavaScript. – Stephen Ostermiller Nov 22 '14 at 22:22
  • Hi Adrian Thanks for your help, but as i mentioned in my post and as @StephenOstermiller mentioned, what comes after the hash is not sent to the server so redirect will not work. – David Nov 24 '14 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.