1

Google is indexing ajax pages with hastags since 2015: https://webmasters.googleblog.com/2015/10/deprecating-our-ajax-crawling-scheme.html

However, is there a possibility to exclude specific URLs with a specific hash tag (because of duplicate content, i.e. sorting parameters)?

Example:

  • example.com/#!explore/world (is OK to be indexed)
  • example.com/#!explore/world:sortby=date (should not be indexed)

Since the page does not get reloaded after the hash tag changes to a new ajax page, it does not make sense to use the <meta name="robots" content="noindex"> tag, since it would count for ALL ajax hash URLs...

Improve this question
1

The best thing you can do is setting the canonical meta tag for all pages with filtered views (sort by, ascendant, descendant, price range, etc), to let bots know which is the original page and which one should be indexed.

So when URL is:

example.com/#!explore/world:sortby=date

Canonical meta tag should be set to:

<link rel="canonical" href="example.com/#!explore/world">

After implementing the canonical, wait some time, maybe a week, to make sure the bots know that the canonical tag is present and then proceed blocking web crawlers via robots.txt.

After waiting for a couple of days/weeks block via robots.txt

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /*sortby=

Note 1: the /*sortby= will match any url containing the string sortby= . Do not use ! as in regex has a specific meaning.

Note 2: It might be longer or less than a week, check the SERP after a while to see if hash filtered urls have been removed.

Note 3: the order is important. Implement canonical, wait, then block via robots.txt. The reason this is important is because you need to allow web crawlers to read the canonical tags, once the access is "blocked" via robotx.txt they wont be able to see the canonical tags.

Improve this answer
5
  • Thank you. But when you set a canonical meta tag for an ajax site. How does this work? Is Google reloading a page when navigating through the ajax pages such that the canonical tag can update too? Or do you need to change the canonical meta tag, everytime a new ajax pages is loaded? Would Google even notice that? – Simon Ferndriger Nov 23 '17 at 15:55
  • @SimonFerndriger It will depends on what technology are you using, but the goal is the same: the canonical tag works like any other meta tag in the head section. You can change it to write the base url according to the main page without filters, if you apply the filters it will always stay the same because it has already being set for that main page or original page. So even if the page is reloaded you need to make sure the canonical is referring always to the original page and not the current filtered URL. You can let javascript set the base URL Or use a plugin if this is a CMS. – Raul Reyes Nov 23 '17 at 21:07
  • Thank you - however, I'm just not sure about the whole concept of having canonical hash tag URLs (which need to be set by JavaScript) in general... is there any documentation/recommendation from Google & Co. about canonical hash tag URLs? – Simon Ferndriger Nov 25 '17 at 9:05
  • @SimonFerndriger hello, the url with or without hash it will always be a URL and Google will crawl the page again every time someone decides to “sort by” the results unless told otherwise via robots file or via noindex, nofollow meta tag. Read about canonical tags support.google.com/webmasters/answer/139066?hl=en. And you can write tags before loading the page, I recommend doing this at the earliest stage possible in the head section. Read tab out running JavaScript before page load here stackoverflow.com/questions/2920129/… . – Raul Reyes Nov 26 '17 at 0:24
  • Thank you. However, on support.google.com/webmasters/answer/139066?hl=en there is no mentioningi about canonical hash URLs. And yes, I agree also that - if you alter canonical with JavaScript - you need to do it as early and immidiately as possible. But still, would be nice to have some sort of official Guidance from Google about this. They somehow left a vacuum after they decided to deprecate their previous ajax fragment solution... – Simon Ferndriger Nov 27 '17 at 7:26
0

Update your robots.txt and disallow the bots from indexing or crawling these dynamic pages. Your robots.txt can be something like:

User-agent: *
Disallow: /*sortby=date*

Also, if you connected your website to a Google Webmaster Tool, make sure to run the robots.txt tester on the dashboard.

And yes, the for the dynamic pages can be used too.

Improve this answer
1
  • Do you have documentation that robots.txt can be used for hash fragments? I don't believe that it can. – Stephen Ostermiller♦ Nov 23 '17 at 10:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.