I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow
and other methods in robots.txt, but, I'm told that whilst these will be respected in terms of the content of the page, the URL itself is still crawled.
The reason I'm so bothered about is due to the crawl budget that is allocated to the site. The site uses layered/faceted navigation, and the combination of filters creates hundreds of thousands of unique links.
I want to make sure Google ignores these links completely, and concentrates on the important pages that I want to be crawled and indexed.
I've been made aware of a couple of options -
- Ajax Checkboxes - replace the HTML links with ajax checkboxes. The problem here is that these aren't any good for accessibility and browsers with JavaScript disabled. I'm also aware, that Google can see this as cloaking, and can penalise appropriately.
- Hashbang #! Urls - From what I can find, if you add a hash bang to a URL, Google will not crawl it. For example
www.myshop.com/shoes#!colour-red
. I can't find any sites using it in this form though, so I'm not convinced.