I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use
noindex/nofollow and other methods in robots.txt, but, I'm told that whilst these will be respected in terms of the content of the page, the URL itself is still crawled.
The reason I'm so bothered about is due to the crawl budget that is allocated to the site. The site uses layered/faceted navigation, and the combination of filters creates hundreds of thousands of unique links.
I want to make sure Google ignores these links completely, and concentrates on the important pages that I want to be crawled and indexed.
I've been made aware of a couple of options -
- Hashbang #! Urls - From what I can find, if you add a hash bang to a URL, Google will not crawl it. For example
www.myshop.com/shoes#!colour-red. I can't find any sites using it in this form though, so I'm not convinced.