3

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told that whilst these will be respected in terms of the content of the page, the URL itself is still crawled.

The reason I'm so bothered about is due to the crawl budget that is allocated to the site. The site uses layered/faceted navigation, and the combination of filters creates hundreds of thousands of unique links.

I want to make sure Google ignores these links completely, and concentrates on the important pages that I want to be crawled and indexed.

I've been made aware of a couple of options -

  1. Ajax Checkboxes - replace the HTML links with ajax checkboxes. The problem here is that these aren't any good for accessibility and browsers with JavaScript disabled. I'm also aware, that Google can see this as cloaking, and can penalise appropriately.
  2. Hashbang #! Urls - From what I can find, if you add a hash bang to a URL, Google will not crawl it. For example www.myshop.com/shoes#!colour-red. I can't find any sites using it in this form though, so I'm not convinced.
2
  • Google will pretty much attempt to crawl anything as its my understanding, you should use robots to attempt to stop Google crawling, and to further prevent indexation use no index on those pages... Google is a bit of an animal when it comes to stopping the crawls, it's a lot easier to stop Google indexing than it is to stop it crawling. Jun 11, 2013 at 9:43
  • Google can indeed crawl, and index hash bang URLs.
    – Max
    Jun 12, 2013 at 10:13

1 Answer 1

4

Using Disallow: in robots.txt will prevent Googlebot from crawling the URLs at all. In fact, robots.txt is more of a tool for controlling your crawl budget than for controlling which pages are indexed. In some (albeit fairly rare) cases, Google chooses to included pages in the index even though they are in robots.txt based on the number of inbound links to those pages and the anchor text of those links.

So if you have a section of the site that you don't want to have Google crawling and using up your server resources create a robots.txt file like this:

User-agent: *
Disallow: /folder1/

Here is Google's full documention for robots.txt. They even support wildcards.

AJAX Checkboxes

I would not rely on javascript to prevent Googlebot from crawling anything. Googlebot now executes some javascript and its ability to do so is likely to improve in the future.

Hashbang #! URLs

Hash bang urls are specifically designed to be crawlable. When Googlebot encounters #! in a URL it makes a request to your site. Using a #! is also going to use up your crawl budget.

Using just a hash without the bang (#) in your URLs might work for you. Googlebot doesn't make requests to your server without the !.

6
  • Hi Stephen, Thanks for your answer. I forgot to include this in my question, as I'd already looked at this route. From what I've been told, using this method will stop Google crawling the CONTENT, but not the URL, so the crawl budget is still used up. If this is incorrect, then let me know because I'm well out of my depth. If I could use this method, then it would be ideal, because I include a keyword in all of my URL's that I don't want to be crawled. Thanks! Jun 11, 2013 at 9:52
  • 3
    Google will not even request URLs that are in robots.txt. Jun 11, 2013 at 9:54
  • Thanks for the clarification Stephen. Do you have any opinion on the other methods that I mentioned? Or is this just the best way? Thanks Jun 11, 2013 at 10:04
  • I added to my answer to address that. Jun 11, 2013 at 10:12
  • @StephenOstermiller how are you feeling in 2021+ about using JavaScript as a solution to limit the URL being changed? Nov 10, 2021 at 12:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.