Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I would like to stop Google crawling certain links on a site. When I say this, I mean completely stop the crawling. I'm aware that I use noindex/nofollow and other methods in robots.txt, but, I'm told that whilst these will be respected in terms of the content of the page, the URL itself is still crawled.

The reason I'm so bothered about is due to the crawl budget that is allocated to the site. The site uses layered/faceted navigation, and the combination of filters creates hundreds of thousands of unique links.

I want to make sure Google ignores these links completely, and concentrates on the important pages that I want to be crawled and indexed.

I've been made aware of a couple of options -

  1. Ajax Checkboxes - replace the HTML links with ajax checkboxes. The problem here is that these aren't any good for accessibility and browsers with JavaScript disabled. I'm also aware, that Google can see this as cloaking, and can penalise appropriately.
  2. Hashbang #! Urls - From what I can find, if you add a hash bang to a URL, Google will not crawl it. For example www.myshop.com/shoes#!colour-red. I can't find any sites using it in this form though, so I'm not convinced.
share|improve this question
    
Google will pretty much attempt to crawl anything as its my understanding, you should use robots to attempt to stop Google crawling, and to further prevent indexation use no index on those pages... Google is a bit of an animal when it comes to stopping the crawls, it's a lot easier to stop Google indexing than it is to stop it crawling. –  bybe Jun 11 '13 at 9:43
    
Google can indeed crawl, and index hash bang URLs. –  Max Jun 12 '13 at 10:13

1 Answer 1

Using Disallow: in robots.txt will prevent Googlebot from crawling the URLs at all. In fact, robots.txt is more of a tool for controlling your crawl budget than for controlling which pages are indexed. In some (albeit fairly rare) cases, Google chooses to included pages in the index even though they are in robots.txt based on the number of inbound links to those pages and the anchor text of those links.

So if you have a section of the site that you don't want to have Google crawling and using up your server resources create a robots.txt file like this:

User-agent: *
Disallow: /folder1/

Here is Google's full documention for robots.txt. They even support wildcards.

AJAX Checkboxes

I would not rely on javascript to prevent Googlebot from crawling anything. Googlebot now executes some javascript and its ability to do so is likely to improve in the future.

Hashbang #! URLs

Hash bang urls are specifically designed to be crawlable. When Googlebot encounters #! in a URL it makes a request to your site. Using a #! is also going to use up your crawl budget.

Using just a hash without the bang (#) in your URLs might work for you. Googlebot doesn't make requests to your server without the !.

share|improve this answer
    
Hi Stephen, Thanks for your answer. I forgot to include this in my question, as I'd already looked at this route. From what I've been told, using this method will stop Google crawling the CONTENT, but not the URL, so the crawl budget is still used up. If this is incorrect, then let me know because I'm well out of my depth. If I could use this method, then it would be ideal, because I include a keyword in all of my URL's that I don't want to be crawled. Thanks! –  JamesAllwood Jun 11 '13 at 9:52
2  
Google will not even request URLs that are in robots.txt. –  Stephen Ostermiller Jun 11 '13 at 9:54
    
Thanks for the clarification Stephen. Do you have any opinion on the other methods that I mentioned? Or is this just the best way? Thanks –  JamesAllwood Jun 11 '13 at 10:04
    
I added to my answer to address that. –  Stephen Ostermiller Jun 11 '13 at 10:12
    
Thanks Stephen. –  JamesAllwood Jun 11 '13 at 10:50

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.