3

If every single link to a given website www.example.com points to a page in a particular subdirectory (i.e. www.example.com/user/[something]), but that directory is off limits as per robots.txt, i.e.

User-agent: *
Disallow: /user/

because I don't want these pages appearing in google search results, am I shooting myself in the foot in terms of pagerank? Does the incoming "link-juice" still count if the target page is forbidden to Google in my robots.txt?

2
  • Are you asking whether or not you should be disallowing user functions in robots.txt, such as log in, log out, profile, etc.? Because I don't see how it's "shooting yourself in the foot" to link to your own login page. I guess the only conclusion is that there's no reason to disallow search engines from indexing the login page. – Lotus Notes Nov 17 '10 at 18:29
  • No, users get profile-pages which they link to. I do not want these pages crawled by google, yet these are likely the vast majority of links that will be pointing to my site – Mala Nov 19 '10 at 2:03
3

Speaking to your question - "Does the incoming 'link-juice' still count if the target page is forbidden to Google in my robots.txt?" - I would say that PageRank is calculated, even for noindex/nofollow URI's:

While Google won't crawl or index the content of pages blocked by robots.txt, we may still index the URLs if we find them on other pages on the web. As a result, the URL of the page and, potentially, other publicly available information such as anchor text in links to the site, or the title from the Open Directory Project, can appear in Google search results.

Google Webmaster Central: Block or remove pages using a robots.txt file

Example: My "working-model.com" domain has had an all-exclusive robots.txt specified for as long as I can remember, however, a Google search for working-model.com (or a Yahoo search, or a Bing search) shows a rank for the domain (probably as a result of a domain WHOIS site linking in).

2

Even if the page is not indexed by Google those pages will still get PageRank assigned to them. This means by linking to them you will "leak" PR as that PR will simply be lost instead of passed to other links. It essentially is the same as using nofollow on a link. So if you are linking to internal pages that are blocked with robots.txt you are essentially diluting the amount of PR you are passing to the allowed pages on your site.

See this blog post for more on this.

1
  • There are very few "allowed" pages on my site: just the home page, and an updates blog. The rest are all user profiles, and forbidden by the robots.txt file – Mala Nov 19 '10 at 2:05
1

Yes, Google does assign PageRank to URLs that are robotted, but no, you're not shooting yourself in the foot by having such URLs or by linking to them. The time you spend on tweaking your perceived PageRank flow is generally much better spent on working on your content instead.

The only reason to watch out for this is if you're using the robots.txt disallows to control duplicate content. Since the robotted URLs can collect PageRank, and since Google can't confirm that they are duplicates (as would be possible if they could be crawled), it can result in Google indexing both the robotted, uncrawled URL as well as the crawlable version with the same content. It's much better to allow crawling of duplicate content and to use one of the usual canonicalization methods (like a 301 redirect or a rel=canonical link element).

0

There are three schools of thought on this.

a. Yes, page rank will pass to the robtos.txt blocked page, it will be lost, find a way not to do it.

b. No, It's an internal link. The way Page Rank flows around a site prevents it being lost to pages that are banned by robots.txt.

c. John Mueller's position(he has actually commented on this thread) that no it won't impact you, but then he goes and muddies the waters saying you'd be better off working on your content instead. Since tech and content teams are able to work in parallel which he well knows, this reasoning is a strawman and not at all useful. It's impossible to tell if he means the impact is so small, focus elsewhere, or that internal linking to pages blocked by robots txt has zero impact.

He has also gone on the public record saying most of what you read on this topic is dated, wrong etc...so who knows if what he said was true in the first place, or true now?

I don't know which is right - so I

a. Assume A is correct. I benefit it its right, I cause no harm if it's wrong. b. I assume B is incorrect. I benefit if its wrong, cause no harm if its right. c. I assume John is not going to give a straight answer that ultimately closes the subject. He has stated his opinion about where time might be best spent, but does not absolutely close the door on no benefit passing from stopping link to blocked by robots.txt pages.

So - I don't link to pages blocked by robots.txt wherever possible.

Also not helpful as he talks about using this to stop duplicate content when its more often used to stop faceted content being indexed, a subset of the main content but not duplicate.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.