0

I have a search-engine based website, and the search functionality itself is not crawlable because of a Disallow: rule in robots.txt. Other content is on the site is crawlable.

If I get incoming links to the search portions of the site, will they be as good for SEO? (Like links to the front page of the site would be.)

Improve this question
2
  • I deleted my answer because I felt it wasn't based on facts and plus for the fact I received a -1. In my findings I found a page that was denied access using robots with a link from a PR6 site (yes when Google was using page rank and it actually worked then) that the page that was disallow remained N/A and also the domain remained 0 which would indicate no JUICE passed or at least back then (2 YEARS AGO). Additionally people refer to spammy links like its some kind of 'WordPress Comments' or 'Forums' – Simon Hayter♦ Nov 12 '13 at 18:10
  • Its far more complex what makes a spammy link and its actually better to have 10 links from 10 poor sites about soap to a soap site than it is to have 10 links from stack overflow about jQuery problems. – Simon Hayter♦ Nov 12 '13 at 18:13
1

Robots.txt does not prevent links from passing pagerank

We know this because sometimes pages that robots.txt disallows end up ranking even when Googlebot can't crawl them. In those cases the "title" of the page that Google uses is usually based on the anchor text from the links. The pages rank based solely on the power of the external links.

I've also done experiments that show when internally linking to pages blocked by robots.txt less pagerank is available to flow to other pages on the site.

Google can't tell what blocked pages link to

So even if blocked pages get some pagerank, they can't directly help the other pages on your site.

Google may re-assign pagerank to uncrawlable pages

Sites that block lots of pages with robots.txt (and link to those pages internally frequently) seem to enjoy good rankings anyway. The same goes for sites that try to use nofollow to sculpt pagerank. The practice doesn't seem to hurt as much as I would expect. This is likely because Google re-assigns "lost" pagerank to other pages on the site. Possibly by distributing it evenly across all indexed pages. Another explanation might be that Google has a concept of "domain authority" that they use in addition to Pagerank that helps in these cases.

Inbound links to blocked pages may help some

So my takeaway would be:

  • All (non-spammy) inbound links (whether to blocked pages or not) should help some.
  • Links to blocked pages are not going to help directly. They are not going to help specific pages rank. Nor is the anchor text going to matter much.
  • Links to blocked pages probably don't help as much as links to crawlable pages (but it's hard to say how much less they help.)

So I would:

  • Gladly accept inbound links to blocked pages.
  • Not go out of my way to build links to blocked pages.
  • Don't try to do anything stupid to claim every last ounce of link juice from blocked links. For example, you wouldn't want to make your site search results crawlable. That would be a direct violation of the Google webmaster guidelines.
Improve this answer
3
  • Hey Stephen how can Google determine if the links are indeed spammy? If the content is blocked. Spam links are determined by the content of the site where the link is coming from and that too the site, This princabple would work on a site that only focuses on one niche. Google no longer takes domain authorithy that seriously. Google wants to stop people building links on high PR sites in the hundreds if not thousands (Stack Overflow, Adobe, MSDN, Mozilla) and so on, all these sites are high authorithy, only way to determine if many of them are spam is relevancy to that of the site or page. – Simon Hayter♦ Nov 12 '13 at 18:19
  • I wouldn't imagine that anchor text spam is much of a problem to pages blocked by robots.txt. It is hard to get them to rank anyway. As far as detecting general link spam, Google will know the subjects of the site and would likely penalize (or simply ignore) links from sites that are not on topic. – Stephen Ostermiller♦ Nov 12 '13 at 18:24
  • Gonna have a think, I can't imagine that the juice flows other than the authority due to my understanding of how the juice flows. For example if you have 10 internal links on the page then the juice is eaten up / 10 of those links so I'm wondering where the juice goes if there's no links. Only way I can see it helping is because its one less link going to the home page which helps loads on link diversty – Simon Hayter♦ Nov 12 '13 at 18:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.