Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

At a question asked previously at this site regarding Baidu's overactive crawling tendencies, one user comments:

I'd only block a search engine spider if you never want to receive any search traffic from them, not just because you currently don't receive any.

What's the logic behind this statement, and does it hold?

In other words...

Is it true that blocking a search engine now puts future page rank in jeopardy should I choose to unblock it later?

share|improve this question
add comment

1 Answer

up vote 2 down vote accepted

Since I made that comment, I should probably explain myself a little.

Blocking a search engine does not jeopardize your future ranking on that search engine. However, it's still unwise to block a search engine simply because you don't currently receive any search traffic from them. The reason being that, just because you don't currently receive much traffic from them doesn't mean it will always be that way, especially in the case of a major search engine like Baidu.

And the reason you shouldn't block a search engine that you might possibly want to rank well on in the future is two-fold:

  1. If you've blocked the search engine, you're automatically taken out of its index, cutting your organic search traffic to 0, creating a self-fulfilling prophecy. If you don't block that search engine, at least it's possible for that traffic source to increase, and you'll be able to see when that search engine might be worth paying attention to.
  2. AFAIK, most search engines still take page age into account. Most ranking factors can be faked, and even domain age can be gamed by buying used domains. However, page age is still a very reliable ranking factor that is difficult for spammy sites and blackhat SEOs to fake. And most search engines are still very favorable towards sites that are consistently maintained and grown gradually over long periods of time. If you invest the time to do this, but you block a search engine, then all that hard work is essentially wasted. All the search engine will see is hundreds of new pages popping up in a single day, and it will treat them as such.

There are cases where one might benefit from blocking a particular search engine. But you need to be very sure that there is no value in being indexed by that particular search engine. If it's a legitimate search engine that's being used by lots of people, then the potential benefits will generally outweigh the cost of being crawled.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.