2

I read a lot about this but I'm a bit confused since it looks like it's not a black or white thing. In some posts you can read that the more backlinks the better and other ones claiming the opposite.

I'm blocking everything I can except some important bots like those from Google, Bing, Yahoo, Facebook, Linkedin, etc.

Am I doing right or it's a completely bad practice?

  • What do back links have to do with bots? Are you also blocking requests that have an external referrer? – Stephen Ostermiller May 29 at 21:04
  • @StephenOstermiller AFAIK some crawlers (most of them?) build maps of links in their websites or add your website info... I mean, they process the information and present it in their own website generating backlinks. In Search console I can see thousands of links from other websites with good DA, and I think the only way they could link to my website is crawling it. I'm not sure those backlinks are good or bad for Google and most of search engines. – Luciano Fantuzzi May 29 at 21:25
  • @StephenOstermiller I'm blocking bots (with ModSecurity) filtering by user agent, ip, country (like Rusia, China, etc), and other criteria, but I believe I'm not blocking user requests. – Luciano Fantuzzi May 29 at 21:28
  • 1
    @Michaeld We are using ModSecurity (in Apache) and several rules to block traffic depending on conditions. It uses Maxmind's free database to determine user country. You can use some Maxmind implementation in your software I guess. – Luciano Fantuzzi May 30 at 19:00
  • 1
    @Michaeld Depending on the language you use, you may be able to find an appropiate SDK. This is the main dev page of Maxmind: dev.maxmind.com and here you can download the v2 database dev.maxmind.com/geoip/geoip2/geolite2 For example, in PHP you may use the method suggested here: php.net/manual/es/book.geoip.php or jus find some package in Packageist: packagist.org/?query=maxmind – Luciano Fantuzzi May 31 at 17:42
2

There are many successful websites that are blocking all bots except for major search engines such as Google, though most websites allow for bots to crawl.

There are pluses and minuses to blocking all unknown or low ranking bots.

Some bots can become abusive and can crawl your site too frequently affecting your server performance. There are also bots that want to scrape your website's content without your consent and not provide you any credit or links. You also have bots that are trying to find exploits in your server's security, and with enough crawls they may be able to find one depending on your setup.

On the other hand, good bots may crawl your content, publish portions of it on the web while linking back to your site. These potential links naturally can help your SEO greatly.

While it's primarily important to look out for the well-being of your own site while deciding which bots to allow and which to ban, it's worth noting that allowing unknown and new bots to crawl your website is in the internet's general best interest. This is because there are good bots out there trying to do new and exciting things, and when enough websites blacklist all bots except for the major ones like Google, it makes new bots and search engine competitors unable to enter the market as they simply can't crawl enough websites.

A generally wise practice is to throttle smaller bots from crawling too many pages too quickly. This will prevent bad bots from trying to break your site while allowing good bots to do their thing. You can throttle bots by setting a "Crawl-delay" in your robots.txt which specifies how many seconds a bot has to wait before loading another page. If you find that bots aren't obeying the crawl-delay when viewing IP addresses in your server logs, you can simply ban the IP address. If a bot isn't obeying a robots.txt crawl-delay specifications, it is considered to be a "bad bot".

  • Thank you for your opinion. I manually blocked so many bots that I'm in a trouble if I have (and I will) to manually check which one of them are good or bad. Since we have too many websites hosted I can't add custom rules to robots.txt (it would be a maratonic task), so I have to find a transparent solution for all of them. – Luciano Fantuzzi May 30 at 19:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.