Tell me more ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.
up vote 5 down vote favorite

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and why?

Which ones have a legitimate purpose in increasing traffic to my site? Many of them are SEO.

I have to say I see less traffic if anything since the bots have arrived in large numbers.

It would not be too hard to block these since they all admit in their User Agent that they are bots.

share|improve this question
Please don't forget to choose an useful answer. Thanks! – Zistoloen Mar 7 at 11:37

2 Answers

up vote 6 down vote

You can block bots but it depends on what you want for your web site.

You can block search engine bots if you don't want to see your web site indexed in particular search engine.
Example: Yandex is russian search engine. You can block it if your business is not targeting Russia.

You can block SEO bots if you don't want to use web analytics of it.
Example: Ahrefs is web analytics solution. You can block it if you don't use this web analytics solution.

Reason to block bots:
- less robots go to your web site and more bandwidth is attributed to real visitors.
- be safe against malwares bots.
- logs size

Reason to not to block bots:
- bots like search engine bots can increase your traffic by indexing your web site.

You can learn more about bots by reading FAQ of robotstxt.org.

If you want to determine several robots to block, you can take inspiration from this web site robots.txt.

Be careful, some bots can ignore robots.txt, more information here.

Conclusion: You can search on internet robots function to determine if blocking them can be useful.

share|improve this answer
Thanks. By the way I am also getting visits from Linode. The UA says it is a Python script. Should I be suspicious? – Frank E Jan 14 at 16:25
If I were you, I didn't block Linode. I don't know what it is exactly by it's about hosting. – Zistoloen Jan 14 at 16:39
What about IPStrada? I see them showing up from time to time. They never sign up for an account. – Frank E Jan 15 at 1:35
Like Linode. Robots scan internet even if you or they never sign up for an account. But don't worry, it's not big deal. – Zistoloen Jan 15 at 8:39
up vote 3 down vote

While attempting to block bots can help in free up resources and clean up your logs it’s important to note that Robots.txt and even using the meta tag on pages noindex does not actually stop bots visiting your site. They can still crawl your site occasionally to see if the denied from robots has been removed. A lot of bots don’t even use a user agent and will use a standard user agent. The bots I’m referring to are typically SEO harvesting bots that scan for backlinks and not the general ones you find from search engines.

Rather than blocking the bots you should just factor in these bots when counting up your visitors, after a while of actively monitoring your site your establish a rough figure which are bots. Most people care about unique visits and this rules out the bots since they are constantly returning. In this day and age there are plenty of servers, shared hosting that can handle these bots so other than pages that you don’t want indexed I see no reason why to block these types of bots. Of course you have harmful bots as well but these certainly won’t use user agent ;)

Personally I believe blocking robots is a waste of time as they don’t use that much resources at all, SEO robots can help as they list your site on PR0 pages which of course increases your PR and there automated so you won’t get punished by them.

Logs Issue

You should use a proper log viewer that enables you to filter out certain requests, this makes it easier when reviewing your logs. Good viewers can filter out lots of things like normal visits, 404s' and so forth.

share|improve this answer
1  
Why do SEO bots scan for backlinks? Is it just because that's something SEO consultants use? Or is there some more sinister purpose? – Frank E Jan 14 at 17:29
1  
Hi Frank, they scan for backlinks and log them so webmasters can see who's linking to them. It is big money for the SEO companies to log backlinks and charge webmasters to discover whos linking to them, when the link was created, if its been deleted, if its follow or no follow - what anchor text they use and so forth. Google, Bing do not provide this information so its a desired service and valuable to most web masters wanting to look at their link network. – bybe Jan 14 at 18:03
It's also worth nothing that a lot of vulablites that people look for are gathered by using search engines and not having a bot scan every single site. For example if a module/plugin is out of date and theres a known exploit people will use Google to locate these sites to exploit them. Blocking robots will not secure your server :P – bybe Jan 14 at 18:05

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.