7

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing.

I want to parse the logs and get accurate statistics as to how many human visitors I get, so I've temporarily updated robots.txt to ban bots on all pages. I know this is only partially effective at preventing bot access, but I'm OK with that.

How will disallowing bots affect users searching for this site? Will it prevent users from finding the page on Google?

Improve this question
3
  • 17
    If you want to "parse the logs so I get as few stats from the bots", then do exactly that and parse the logs. (?!)
    – MrWhite
    Oct 28, 2014 at 8:52
  • 5
    Following on what w3d said, all of the major search engine bots will clearly identify themselves with the User-Agent header.
    – Brendan Long
    Oct 28, 2014 at 14:44
  • 2
    If you tell Google to go away in robots.txt, they will take your word for it that you don't want to be indexed and your ranking will disappear. Getting your ranking back may take some time after pulling such a stunt.
    – Fiasco Labs
    Oct 28, 2014 at 14:51

7 Answers 7

Reset to default
21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt.

Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.

Improve this answer
8
  • 6
    +1 and your logs will still be full of bot traffic. Only specific denial in .htaccess will keep them out and you still get a 403 for every link they know about from external reference, they just can't suck in a page and start crawling. The list gets mightily long...
    – Fiasco Labs
    Oct 28, 2014 at 15:40
  • 1
    +1 this is the answer. There is no stopping bots, especially malicious bots.
    – Qix - MONICA WAS MISTREATED
    Oct 28, 2014 at 21:18
  • 2
    @Qix I have seen a few websites that issued an IP ban against anyone that visited a particular URL (that was both forbidden in robots.txt and clearly labeled with "don't click this link or you will get banned" for humans) - that strikes me as a fairly effective way to target unscrupulous crawlers. Granted, it doesn't do much against a bot specifically designed to target your site.
    – Brilliand
    Oct 29, 2014 at 18:02
  • 1
    @ChrisMarisic and clearly labeled with "don't click this link or you will get banned" for humans
    – Qix - MONICA WAS MISTREATED
    Oct 29, 2014 at 19:03
  • 2
    @Brilliand Sound like a fun way to ban other users via CSRF.
    – CodesInChaos
    Oct 30, 2014 at 8:40
13

Banning bots will not let any search engine get the content of the site.

Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic.

Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like Google, Yahoo & Bing follow.

Improve this answer
5
  • That's what I figured.
    – Tensigh
    Oct 28, 2014 at 9:10
  • 12
    Can you please update your answer to explain that robots.txt does not "ban bots". It politely asks bots not to crawl the site and for search engines not to index it. There's absolutely nothing stopping someone from crawling your site anyway.
    – ErlVolton
    Oct 28, 2014 at 15:27
  • 1
    To add to what @ErlVolton said. The malicious bots you'd presumably want to ban won't pay any attention to your robots.txt "ban" on bots. So the only affect this ban will have will be to hurt your SEO.
    – Dennis
    Oct 29, 2014 at 20:02
  • @ErlVolton +1 for "politely ask bots"
    – Archy Will He 何魏奇
    Oct 30, 2014 at 9:20
  • I think the technical term is "Disallow" which IMO is still not a very good representation of what's happening. Maybe "RequestNoCrawl" or something would be better :)
    – ErlVolton
    Oct 30, 2014 at 15:33
3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt

Improve this answer
2
  • "Google may still crawl pages ignored by robots.txt" - By "ignored", do you means URLs that aren't blocked by robots.txt?
    – MrWhite
    Oct 28, 2014 at 9:42
  • 1
    That sentence in the "Does Google ignore robots.txt" answer is also wrong. Google may index pages linked by robots.txt even when they are disallowed in robots.txt, but it won't crawl them.
    – Stephen Ostermiller ♦
    Oct 28, 2014 at 16:29
3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to.

However it is possible your site will still be displayed in search results, particularly if a high-ranking site has a link to your site. Google and possibly other engines may use information from the link alone to decide to show your site in their results pages.

Improve this answer
1
  • This is perfect information. This page is actually a subdomain of our main page which is where I prefer the search engine rank comes from. So knowing that our main page will get the page rankings is perfect. Thanks!
    – Tensigh
    Oct 28, 2014 at 23:21
1

The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN.
Analyzing page views is a task for page-tag based analytics.

Improve this answer
0

The correct answer is to not mess with robots.txt and instead parse your logs, looking at the User-Agent header as mentioned in the comments. Google, Yahoo, etc. should identify themselves as bots using this header, and disallowing bots via robots.txt would be like driving a truck through your search engine ranking. As @adria said, there are tools out there that can do this for you. A very popular one is Google Analytics, here's how they handle crawler traffic.

Improve this answer
0
0

Since your stated intention is to

get accurate statistics as to how many human visitors I get,

The approriate solution is to use a service like Google Analytics, or New Relic, once signed up you insert a snippet of javascript into your page (many engines such as wordpress can do this automatically or with a plugin) which sends information to the monitoring service. Once set up such a service will give a wealth of information about your visitors. Google Analytics is quite amazing in the detail with which it tracks user interactions with your site.

These services are implemented in such a way as to only track real humans and it would be folly to try and re-implement what they already do so well, and they are so useful it's almost folly to not use such a service.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.