7

I'm running Apache 2 and a large part of our page views come from bots. Most of those are legitimate ones such as Google and Bing.

I want to parse the logs and get accurate statistics as to how many human visitors I get, so I've temporarily updated robots.txt to ban bots on all pages. I know this is only partially effective at preventing bot access, but I'm OK with that.

How will disallowing bots affect users searching for this site? Will it prevent users from finding the page on Google?

  • 17
    If you want to "parse the logs so I get as few stats from the bots", then do exactly that and parse the logs. (?!) – MrWhite Oct 28 '14 at 8:52
  • 5
    Following on what w3d said, all of the major search engine bots will clearly identify themselves with the User-Agent header. – Brendan Long Oct 28 '14 at 14:44
  • 2
    If you tell Google to go away in robots.txt, they will take your word for it that you don't want to be indexed and your ranking will disappear. Getting your ranking back may take some time after pulling such a stunt. – Fiasco Labs Oct 28 '14 at 14:51
13

Banning bots will not let any search engine get the content of the site.

Ultimately you will not rank for any keywords. It would be next to impossible to find your page on Google. You might get referral traffic but no organic traffic.

Note: Robots.txt does not ban bots but ask them not to index and crawl the site. Which major search engine bot like Google, Yahoo & Bing follow.

  • That's what I figured. – Tensigh Oct 28 '14 at 9:10
  • 12
    Can you please update your answer to explain that robots.txt does not "ban bots". It politely asks bots not to crawl the site and for search engines not to index it. There's absolutely nothing stopping someone from crawling your site anyway. – ErlVolton Oct 28 '14 at 15:27
  • 1
    To add to what @ErlVolton said. The malicious bots you'd presumably want to ban won't pay any attention to your robots.txt "ban" on bots. So the only affect this ban will have will be to hurt your SEO. – Dennis Oct 29 '14 at 20:02
  • @ErlVolton +1 for "politely ask bots" – 吖奇说 Oct 30 '14 at 9:20
  • I think the technical term is "Disallow" which IMO is still not a very good representation of what's happening. Maybe "RequestNoCrawl" or something would be better :) – ErlVolton Oct 30 '14 at 15:33
21

Banning bots is a fruitless activity. The only bots that will obey robots.txt are helpful bots like Googlebot and Bingbot. Malicious bots or even less scrupulous search services' bots will ignore your robots.txt.

Banning bots is only a sure way to lose all page ranking with the major search providers AND your logs will still be full of bot traffic.

  • 6
    +1 and your logs will still be full of bot traffic. Only specific denial in .htaccess will keep them out and you still get a 403 for every link they know about from external reference, they just can't suck in a page and start crawling. The list gets mightily long... – Fiasco Labs Oct 28 '14 at 15:40
  • 1
    +1 this is the answer. There is no stopping bots, especially malicious bots. – Qix Oct 28 '14 at 21:18
  • 2
    @Qix I have seen a few websites that issued an IP ban against anyone that visited a particular URL (that was both forbidden in robots.txt and clearly labeled with "don't click this link or you will get banned" for humans) - that strikes me as a fairly effective way to target unscrupulous crawlers. Granted, it doesn't do much against a bot specifically designed to target your site. – Brilliand Oct 29 '14 at 18:02
  • 1
    @ChrisMarisic and clearly labeled with "don't click this link or you will get banned" for humans – Qix Oct 29 '14 at 19:03
  • 2
    @Brilliand Sound like a fun way to ban other users via CSRF. – CodesInChaos Oct 30 '14 at 8:40
3

Google may still crawl pages ignored by robots.txt and may even list them see Block URLs with robots.txt and Does Google ignore robots.txt

  • "Google may still crawl pages ignored by robots.txt" - By "ignored", do you means URLs that aren't blocked by robots.txt? – MrWhite Oct 28 '14 at 9:42
  • 1
    That sentence in the "Does Google ignore robots.txt" answer is also wrong. Google may index pages linked by robots.txt even when they are disallowed in robots.txt, but it won't crawl them. – Stephen Ostermiller Oct 28 '14 at 16:29
3

It is likely to make your site very difficult or impossible to find in search engines, as the search engines won't send their robots to see what's on your site. They won't know what words you use so it will be hard for them to tell what searches your site might be relevant to.

However it is possible your site will still be displayed in search results, particularly if a high-ranking site has a link to your site. Google and possibly other engines may use information from the link alone to decide to show your site in their results pages.

  • This is perfect information. This page is actually a subdomain of our main page which is where I prefer the search engine rank comes from. So knowing that our main page will get the page rankings is perfect. Thanks! – Tensigh Oct 28 '14 at 23:21
1

The data from server logs is limited, and will unavoidably have a high noise to signal ratio, thanks to factors such as bots, caching, CDN.
Analyzing page views is a task for page-tag based analytics.

0

The correct answer is to not mess with robots.txt and instead parse your logs, looking at the User-Agent header as mentioned in the comments. Google, Yahoo, etc. should identify themselves as bots using this header, and disallowing bots via robots.txt would be like driving a truck through your search engine ranking. As @adria said, there are tools out there that can do this for you. A very popular one is Google Analytics, here's how they handle crawler traffic.

0

Since your stated intention is to

get accurate statistics as to how many human visitors I get,

The approriate solution is to use a service like Google Analytics, or New Relic, once signed up you insert a snippet of javascript into your page (many engines such as wordpress can do this automatically or with a plugin) which sends information to the monitoring service. Once set up such a service will give a wealth of information about your visitors. Google Analytics is quite amazing in the detail with which it tracks user interactions with your site.

These services are implemented in such a way as to only track real humans and it would be folly to try and re-implement what they already do so well, and they are so useful it's almost folly to not use such a service.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.