Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've read that a huge percentage of traffic comes from bots - legit bots like search engine spiders and illegitimate bots like spammers and ones looking for vulnerabilities.

I'm about to launch my website. The problem is that I am about to sign a copyright license for some of the content on the website, and the contract says that I have to pay a certain sum of money depending on how many visits the site gets (per month). The more visits, the more it will cost.

So I have an incentive in limiting traffic to only legitimate hits by blocking the crap. I don't want the website to get popular, so maybe I don't want the search engines to do too much indexing either.

What should I be looking into?

share|improve this question
You can try to block bots (search list on Google) with a robots.txt file. –  Zistoloen Sep 1 '13 at 8:51
How are you/they counting the visits? –  unor Sep 1 '13 at 10:25
@Zistoloen - that would work for the legitimate bots that actually look at the robots.txt file, sure. But I'm wondering what % of bot traffic that will be. –  user1887049 Sep 1 '13 at 10:49
@unor - The website isn't uploaded yet, and I haven't added that functionality. I just assumed I would use some JavaScript log counter or HTTP log file analyzer, and then report the numbers to them. –  user1887049 Sep 1 '13 at 10:50
Most spambots don't execute javascript so they won't show up in Google Analytics. –  ZippyV Sep 1 '13 at 15:06

3 Answers 3

up vote 2 down vote accepted

"Bad" traffic will ignore your robots and constantly spider you. It will always be changing ip addresses and agent combinations and that is a fact of life. You may be able to look into "honeypot" type solutions to stop bad repeat traffic that ignores your robots but it can become impractical to manage blocking it yourself unless you love the game "wack-a-mole".

In addition to authentication already mentioned, consider starting with one of the services that sit in front of your site giving you basic CDN and filtering out "bad" traffic based on community metrics. The two most popular ones that I am aware of are CloudFlare and Incapsula.

share|improve this answer

It sounds like the copyrighted content (the content that costs you money when someone visits that page) should be behind some kind of (free?) authentication/login? That would certainly help limit the number of visits to real visitors.

You could perhaps have a non-copyrighted snippet or summary that can be indexed by all that linked to the full article in order to capture the organic search traffic?

share|improve this answer
I think this solution, or a similar one in which you negotiate a separate rate for visits by bots, is the better way to handle this problem. Maybe both. –  Nick Sep 3 '13 at 14:11

You can trace the location of the spammer mostly they are from a group of IPs. Block those IPs.

For Google bots you cannot do anything. By using robots.txt it will disallow your whole domain or page for audience coming from search engines.

While submitting sitemap set the page change frequency to "NEVER". Surly it will reduce the bot traffic.

share|improve this answer
A suggestion: Linking to information on the IP range would be a way to make this answer much more useful. I'd suspect that spammers are not confined to a defined block of IPs, and that it changes regularly. Is there a service you use to keep up to date on the set of IPs in use by spambots (a la Akismet?). –  Nick Sep 3 '13 at 14:14

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.