I have a GitHub site and I hate web-crawlers that disobey or ignore robots.txt. How would I set up a Spider-Trap on a GitHub site that the robots.txt disallows the trap, so the good bots aren't trapped but the bad ones are?

PS I have no money for a server or a web host.(I'm 13)

  • You will never successful stop bad bots using robots.txt or .htaccess because they will appear as a real user. If you want to block bad bots then you need your own dedicated hosting that allows things like Fail2Ban which throttles or blocks users when they request too fast. – Simon Hayter Jan 28 '17 at 20:55
  • And to block crawlers that scrap content see github.com/JonasCz/How-To-Prevent-Scraping but again... you will need proper hosting and not a limited one provided by GitHub because it requires server-side processing. – Simon Hayter Jan 28 '17 at 20:57
  • What is the specific problem you're encountering from crawlers visiting the site? If it's a GitHub site there is no bandwidth usage to be concerned about. – DisgruntledGoat Feb 13 '17 at 1:35

This is incredibly complex and not something that you will be able to do with a zero budget as it requires access to server side data (which GitHub Pages does not support) and is something that will need ongoing tweaking.

Unfortunately as a rule once you put content on the web it is accessible to all. A robots.txt file tells legitimate known crawlers (like Googlebot, Bingbot, etc) where they are blocked from accessing. It is loosely based on the honour system in that these companies agree to have their bots comply with a robots.txt directive and you trust them to do so, there is no easy way to enforce it.

Some techniques done in the past with limited or no success have been...

IP Address Blocking
Even most legitimate crawlers will come from an extensive IP netblock and the netblocks change frequently so trying to block based on IP can be a never ending battle. On top of that illegitimate scrappers (private people or companies that scrap your site content for their own nefarious purposes) come from standard IP addresses which frequently change and are not associated with any known bots, as such the only bots you could effectively block even for the short term are bots which would respect your robots.txt file anyway.

Rate Limiting
You can implement rate limiting on your site so that a single IP address can't access your site any faster than a predetermined number of times in a second. This is reasonably good as it won't block human users unless they are speeding their way through your site, but on the flip side this can only be done with server side software and configurations and so won't work in the GitHub Pages environment.

Monitor Logs For Unusual Activity
Regularly check your server logs for unusual activity such as a large number of pages being accessed in a relatively short amount of time from the same IP. Once again this depends on access to server logs, as well it is not a hard and fast dependable method as IP addresses can change frequently, and many companies have a large number of users funnelled through a very small number of IP addresses so a group of a few hundred users could appear to the rest of the world to have the same IP address as everyone else.

Monitor Time For User Activity
As IP address monitoring is not the best you need to depend on other indicators. Some that you can use are how long it takes to fill out a form from when it is loaded to when it has been filled in and submitted. Check if a button has been pressed to submit the form or if the submit has been done programatically.

Use a CAPTCHA To Protect Forms
If your concern is around forms being submitted by bots then you can add a CAPTCHA to protect those forms. Not a guarantee of a human but a reasonably good test to block most of the poorly written scrappers out there.

  • what should I use for the web crawlers that ignore robots.txt? – user74362 Feb 14 '17 at 15:10
  • There is nothing that you can use from Github Pages or for that matter any free or shared web hosting service to deal with web crawlers that ignore robots.txt files. You need a level of control over the server and th ability to install software to the server such as Fail2Ban. Without that sort of control there is not much that you can honestly do. – Chris Rutherfurd Feb 14 '17 at 22:56

Kudos to you for being concerned about this at 13!

I'm going to ignore your lack of resources and pretend you could spend $5/mo for a Digital Ocean droplet (effectively a VPS of your own, where you could control all this stuff)...

Let's think about this in another light: What if you hated traffic from social media?

You'd make a list of social media sites that users from (referral traffic) were banned from visiting your site, right?

OK, so what sites are on that list?

Ignore the first 20 sites that came to mind, those sites are already on the blocked list. What other sites are on that list? Dogster? Ning? LifeAt?

Now do you get an idea of how hard it would be to block social traffic?

Here's a list of user agents dedicated to blocking of simple bots that I regularly use via a WordPress plugin.

In the context of a webform, do you know what a 'honeypot' is?

  • No I don't know what a honeypot is. – user74362 Feb 14 '17 at 15:09
  • A honeypot is an anti-spam measure for webforms. Basically, it's an input field that you add to a form and then you hide it via JS or CSS, so that people don't see it. Because that field is part of the form's HTML, bots will fill it out. If a form is submitted with a value in that field, it's assumed to be from a bot and it discarded. On a server you control, you could create a page that redirected to a trap and then ban all bots from it in robots.txt, effectively creating another type of honeypot. – adam-asdf Feb 14 '17 at 18:43
  • Wouldn't the bot just ignore the robots.txt and the fact that it's banned, then continue crawling? – user74362 Feb 15 '17 at 18:31
  • Maybe, but there's really no way to control that on a site that is publicly available. Really sophisticated bots are probably indistinguishable from actual users. By hiding a link (from humans) to a directory on your site that is prohibited in robots.txt you can be pretty sure any visitor to that directory is a bot. You can then (try to) ban them or you can then (try to) redirect them. Most bots don't have countermeasures for that kind of thing...here's more info: goo.gl/t4tEK3 – adam-asdf Feb 16 '17 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy