2

We have a self generating honeypot that is designed to fill crawlers with junk data (among other things). Its protected with the right "treaties" and headers so 99% of good bots stay away. Today though it seems as if SEMrush has found its way in and has seen thousands of pages of trash data.

In theory, this would be awesome since it would warp their stats to any competitors trying to sniff our site, but we actually use SEMrush. How do I keep useful badbots like SEMrush from crawling into this honeypot? Seems robots.txt and nofollow have no effect. Here is how its set up:

  • Honeypot file is named as wp-admin (Wordpress) so stuff should not be hitting it (we don't use WP)
  • Robots.txt says that all traffic should not visit url example.com/wp-admin
  • On every page a hidden off-UI display:none link with noindex/nofollow points to example.com/wp-admin
  • As honeypot is loading, a 403 forbidden HTTP header is set for client
  • In honeypot, the includes a meta header for nofollow/noindex
  • After honeypot is loaded there is a CSS overlay to block certain things, or explain what this is to any humans that find themselves there.

So how do I make sure that SEMrush or other tools like it do not get hung in the honey?

Improve this question
7
  • This happens from time to time. Even Google can occasionally accidentally ignore robots.txt and hit my trap. I do not know what honeypot you are using, but if you can white-list by domain name or IP address, that would be helpful.
    – closetnoc
    Oct 10, 2014 at 15:16
  • @closetnoc It's one we made, just some simple stuff (its more of a tarpit). Good thought though, whitelist to abort the script seems most safe. Would prob do it with user agent. Partially figured it out in SEMrush: Under Tools > Site Audit > Edit Audit there is a spot to include/exclude domain paths. Although, this doesn't prevent other people/competition from triggering the same flood when they set up an audit, so its still an issue. Wish there was just a magical header or something instead hah.
    – dhaupin
    Oct 10, 2014 at 16:10
  • 1
    User agent is faked so often that it is completely unreliable. Use the domain name and IP address for the obvious like Google and Bing. I can help with some of that.
    – closetnoc
    Oct 10, 2014 at 16:17
  • What exactly do you mean by "fill crawlers with junk data"? What are you trying to prevent happening?
    – DisgruntledGoat
    Oct 13, 2014 at 14:07
  • 1
    Could you configure honeypot to look at the user-agent string and not push out the junk data for certain user agents?
    – Stephen Ostermiller ♦
    Oct 24, 2014 at 19:05

1 Answer 1

Reset to default
1

If you're using Apache web-server you could use an .htaccess configuration to white-list by User Agent and prevent genuine bots from reaching your 'tarpit':

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} SEMrush [NC]
RewriteRule .* - [F,L]
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.