I run a website that provides meta-data from specialized electronics (data like weight, dimensions, applications, voltage, etc). It also compare products using all this data so user can see the differences between two products.

I have a lot of work to research that data and place it in a structured database for my website.

A competitor made a "crawler robot" that scans my website automatically, grabs all the structured data and replicate it on their own website. He reads the HTML code, parse the tables, get the values, he gets even the "tooltip descriptions" I wrote for every property, and with all this data, he feeds his own database so he can display the data as he wants.

I place a new product on my website, and the product is on the competitors website few hours later. It's not a "simple HTML copy". He parses all the data and saves it in a structured way. He even downloads all the product images and host them in their own server. It's a quite advanced crawler robot.

I want a way to prevent my competitor to do that.

I was planning to just block his entire country on my WebServer, but I guess it won't stop him for a very long time. He probably will find a way to go around that (I would).

Also, I can't just "image-encode" my data because it's very SEO unfriendly. And I need Google to find me.

What do you suggest?

  • "so he can display the data as he wants." - So the "cloned" site does look physically different in the HTML structure; more than just changes in CSS?
    – MrWhite
    Jan 23 '19 at 1:22
  • 3
    @DanielRibeiro You've got some fair answers below. I'd suggest studying your server logs and try to identify some telltale characteristics/footprints of their bot, perhaps by setting up a honeypot with data that only they'd be likely to request. Based on the server log for that request, you'll likely spot a pattern of IP's, user-agent, duration (bots are often much quicker than human visitors), successive/rapid requests, etc... Once you identify a consistent pattern, add and configure a web server security module.
    – dan
    Jan 23 '19 at 6:08
  • 1
    Yes @MrWhite, the site "looks" physically different... The problem is the copy of my contents, the structured data... I have no problem with my layout or my design... The problem is the business data. Jan 23 '19 at 12:21
  • "He reads the HTML code" - are you using any JavaScript to generate your content? Just wondering how sophisticated the crawler is... is he literally just scraping the HTML response, or is he using a headless browser to scrape the rendered output?
    – MrWhite
    Jan 23 '19 at 14:08

As a general rule, anything visible to the public can be crawled by sophisticated bots. Locations can be faked, IP addresses can be masked, captcha-solves can be purchased.

Your best recourse in this case may be through legal. Start by sending a DMCA takedown request to Google to get his site de-listed from Google Search, which should slow him down. If the competitor is in a jurisdiction you can legally touch, you could also consider talking with a lawyer about creating a fictitious entry copyright trap.

This similar thread has advice for if a competitor steals your content and ranks above you in Google Search, which could be helpful:

Another website is mirroring and ranks above my site in search results

I am not a lawyer and this is not legal advice.

  • 1
    There is a possible sticking point with the copyright trap idea if the Wikipedia article is correct. "Fictitious entries may be used to demonstrate copying, but to prove legal infringement, the material must also be shown to be eligible for copyright. However, due to the Feist v. Rural, Fred Worth lawsuit where the Supreme Court ruled that "information alone without a minimum of original creativity cannot be protected by copyright, there are very few cases where copyright has been proven." Jan 22 '19 at 23:46
  • Good point, @ChrisRogers. It seems in that case to depend on whether the data in question has an amount of original creativity. Jan 23 '19 at 0:41
  • @ChrisRogers The lawsuit you referenced was related to phone books and has nothing to do with Copyright Laws in relation to the Internet.
    – elbrant
    Jan 23 '19 at 6:34
  • Can you explain how a fictitious entry copyright trap could be related to the OP's data scraping issue on his website?
    – elbrant
    Jan 23 '19 at 6:38
  • 2
    @elbrant OP creates a fictitious entry, competitor copies the entry, then OP has hard proof that the other site is simply copying his, and they could not construe that they independently gathered the data from other sources. Jan 23 '19 at 7:18

Jerks like this came online almost the moment the Internet was born. A couple options worthy of thought:

  1. You could write code in CGI/Perl, or PHP that calls your data (and pages) "on the fly". Putting the data into product files, or dataBases, instead of in standard HTML, may inhibit their bot from being able to steal the data. Granted, they could have a staff member come to your site and get the data, but that won't be as appealing as a bot program stealing the data. Making that happen means that the pages are produced at that moment, so there are no "static" pages to steal from. Not perfect, but it has potential.
  2. You could use .htaccess to try: Blocking Offline Browsers and BadBots , or to Deny visitors by IP address. That might hold them off for a little while.
  3. You could copyright your website. You have to understand that the (dimension, weight, size, etc) specs for the product can not be copyrighted. Anyone featuring the product on their site is entitled to use that information. However, you indicated that you author "tooltips" and that those descriptions are being harvested by the bot as well. That is something you can site as copyright infringement. Look into it. (Note: Copyright rules vary by country.)
  • 1
    CGI/Perl/PHP runs server-side and then sends standard HTML to the client anyways, so that won't work... As far as using AJAX, it's very easy with modern tooling to write a bot that parses and executes JavaScript. Jan 23 '19 at 0:39
  • As @MaximillianLaumeister points out, using PHP and the like will not stop someone scraping data from the resulting page using JavaScript etc. +1 for the suggestion on using .htaccess though. Jan 23 '19 at 8:45

Technically you are not able to prevent this action.

Though, you can do things in order to kinda "defuse" those duplicate or low-quality pages that may hurt your rankings.

  • You can put a canonical link inside your body tag and inside the section of the main content.
  • You can put a branded link inside your main content so that cloned content will have a link to your own website.
  • 2
    This will not work as the competitor's robot is smart enough to replace all links from my data. Also, he's not copying the "page", he is copying the "data"... If I have a product telling "Working voltage: 24V", he will parse this information and save only the "24" in the "Working Voltage" property of that product. Jan 23 '19 at 12:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.