I'm working on a site with a large music content database, and recently some app developer looking for a bit of a reputation boost has launched an app that completely piggybacks on our database: a user will search for a track in the app, the app will send the request to our search page and scrape the results, returning the top result to the user. Here are some things I've tried:

  • Returning 444 for the app's user agent (but they changed it to a legit browser string).
  • Detecting a referrer on the search page, and returning 404 if no referrer is found (but it would be easy enough for them to spoof a referrer).

One idea I thought of is some sort of token sent along with the search query any page with the search box, and the search result validates this token and, if invalid, returns 404. Are there any tried-and-true techniques that do this kind of thing? Or anything else I can do to prevent this data thief? He's completely unapologetically stealing our data and crippling our site!

By the way I'm using Django, in case there's anything built in to there that could help me.

  • 1
    Your token idea would work if you generated a unique number which you stored in the database temporarily. You are obviously accessing the database for the search results so the overhead would be next to nothing, only really the call to generate it. All databases I know have built-in methods to generate random GUIDs – Geezer68 Jan 23 '14 at 12:30

Your token idea would work. You could change it periodically to make it harder.

You could also change the parameters on your search form. Start using s= instead of q= for the search term. That would force this developer to keep up with those changes.

You could also implement Captcha and not show search results unless the user proves they are a human. Google just launched a version of recaptcha that is just a checkbox: http://googleonlinesecurity.blogspot.hu/2014/12/are-you-robot-introducing-no-captcha.html

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.