Most website content is only accessible after logging in. 3rd party sites (and apps) provide alternative interfaces to the site by passing on the login credentials and scraping content. How do I make sure users only log in from my website and app and not from (or only specific 3rd party) sites? The goal is to protect the login, not generally prevent scraping.

Blocking access by IP is not working, the 3rd party sites use changing IPs for access. Also I would not like to use a captcha as it is annoying to users (and might be passed on as well?)

  • Captchas can be annoying although its worth revisiting as Googles modern recaptcha can very often provide verification without any input, or with only a tick box. Also, probably doesn't make users life easier but 2FA could help as well (at least to frustrate syndication)
    – davidgo
    Nov 14 '20 at 22:41
  • Aside: "the 3rd party sites use changing IPs for access" - presumably you are referring to users logging in to your site? A legitimate 3rd party "site" should have a reasonably static IP address. (?)
    – MrWhite
    Nov 15 '20 at 0:23
  • Thank your for the proposals. Legitimate 3rd party sites either use a static IP or a secret key. I'm reading up on 2FA on the web, but so far it doesn't seem very user friendly and doesn't seem to prevent passing on authentication
    – L L
    Nov 19 '20 at 8:10
  • I wonder if requiring javascript to work for the site to load (maybe using AJAX) would frustrate most of these scrapers. No doubt it would be possible, but you could probably also use the javascript to collect additional data to help you work out if its being emulated - using some of the markers shown in panopticlick for example could help you effectively fingerprint the javascript emulator//scraper to facilitate blocking.
    – davidgo
    Nov 19 '20 at 8:15

It is unclear how your login procedure is currently working, but its broken and needs fixing - particularly if - as hinted at - it passes the login and password back and forth between the client and server on each page.

The standard way of handling logins is to perform a login and then set a cookie which the user passes back and forth - in this way once the login is performed the server knows who the user is without sending the credentials multiple times. Most programming languages used for web programming have a concept of a session - which does this for you. A session is particularly useful as it can remember settings/variables between page views, so providing a lightweight and convenient way of handling user preferences. Logging out becomes as simple as destroying the session. Because its done by way of cookies it can easily be set to prevent third parties involvement in the login process In order to support third party logins you will need to implement something like SSO (Single Sign on) or equivalent with those parties.

Not a recommendation as its not a best practice - but If you dont want to change the current mechanisms as above another mechanism you can use is to ensure that before displaying pages other then the login page is to check the REFERER header - which will allow you to make sure the login was completed on your site. This would, however, prevent users from logging in, then opening a new tab/window and browsing your site from that Window - they would need to log in again.

Her another bad way to do this - which will frustrate general scrapers but is not fullproof - is to create a table of IP addresses associated with logged in users and deny access to requests coming from other IP addresses until logged in. Things like carrier-grade NAT means its not a robust solution - but can validly be selectively used to provide login-free access to an organisation per static IP in addition to the cookie/session mechanism above.

  • Your suggestions are good for preventing 3rd party sites from logging in. Those operate in a browser which maintains some cross site controls. An app that logs in is a different story entirely. Apps can control their referrers, cookies, and anything else you can do on your own site. Nov 14 '20 at 19:41
  • Thank you for the answers. Session cookies are used, but 3rd party sites act as man-in-the-middle, they pass on the login and password, store the session cookie and use it thereafter as any browser would do. As such they can fake the referer header as well. It's a public site therefore restricting users to certain IPs is not feasible.
    – L L
    Nov 19 '20 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.