1

I'm supporting a larger web database that contains many hundreds of articles with consumer grade and technical tests of hardware and electronics. Our policy has always been to provide open access to these resources without the constraints of a paywall, enabling users to freely engage with our content. The articles are free to read, no paywall, recently however we debated the possibility of protecting the articles from misuse as training data in AI models without our consent.

Question is - is there any viable way to achieve this? If possible, with minimal effect on SEO. We wouldn't want to go with a paywall or members-only content section as this would affect the purpose of the database. Any viable suggestions or approaches are greatly appreciated.

There are certain ways like robots.txt to achieve some protection from search indexing, however this has direct effect on SEO. Not sure how to handle this.

Improve this question

2 Answers 2

Reset to default
1

short answer: You can't prevent it.

any freely accessible information on the web is (simply) available and can be included to any database, LLM or ML training data source etc. If someone wants to include it, they will. There are some standards in the making like https://mlnoindex.org that address this but they all rely on cooperation and goodwill of AI providers to respect them.

Improve this answer
1

The MLNoindex attribute, like robots.txt, is merely a polite rquest that people do not use the data from your site for their own benefit via automated crawling. It requires the crawler to opt-in to your request.

Other things people have tried (which don't work) are:

  1. blacklisting addresses - the list is not well defined, large and has VERY hgh churn rates.

  2. Using CAPCHAs to gate access - e.g. Cloudflare thought they were smarter than AI, it turns out they weren't.

  3. Rendering content via Javascript - this was successful in blocking bots for about a week, but it is really difficult (impossible?) to do this without impacting the user experience for human beings

  4. Blocking via user-agents - although this does not have the issues of blocking by IP address, it still depends on the crawler being honest about their intentions

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.