Is there a way with robots.txt or page headers to inform the managers of web crawlers they must agree to a terms of service prior to indexing any crawled pages?
-
Please note this is not intended to be a question about trying to keep malicious bots from being malicious.– PaulCommented Nov 23, 2020 at 14:46
-
Bots can't read. How do you expect them to understand your terms of service?– Stephen Ostermiller ♦Commented Nov 23, 2020 at 17:27
-
@StephenOstermiller The bots are controlled by humans.– PaulCommented Nov 23, 2020 at 19:49
-
2What do you have in your terms of service that would require that type of individual attention from search engines?– Stephen Ostermiller ♦Commented Nov 23, 2020 at 19:58
-
2I'm trying to ask what you're really trying to achieve. Even if there is no way to get crawlers to agree to a terms of service there might be a way to make crawlers behave in the way you want through some other means.– Stephen Ostermiller ♦Commented Nov 24, 2020 at 17:13
|
Show 3 more comments
1 Answer
The robots.txt spec does not allow for this - and even if it did, there are legal issues - a robot can not provide informed consent,/it does not have a mind so there can be no meeting of the minds. (If there is no meeting of the minds there is no contract)
Its likely to be an exercise in futility, but you could set up pages with arbitrary headers requiring a particular action described in the headers to accept terms and conditions - of-course robots would not consent because they can't.
Even if you could do do the trick you are attempting, contracts of adhesion (ie take it or leave it, with no ability to negotiate) are more difficult to enforce then negotiated contracts.
