Timeline for Is it possible to tell web crawlers to agree to a site's TOS before indexing?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Nov 24, 2020 at 17:13 | comment | added | Stephen Ostermiller♦ | I'm trying to ask what you're really trying to achieve. Even if there is no way to get crawlers to agree to a terms of service there might be a way to make crawlers behave in the way you want through some other means. | |
| Nov 24, 2020 at 17:04 | comment | added | Paul | @StephenOstermiller You are asking a purely subjective question framed to make me look overly-self-important. Stop doing that, it's disrespectful of your users, even though it does earn you brownie points with the site's cheerleaders. | |
| Nov 23, 2020 at 19:58 | comment | added | Stephen Ostermiller♦ | What do you have in your terms of service that would require that type of individual attention from search engines? | |
| Nov 23, 2020 at 19:55 | comment | added | Paul | @StephenOstermiller The managers could program a rule that alerts the managers to a site requesting the managers acknowledge reviewing the site's TOS before indexing. | |
| Nov 23, 2020 at 19:52 | comment | added | Stephen Ostermiller♦ | Search engine bots don't have humans directing their every move. They automatically crawl based on pre-programmed rules. | |
| Nov 23, 2020 at 19:50 | history | edited | Paul | CC BY-SA 4.0 |
added 16 characters in body
|
| Nov 23, 2020 at 19:49 | comment | added | Paul | @StephenOstermiller The bots are controlled by humans. | |
| Nov 23, 2020 at 18:15 | answer | added | davidgo | timeline score: 1 | |
| Nov 23, 2020 at 17:27 | comment | added | Stephen Ostermiller♦ | Bots can't read. How do you expect them to understand your terms of service? | |
| Nov 23, 2020 at 14:46 | comment | added | Paul | Please note this is not intended to be a question about trying to keep malicious bots from being malicious. | |
| Nov 23, 2020 at 14:46 | history | asked | Paul | CC BY-SA 4.0 |