I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" tagged on to the end.

Is this correct behaviour? If I write something which fetches one file from a web site, at the user's discretion, is it acceptable to not fetch "robots.txt"? Or should any automated program always obey robots.txt?

Please note this is not a question about tineye.com.


Using robots.txt is only ever voluntary, there's no international law that says you should.

| improve this answer | |
  • I pointed them to my web site and checked the log immediately afterwards, and there was no record of them looking at it. – delete Apr 27 '11 at 12:51
  • PS this doesn't answer the question, it should have been a comment. – delete Apr 27 '11 at 12:52
  • You asked if it was correct behaviour. The answer is there is no law that says if people should. I've taken out the bit about TinEye saying they do read it. – paulmorriss Apr 27 '11 at 12:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy