Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" tagged on to the end.

Is this correct behaviour? If I write something which fetches one file from a web site, at the user's discretion, is it acceptable to not fetch "robots.txt"? Or should any automated program always obey robots.txt?

Please note this is not a question about tineye.com.

share|improve this question

Using robots.txt is only ever voluntary, there's no international law that says you should.

share|improve this answer
I pointed them to my web site and checked the log immediately afterwards, and there was no record of them looking at it. – delete Apr 27 '11 at 12:51
PS this doesn't answer the question, it should have been a comment. – delete Apr 27 '11 at 12:52
You asked if it was correct behaviour. The answer is there is no law that says if people should. I've taken out the bit about TinEye saying they do read it. – paulmorriss Apr 27 '11 at 12:59

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.