Should a single page fetcher like TinEye obey robots.txt?

Question

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" tagged on to the end.

Is this correct behaviour? If I write something which fetches one file from a web site, at the user's discretion, is it acceptable to not fetch "robots.txt"? Or should any automated program always obey robots.txt?

Please note this is not a question about tineye.com.

paulmorriss · Answer 1 · 2011-04-27 13:00:12Z

up vote 3 down vote

Using robots.txt is only ever voluntary, there's no international law that says you should.

edited Apr 27 '11 at 13:00

answered Apr 27 '11 at 8:30

paulmorriss
12.9k11339

	I pointed them to my web site and checked the log immediately afterwards, and there was no record of them looking at it. – delete Apr 27 '11 at 12:51
	PS this doesn't answer the question, it should have been a comment. – delete Apr 27 '11 at 12:52
	You asked if it was correct behaviour. The answer is there is no law that says if people should. I've taken out the bit about TinEye saying they do read it. – paulmorriss Apr 27 '11 at 12:59

asked	2 years ago
viewed	137 times
active	2 years ago

Should a single page fetcher like TinEye obey robots.txt?

1 Answer