Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I've noticed that tineye.com does not attempt to fetch "robots.txt" from my website. It also sends a "user agent" string consisting of the user's browser with "TinEye/1.0 (via http://www.tineye.com/)" tagged on to the end.

Is this correct behaviour? If I write something which fetches one file from a web site, at the user's discretion, is it acceptable to not fetch "robots.txt"? Or should any automated program always obey robots.txt?

Please note this is not a question about tineye.com.

share|improve this question
add comment

1 Answer

Using robots.txt is only ever voluntary, there's no international law that says you should.

share|improve this answer
    
I pointed them to my web site and checked the log immediately afterwards, and there was no record of them looking at it. –  delete Apr 27 '11 at 12:51
    
PS this doesn't answer the question, it should have been a comment. –  delete Apr 27 '11 at 12:52
    
You asked if it was correct behaviour. The answer is there is no law that says if people should. I've taken out the bit about TinEye saying they do read it. –  paulmorriss Apr 27 '11 at 12:59
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.