I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex.
i dont really trust robot.txt. Because they can simply ignore it and continue crawling.
I would use rails as my web framework.
How can i do it, atleast to decrease the posibility destructive action they do by simply crawling a specific page.
Is there any special mark or identifier on them while they are crawling a site ?
This question marked as duplicate with Robots denied by domain is still listed in search results
Wich is i guess a mistake. Above questions is asking specifically why. I dont really care why google crawling my site. I just want to block it. Show 404 status.