I want to block search engines from crawling my websites. Atleast for Msn, Yahoo, Google, and Yandex.

i dont really trust robot.txt. Because they can simply ignore it and continue crawling.

I would use rails as my web framework.

How can i do it, atleast to decrease the posibility destructive action they do by simply crawling a specific page.

Is there any special mark or identifier on them while they are crawling a site ?

This question marked as duplicate with Robots denied by domain is still listed in search results

Wich is i guess a mistake. Above questions is asking specifically why. I dont really care why google crawling my site. I just want to block it. Show 404 status.

| improve this question | | | | |

An elegant way to specify search engine crawling instructions for non-html files without having to use robots.txt. This comes first in listing. Some other crawling control methods listed below:

Use noindexpage meta tags:

<meta name="robots" content="noindex" /> 

Use index in page meta tags with searchengine name to crawl:

<meta name="googlebot" content="index" />

Nofollow: Tell search engines not to spider some or all links on a page

<meta name="robots" content="nofollow" />

To specify nofollow at the link level, add the attribute rel with the value nofollow to the link:

<a href="mypage.html" rel="nofollow" />

Use X-Robots-Tag in your http headers which is accepted only by yahoo.

| improve this answer | | | | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.