Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?

share|improve this question

closed as not a real question by John Conde Nov 10 '12 at 1:14

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

add comment

1 Answer 1

Yes, you can still access the page through your browser. robots.txt is a suggestion to search engines about pages or directories you would like them not to crawl. Most major search engines such as Google honor these requests, but there is nothing that physically forces them to or guarantees that they or any other search engine will.

share|improve this answer
    
+1 good answer. –  alfasin Nov 9 '12 at 23:29
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.