What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
closed as not a real question by John Conde♦ Nov 10 '12 at 1:14
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.
Yes, you can still access the page through your browser. robots.txt is a suggestion to search engines about pages or directories you would like them not to crawl. Most major search engines such as Google honor these requests, but there is nothing that physically forces them to or guarantees that they or any other search engine will.