What is the purpose of robots.txt? Is it to prevent bots from crawling pages? And if so, does that mean that I can still access the page through my browser?
Tell me more
×
Webmasters Stack Exchange is a question and answer site for
pro webmasters. It's 100% free, no registration required.
|
|
closed as not a real question by John Conde♦ Nov 10 '12 at 1:14
It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, see the FAQ.
|
Yes, you can still access the page through your browser. robots.txt is a suggestion to search engines about pages or directories you would like them not to crawl. Most major search engines such as Google honor these requests, but there is nothing that physically forces them to or guarantees that they or any other search engine will. |
|||
|
|
