Robots.txt is text file used by Website owners to give instructions about their site to web robots. Basically it tells robots which parts of the site are open and which parts are closed. This is called The Robots Exclusion Protocol.
Stats
created |
2 years ago |
viewed |
12 times |
editors |
0 |
Recent Hot Answers
Files I should have on every websiteDoes Google cache robots.txt?
How to stop certain urls from being indexed
600 visitors per day, 20 backlinks but still not referenced by Google
Robots.txt: do I need to disallow a page which is not linked anywhere?
more »
Related Tags
seo × 52google × 29
googlebot × 23
google-webmaster-tools × 22
search-engines × 18
robots × 13
noindex × 9
web-crawlers × 8
sitemap × 8
google-search × 8
htaccess × 7
indexing × 7
html × 7
crawling × 7
search × 6
php × 5
redirects × 5
google-index × 5
wordpress × 5
duplicate-content × 5
facebook × 4
user-agent × 4
404 × 4
nofollow × 3
mobile × 3
