Timeline for Should I use both Robots.txt and META NAME="ROBOTS"?
Current License: CC BY-SA 3.0
3 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 26, 2013 at 15:49 | comment | added | sam | i stand corrected | |
| Apr 26, 2013 at 13:25 | comment | added | MrWhite | The HTML META index tags have not been deprecated in favor of using a robots.txt - they serve a related but different purpose. A robots.txt file prevents pages from being crawled, but can still be indexed if pages link to the page in question. A robots META tag (or corresponding X-Robots-Tag HTTP response header) on the other hand prevents the page from being indexed. | |
| Apr 26, 2013 at 11:06 | history | answered | sam | CC BY-SA 3.0 |