103 reputation
4
bio website
location
age
visits member for 1 year, 3 months
seen May 23 '13 at 22:30

Jul
10
awarded  Announcer
Jan
8
comment Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
@Kris: I actually asked my question to get information for another webmaster: he currently blocks ia_archiver from his massive and useful site. My website is composed of just one solitary webpage (/index.html), so I can't really test Crawl-delay. :) But anyway, the Crawl-delay directive has never been part of the official Robot Exclusion Standard. Even though ia_archiver follows the entire standard, it still might ignore Crawl-delay.
Jan
7
awarded  Excavator
Jan
7
awarded  Editor
Jan
7
revised My old webhost has gone out of business. How can I recover the text of my website?
Revisions 2 and 3 were wrong. The asker isn't missing robots.txt -- the asker is missing his/her website's content. The Wayback Machine is reporting that it can't find robots.txt not because robots.txt is down, but because the asker's entire website is down.
Jan
7
comment Control over the Internet Archive besides just “Disallow /”?
Related: "Is there a way to make Alexa's ia_archiver slow down its crawling of my website?"
Jan
7
comment Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Related: "Control over the Internet Archive besides just 'Disallow /'?"
Jan
7
asked Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Jan
7
awarded  Supporter
Jan
7
suggested suggested edit on My old webhost has gone out of business. How can I recover the text of my website?