A computer program that accesses web pages for various purposes (to scrape content, to provide search engines with information about your site, etc.).
8
votes
3answers
291 views
Tactics for dealing with misbehaving robots
I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site.
Obviously we've had ...
3
votes
1answer
482 views
How To Track Down and Stop Rogue Bots?
Most of the bandwidth of one site is being consumed by an unidentified bot. According to AWSTATS it says:
Unknown robot (identified by 'bot*') consumed 164 GB this month.
By comparison, Googlebot ...
5
votes
2answers
143 views
Which meta “robots” tag gets preference?
My wife works at a high school in Germany. I recently noticed that it's extremely hard to find that school's homepage using Google. I looked at the source code of the page and I believe I've found the ...
3
votes
2answers
259 views
How often are sitemap.xml checked for updates by crawlers?
How often do crawlers check for updates in submitted sitemap.xml? Every hour, day, week, month, more?
2
votes
2answers
4k views
Googlebot crawler: URL restricted by robots.txt - how to fix the entries in robots.txt?
I noticed recently that Google is not caching all of the pages on my website. Upon using the Google webmaster diagnostic tool, I realized that some of my pages were being restricted by entries in my ...
0
votes
0answers
145 views
Is there a way to make Alexa's ia_archiver slow down its crawling of my website?
Alexa's ia_archiver bot is the main contributor to the Internet Archive's "Wayback Machine" web collection, and there are advantages to having my website included in that collection. There are other ...
