Tagged Questions
2
votes
1answer
39 views
What does “all” mean in meta robots tag?
Several websites have the following meta tag aimed at search engines like Google, Bing, etc.:
<meta name="robots" content="all" />
What does it do?
1
vote
1answer
49 views
What dangers await if I block non-standard, non-major-usa search engine bots from my USA only website?
I noticed tons of bandwidth being used by non-USA search engine bots, so I began blocking them in an effort to save bandwidth and cpu cycles for actual users and the search engines they come from ...
3
votes
1answer
527 views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
7
votes
2answers
1k views
Ask.com Sitemap Crawler Down (for good)?
I have been using Ask.com as one of the search engines to ping my sitemap. When I went to ping my sitemap today to have my sitemap re-indexed, the website (http://submissions.ask.com/ping) says it no ...
2
votes
2answers
121 views
Does Yahoo Slurp crawl hammer a site?
A lot of robots.txt examples seem to feature Crawl-Delay for Yahoo Slurp for up to 2 hours, is this because Yahoo Slurp tends to hammer a website and increase server load?
6
votes
7answers
1k views
Is there an index of the IP addresses used by indexing bots?
I have a page that gets minimal traffic, but I set up static notifications when it gets hit. Now, I want bots to get ignored, so what I'm doing now is adding bots I see to a "no notify" list.
Is ...
