7
votes
1answer
71 views

Combine user-agents in robots.txt

Can user-agents be listed together, followed by their common rules like so in a robots.txt? User-agent: Googlebot User-agent: ia_archiver #Alexa User-agent: BingPreview #Microsoft User-agent: bingbot ...
1
vote
3answers
138 views

Can web crawlers that visit my site fake their user agent?

I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest. I want to know if I add User-agent: Googlebot, will fake ...
2
votes
2answers
128 views

Plus sign in front of URLs in user agents

I run a small web crawler and had to decide on what user agent to use for it. Lists of crawler agents as well as Wikipedia suggest the following format: examplebot/1.2 ...
7
votes
2answers
2k views

Should I block bots from my site and why?

My logs are full of bot visitors, often from Eastern Europe and China. The bots are identified as Ahrefs, Seznam, LSSRocketCrawler, Yandex, Sogou and so on. Should I block these bots from my site and ...
3
votes
1answer
1k views

Preventing high bandwidth usage from Yandex

Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
0
votes
1answer
1k views

BingPreview user agent

I'm getting several invalid requests from a "BingPreview" user agent. The complete user agent header is: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/534+ (KHTML, like Gecko) BingPreview/1.0b The ...
1
vote
1answer
93 views

In User Agent strings, what does “AskTbTRL” denote?

Here are a bunch of examples: http://www.botsvsbrowsers.com/listings.asp?search=asktbtrl I would've guessed an Ask.com-related bot, but apparently not according to their FAQ.
8
votes
3answers
560 views

Tactics for dealing with misbehaving robots

I have a site that, for regulatory reasons, may not be indexed or searched automatically. This means that we need to keep all robots away and prevent them from spidering the site. Obviously we've had ...
1
vote
1answer
101 views

What is user agent Issbot

I am seeing a lot of traffic to my website from user agent Issbot. This certainly looks like a bot, but is not listed in the www.user-agents.org listing, and I have not been able to find anything ...
5
votes
3answers
1k views

What user agent should I set?

There is Ask bot, which sets this header: Mozilla/2.0 (compatible; Ask Jeeves/Teoma) Considering this, I have the following questions: If I'm writing a web crawler named Goofy, what user agent ...
22
votes
4answers
825 views

Is Yahoo Search the same as Bing Search now?

We had some serious issues with Yahoo's (incredibly) badly written spider in the past, and as a result we blocked them. Marco Arment of Tumblr also shared his frustrations with us on 8/31/2009, which ...