All Questions
4 questions
4
votes
1
answer
1k
views
Are user agent names case-sensitive in robots.txt?
I'm blocking various bots in robots.txt and I was wondering if their names are case-sensitive.
For example:
User-agent: grapeshot
Disallow: /
Would it matter if I use grapeshot or GrapeShot?
- 85
11
votes
1
answer
2k
views
Combine user-agents in robots.txt
Can user-agents be listed together, followed by their common rules like so in a robots.txt?
User-agent: Googlebot
User-agent: ia_archiver #Alexa
User-agent: BingPreview #Microsoft
User-agent: bingbot ...
- 395
3
votes
3
answers
3k
views
Can web crawlers that visit my site fake their user agent?
I want to write a robots.txt file for my website and allow the famous bots (Google, Bing, and Yahoo) to crawl my website, but deny the rest.
I want to know if I add User-agent: Googlebot, will fake ...
- 131
3
votes
1
answer
3k
views
Preventing high bandwidth usage from Yandex
Although I put a row for Yandex into my robots.txt file, sometimes Yandex indexes my website aggressively. So I hard coded a part and check for user agent, and serve cached file if user agent is like ...
- 405