Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I have a Twitter application so users of my application shares links of my webpage inside their tweets. It seems like bots follow these links and some of these bots create high bandwidth usage. And most of them doesn't provide me any hit. So I want to disallow them with robots.txt, or .htaccess file.

When I check access.log I see following bots below.
My conecern is will it be a problem to ban Twitterbot ? Who owns this bot ? Twitter.com or other website? What would be the drawbacks to disallow it ?

No  Bot name    Daily hits
1   Twitterbot              1,499
2   MJ12bot                 1,490
3   Google AdSense Robot    774
4   ShowyouBot (http://showyou.com/crawler) 655
5   Googlebot               595
6   Bing Robot              204
7   Yandex Robot            186
8   Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)    148
9   Apple RSS Robot         126
10  Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php) 76
11  FaceBook Crawler        62
12  Alexa Robot            48
13  QuerySeekerSpider ( http://queryseeker.com/bot.html )   37
14  Google Feedfetcher      28
15  Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C; .NET4.0E; .NET CLR 1.1.4322; Tablet PC 2.0); 360Spider  17
16  Ezooms Robot            14
17  AhrefsBot   10
18  Mozilla/5.0 (Windows; U; Windows NT 5.1; zh-CN; ) Firefox/1.5.0.11; 360Spider   9
19  Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html) 8
20  Baidu Spider    7
21  Yetibot 3
22  Exabot  2
23  FeedBot 2
24  Mozilla/5.0 (compatible; SISTRIX Crawler; http://crawler.sistrix.net/)  2
25  SeznamBot   2
26  Yahoo! Slurp
share|improve this question
1  
Do you pay directly for bandwidth? A couple thousand hits a day isn't anything significant and shouldn't be costing you much. –  DisgruntledGoat Jan 20 at 18:03

1 Answer 1

Twitterbot is owned by Twitter. Basically it comes along to index the content of any given URL (like Google does). I'm not sure, but I think they mostly use this data for the snippets (Twitter Cards) shown that go along with a Tweet with a link ie. the page title, the description, and an image (if present).

So the only downside I see to blocking it would be that links to your site wouldn't have Twitter Cards associated with them to other users. This could of course result in a lower click-through rate for links to your website, however.

share|improve this answer
    
Thank you. I only have a Twitter card for my android app. My website isn't suitable for creating different cards for each page. Because only authorized user can see his/her data. –  trante Jan 13 at 22:45
    
Then it's probably safe to go ahead and block Twitterbot. –  nathangiesbrecht Jan 14 at 3:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.