Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?

share|improve this question
3  
Don't dupe questions across sites. You got your answer over there, by the way. (See Jeff's bio.) –  Su' Oct 19 '11 at 5:00
    
Boo... thanks @Su' - now how do I withdraw my bounty??? –  Mike Hudson Oct 19 '11 at 6:59
add comment

2 Answers

Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API.

share|improve this answer
    
Not a bot, a scraper. What evidence do you have for your claim? There's data disputing your claim here: stackoverflow.com/a/9165191/63094 –  artlung Nov 13 '12 at 17:25
    
Evidence, is 100s of log files. If you dont use facebook on your site, facebook will bot bot or spider it. The answer of Stack overflow is exactly the same. Facebook connects to your site when it is shared or if you use facebook FBML or other facebook code. –  Frank Nov 20 '12 at 4:32
add comment

This same question was asked on Stack Overflow.

The answer is that the Facebook scraper does not obey robots.txt or support a crawl delay. The best that you can hope for is implementing some sort of server side throttling. Some code for doing that has been provided as one of the answers on Stack Overflow.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.