Webmasters Stack Exchange is a question and answer site for pro webmasters. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?

share|improve this question
Don't dupe questions across sites. You got your answer over there, by the way. (See Jeff's bio.) – Su' Oct 19 '11 at 5:00
Boo... thanks @Su' - now how do I withdraw my bounty??? – Mike Hudson Oct 19 '11 at 6:59

This same question was asked on Stack Overflow.

The answer is that the Facebook scraper does not obey robots.txt or support a crawl delay. The best that you can hope for is implementing some sort of server side throttling. Some code for doing that has been provided as one of the answers on Stack Overflow.

share|improve this answer

Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API.

share|improve this answer
Not a bot, a scraper. What evidence do you have for your claim? There's data disputing your claim here: stackoverflow.com/a/9165191/63094 – artlung Nov 13 '12 at 17:25
Evidence, is 100s of log files. If you dont use facebook on your site, facebook will bot bot or spider it. The answer of Stack overflow is exactly the same. Facebook connects to your site when it is shared or if you use facebook FBML or other facebook code. – Frank Nov 20 '12 at 4:32

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.