Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?


2 Answers 2


This same question was asked on Stack Overflow.

The answer is that the Facebook scraper does not obey robots.txt or support a crawl delay. The best that you can hope for is implementing some sort of server side throttling. Some code for doing that has been provided as one of the answers on Stack Overflow.


Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API.

  • Not a bot, a scraper. What evidence do you have for your claim? There's data disputing your claim here: stackoverflow.com/a/9165191/63094
    – artlung
    Nov 13, 2012 at 17:25
  • Evidence, is 100s of log files. If you dont use facebook on your site, facebook will bot bot or spider it. The answer of Stack overflow is exactly the same. Facebook connects to your site when it is shared or if you use facebook FBML or other facebook code.
    – Frank
    Nov 20, 2012 at 4:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.