Do Facebook's webcrawling bots (for example facebookexternalhit/*) respect the Crawl-delay: directive in robots.txt files?
-
3Don't dupe questions across sites. You got your answer over there, by the way. (See Jeff's bio.)– Su'Oct 19 '11 at 5:00
-
Boo... thanks @Su' - now how do I withdraw my bounty???– Mike HudsonOct 19 '11 at 6:59
Add a comment
|
This same question was asked on Stack Overflow.
The answer is that the Facebook scraper does not obey robots.txt or support a crawl delay. The best that you can hope for is implementing some sort of server side throttling. Some code for doing that has been provided as one of the answers on Stack Overflow.
answered Nov 22 '13 at 11:45
Facebook does not have a bot that spiders the internet. They only track events on your site when tied with their Graph API.
-
Not a bot, a scraper. What evidence do you have for your claim? There's data disputing your claim here: stackoverflow.com/a/9165191/63094– artlungNov 13 '12 at 17:25
-
Evidence, is 100s of log files. If you dont use facebook on your site, facebook will bot bot or spider it. The answer of Stack overflow is exactly the same. Facebook connects to your site when it is shared or if you use facebook FBML or other facebook code.– FrankNov 20 '12 at 4:32