It depends on the permissions you are asking when they connect. I often use the "Facebook" registration option on sites, as it saves time entering in all of my info.
However, I regularly see sites asking for a lot more permissions than they need for what you are trying to do. For example, a lot of sites ask for permission to post on your Facebook profile. Why would this permission be needed to allow you to register on the site? If a tool of feature of the site requires this, the permissions should be asked at the time that this feature is used, not when first connecting.
I say close to half of the sites I try using the Facebook Connect method I end up closing because it is asking for excessive permissions. Depending on if it is a website I really want to use I may sign up using the non-Facebook method, but in a lot of cases it is not worth the trouble so I move on.
So, while I'm not sure if this is the case in your situation, it may be something to consider. I'm sure I'm not the only one that doesn't like allowing a site to do all sorts of things just to register or login.