Is there any downside to registering a Facebook page custom URL or Google Plus custom URL?
No, this will help users more readily identify links and increase their likelihood of clicking on them.
Is there a risk of these social media custom URLs competing with my own domain's website and cannibalizing my own traffic?
Not likely, the domain would still be facebook.com
or plus.google.com
so content would be indexed under those sites, not www.example.com
. This is not any different than having a user string instead of a custom URL, and that's typically not an issue with those either. Unless you have a large amount of duplicate content on both these social sites and your site, there wouldn't be any cause for concern.
Or do the benefits outweigh the risks?
There really aren't any risks, it's just a different URL structure that's more easily recognized and remembered by users.
If you want your own site to rank higher than your social media pages (custom URL or not), then you'll need to provide more relevant content and external links to it (including links from these social sites, which are signals to search engines).