I'm planning to extend an existing website by a version which is optimized for mobile devices. Would it be a good idea to register a .mobi domain or is it better to use a subdomain (for example:http://m.////.TLD) for this site?
If you can, the best option is to use responsive design, and make your site work well in all modern mobile devices on the same domain and URL.
However, there are some situations where a different site is a better option. For example, when most of your audience has old mobile equipment, you think your users will prefer a different experience, or there is heavy use of flash in the desktop site.
m.example.com subdomain is more usual today, and users will be more confident as they can be assured that the mobile site is not a phishing site, created by spammers using the same brand name/colors/logo.
In terms of SEO, if there's any difference (I think there isn't, but you never know whether search engine algorithms will change tomorrow), it will favor sub-domains.