I'm planning to extend an existing website by a version which is optimized for mobile devices. Would it be a good idea to register a .mobi domain or is it better to use a subdomain (for example:http://m.////.TLD) for this site?
If you can, the best option is to use responsive design, and make your site work well in all modern mobile devices in the same domain/url.
However there are some situations where a different site is a better option, like for example when most of your audience has old mobile equipment, you think your users will prefer a different experiences or heavy use of flash in the desktop site.
An m.samedomain.tld is more usual today and users will be more confident as they can be assured that the mobile site is not a fishing site, created by spammers using the same brand name/colors/logo.
In terms of SEO, if there's any difference (I think there isn't, but you never know if search engine logarithms will change tomorrow), it will favour sub-domains.