I start a Web server for my company for hosting about 15 website and I didn't install a Control Panel, I do everything by command and i'm pretty comfortable with it but do you think i'm losing time by doing all the task by command ???
To be honest, No.
I am currently helping out friends setup their hosting using different CPs and also manage the webserver for my organization.
I find that doing things the console way is way faster once you get the hang of it. A CP ensures that you do not shoot yourself in the foot when you are on the console.