There's a few things you need to do so I'll try and break it down in steps in the same order as you have asked.
Pointing your new domain to the VPS using GoDaddy
GoDaddy has their own dns manager and getting it to work with your domain is really simple, simple change the A record within the DNS manager to point to the IP address of your VPS. Below is a picture to display what setting I'm referring too.
Setting up Email Though the Domain
You mention that you want to receive emails and send emails using the domain, though mentioned that you want to host the emails this can be resourceful for your VPS and demanding how many email accounts you want them a better way is to have a email server hosted by someone else, not only because they tend to be more reliable but the fact is your being eating resources. As you know Google have a free email service called Gmail but they also do a Google APPS service which is exactly the same as Gmail but you use your own domain with their service. You don't need to use the web login to read emails as the same as the standard accounts you can throw them into outlook if you choose. Google apps is free upto 10 users, and then you need to start paying - it'll say free trial when you head over too www.google.com/apps ignore this if your below 10 users its actually free for standard use.
Below is a picture of what you need to do with the DNS with GoDaddy to send emails and receive though your server.
To send emails from the website you can use a PHP SMTP method or you can use a standard method by just using sendmail (Apt-get install sendmail) which won't login to your Google apps and send the email as SPOOFED which most people use. But SMTP avoids spam filters better.
Optional - Setting up a FQDN
Should at any stage you be asked to enter the a valid FQDN you should add it to prevent further issues. Since you have not mentioned what the VPS is running on I’ll give you a few pages that will help on multiple distros and assuming that your using linux. You want to add the FQDN to your hosts via the /etc/hosts file. References below.