The solution would be the same as for Block a random domain pointing at my dedicated IP. I posted an answer there with configuration and a script.
Basically, you need to create a default virtual host entry to handle the IP address and any other random addresses. Then you can have your real site in a second virtual host directive that is specific to your site.
