Image Duplication Detection is not reliable nor is Video
As far as I know there is no online web application which finds video duplicates or minor edits and with this said Google image search isn't 100% accurate either as most often edits of images appear as non duplicate. There are applications available that scan the fingerprints of local video files but in your case this would be useless.
Your audiences are the best judges
Normally people who spend a lot of time making these videos for contests at premium levels are your best judges and you'd be surprised how easily they can spot duplicates and minor edits. Even if their was a online web app which detected duplicates it would not delete people using FRAMEWORKS with minor edits. Take a look at http://videohive.net/ for example. These video's are extremely easy to adapt and these types of videos will not display as DUPLICATE since they contain heavy edits with little work.
Comments and Reporting Buttons can help
Enabling things like user comments and report buttons are good if you allow submissions to be viewed by submitters and site visitors, when you have narrowed the top 5 down to potential winners then you should of course do your own checks checking various sites such as videohive, vimeo, YouTube etc.
Big Prizes - The top 5 most likely will be unique videos
In all honesty your find if its a BIG prize on offer your top 5 will be unique entries. You should post your competition on Dribbble, and other sites that have skilled people.