Yes a video background does effect your SEO see https://support.google.com/adsense/answer/6196932?hl=en and http://searchengineland.com/mobilegeddon-beginning-not-ending-220512
The slower your site loads the farther Google pushes you down the SERPs. The biggest concern for Google is mobile devices.
So what should you do instead of adding a video background? Research Sales Copywriting, customers care about your website solving their problem more than anything else. It's hard for a video that's a background to solve any problem unless that customer is looking for flash.