From Modernizr installation page:
Drop the script tags in the HEAD of your HTML. For best performance, you should have them follow after your stylesheet references. The reason we recommend placing Modernizr in the head is two-fold: the HTML5 Shiv (that enables HTML5 elements in IE) must execute before the BODY, and if you’re using any of the CSS classes that Modernizr adds, you’ll want to prevent a FOUC.
Modernizr is pretty fast and lean, especially if you have used their customizer to trim out the features you don't need to use