The HTML specs say that user agents that don't understand an HTML tag are to ignore it. Assuming that search engine bots follow this spec, and they do, then using HTML5 tags that aren't yet recognized by the search engines are ignored and shouldn't hurt your rankings.
In your example the
<h1> tag is not being used incorrectly. It just is wrapped in a new tag that may or may not be recognized by the search engines. But that won't hurt you as that tag simply has no semantic meaning for now. But when it does you're rankings may benefit because of the increased semantic meaning. So use it without worry and with the expectations that in the future this will help you.
(One possible exception is the proposed multiple heading tag thing where you are allowed to have multiple
<h1> tags on a page. I'd avoid this for now since that could cause issues as doing this previously could only be construed as an error in markup or manipulation of the SERPs. Obviously that may change).