Yes you would. You are telling Apache to take any reference to contact and rewrite the URI to contact.php then telling Apache to take any contact.php reference and change it to /contact which gets captured again by the first rule.
Here is what you are missing. For each rewrite or redirect, Apache makes the change then starts the whole matching process over again. So after your RewriteRule, Apache is converting the request to contact.php and after your Redirect, Apache is converting the request almost back to what it was originally.
If you are using the file name contact.php for SEO purposes, I would drop this concept. File names do help as much as the (apparent) directory name. As well, single keyword file names do not seem to matter much at all especially when it is such a common word. There are exceptions of course. Brand names with product names seem to matter. In this case, just do things the old fashioned way- a proper link and no fancy stuff.
Your redirect is correct. I would use it, however, I would not use specifically named PHP files such as contact.php. It just confuses things. Instead, I would stick with index.php which should be your DirectoryIndex. Unless you have several files within your /contact directory, this should work okay. If you do this, you can remove your RewriteRule and everything should work fine.
Clear as mud?