Yes and no. I have a website that has two versions of the site. One for desktop, one for mobile. The mobile website has a fixed viewport of 380 and according to Google's Mobile Ready TestMobile Ready Test my website is mobile friendly. It also shows up in mobile search results as being mobile friendly.
Google says that a fixed width viewport is accepted but it is not recommended.
So, no. You will not be penalized for a fixed viewport as long as the viewport is small enough. Most mobile viewports that I've seen are set to 380.