We are going to update our website from well-formed XHTML 1.0 Transitional to HTML 5. However, as I see, we will be doing this step-by-step. This means we will have some pages in HTML 5 and others in XHTML at the same moment. Can it affect our current SERP positions and the results of other SEO works we have already done?
Unless you're going to dynamically serve the relevant doctype for either HTML5 or XHTML then you'll likely have a lot of W3C invalid markup during the update process and whether this affects the performance of your website (SEO) is answered here.
What you should be doing is setting up your website as a development stage and completing all the work before pushing it live. This avoids countless potential issues.