The XHTML doctype is for XHTML web pages and, unless you are serving your page as application/xhtml+xml, it ain't XHTML. I'm betting you're not doing that.
Writing HTML as XHTML is called "tag soup" and browsers will ignore what you wrote and do the best they can with it cause it won't make sense to the HTML parser. So that's a waste of time.
Unexpected issues is what will happen. Some HTML will get rendered oddly, some css will behave unexpected, not sure what JS will do, but I would not be surprised that that'll result in something quirky also.
There really isn't a reason not to do so (apart from some very very very edge cases). Everything might work exactly as you want to. And it could also ...
Nowadays browsers will auto correct a lot of issues found within the markup of the webpage, however older browsers not so much. So while you see no problem is not necessary true for other visitors.
So I advise that you have valid code to ensure that your website is displayed correctly not just to some visitors, but to all.
There is clearly no problem about it. Every request returns one document that has its own independent doctype. This is not affecting any other sites.
According to https://stackoverflow.com/questions/10380346/css-styles-not-applied-properly-if-use-doctype different doctypes could define different default CSS property values of specific (meta) elements. So ...
What you are asking is if HTML document type could affect SEO, right?. Based on my experience and logic I'll say no.
A site built to standards (valid HTML and CSS) is often easier to access and crawl by searchers but using different doctype will not boost your ranks cause accesibility it's not a direct SEO factor.
As long as you can check that Google ...
If you mouse over the doctype you'll see that Firefox is rendering the page in "Almost standard mode".
The DOCTYPEs that will trigger "almost standards" mode are those which
The public identifier "-//W3C//DTD XHTML 1.0 Transitional//EN"
The public identifier "-//W3C//DTD XHTML 1.0 Frameset//EN"
The public identifier "-//W3C//DTD ...
It is always recommended to define the doctype at the beginning.
SEO Aspect: Google does relatively good job with reading incorrect syntaxes.
There is something called "Best Practice" and it's always good to implement best practice just to ensure you may not have any issues.