If the content and internal link structure is different between both servers (I am not talking about the presentation), bots will notice the changes and will likely increase their visits to your site.
They will be very confused if they are being served different content back and forth and this may trigger a manual review. Google wants to be sure about the content being served to users. If algorithms have doubts, they tend to remove sites from search results automatically too.
To avoid such issues, make sure that a given user is always served content from the same server, by setting a cookie for example. Keep in mind Google bots don't use cookies, but you can detectdetect them and send them to the original config server.
If you do that, you will be SEO safe.