Server-side rendering for search engines only (SEO)

Question

I have a heavily JavaScript based web app which I want to make it indexable.

I know that google bot and others agents can already see JavaScript, but I don't trust they will capture my content correctly and the problem is even bigger with asynchronous JavaScript. For examples: there are sections that are loaded only when the user scrolls down. There are also crawlers that can't see JavaScript, like Facebook's crawler.

Can I just send pre-rendered HTML, using phantomjs or other headless browser, to the google bot? This would be exactly what my end-users sees, but my fear is I may suffer penalties for serving different content for the bots.

Answer 1

Can I just send pre-rendered HTML, using phantomjs or other headless browser, to the google bot?

In short... no you shouldn't. Google's stance on differing versions for visitors and search bots is clear and regardless of your intentions it is classed as a cloak. Google also goes to great lengths to activity look for sites that specifically treat the Googlebot differently to other website visitors.

All sites should render the same with both Googlebot and Users

Every site regardless of the JavaScript should be render the same by both users and search engines, if it does not then there is something wrong with your site.

Emulating what Googlebot sees...

You can emulate and render what Googlebot sees when visiting your site by using the Google's Fetch Test with Google search console, with the results provided you can debug your site and ensure that its read and renders as expected for both search bots and users.