How can I get a web crawler to see dynamic data injected by JavaScript in 2019?

Question

Originally posted this on stackoverflow and it got put as off-topic so posting here.

I recently read this google webmasters article from 2015 on accessibility and SEO for web pages using AJAX. It states the deprecation of their old crawling scheme using the hashbang method, and then states the updated proposal:

Times have changed. Today, as long as you're not blocking Googlebot from crawling your JavaScript or CSS files, we are generally able to render and understand your web pages like modern browsers. To reflect this improvement, we recently updated our technical Webmaster Guidelines to recommend against disallowing Googlebot from crawling your site's CSS or JS files.

Any research I've done on this topic has led to old articles that point to and reference the deprecated article/method. So, consider this simple data binding snippet (not AJAX just plain JSON):

https://codepen.io/tfisher9180/pen/qGRbVd

The select box populates a table from JSON data.

Obviously the source of the page contains no table data. So, what is the recommend approach to expose each of the option's table data to a web crawler?

Answer 1

Googlebot does know how to execute JavaScript these days, however there are limitations:

• Googlebot renders pages sometime after crawling them. It could take extra months for Google to get around to rendering and find extra content written into the page by JavaScript.
• Googlebot doesn't click on anything. Any content that is only shown after user interaction doesn't get indexed. (Googlebot does however still find links in dynamic content and will later crawl and render linked pages.)

In your example, the content is only shown after a user interacts with a select box on the page. That means that Googlebot will never see it. You have two options:

1. Write all the data into the page on page load. Rather than use a select box to get to the different areas of content, let the user scroll to them.
2. Create separate pages and separate URLs for different content. Instead of a select box to change the content, use links. You could even put the links in a pull down menu that looks and acts a lot like a select box. For users, you could intercept the link clicks, show the content in the current page and use pushState to change the page URL without actually loading a new page. Just make sure that when the page is loaded with that different URL that the appropriate content is shown by default.

Your example doesn't use AJAX, but it doesn't change too much to use AJAX. Googlebot now downloads supporting resources such as external JavaScript, CSS, and AJAX when it renders pages. Again that AJAX download has to be triggered during the page load, and not as a result of user interaction. Googlebot uses a fairly short timeout when fetching resources, so your server has to be able to return them to Googlebot relatively quickly when Googlebot tries to fetch them. Other than that, Googlebot treats inline JS data almost the same as AJAX fetched data.

You might be interested in watching this video that Google just released that goes through a lot of this. Google Webmaster Analytist Martin Splitt has an entire video series on the topic.

Answer 2

The most problem i see in your implementation is the same url for any content. Rendering content driven by user action makes things just worse.

This implementation violates the paradigm, which drives the whole internet - the paradigm of the URI/URL. It means, a document has its very own, unique address (resource locator).

To make such applications visible to bots, one pre-renders them (somebody calls it warm cache).

It works like: you have a browser instance and a kind of clickbot. You load your (main) page in the browser and let the clickbot click on everything. After each click (changing state, loading new content into certain div etc.) you save the page as static HTML with an own URL and save all HTML linked one to one in a kind of cache directory. Then you let bots and visitors visit yur static page variant. To make static variant equal dynamic you run the clickbot after each page edit.