7

According to google itself and another question here, google's bots execute JavaScript on crawled sites and index the rendered content.

However,

[a]ccording to [the] experience [of at least this one company], while most of the JavaScript-based functionality is now understood by Googlebot, it consistently fails to crawl the content retrieved by XMLHttpRequest API from an external source - or anything built on top of, or related to this API. This behavior is present in pure JavaScript, jQuery, AngularJS, or other modern JavaScript frameworks. Whenever you need to pull content from an "external" URL or call a REST API endpoint to fetch some data, chances are that it will not be crawled and indexed properly.

(emphasis mine) This behaviour we are now facing as well:

  • We have a react app that integrates into a client's website, whose landing page is a list of things with links to a subpage for each (<a data-reactid="0.1.2.3.4" href="/some/site/itemid/9876">)
  • The landing page is rendered correctly in the Google search console ("This is how Googlebot saw the page" visually equals "This is how a visitor to your website would have seen the page")
  • Content of the landing page can be found in google's search results.
  • No content of any subpage can be found via google in the near term (weeks later, content of a fraction of the subpages is).

Here is a demo react app with react/js-generated links and subpage that were correctly indexed. Key difference is, as mentioned, that we retrieve the content of the subpages from an external server. There is no way around that - we have the content, but the website that shall render it only includes our js script and an element to render it into, we have no further control over the website, much less the server it runs on.

Is the above quote correct or am I missing something?

Improve this question
4
  • maybe you can use a proxy, so it seems like the request goes to your server? Like: /our/site/proxy/http%3A%2F%2Fdifferent.com%2Fcontent%2F2
    – Milo Tischler
    Aug 30, 2016 at 23:03
  • @milo: Sorry, I was imprecise. The user's browser and googlebot actually visit our-clients-site.com/some/site, which loads oursite.com/app.js, which then fetches data from oursite.com/api/item... and renders the page. So we actually control the API site, but not the actually visited one. (Sorry, missed your comment last week. Thank you anyway. :-) )
    – Titanic Tub Tree Tiger
    Sep 9, 2016 at 12:40
  • One and a half month later, google has actually indexed a handful of item subpages, so it can do so in principle. I am not sure what prevented it from indexing the other two dozens or so item pages, though. Our working hypothesis is that the API did not respond quickly enough, although each item page contain some static content around the stuff that comes via the API, which does not appear in search results either (except for the aforementioned handful of pages, +1 for the landing page).
    – Titanic Tub Tree Tiger
    Sep 9, 2016 at 12:48
  • The quote is wrong, you proved it ;) Using the History API with pushState() correctly is a hard requirement, though.
    – Jonas Eberle
    Oct 1, 2016 at 12:45

1 Answer 1

Reset to default
2

As you yourself have proven @titanic rub tree tiger Google is in fact able to parse content from the XHLHTTPRequest API. The issue here isn't to do with parsing the site rather it is to do with how long it takes for the content to be updated to the search index. Based on your comment it would appear 6 weeks later the content has started being indexed and listed on Google which is not completely unreasonable.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.