5

I have a a big problem that I'm only considering after I have finished development of a webpage. My page is written with only a skeleton HTML. All the actual content of my site is sent through websockets. The client-side JavaScript then captures this information and dynamically builds the page.

I did this for the purpose of having everything being able to be changed quickly and in real time via my websocket instructions.

All the code functions without errors. However I have just found out that when I fetch with Google crawler, all that the bot sees is the base HTML with no content.

I'm not sure what ways I can use to get around this? Am I stuffed because of this approach I used? Changing this will mean a whole overhaul of my architecture. Can anyone please give me some suggestions as to how I can get the Google cralwer to pick up on this dynamic content? Is there a way I can tell when Google crawls my site and render things on the get request rather than websocket?

Improve this question

1 Answer 1

Reset to default
3

All right let's understand this problem slightly more in detail.

Google does relatively good job indexing JS / AJAX content in compare to other search engines but it will also depens on the implementation as it can impact how Google sees your content.

Honestly, if you got a long term vision for your website to perform well in search engines then you might have to think about your implementation more seriously.

One thing is for sure, even AJAX content could be crawled, it might be very hard to rank considering other site might better structure.

Another point to consider is, Google may do fine job reading AJAX content but other search engines may not do that well.

It's an old video but you may wanna watch:

https://www.youtube.com/watch?v=_6mtiwQ3nvw

You might have to explain slightly more in detail, how your dynamic content loads in the page to see the possibility getting it indexed.

Improve this answer
8
  • Hi TopQnA thanks for the response. I have to say I'm quite disappointed in 2017 that something like this isn't working. I believe google being such a large search engine has a responsibility to provide cutting edge indexing capabilities with the latest technologies. Anyway rant over. To better understand how my page work you can think of it as a mini protocol layer over websockets. It starts of by establishing the websocket, checking and registering the session ID of the page loaded, then making an ACK request for content to the server. This ACK request is then responded to by the server that
    – johhny B
    May 21, 2017 at 9:33
  • will send out a certain type of packet over the websocket. These packets are picked up in the javascript the content taken from the packets and then rendered and inserted. Jquery is used. So it's basically like a requesting and sending packets of information over the WS. Is there some workaround to this without changing my code substantially? I honestly think the world should be moving away from static HTML to a more dynamic environment anyway. Although i guess that's my own personal thoughts... In the meantime while the world catches up. Can anyone suggest some viable alternatives
    – johhny B
    May 21, 2017 at 9:37
  • 1
    Can the use of sitemaps or somehow detecting the google crawler be used here?
    – johhny B
    May 21, 2017 at 9:41
  • Agree about moving technology and the role of JS. And Google understands this better than any other global company and example is creating Google Chrome which is the leading / best browser to render complex AJAX applications. Even their AdWords system which returns maximum revenue relies heavily on some heavy AJAX. From search engine side we also need to understand there are so many situations they need to consider.
    – TopQnA
    May 21, 2017 at 12:10
  • And Google's goal is to do what best for the user. Think in this way Google got a massive number of issues to address to ensure the user gets the best experience knowing currently available technologies and also pushes hard for something which they truly believe in and example is converting entire web to HTTPS. So, far it has been seen AJAX is amazingly good for building complex web applications. But as soon as it comes to normal web it's totally different game. Google expects web to be a rich, fast and spam free experience. AJAX got its own limitations.
    – TopQnA
    May 21, 2017 at 12:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.