1

On our page users are opening posts in a modal (like facebook or twitter).

The links look like this

<a class="showpost" data-mpid="416246" href="/p/416246/">Open Post</a>

We catch the click, execute an ajax call and convert the JSON response to HTML. Even I read that Google can execute ajax for some time now, none of our posts gets indexed. If I make a "fetch-preview" with render in the search console it shows the website correctly with the post opened.

We have more than 500k posts, most of them with very long and good content (reviews of products) and it's really annoying that none of them can be found in search engines.

Any ideas what we are doing wrong?

EDIT: After some days of further research I still can't find the solution. The sitemap status for these links is "Discovered - currently not indexed".

Is it possible that Google renders it correctly but the added content to the DOM is too small to get "noticed"? Its an image, some paragraphs and sometimes comments added to an existing DIV and then we show this DIV with css. Should I hide everything else on the page when opening the post?

Improve this question
4
  • You say you "catch the click", so users don't see the href directly. Does that URL work if you visit it manually? Do you change the page URL using pushState to match the href of the link?
    – Stephen Ostermiller ♦
    Mar 28, 2018 at 20:18
  • 1
    They see the href. Yes we use pushState and the URL works if opened manually. We check the URL and fire the same function as the click on the link,.
    – Kornel B.
    Mar 29, 2018 at 8:42
  • How long has this been going on for? How long have these pages been live? Do you also use an XML sitemap? Is /p/416246/ (ie. the contents of the HREF attribute) an example of a URL that you would want Google to index and the URL that appears in the browsers address bar?
    – MrWhite
    Mar 29, 2018 at 9:08
  • 1
    The oldest posts are about a year old. Yes we have a XML Sitemap and yes we want this URL to be indexed and show up in the results.
    – Kornel B.
    Mar 29, 2018 at 10:30

2 Answers 2

Reset to default
1

There's a good chance that Google is not indexing these AJAX pages because it's having trouble reading your script that catches the link.

Let's say for instance that the AJAX page loads /ajax/page.html. Because your link has href="/p/416246/", Google is going to pass your link juice to /p/416246/, when it sounds like you want it to pass it to /ajax/page.html.

The href plus the ajax URL is may be too many pages for one link for Google to crawl.

But the most likely issue for Google not crawling these AJAX URLs is that the Javascript code is a little complex for Google to correctly understand. For instance, if your AJAX has URLs in it like /ajax/page.html then Google will likely crawl this. If on the other hand your AJAX looks something like var z ="page.html"; var url = '/ajax/' + z;, or even something more complicated, then this could be a bit much for Googlebot to process accurately.

Improve this answer
3
  • Thanks for your answer. How about making an own html for each post and if its a bot I dont execute the script but instead redirect to the html page. Do you think this would work to get all posts indexed?
    – Kornel B.
    Mar 29, 2018 at 8:57
  • I think it's risky. Because if you show Googlebot one version of the site and a user another version, Googlebot will think of this as cloaking. Google needs to know how your site will appear to users, and what will happen when links are clicked. You're not trying to trick the bot, but you are making it difficult to see how your site works.
    – Michael d
    Mar 29, 2018 at 11:06
  • "Because your link has href="/p/416246/", Google is going to pass your link juice to /p/416246/, when it sounds like you want it to pass it to /ajax/page.html." - Although the OP has confirmed in comments above that /p/416246/ is the actual URL that renders the content.
    – MrWhite
    Mar 31, 2018 at 0:06
0

Google or any other boat will not get your data since you are not allowed it. Yes true. A bot cannot be interact with your page like click or mouse event. It is true that boat will start reading js but they not said that they will start click or hovering on your screen. And in your case the html is generated after a user click.

This is the limitation to the boat that it cannot perform user action. So, if you want to index that data on same page you have to call it before rendering. Whether it is hide from the user without an action such as user click, bots are smart enough to read it.

Improve this answer
3
  • "...since you are not allowed it." - what do you mean by this?
    – MrWhite
    Mar 29, 2018 at 9:05
  • there is the limitation of the boat that it does not fire any user event such as click, scroll or hover event.
    – Anurag Jain
    Mar 29, 2018 at 9:19
  • From the additional comments from the OP, it would seem the "firing of the user event" would not be necessary for Google to index these pages. Simply crawling these links/URLs in the "normal" way would be sufficient - since the canonical URL is already given in the HREF attribute. (Although the page content is still generated with JavaScript/AJAX it seems).
    – MrWhite
    Mar 31, 2018 at 0:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.