5

In a website that has posts system similar to Facebook, where the user has option to:

  1. Keep his posts private (needs login to view)
  2. Make them public (anyone can see them)

In case #2 I want to allow Google to index these posts, but there are problems:

  • There are no direct links to the posts
  • The posts are loaded with infinite scroll style without pagination

Note that each post also has a permanent link that points to separate page with the post and all the comments on it.

Is it good strategy to index public posts? What is the best method to achieve that?

Edit: I will accept 2 answers from Yhorian & Stephen Ostermiller.

Stephen Ostermiller delivered great point on not relying on dynamic site map for ranking in google, where he linked to The Sitemap Paradox, and his advice was great on creating links patterns between the pages.

Yhorian recommended a dynamic xml map for undiscoverable content.

I will use both solutions, a proper linking between pages and a dynamic site map to cover missing links.

Improve this question
2
  • How does a sample post permalink look like? (no need for domain name, I'm asking whether its based on params or not)
    – Bartek
    Apr 11, 2017 at 9:40
  • @Bartek it is basically domain.com/postID so it is database driven
    – DeepBlue
    Apr 11, 2017 at 10:31

4 Answers 4

Reset to default
2
+50

Your current setup isn't good enough for SEO. To get Google to index content you have to:

  • Put each post on its own URL so that Google can deep link to it
  • Link to each post from other pages

Other answers suggest creating an XML sitemap. That isn't sufficient to solve the problem either. Google usually chooses not to index pages found only in an XML sitemap. Even if Google does index them, they won't rank well.

I would recommend:

  • Linking to the permanent URL of each post in the infinite scroll
  • On the permanent page include next and previous links
  • Link each post to 10 other related posts. Use criteria such as subject, tags, author.

Once the posts are well linked to each other, Google will rank all of them very nicely.

See The Sitemap Paradox in which this site had pretty much the same problem. When that question was asked, the site only linked to a few questions from the front page. Jeff Atwood was hoping that XML sitemaps would work better than they do. He created the "Related" links section that appears in the right column of this page. It is a solution that works very well. StackExchange questions tend to rank very well in Google now.

Improve this answer
6
  • I was thinking to add pagination (1,2,3) in the top of the page to navigate through posts but make it hidden to normal users using css. This way the bot will follow all the pages, and I can keep the infinite scroll. is it good idea?
    – DeepBlue
    Apr 11, 2017 at 11:29
  • No. Google calls hiding links from users cloaking and could penalize your entire site for doing so.
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 11:41
  • You can keep infinite scroll, just each post should have a link to its own page. Such a link is often labeled "permalink". That is what WordPress does.
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 11:42
  • Also, pagination is not a very good solution for getting things indexed and ranked. It is 100 times more powerful to link the content together directly.
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 11:43
  • I will add links to each post (good idea).. but will google bot actually navigate using infinite scroll? Otherwise how can I make old posts discoverable?
    – DeepBlue
    Apr 11, 2017 at 13:14
1

Content served with infinite scroll usually is not directly indexed because content loaded by javascript is ignored by crawlers. If you have for each item the infinite scroll is loading a details-page you still can create a google-index in XML where the detail-pages are listed, like that the detail-pages are found only by the XML-page which is usually read only by crawlers.

Improve this answer
3
  • 1
    "content loaded by javascript is ignored by crawlers." - Google does a reasonable job of indexing JavaScript generated content these days.
    – MrWhite
    Apr 8, 2017 at 10:41
  • You're right, more about it here: webmasters.googleblog.com/2014/10/…
    – David
    Apr 8, 2017 at 11:05
  • Content loaded by JavaScript onload can be indexed. Content loaded by user action such as scrolling would generally not be indexed.
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 10:36
0

From your description, these are the recommendations I would make:

  1. Create a sitemap with links to the unique post/comment links.
  2. Submit the sitemap to Google Search Console/Bing Webmaster Tools.
  3. Don't worry about ranking the AJAX infinite scrolling presentation. Make it optimal for users rather than search engines.

To explain further: This strategy makes sure all your content is indexed and ranked optimally.

Google struggles with dynamically loaded content because it can't be sure what it's presenting to the user. Incosistancy means it would be unsure which queries to present your AJAX scrolling page to.

Fixed URLs are easier for it to index and rank. This content will lead people to your site to view the content they've searched for. The infinite scrolling 'front page' for this content can remain the way it is, don't touch it or try to 'noindex' it. Google will be able to understand it's a navigational page and rank it appropriately.

Improve this answer
6
  • But the content is database driven, so I have to make dynamic xml map and send it from inside the PHP code. Also I have to update the site map when new users add new posts. How to do all of this?
    – DeepBlue
    Apr 11, 2017 at 10:34
  • 1
    A sitemap doesn't help content get indexed or ranked. See The Sitemap Paradox
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 10:39
  • This entirely depends on your code base. It's a very difficult question to answer in a Webmaster context, and if you're talking about code specifics you may be better asking on stackoverflow.com.
    – L Martin
    Apr 11, 2017 at 10:41
  • @StephenOstermiller - from your own link, a Google collaborator states sitemaps are used for: "Discovering new and updated content (I guess this is the obvious one, and yes, we do pick up and index otherwise unlinked URLs from there too)". A sitemap isn't considered for ranking but it categorically is used for content discovery. User generated content should be logged and submitted via a sitemap if the navigation page is dynamically generated.
    – L Martin
    Apr 11, 2017 at 10:45
  • In practice that doesn't really work. Google usually decides not to index pages found only in a sitemap, especially if your site has lots of pages. Even if they are indexed, they don't get the link juice they need to rank competitively.
    – Stephen Ostermiller ♦
    Apr 11, 2017 at 10:47
-1

  1. If a post has a description then you can allow google to index that posts by descriptions except that private post have image or video that can't read by any site.

  2. An infinite scroll is not a big deal you can download this type of code by just googling and in codepen or anywhere or you can do by giving load more button(thanks to JSON!).

Improve this answer
2
  • What is they method to achieve crawling?
    – Abu Nooh
    Apr 8, 2017 at 9:48
  • @AbuNooh I'm not an expert here but, As Per some SEO books and other websites, it's call spider that came to your site and takes data (means whatever you write). now, when people search word and if that same words or content you have on your site then google give your page into its list.
    – Arjun Paneri
    Apr 9, 2017 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.