4

I have a website with a lot of user profiles (like LinkedIn or the like). I need Google to index each public user profile page. I have a few things on my mind:

1- There is no user listing on my page, and also I don't want to show all profile pages listed on the website. How can Google still index all profile pages?

What should I do?

| improve this question | |
  • 1
    Don't really follow what you are on about but if you want pages indexed, then link to them so Google can find them... – zigojacko Apr 3 '15 at 9:21
  • I feel like i see this question asked weekly: "How to get Gbot to index my huge amount of city/forum/user pages". Seems like this should be flagged duplicate and un-bully all the others that are constantly flagged for obscure reasons :) – dhaupin Apr 3 '15 at 14:06
  • See also: Make Google index over 90,000 user profiles which addresses the situation for an order of magnitude less. – Stephen Ostermiller♦ Apr 3 '15 at 15:22
7

Google is only going to index millions of pages from a site with lots of reputation. I'd estimate that your site would have to have at least PageRank 6 before Google were willing to index that many pages on your site. That is especially true when the pages are only listed in a sitemap (as tillinberlin suggests).

User profile pages are often low quality. On most of my sites, many users tend to sign up without creating any activity. In many cases I have little more for them than an email address and password. A page for such a user would not be useful in search engines. Furthermore, having lots of low quality pages on your site indexed can cause Google's Panda algorithm to identify your entire site as low quality and penalize your entire site.

I would start with trying to get the 10,000 user pages that have the most content indexed first. If that works well, expand it to 20,000 then 40,000, then 80,000. Stop when you are having trouble getting the pages indexed, or when there isn't enough content on the pages.

| improve this answer | |
  • 1
    Agreed. Keep in mind, Google's goal is to serve searchers, not webmasters. If pages on your site are likely to be of limited interest to the general public (i.e. never rank in the top ten on any search), then Google has little incentive to index them, and is likely skipping them deliberately. Add useful content to each page (or allow users to do so) and you may find more of them being indexed. – Charles Jaimet Nov 26 '16 at 19:00
2

I would suggest you use a sitemap – if you don't use one already:

  • You can list large number of individual pages.

  • Google (and other search engines) do like to crawl sitemaps.

  • Pages listed on sitemaps don't have to be 'exposed' on the website – so you don't have to have a page listing all user profiles.

  • Building sitemaps can be done programmatically – either you write your own script, or you can use one of many free services to write the actual sitemal file (although I think most free services are restricted to some thousands of pages listed).

This page "Manage your sitemap" at support.google.com/webmasters can be a good starting point. However – using sitemaps doesn't guarantee you anything – and getting google to index literally millions of user pages through sitemaps actually sounds close to impossible for various reasons (as also Stephen Ostermiller points out).

| improve this answer | |
  • 2
    This may work some, but generally not well. See The Sitemap Paradox. Stack Exchange found that they had to link to questions on the site to get them indexed. Just putting them in a sitemap is not enough. – Stephen Ostermiller♦ Apr 3 '15 at 10:25
  • Work adding that even with links and sitemaps Google still won't index all user pages if the domain authority is low, or the user pages are thin. – Simon Hayter♦ Apr 3 '15 at 10:35
0

you will surely penalized by Google panda, If you are adding profile pages into sitemap, and you also be penalize by Google Penguin if you have no control on outbound links. So Don't forgot to learn about nofollow attributes.

In my real advice, Don't add every profile url into sitemap, simple disallow your /profile directory from robots.txt

Just look out facebook.com/robots.txt there are many things are blocked from spider, and hence Google will not index all the items, also Google will not crawl that page directly from Facebook.

Now suppose user linked the Facebook profile on own site, Then Google will crawl that page from user site, not from Facebook site, because it is blocked by Facebook robots.txt. Now in reality Facebook blocking profile page to index, But some of people mention Facebook profile on web pages, hence Google index that pages on search result. Just look out Facebook Home Pagerank, It is 9 and just look out someone profile pagerank is 0. I hope you got it me.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.