2

I'm developing a social website for book readers, with public user profile pages. For each user, there are several pages available:

  • The main user page (about me, last activity, ...)
  • Several book listing pages:
    • Personal library
    • Wishlist
    • Reading list
    • ... and more

I already have close to 10,000 registered users, many of which have little or no activity.

I know that Google cannot index millions of user profile pages from day one, and I don't want it to index useless pages that basically contain the user name and "This user has no activity". Google is not willing to index many of my pages at the moment, and I would like to be able to give it a hint as to which pages are relevant.

Should I explicitly prevent Google from indexing the empty profile pages? I was thinking of a noindex, follow robots meta tag, that would basically tell Google that it's OK to grab links from this page, but that its content is of little value.

I know Google will not magically find empty profile pages if they're not linked from anywhere (and I won't make the mistake to put them in a sitemap); however I'm more concerned about the "almost empty" profiles: someone writes a single review, his profile page is linked from the book page, and suddenly GoogleBot finds his almost empty profile page, and his fully empty book listing pages. I don't want it to index those until they contain some content.

Is it a good idea to put some lower limit on the page contents (for example, a personal library page with at least 10 books, or a profile page with a long enough About Me and some activity to display), and only explicitly allow bots to index these pages?

| improve this question | |
  • If there are only few profiles then it is fine to not to use noindex meta tags, 5 low quality pages and 500 quality pages on user generated website is totally fine. – Goyllo Jun 27 '17 at 18:31
  • It's quite the opposite, actually. There are many (tens of thousands) of low quality pages, for a few hundreds high quality ones. – Benjamin Jun 27 '17 at 19:49
  • Then I would go for noindex meta tags. And It will be great if you can do programming and add noindex meta tags only on low quality pages. Like profile have no activity. Or Profile have empty bio details. – Goyllo Jun 28 '17 at 7:47
  • The meta tag would definitely be programmatically generated based on the page contents! – Benjamin Jun 28 '17 at 10:46
1

You can deal with this in different ways.

1. Using robots.txt

Let's say your website is: example.com

And user profile is:

Structure 1: Well designed Profile

 - example.com/user/user1 => Main Profile
 - example.com/user/user1/other-page => Other Page

Structure 2: Basic Profile

You can create different URL structure like:

- example.com/minuser/user1 => Basic Profile 
- example.com/minuser/user1/other-page => Basic Other Page

You can create condition, for example: If user got 5 things ticked then have structure 1 else structure 2 (Not recommended to calculate at run time).

Note: While designing the algoritm you need to think about many things.

BLOCK: You can block /minuser using robots.txt.

2. Using noindex meta tag

As discussed above, you can try this but may not look clean to user.

Warning

Worth reading about noindex vs robots.txt block issue here: https://support.google.com/webmasters/answer/93710?hl=en

| improve this answer | |
  • I would not go for a different URL structure for "basic" and "well designed" profiles, as Cool URIs don't change; with your approach, when someone finally fills in their profile, their profile URL would change. I know quite well the subtle differences between robot.txt and the robots meta-tag, and I think that if I follow this approach, the meta tag is superior anyway (Google is allowed to visit the page, but asked not to index it). The question really boils down to "do I explicitly block these pages or not?" – Benjamin Jun 28 '17 at 10:43
  • When you say that "using noindex meta tag (...) may not look clean to user", what do you mean? How will the user ever be affected by a meta tag? – Benjamin Jun 28 '17 at 10:44
  • 1
    URL Change: That should be OK as old URL can redirect to the new one. It's absolutely OK to change url if needed. What is important is how it is handled. There are 10 reasons why a URL can change as the web is changing. Just imagine what has happened in last 2 years, people are switching to HTTPS.... So, think ahead and implement a proper solution. Cool URL: Not relevant these days i guess. – TopQnA Jun 28 '17 at 12:29
  • Clean to User: What I meant is, the user wouldn't know about noindex, but having clear different URL structure, some may notice. And you can also encourage people to have structure 1 url. See what Google did, G+ came with user ID and based on the quality of profile Google offered custom URL. – TopQnA Jun 28 '17 at 12:36
  • 1
    Google can crawl NoIndex pages as like normal webpages, it just didn't show up in search result. Google provide custom username to only those users who really use their platform, so good username can available for others. – Goyllo Jun 28 '17 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.