Take the 2-minute tour ×
Webmasters Stack Exchange is a question and answer site for pro webmasters. It's 100% free, no registration required.

I'm going to be creating a website that allows users to upload and vote on each others' content (specific kinds of images), and I want a simple rating system in place.

I do not care about the users and examining their rating habits. I only want to make sure that 'good' content floats to the top, and 'poor' content floats to the bottom.

I was trying to determine what sort of rating system to use:

  1. A binary upvote/downvote system like StackExchange or Youtube.
  2. A 5-star or n-star rating system like Netflix, or the previous rating system on YouTube.
  3. An upvote-only ("like" but no "dislike") system like Digg or, I guess, Facebook.

My concern with a 5-point rating scale is that some users will tend to be 'extreme' voters, voting all 5s and 1s. However, I wonder that with a unary rating system, I won't have all of my users voting 'yes' to everything.

I know this is an extremely subjective question, so perhaps I should not be asking it. I am simply interested in reading a thoughtful and informative comparison on the pros and cons of these choices.

I've found one article which discounts the five-point rating system because they are often poorly weighted -- the average weighting being far too high to be useful:

Unfortunately, sites discovered that results from the 5-point scale weren’t meaningful. Across a wide range of applications, the majority of people people rated objects a “5″ – the average rating across many type of sites is 4.5 and higher.

share|improve this question
    
Given that these are all highly subjective items of information which are being rated, how would you ultimately determine an objectively better or worse system of rating? –  danlefree Nov 20 '10 at 3:01
add comment

1 Answer

up vote 4 down vote accepted

Define "effectiveness". If by "effective" you mean effective at capturing the community consensus, then they're all effective. If by "effectiveness" you mean the richness of the data returned, then using a point scale is going to be more "effective" than using a binary system. If by "effectiveness" you mean ease implementing a collaborative filtering system, then a binary system is preferable.

This question is too vague and it also depends on the type of rating (data) you are looking for and what is being rated. They each have their place.

The article you cited is concerned with poorly weighted point scale systems. Just because most people who rate a movie tend to be fans of the movie/genre doesn't mean the system is useless. That just means you need to scale/calibrate the rating system appropriately. If the median rating of all movies is 3.5 instead of 2.5, then you need to calibrate the rating system so that a 2.5 star is actually a 3.5 average. Similarly, you need to take the standard deviation into account and scale the star rating appropriately. If you do both of these things, then the precision of the 5 star rating system is very real and very valuable.

Another benefit of the point scale system is that it's not just a community consensus. You can actually see how much an individual likes or dislike a film and the rating distribution for different demographics. With a simple binary system, you lose all of that. You only know that X% of girls like this film, not the degree to which they like it. And even that may not be accurate since people who don't strongly like/dislike a film probably won't vote with such limited options.

With a unary system the data quality is even worse—there's no distinction between a movie that has very few votes because it's not well known versus a movie that has very few votes because it sucks and everyone hates it. A movie which is loved by 50% of the viewers could have the exact same rating as one which is loved by 100% of the viewers, and there'd be no way to tell them apart.

OTOH, there are all kinds of cool things you can only do with a point scale system, such as throwing away votes that are way outside the standard deviation or diluting the votes of users who consistently vote at one extreme or another. You can also make better recommendations based on more detailed individual voting data. E.g. not just group users based on which items they vote on, but also the relative ratings they give to movies: a user who gives the Boondock Saints a 5/10 and Basketball Diaries a 9/10 is going to have very different tastes from a user who gives Boondock Saints a 9/10 and Basketball Diaries 5/10.

So do you want to be able to see the degree to which an individual likes/dislikes a photo, or do you only want to collect data for collaborative filtering? Is the item being rated of enough qualitative nuance and diversity that users will even attempt to rate it properly on a 5-point scale (e.g. movies versus youtube videos). Even the unary system is useful if your primary goal is to maximize user participation or you're just measuring popularity (e.g. holding a vote).

On a side note, another alternative is a relative rating system in which users are shown 2 items simultaneously and pick which they prefer. There's some slightly more complicated math involved to rank items in this way, but it offers the simplicity (for users) of the regular binary system along with quality of data of a point system.

Lastly, there are all kinds of tweaks you can make to voting/rating systems to account for the psychology of voting and encourage/discourage certain behaviors. For example, research has shown that users vote very differently depending on whether a vote is "for keeps" or inconsequential. Similarly, if users have a limited number of votes (which they need to earn), they will vote more wisely. And you can force users to expend the same number of positive votes as negative votes to compensate for the tendency of users not voting at all for items they dislike. Similarly, if giving a movie 5 stars costs 5 voting points, you'll probably see less polarized rating patterns from users. And would users vote more diligently if their votes were broadcasted (e.g. on Facebook)?

share|improve this answer
    
An amazing answer. I appreciate the amount of thought you put into it. You're right - my question is far too vague. –  Pandincus Nov 20 '10 at 5:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.