Online five-star rating systems blur the lines between ‘good’ and ‘bad’ products

An easy-to-use online rating system, such as using a star rating without a written review, decreases the informativeness of the ratings and blurs the lines between ‘good’ and ‘bad’ products, finds new research from Frankfurt School of Finance and Management.

Online platforms such as Amazon, Google, and Glassdoor, use rating systems to indicate product quality to customers. In 2020, Amazon changed their rating policy, as critics said ratings were too high and that high paid ratings had become a problem. Previously, it was only possible to rate a product if the customer also provided a written review. However, it is now possible to submit a rating on Amazon by simply tapping a scale from one to five stars, requiring less effort from genuine customers.

Industry experts expected this to increase the number of authentic reviews, decreasing the influence of inauthentic “bought” 5-star reviews on the overall rating of a product. If this was true, then average ratings should decrease, and standard deviation increase – meaning that ratings for different products are less similar to each other – across products.

Matthias D. Mahlendorf, Professor of Managerial Accounting from Frankfurt School of Finance and Management, and Oliver Hegers from Amsterdam Business School, investigated how making rating easier has affected ratings and their informativeness.

They focused on the rating of books purchased through Amazon. To study the effect of the policy change on the informativeness of ratings, they investigate whether average ratings and standard deviation have increased or decreased across products. They obtained 70 million product-week observations for around 400,000 books, and for a control group not affected by the policy change, used the ratings of around 31,000 books from the website LibraryThing.com. 

They found that products were actually rated higher after the policy change – the average product held a 4.46-star rating versus 4.37 before the policy change – and there was a decrease in standard deviation across products. This means different products were rated more similarly to each other. Further analyses also confirmed that higher ratings for a product were associated with more sales. Specifically, there was a significant increase in sales from 3.9- to 4-star rating shifts, suggesting thresholds for differentiating between ‘good’ and ‘bad’ products.

Professor Mahlendorf says,

“Customers heavily rely on the 5-star rating system in online marketplaces, especially those where choice is extremely high, and there are little differences in prices. Especially as we lead up to Christmas, customers will rely on these systems even further, to distinguish between ‘good’ and ‘bad’ products. The current ratings system on sites like Amazon needs to be improved. Simply making rating easier is obviously not the solution.”

This research shows that making rating easier can cause the average rating to rise. This makes it increasingly difficult for Christmas shoppers to distinguish ‘good’ from ‘bad’ products. Given the results of this study, requiring reviews could improve the informativeness of rating consensus of other platforms where paid incentives for ratings also occur. Ensuring the reliability of customer satisfaction information is not only important to protect consumers, but also investors who may use ratings to price assets.