How User Ratings Became Worthless on the Internet

There was a minor controversy early this week when it was revealed that Amazon was delaying user reviews of The Lord of the Rings: The Rings of Power in order to weed out trolls.

It is worth acknowledging up front that this is inherently self-serving. Amazon invested $715 million into the first season of The Rings of Power, and reports suggest that the show will be make-or-break for the company as a streaming service. Amazon’s primary motivation here isn’t to protect the cast and crew members being targeted for racist abuse; it is to protect its own financial investment. It is unlikely it would provide the same coverage for another lower-profile show from another studio.

It is also worth acknowledging that this is not an infringement of freedom of speech. People have the right to freedom of speech, but they don’t have the right to a platform or even the right to be heard by the public at large. Amazon’s user ratings are a service provided by a private enterprise, not a town square. As such, they are not subject to these sorts of protections, any more than being banned or blocked on social media can be said to impugn an individual’s freedom of speech.

However, setting aside these basic but necessary acknowledgements, this decision underscores something increasingly obvious to even the most passive of observers: User scores on the internet are inherently worthless. In many ways, this is a microcosm of the broader issues playing out across the internet. The world wide web was initially conceived as a virtual space that was truly egalitarian, that gave everybody a voice in the hope that the cream might rise to the top.

Theoretically, the internet abolished the gatekeeping of old media. Now, a young movie fan in Ohio could publish their own opinion just as easily as a journalist in New York. Institutional barriers to entry were abolished. In 2001, in Kevin Smith’s Jay and Silent Bob Strike Back, Holden McNeil (Ben Affleck) summarized the utopian ideal of the internet as “a communication tool used the world over where people can come together to bitch about movies and share pornography with one another.”

It is debatable to what extent user review metrics were ever reflective of general audience tastes. These polls are anonymous and self-selecting, and so they are only reflective of the opinions of those who vote in anonymous and self-selecting polls. This is obvious even looking at the demographics involved. Voters on IMDb are disproportionately male, particularly considering that 51% of the cinema-going audience is female.

There are other procedural concerns baked into these metrics. On most of these sites, all it takes to vote is an account, and it is possible for a single user to inflate their vote by creating multiple accounts tied to different addresses. There is rarely any validation to ensure that the user in question has actually seen the film that they are rating. In most cases, the only requirement is that the person voting must want to rate it. It is hardly a foolproof system.

It’s difficult to gauge how the general audience feels about a movie. CinemaScore is a much better metric than online polling, if only because it is conducted using a random sampling of actual audiences with something approaching a scientific methodology. Even then, the audience is self-selecting people who actually went to the theater on opening weekend in Los Angeles to see the movie. It’s more reliable and verifiable than online polling, but it’s still only a subset of the audience.

It is quite rare for magazines or polling organizations to commission actual research into audience opinions. The Hollywood Reporter did run a poll on the Game of Thrones finale and found, contrary to online outrage, that most viewers had no real problem with it. In the United Kingdom, YouGov polls periodically check the public’s opinion on the Star Wars and Star Trek movies, often contrasting with “accepted wisdom” online. It is a reminder that the internet is not real life.

Still, while there were very obvious structural issues with these sorts of user ratings, they were a handy enough tool to have while they were used in good faith. At the very least, a casual browser could assume that the majority of the self-selecting user base voting had seen the movie and were responding to it on its own merits. These ratings were also a handy way to drive traffic. Clicks and engagement go hand in hand in the attention economy, and user ratings generate both.

Much like the internet itself, this model found itself tested when bad-faith actors discovered that they could weaponize these metrics as part of campaigns that had little to do with the quality of the films themselves. In 2014, Indian movie Gunday was targeted for downvoting on IMDb by Bangladeshi nationalists. In October 2016, The Promise was downvoted by over 86,000 IMDb users after only three public screenings, likely due to its acknowledgement of the Armenian Genocide.

In recent years, that problem entered the mainstream blockbuster discourse. In December 2017, observers noted curious behavior around user scores on sites like Rotten Tomatoes for The Last Jedi. In February 2018, something similar happened around the release of Black Panther. In both cases, these blockbuster films in populist franchises ended up being folded into a larger culture war. These arguments weren’t about Star Wars or Marvel movies; they were about society itself.

Historically, IMDb and Rotten Tomatoes operated by a logic similar to that of social media giants like Twitter or Facebook, avoiding responsibility for how their users behaved. They were platforms, not publishers. However, like how Twitter and Facebook eventually had to take responsibility for the abuse of their networks, IMDb and Rotten Tomatoes had to tidy their own houses. IMDb closed its forums in February 2017, admitting they were “no longer providing a positive, useful experience.”

In February 2019 — coinciding with the release of Captain Marvel — Rotten Tomatoes announced that, due to an “uptick in non-constructive input”, it would no longer allow users to leave comments on movies before release. Paul Yanover, president of Fandango, described the move as necessary “noise reduction.” Three months later, the company introduced verification for users who had seen what they were reviewing, to protect the site’s reputation as “a trusted tool” against “bad actors.”

Obviously, everybody is entitled to their opinion. More than that, taste is subjective and mileage varies. However, it seems safe to question the validity of any metric that fails to capture the universal acclaim of a movie like Black Panther — with its A+ CinemaScore, its critical raves, its Best Picture nomination, its box office and home media success, and its level of cultural impact. Of course not everybody likes Black Panther, but — taken as a group — moviegoers adored Black Panther.

Similarly, there is something more than a little bit suspicious about Captain Marvel receiving 58,000 user reviews by 8 a.m. on the morning of release, more than Avengers: Infinity War received over the course of its entire theatrical run. Anybody with any grasp of statistics and user behavior would understand that the resulting metric was not to be trusted and would be equally wary of any user score generated even after Rotten Tomatoes reset the count.

So much of what is terrible about the modern internet is rooted in bad-faith actors discovering that they can weaponize the public nature of the web. Public platforms are a means of spectacle, with systemic weaknesses to be exploited as part of cynical promotional strategies. These ratings are no longer about providing a weary internet user with recommendations for particular movies or shows. They are now a theater in the culture war, allowing these actors to play to the gallery.

Anonymous self-selecting user scores are easily gameable metrics, gateways through which insidious ideologies can worm their way into mainstream conversation. They turn nonsense culture war points into free advertising. How much of the discussion around The Rings of Power has had to delve into the weirdly insistent racism held by an extremely vocal minority of the audience? These voices are trapped in their YouTube echo chambers but are hoping to get their siren call into the mainstream.

There is ample evidence that deplatforming works. There’s a reason that Donald Trump is so desperate to get back on Twitter. If you deprive these bad actors of oxygen, they will skulk back to the shadows. While many readers might be too young to remember, pop culture didn’t use to be like this. Even in the early days of the internet, these extremist voices just weren’t part of the conversation.

Peter Jackson’s The Lord of the Rings trilogy could bulk up its female parts and have Éowyn (Miranda Otto) cite her feminist bona fides before impaling the Witch-king (Lawrence Makoare) through the face without being called “woke” or “a Mary Sue.” There are interesting things to talk about with The Rings of Power, including the extent to which it may or may be a spiritual adaptation of Cats, but all of it gets drowned out in this bad-faith nonsense.

Even leaving aside these more overt campaigns, modern fandom has become increasingly zealous. Rival fandoms will vote up the objects of their affection and vote down the targets of their ire. The most recent major blockbuster to appear on IMDb’s Bottom 100 is Catwoman from 2003. Devoted fans work hard to keep movies like Terminator: Genisys, Joss Whedon’s Justice League, and Jurassic World Dominion from ever ranking too low.

Much is made of how there is a bigger gulf than ever between critics and audiences, but these arguments are often anchored in these easily manipulated metrics. In reality, it seems likely that both audiences and critics would agree that Top Gun: Maverick is the blockbuster of the summer. There is something to be said for trusting qualified expertise, particularly in turbulent and uncertain times. Critics may not be perfect, but they are generally more trustworthy.

Sites like IMDb and Rotten Tomatoes are unlikely to ever do away with user ratings. They drive too much traffic and engagement. However, it is good to see companies like Amazon acknowledging the way in which these systems can be exploited and manipulated, even if they are acting in their own self-interest. In the meantime, these metrics are easily gamed by those looking to settle scores.