VG Tribune

Scoring “Reviews-In-Progress” Is Irresponsible

May 3, 2016 / 10:28 AM

By: Poorna Shankar

On Monday, IGN announced that they would begin scoring their reviews-in-progress. As soon as I read this, frustration and anger swelled inside me. This is irresponsible. This is wrong. This is not in the best interest of consumers. And none of this is helped by the current environment.

The environment of which we speak is this: the need to be first is prioritized more than the need to be right. When informing consumers is your responsibility, it’s clear that being right is more important than being first. But this is simply not the case in the current media environment.

However, when taking a holistic approach to this matter, scoring reviews-in-progress is simply the wrong decision for not only consumers, but for journalists and the industry as a whole.

Not following me? Fair enough. Let’s take a look at exactly what Dan Stapleton stated in the article.

First, let’s take a look at the good:

“We insist on giving every game we review a thorough playthrough, including testing on live servers when applicable, before rendering a final verdict.”

This is relatively straightforward, and is a genuinely pro-consumer attitude the media should take. Games should be reviewed in the same environment that real consumers will play them. That is to say, games should be played on live servers in real-life conditions. Why? These are the same conditions that consumers will experience, so it stands to reason to review the game in those same conditions.

Stapleton continues,

“The downside to this diligent approach is that in recent years more and more games have been dependent on multiplayer servers that aren’t available before launch, and some come in so hot off the presses that we don’t have time to play them to completion. More and more frequently we haven’t been able to have a score ready for you when you need it to help inform your day-one buying decision one way or the other, which has frustrated us and disappointed some of our community.”

Again, within the context of what is being said, this is a logical argument to make. Games have become more and more dependent on servers that aren’t available until launch. This naturally means the press has less time to review them and provide information to readers on day-one.

On the other hand, the media may actually have a long lead time. Halo 5: Guardians had a two week lead time with servers that performed in a highly controlled optimal environment. This environment is not indicative of real-world application.


Image via GameSpot

Following this logic, readers will become frustrated when this information isn’t available to them and when that information is not representative of the experience they will receive.

So the logic is there, but it’s nested in the illogic of the current environment. In other words, in this current illogical environment, the argument presented here makes logical sense.

Therefore, zooming out, we can see how the argument actually does not make sense. He discusses how games becoming unavailable to play until launch is a “downside”. One must immediately ask, why is this a downside? Don’t we want our media – the people who are supposed to inform us – to play and review games in the same conditions we will be?

The argument can be made that this puts stress and pressure on the media to get some coverage of the game out before their competition. Embargoes are designed to give all outlets an “even playing field” so that no one outlet will have an advantage over the other. Ironically, this simply adds to the stress in a race to not be left out, regardless of embargo.

However, stop and think for a second and ask yourself, does this benefit me? The answer is no, it does not benefit you. In an effort to “be first,” so often, mistakes are made which are then published. These mistakes are then consumed and taken as truth by the reader.

In fact, Stapleton cites the need to inform readers of their “day-one buying decision.” I have a feeling his heart is in the right place, but the argument isn’t fully thought through. Rather than trying to inform people of a day-one buying decision, media should focus instead on informing consumers to wait for complete information.

In other words, media should stop encouraging day-one buying decisions and instead encourage consumers to be more knowledgeable about a game before spending real money on it. Media should encourage consumers to do their research and not make a decision based on an incomplete day-one reactionary opinion.

Again, I understand why IGN is making this decision. They cite reader feedback as the driving force. However, readers don’t always know what’s best for them. If they did, why would they constantly seek out reviews and other information?

It’s the function and responsibility of the media to educate and inform the otherwise uninformed readers. Therefore, appeasing to readers and saying, “OK, we’ll give you what you want” isn’t in the best interest of readers. Not informing readers is not just irresponsible, it’s also dangerous. Media must educate consumers. It’s their duty.

Media like IGN should know better.

There are a few suggestions I can make here, not least of which is this: don’t score reviews. Consumers place far too much importance on a single abstract number meant to boil down all the nuances of a game. That score, that single number, can not convey everything about the the game.

That single number simply isn’t enough information. A single metric to describe a complex multifaceted game isn’t logical.

By scoring unfinished reviews, you are implicitly telling your readers that it’s OK to treat these unfinished reviews as finished ones. This is exactly how readers will treat these scored unfinished reviews.

Consumers are going to place even greater importance on these unfinished reviews even more so to make purchasing decisions. By scoring unfinished reviews, you are simply enabling that behavior.

I’ve always said – and will continue to say – the media’s responsibility is to readers and consumers. And this means giving those consumers all necessary information.

All. Necessary. Information.

Scoring unfinished reviews is not giving readers all the necessary information – and IGN knows this (emphasis added):

“More often than not we expect these temporary scores to stick, or at least to be very close to the final verdict. But sometimes the final score will be different; circumstances where we’d see a significantly altered score would be a game that functioned well on pre-release servers but fails at launch, or a game of epic length that arrives too late for us to complete before the review embargo lifts and the ending ends up either thrilling or disappointing us. Conceivably it could even change more than once, such as with an MMORPG that requires weeks of play to review.”

Let’s examine this quote and dissect it using a hypothetical – but still realistic – scenario. IGN has a review-in-progress, and part of the game they’ve played so far is awesome. Therefore, for this review-in-progress, they score it a 9/10.

However, since this is “in progress,” they haven’t completed the game yet, therefore this 9/10 score does not take into account the end of the game. Thus, we can conceivably imagine a scenario where the end of the game is horrendous compared to the beginning of the game. It could have major bugs, or it could simply be boring.

For example, Mass Effect 3 was considered a brilliant game by many, right up until the ending. Fans and media alike were disappointed, many feeling this marred the whole game. A game like this is a clear example of a game which could suffer from scored reviews-in-progress.


Image via Engadget

Whatever the case, the ending could drag down the entire game — however this isn’t reflected in the “day-one” score IGN provides their readers. At this point, it’s already too late. The damage has been done. Their readers didn’t know the end of the game was horrendous and have already made a purchasing decision based on incomplete information.

This is wrong. This does not serve the best interest of consumers. This speaks to the larger issue among the media where being first is more important than being right. And it is within this illogical environment where IGN’s decision to score reviews-in-progress makes logical sense.

But in the real world, where real consumers will spend real money, the media must educate these consumers who simply aren’t aware of such issues. Appeasing these uninformed consumers without looking at the bigger picture is simply irresponsible. It is a failure on the part of IGN.

The media must educate its consumers. They must understand the bigger picture. Not doing so is simply dangerous. Consumers receive incomplete information. And they make actual purchasing decisions based on this incomplete information. This incomplete information is because of decisions to score reviews-in-progress. Scoring reviews-in-progress is simply wrong.  

In the end, consumers lose.

About the author /

A highly opinionated avid PC gamer, Poorna blindly panics with his friends in various multiplayer games, much to the detriment of his team. Constantly questioning industry practices and a passion for technological progress drive his love for the video game industry. He pulls no punches and tells it like he sees it.

Related Articles

Post your comments

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.