The Parthenon in Athens, Greece. Photo courtesy of wallyg.

Are reviews primarily a consumer guide, or should they serve another purpose? Do review scores deter intelligent discussion of videogames? Is the presence or absence of a review score the only difference between a reviewer and a critic? What is the role of the reviewer when the Internet is democratizing published opinion? How should reviews and reviewers evolve in light of the emergence and growth of Flash games, small games, indie games and user-generated games?

These questions and more were on the mind of N'Gai Croal, John Davison and Shawn Elliott last summer when they decided to expand their conversation to a number of noted reviewers, writers, bloggers and reporters for a published email symposium on game reviews. (See below for the full list of participants.) The planned list of topics include Review Scores; Review Policy, Practice and Ethics; Reader Backlash; Reviews in the Age of Social media; Reviews in the Mainstream Media; Casual, Indie, and User-Generated Games; Reviews vs. Criticism; and Evolving the Review.

The topic for Round 1, which will be published here in installments over the next several days, is Review Scores. Previously, we published Part I, Part II and Part III; today, we conclude the Review Scores portion of our symposium with Part IV.

Participants

• Leigh Alexander, Gamasutra/Sexy Videogameland/Variety
• Harry Allen, Media Assassin
• Robert Ashley, freelancer
• Tom Chick, freelancer
• N'Gai Croal, Level Up/Newsweek
• John Davison, What They Play
• Shawn Elliott, 2K Boston
• Jeff Gerstmann, Giant Bomb
• Kieron Gillen, Rock, Paper, Shotgun
• Dan Hsu, Sore Thumbs Blog
• Francesca Reyes, Official Xbox Magazine
• Stephen Totilo, MTV News

***

Shawn Elliott, 2K Boston: To finish this section and finally return to Stephen's point that scores, despite 'damaging the discourse about games' and 'obscuring the value of words,' aren't an actual problem, I'd like to ask one last question.

Review writing carries real consequence because some publishers do base developer and PR bonus pay on aggregated ratings. This shouldn't concern critics, but once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background--especially among members of the enthusiast press--or even enter our thoughts as we write or edit reviews and assign reviewers or scores? The stock answer says, "Only if you're a bad apple, and I'm not." But isn't the seeming impropriety of business in a bad barrel a problem in itself?

Consider special instances such as the Gamespot Kane and Lynch review that N'Gai cites. Jeff, are you contractually able to discuss that episode in any detail? John, Francesca, and Dan, as well, serve/served as EICs of enthusiast publications and presumably face/faced such pressures and repercussions.

***

Dan "Shoe" Hsu, Sore Thumbs: Exclusive reviews stink...it’s a lose-lose situation. If you score high, then readers call shenanigans...even if there were none. If you score low, then the game makers get mad and won’t want to work with you in the future.

Plus, most companies want a guaranteed score or range of scores. We did play with this fire once at EGM...the very first time I was faced with it. I thought I could deal with it by protecting the reviewers from that discussion--I’d let them review the game independently and then see if the scores  were high enough for us to secure that exclusive. But because I had final say on all magazine content, I’m still a part of the reviews process. In the end, I didn’t feel comfortable promising any certain scores, so I backed off and made a rule to never entertain these offers again. If we don’t even have those discussions to begin with, then we’ll have preserved the integrity of the reviews from start to finish.

Besides...if the reviewer caught wind of the deal, how can it not spoil things, however so slightly? Let’s say the reviewer’s 50-50 wavering between an 8.5 and a 9.0, but way back in the deep recesses of his mind, he knows that a 9.0 will get his website or magazine the exclusive review, which translates into revenue-producing traffic or sales. Will that load the dice? Maybe not for everyone, but that’s not the point. Once that process is tainted, it’s tainted. Just ask the reader what he or she thinks.

***

Leigh Alexander, Gamasutra/Sexy Videogameland/Variety: I think in Jeff Gerstmann's case, the fact that the editors took liberty with the final text and scores are evidence of the "Rolling Stone System" not working in practice--again, I support the idea of editors protecting their writers (as Shoe tried to do with his folks, it seems), and doing whatever they can do to keep the heat of publisher response to scores out of the way so that the writer can do his best work. The "protection" obviously didn't take place with Jeff, and while I've no inside knowledge of that situation it sure looks to me like they let him take a bullet, which is at the very least evidence that making any kind of bargain with a publisher regarding a review--its exclusivity or otherwise--is a BS proposition.

I hope that this reason, coupled with the evolution of Internet media, means that exclusive reviews are going to go the way of the dinosaur (or of the console exclusive, har har). I wholly believe we now work in an environment where quality and depth can be the primary competitive advantage, and less timeliness.

I also can't speak for Totilo, and don't mean to put words in his mouth, but when I agreed heartily with his response, I was agreeing with the idea that scores only cause problems for the industry and for us--neither of whom is the audience we serve, and therefore that they create stress is not a compelling argument against the fact that plenty of readers find them useful. Hell, I hate scoring as much as most of us do, and I prefer scoreless criticism and blah blah blah--but in all honesty, when I really want to know whether I should pay attention to a game, the very first thing I do is go to GameSpot and see what number they gave it, for whatever it's worth.

I never buy based on a number (let's take bets on how many--or how few--year-end top ten lists the 10-rated MGS4 or GTA IV get), and I don't think readers do either, which further takes the piss out of this idea of their damages. Believe me, I'd love to stop scoring. I'd love if we could enforce an industry-wide moratorium on scores, so that we wouldn't have to think about 'em and so that our audience could re-learn to focus on the words we write.

But I think we hate scores because of the undue importance that has been placed on them and the ways they've distracted from our work, and this is credited in part to editors balancing the needs of the publishers with their management of irrationally hostile reader reactions, neither of which should be their focus. None of this means they in and of themselves cause harm to the readership or to the art of the review.

***

Shawn Elliott, 2K Boston: Dan, do you see where this can apply to cases less extreme than exclusive reviews? Let's say that a publisher puts one game in the market this month and is also looking for an outlet to announce a forthcoming game a few months from now. No conspiratorial conversations or conniving occurs. Nonetheless, you have to know that by giving the available game a negative review, you could risk reducing the likelihood that the company will allow you to reveal their other title. Scruples aside, the cost-benefit equation is simple: A first look at the unannouced game is probably better for business than the bad review, provided nobody calls bulls--t. And comprise comes in shades. An EIC can always assign the review to a specific staff writer who he suspects will appreciate the game more than his peers, and argue afterwards that he hadn't touched the score itself. I have no idea whether or not this happens. I do, however, know that commercial publishing is cutthroat.

***

N’Gai Croal, Level Up/Newsweek: From the outside looking in, I’ve been conditioned to expect that review cover = exclusive review = 8 or above. That led to a bizarre reading experience last holiday when I picked up a copy of Official Xbox Magazine with Turok on the cover, touting the first review. I can’t lie, I immediately flipped to the end of the review to see what score had been handed out…only to be shocked to see that it had been tagged with a 7! What went through my mind next was:

a) Damn, that took guts for Francesca to stick to her guns and let reviewer Paul Curthoys call it like he saw it.

b) there goes Christmas for Josh Holmes and the rest of the guys over at Propaganda Games, to say nothing of Graham Hopper and the folks at Disney Interactive.

c) I would not want to be the publicist who brokered that exclusive. Her/his job is to suss out the most favorable outlet when determining who’s going to get that first review, because you only get one chance to make a first impression. But a first review score of 7 means that something went terribly wrong--and from PR and marketing’s perspective, the “something went terribly wrong” in this kind of situation is /not /the finished game’s quality.

At last glance, the Metacritic rating for the Xbox 360 version of Turok was 69, so OXM’s review was right in line with the critical consensus as calculated by that aggregation engine. I’m sure that was little consolation for Propaganda and Disney, though.

John, didn't you write about the pressure surrounding review exclusives when you first started blogging?

***

Dan "Shoe" Hsu, Sore Thumbs: Shawn, that’s why this business is so messed up! Publishers want good reviews. Editors want exclusives. Magazines and websites want advertising. Advertisers want good reviews.

Sometimes, the companies know their products aren’t that great. They have internal research (via their own gut checks or “mock reviews”--early evaluations done by outside consultants) that give them a rough idea of how a game may score. If you’re dealing with a reasonable PR team and you’ve scored within their expected range, then you’re usually spared the grief. But woe is the outlet who’s at the bottom of the Game Rankings or Metacritic lists. We’ve had PR people complain to us that we’re “outside the average” (on the low end) on those sites, which John and I would always laugh at. Of course someone has to be outside of the average. If everyone was ON the average, there’d be no point in averaging them!

***

Jeff Gerstmann, Giant Bomb: Exclusives are yet another by-product of this business' print roots. Between magazines stories getting scanned, website text getting pasted into message boards, and the way that most of a site's traffic doesn't come in through the front door (where the big promotion of the exclusive always is), exclusive stories are waaaaay more trouble than they're worth. Exclusive reviews doubly so. On top of that, in my experience, most big exclusives don't really move the needle that much, traffic-wise. They've always felt like a huge waste of time to me. Attracting users by differentiating your coverage from the pack has always seemed like a better goal.

But are we talking about a problem that doesn't really exist anymore? Isn't IGN the only publication still doing exclusive reviews? And if so, they're probably getting them because they're the only ones asking for them, right? I think they've deflected questions about their integrity with the ol' "well, we're not on the take, case closed" routine, which the majority of the public seems completely fine with.

***

Stephen Totilo, MTV News: Many of you were struggling to make a decent argument for the elimination of review scores. Here, I think you just did. Don't put scores on your exclusive reviews in your magazines. Add them via your website two weeks later when everyone else's reviews run. Then everyone's happy.

Or…just don't take exclusive reviews. This would be a challenge for a magazine that needs to secure something notable months in advance of press time. But for websites, is there really a need? You can get the game when it comes out, play through it and have a review up within a week of it's release. No tricky politics. No stink of compromised values. And your readers are served.

The threat of publishers withholding cooperation in the future because of a review strikes me as absurd. I don't doubt that it's a real threat, but there are countless ways to report and edit around any blackballing. Besides, getting blackballed usually makes for a good story.

***

Robert Ashley, freelancer: Just to add to the list of reasons why reviews create editorial drama, it's my understanding that retailers take into account review scores when stocking new games, which means that scores can directly affect sales. I had a PR rep from Deep Silver tell me last week that Target bases their game orders exclusively on Gamespot scores (so, of course, he was bitching about all the snooty divas at Gamespot). I can't vouch for his claim, but just having the guy go off in a room full of other pro enthusiasts noticeably changed the mood of everyone in earshot.

While reviewing for EGM, I felt like there was always barrier between me and PR, thanks to the reviews editor. But when I would venture out to write other stories, visiting studios and going to game reveal events, I would often get snippy comments from people about my reviews. I was once accosted by a developer from the True Crime: New York team about my review of his game. We had, by chance, ended up in the same two-person sleeper car on a train from Paris to Venice. And I was on a family vacation, not some gaming press junket.

***

Dan "Shoe" Hsu, Sore Thumbs: Stephen, getting that first review up the weekend before a game’s out versus a week after launch can be a huge, huge difference in traffic. So most website editors would find an exclusive review very appealing. Can’t blame them.

Those threats from publishers can come in many different forms. Sometimes, it’s just one very specific team or division that wants to blacklist you. For example, we were supposed to get an online exclusive as part of a package deal with our Saints Row 2 cover story, but after they read Robert’s article, THQ told us that the developers didn’t want to work with us anymore. So that online exclusive went to someone else instead.

The Mortal Kombat team and Sony’s sports division banned us as well. But the interesting thing is, these three blacklistings didn’t carry over to the rest of THQ, Midway, or Sony Computer Entertainment. They were very specific to those specific products, because they felt we had it out for them.

Then there are situations where the blacklistings go company-wide, like with Ubisoft after our Assassin’s Creed reviews...and after we wanted to do a story about how outlets were allowed to break the universal AC reviews embargo if they scored the game high enough. That “non-cooperation” on our part was the last straw for Ubi, and when I reported on this blacklisting in EGM, it just further cemented their hatred for me as an individual (even though Sam Kennedy, Patrick Klepek, and just about every other editor wanted to run those stories as well).

Now, I don’t really think Ubisoft or any of those other companies have done anything wrong here. It’s their right to work with whomever they want. But it just goes to show how uncomfortably and inappropriately cozy the industry and enthusiast press are expected to be. You cooperate, you benefit. Simple as that.

***

Kieron Gillen, Rock Paper Shotgun: *Yeah, Shawn, regarding exclusives, on the UK side it's pretty similar. On the print side, perhaps even more intensely. There's just more magazines and the competition has always been fiercer. The former is primarily due to the size of the country - the shipping costs don't cripple you, and a smaller magazine can be more profitable. The second is due to how British games magazines are sold. The vast majority of magazines are sold at newstand, not subscription. Subscriptions are cheaper, but not a straight loss leader for ads. When I was on Gamer, the majority of our money came from actual people buying the bloody thing rather than via advertisers. On the bright side, this abstractly means that in a real way, the bills are paid by people who want to read your mag. On the bad side, since there's an enormous floating readership which you have to fight for every single issue, meaning even more importance is placed on exclusives. An enormous amount of effort is thrown into chasing them, and it can totally lead to the sort of issues others have picked up on.

When I'm chatting to modern day magazine staff, one of my favorite stories to tell is about an old pre-me Future major-mag editor. He didn't speak to any PRs. The Dep Ed did all of that. If any of them tried to speak to him, he just blanked them. He just made the magazine. And when I roll out that anecdote, the look of disbelief which it's always greeted with is akin to what I'd have got if I claimed he was capable of flight.

As I said, they're an enormous waste of energy--I always remember that it seemed that due to the inevitable pissing off of PRs, the exclusives we chased went to our major rival as much as not, and the ones that they went for went to us. But it's also a game which I think magazines--and publishers--are loathe to get out of it. Because as long as one other organ in the room is making exclusives, you are at a distinct disadvantage. And it works both ways. One major games publisher, working on similar logic to me above, stopped doing exclusives for a few year. The amount of coverage suffered--because their rivals were still making deals and they weren't--and they went back on it.

That's the problem with stopping exclusives. It requires a conspiracy of doves for it to hold.

Regarding changing marks and having marks changed...much like others have said, I've done it. It's really not a problem per se. Discussing it with the writer is fine (And an art form in and of itself). Explaining how they're not marking to your mark-scheme is fine too. And most commonly, the review and score just don't match up, where you have to ask that one or the other is changed. If a reviewer is having fun comparing part of the game's mechanics to cancer and still give it in the sixties, either the overwrought writing goes or--if they actually mean the overwroughtness--the mark needs to drop like a stone. In my experience, I've argued far more marks down than up.

On the other side, I'm fine with that too. Sometimes I really don't agree, in which case I ask for my name to be removed from the review. Not in a prima-donna way--just because I don't believe in it, and ultimately, your name is all you've got and you're going to give enough review scores the readers will tear you apart on your lonesome without someone else making them for you. This is one of the first things I say to any writer getting into the business, because--bless 'em--not many are even aware that it's something they can ask to do. And if the editor says no, it's about as good a sign that you should get out of that job sharpish.

And, of course, some editors do change marks without asking and keep your name on it. Just don't work for them again.

***

Harry Allen, Media Assassin: Greetings, everyone.

I just want to say that it's really a privilege to be able to read all of your thoughts. I'm a fan of you all. I read your words and listen to your podcasts, in many cases. So, it's great for me to hear your thoughtfully written ideas about this very trenchant aspect of your work.

I don't know that I have anything to add. Much of what I was thinking has been said by other people, and I know we're about to change topics. For example, a very early concern of mine was the relationship between scores and advertising, so I'm glad that some of that was addressed recently.

I will say that one of the first statements I read that directly connected to my own work was N'Gai's comparison of the reviewer and the critic. Though I use different language, I know that when I started writing about hip-hop professionally, in the late '80s, I made it my objective to never talk about an album in terms of whether I liked it or not. Instead, I always saw it as my job to explain the artist's intent to the readers. I've never called myself a "music critic." I've always said that I am a Hip-Hop Activist.

To me, number scores are mostly an attempt at giving the illusion of numerical precision to functions that cannot possibly bear the same.

For example, take a look at Robert Motherwell's 1961 painting, "Elegy to the Spanish Republic, 70," here:

Is that a 10? An 8.5?

Most curators and art historians would call the bold canvas "a masterpiece," and leave it, more or less, at that. If they did quantify it, or one of the others in the series, they would only do so in extremely hushed whispers, among themselves, if at all, but never do so for public consumption. This is actually part of the way that they frame their field, which is commercial, but which is mostly a repository of serious, high culture: By avoiding quantifying numbers ("It's an 8"), which would be seen as crass.

Quantifying a Motherwell, then, would be analogous to asking how much the Maybach is: If you have to ask, you can't afford it. They're not saying you can't afford the Motherwell financially, but you "can't afford it" aesthetically. In other words, the experience of the painting is far too rich to reduce to a digit, and if you don't know that, you shouldn't be here.

I think a reason similar to this is also why I always resisted, during that brief period of my life, when male friends would ask me to assign a number to a girl I've seen. In hip-hop / Black slang, a "dime" is a girl who's a 10...but what does that mean? According to what objective scale?

And indeed, isn't that the core idea that disproves the fantasy: That without an actual 10 to which one can point--the theoretical perfect game--the numbers become meaningless?

That is, on a foot-long ruler, "4" only means something because there's a "7," and a "9," but, most of all, because there's a "12." However, "12" only means something because there's a "13" and "25"; an agreed-upon metric, in other words.

When it comes down to it, game numbers facilitate the purchasing decisions of the buying public, and advertising, and I think that's it. I could have told John Davison that his CGW readers would revolt. It's like all those Americans who claim they only watch PBS and the news, yet, somehow, reality shows are a phenomenon, and porn even more so.

But, clearly, the focus on numbers is leading people to overlook something important, namely the subtle interplay of parts that is a videogame.

I've not played Gears of War 2 yet, though I did have the nerve to write about the game's marketing on Media Assassin, here.

But when I look at stills and video from it, and hear descriptions of what Cliff Bleszinski and crew were trying to do, it's clear to me, Roger Ebert be damned, that we're dealing with a moment here. Gears is a work of art expressed in the videogame medium. It's that simple.

Here's a prediction: A hundred years from now, collectors will purchase copies of Gears, Katamari Damacy, System Shock, Crysis, Ico, Dead Space, and every Mario iteration the way many, today, buy coin banks, first edition books, fine watches, and music boxes.

I'm sorry to say this, but, by that point, those collectors will not care if these games were given sixes or nines. They will collect them as beautiful art expressions of human craft and intellect. For many of the  artists who work on these games, it will only be then that, aside from the thoughtful critique of people like you, they get their just due.