Time is Money: A Response to ‘Review Code’

In a rare bit of candor, a major digital culture website has deigned to pull back the curtain a little bit and show us in the audience the reasoning behind why it writes so many reviews that seem to score in the mid-to-high range, and why it tends not to write that many reviews on the lower end of the ten-point score spectrum.

IGN’s Reviews Director Dan Stapleton writes, “When you consider the main purpose of a review is to answer the question of whether something is actually as good as it appears to be in ads and previews before you decide to spend your time and money on it, there’s no greater waste of everybody’s time and effort than telling people that something they’ve never heard of isn’t good.”

Stapleton stakes a very interesting claim here in this first sentence, but before we analyze this, it’s worth saying that this level of transparency into their process is a positive thing for everybody on the whole, and they should keep doing this despite any criticism they might receive for it.

Right off the bat, Stapleton defines what a review’s purpose is in probably the most controversial way I could imagine someone defining the purpose of reviews. His is a purely utilitarian view, one with no room for criticism-for-criticism’s-sake. But it does make some sense from his perspective as the one planning the “1000+ reviews” across games, television and other media that IGN posts per year.

As he goes on to say, in IGN’s view, most things are universally acknowledged as bad and not worth mentioning, a few things are mediocre-to-fine and widely understood to be so, and a vanishingly small number of things are very good and deserving of accolades — and all of this is complicated by the Schrödinger’s Cat-like existence of the “surprise hit” that “comes from nowhere.” If I’m understanding his perspective correctly, it would be a bad business decision to write about things from the big pile of largely-ignored things that suck because that would take time, energy, money and other resources away from writing about things from the much smaller mediocre-to-fine, great, and unexpectedly fantastic piles, and it would waste the reader’s time on top of that.

But there are a couple of things worth questioning in Stapleton’s opening paragraphs as well as the rest of this piece, like “who really defines ‘review’ in such a fashion,” and “who or what determines what goes in each pile attributed to quality?”

Why do we write reviews, and for whom?

In my years writing and reading criticism I have heard criticism itself defined in a number of different ways, but the one thing that I hear even more constantly is that folks primarily don’t read reviews for the purpose of guiding their purchasing decisions anymore. They read reviews instead to confirm their preexisting biases, discover new media, or to hear from differing perspectives after they’ve already consumed the media in question.

When Stapleton says, “the main purpose of a review is to answer the question of whether something is actually as good as it appears to be in ads and previews before you decide to spend your time and money on it,” he is making the assumption that people are dutifully coming to IGN and performing their Homo economicus role in this relationship, and not, say, reading a bunch of reviews from across the games media just to see what those writers’ thoughts were, or to clown on a game, or to clown on their friends for not giving a game a chance.

Stapleton’s statement also serves to constrain the forms criticism can take at IGN. Because the view is that reviewing is meant to serve a sort of “Consumer Reports”-style purpose (much love to my friend who actually writes for Consumer Reports, this is not a dig on y’all) where the text must purely be informative, this means that by and large, all you will get from IGN is a kind of generalist sweeping overview of a given piece of media. And because of the next part of Stapleton’s claim — “there’s no greater waste of everybody’s time and effort than telling people that something they’ve never heard of isn’t good” — these generalist overviews will always have affixed to them the same narrow range of opinions: that the cultural object in question is either meh, fine, good or great, and also that it deserves to be in the public eye.

It bears mentioning that this is not how everyone writes or approaches criticism. While the number of outlets that’ll let writers actually give their opinions on a given piece of media dwindles by the day (RIP Gawker 2.0), plenty of alternative approaches to Stapleton’s utilitarian view exist. There are critics who operate in a purely historical or archaeological mode, documenting and highlighting games and media from a past that is quickly slipping out of our fingers due to tech rot or corporate enclosure.^[1]A great recent example of this is the whole Cookie’s Bustle fiasco. There are critics who approach media from a purely experiential perspective, writing about games or movies or music based wholly on how it made them feel in the moment. There are feminist critics, queer critics, religious critics, metacritical critics, and critics who do criticism as a kind of a bit (hello). Every one of us probably has a different definition of what criticism means, but most, if not all, of us are doing criticism well past what sites like IGN seem to value or prioritize.

For me, criticism is an opportunity to take two or more ideas about a cultural object and smash them together. My favorite kinds of criticism are the pieces that play with the form and audience expectations for what criticism should be, operating somewhere between critical metacommentary and a big fucking joke.^[2]Unsurprisingly, I am indeed a Tim Rogers/Action Button YouTube fan. Some of my favorite pieces of criticism that I’ve done are those that manage to combine seemingly disparate pieces of media in interesting ways. I like essays with soundtracks. I recognize not everyone feels the same way; I recognize I’m not likely to write a pitch for IGN any time soon. But back on topic: where does IGN get its ideas about what’s good and bad anyway?

It’s always the algorithm

Here is Stapleton on how IGN determines its coverage:

When it’s not obvious that something is a big deal, like a Grand Theft Auto or an Avengers movie or a Game of Thrones-level show, we use metrics like traffic on IGN for news, trailers, and previews to see if the wider audience is interested. We also use publicly available tools like Google Trends and YouTube. Did a lot of people check out the trailer for a new movie? It’s a safe bet they’ll be interested to know more. Did just a few thousand view it on YouTube? Maybe it’s just not clicking with IGN’s audience and a review would suffer the same fate. When those raw numbers leave us uncertain – or even sometimes when they tell us most people aren’t interested, but we are – we often take a chance on something we think is special and should be highlighted, even though it probably won’t do a lot of traffic for us. That’s when you’ll see smaller things make it onto our review list.

In other words, the algorithm typically “decides,” but there’s enough human input to make adjustments or exceptions. I don’t agree with this approach, but I also understand it in the context of a larger, mostly metrics-driven media industry. It looks at popular culture and figures out what people are already talking about in order to create ~*content*~ Taylor-made for that demographic. The problem with this approach is that it makes it very difficult to break out of the cycle of hype for a given media property – hence the need to occasionally “take a chance on something we think is special and should be highlighted, even though it probably won’t do a lot of traffic for us.”

We didn’t need Stapleton to tell us that sites like IGN typically stick to bigger games and media because it’s more profitable. Other folks at other outlets have been saying it for a long time: covering smaller AA titles and most indie games doesn’t pay the bills because it doesn’t tickle Google’s search engine optimization algorithm the right way, for example. When we see games sites gutted or straight up destroyed, it’s typically not the content that does well in the search engine that gets cut, like guides, but the stuff that is much more intellectually or culturally satisfying but plays like dogshit on Google. And even when a games site is made with the intended purpose of making intellectually or culturally satisfying shit, like WaPo’s Launcher, it can still be shuttered simply because it’s “not as important” as hiring an umpteenth transphobic opinion columnist, even if it’s doing much better than expected.

No, the problem with Stapleton’s perspective and IGN’s reviewing principles isn’t that it’s critically stunted, which it is, but that it might be the only way for criticism to survive and change nothing else about the current media industry. You can’t argue with results, even… though…. IGN itself had its own round of layoffs the day of The Game Awards. Ah, well, nevertheless.

Prospects for the future

I do genuinely appreciate Stapleton’s willingness to be open about how the sausage is made at IGN, no matter how much I think it’s a stale way to approach criticism. It is helpful to know the ways in which IGN purposely leaves holes in the coverage and criticism of digital culture, so that we might fill them ourselves. We will always be at a disadvantage to operations that have budgets in the millions of dollars, of course, and the work of filling in those coverage gaps will by design never be profitable. But it’s far from a waste of time to spend time with games or other media “no one” has heard of, and it’s even less of a waste of time to do so if the thing happens to be bad, in my estimation — especially if you’re trying to do some Consumer Reports-style shit.

One of my favorite pockets of criticism comes from a comedy livestream hosted by LoadingReadyRun, called Watch+Play. They periodically take a look at some of the least-put-together games on Steam, but lately they’ve been doing something different: they’ve been looking at those trash ass mobile games whose ads always look different from the gameplay and offering their critical commentary.

This is something I don’t see anyone doing, even though I’ve seen lots of folks gripe about it on social media for years. Why do ads for games like Evony and Hero Wars look different from the games themselves? What are these games even supposed to be doing? In lieu of any “official” coverage of these games in the games press, alternatives like Watch+Play serve to fill in those coverage gaps.

Other media, like Uppercut Crit’s Indie Mixtape Podcast and Kritiqal’s Kritiqal Care Podcast are just two examples of ways in which sites can “waste time” looking at small or relatively unknown indie games and creators in a fashion that benefits the creators and the audience. Canon Fire imagines an alternative history of games, and presents hidden gems to us on a regular basis. Another of my favorite folks, Pizzapranks, does INDIEPOCALYPSE, a zine which collects indie games and puts them in front of your eyeballs. (Andrew does a radio show every Saturday, called Indiepocalypse Radio.)

What I’m trying to say is that we don’t necessarily need to imagine a wholly new alternative to the IGN model of crusty criticism, because it’s already here. There is a vibrant and tenacious indie crit scene out there, and it exists in direct argument against IGN’s review model. What every single outlet and creator mentioned here (to say nothing of the dozens of other interesting projects out there) needs is support to keep doing the thing.