Yesterday I confirmed my hunches empirically: Valve makes good games.

What I did rather extreme: I got a few totally n00b friends to test Portal. And then I got them to test Crysis and Half-Life 2. Out of the 5 test subjects, 4 of them never touched a single computer game in their life (Minesweeper and Solitaire asides). One of them had some experience in computer games, but not in FPSes (I lured him into saying it’s one of my alcohol reflex tests as a control group).

I know, I know, don’t harp on me using a small population sample. I didn’t have many friends who never played computer games. It also means CLT cannot be used (though technically the average approaches normal after n>4). Also, another assumption made is that the gameplay skills of the test subjects are identically and randomly distributed (they are obviously independent of one another, since neither knows each other).

All my test participants were a mix of males and females (3 males, 2 females), aged 23-25, and an IQ range of 130-150. This experiment negates the difference between sex and age related reaction times, simply because I realized, after a series of multivariate play testing, it wasn’t important (or in other words, I suck at even simple multivariate regression testing, something I better buck up before next semester starts).

In simple terms, I measured empirically, the frustration times of the players - i.e. how long does a totally new player gets frustrated with the game and gives it up.

Because they all had no prior experience in first person shooters, I let them play the most ‘guiding’ game - Portal. Having had some experience in first person shooters, I let them play both Half-Life 2 and Crysis. Not at the same time, of course. I let them choose which game to play. Most chose HL2, while 1 chose Crysis first. If I had a larger sample size, I’d make a correlation between choosing either game first, but oh well…

So, anyways, here’s a brief overview of the results:

Notes:
* He escaped after some hints were provided by me. Especially the part where he has to break the glass tube.
† Didn’t want to give up the game. I took the computer away from him

Results and Explanations
First, some explanation of the table above. As these test subjects were totally new to the concept of FPSes, I measured their FPS familiarization time. I measured this in Portal. I noticed that all the test subjects tended to do the same thing when they were first introduced to the concept of a first person shooter. When the instructions in Portal came up: W for moving forward, S for moving backwards and et cetera, I noticed that all five participants were very cautious with their button pressing. In other words, they tap the keys to move. I timed them to a point where they are totally confident with how an FPS work - mouse look, and not tapping the keyboard.

Portal Give Up Level is a measurement of when they gave up the game in Portal. I didn’t use time, because Portal is pretty much a puzzle game. Players give up because they are frustrated with the puzzles.

Crysis Give Up Time is a measurement of when they gave up the game in Crysis. It’s measured and rounded to the nearest minute (because decimalization of time would cause confusion). Longer times is better, because it means that the player takes much longer to get frustrated with the game.

Half-Life 2 Give Up Time is a measurement of when players get frustrated with the game in Half Life 2. Similarly, it’s measured in minutes, and longer time is better.

Results: Generally, Half-Life 2 has a longer playtime than Crysis. This means that players would play Half-Life 2 longer than Crysis before being frustrated.

Based on these results, I’m 90% confident that Half-Life 2 is 1.34 times (± CI: 1.62) better than Crysis 50% of the time.

What does this mean in layman terms? It means Valve is a considerably better game maker than Crytek.

Criticisms
Some of you may have criticisms about certain things with my experimentation. Let me address those issues here. Firstly, gamers would immediately point out that I use complete newbies to a game, and hence not a good judge how good a game is.

I beg to differ, however. There are reasons why I used complete newbies in the experiment:

1. Being ‘virgins’ to FPSes, they do not have any outside experience that will influence their judgement. Their focus is only on the three games I serve up to them
2. Being ‘virgins’ to FPSes also means that they get frustrated more easily, and it would be easier to measure their frustration. Experienced gamers tend not to break down when they get frustrated with a game (well, except the Angry German Kid), so it is very difficult to measure their frustration.
3. Using newbies also means that a game’s design is rigourously tested. If it is foolproof to newbies, it should be foolproof to experienced gamers. Newbies, you learn, do really ridiculous stuff (like getting trapped in Test Chamber 17 without the Weighted Companion Cube)
4. You may argue that certain games like Crysis was designed for experienced gamers only, but trust me, I’ve seen more ridiculous stuff that completely fresh newbies can do that experienced gamers would never have thought of doing, and in the process, bring the entire game to its knees
5. Using newbies to FPSes eliminate the need to factor in experience levels when crunching the numbers (trust me, multivariate test + combinatorics isn’t a bit fun)

A second criticism that some of you more statistically inclined would point out is my sample size is only 5, and hence my figures are not very reliable. I do not have any defence for that. It’s really difficult to find 30 people with no prior experience in FPSes (but experienced enough to use a computer for daily work). So, yea, my GIANT Confidence Interval is one thing I cannot help (unless you’re willing to pay me to go to some remote village in Cambodia to find 30 youngsters to play Portal).

Good Experience
A good game, as the Game Drone had mentioned in my blog, is one that is fun. It doesn’t necessarily have to be innovative, but it has to be fun, else why make a game?

Gaming all boils down to a good experience. The Wii may be gimicky (I still think it’s nothing more than gimmicks), but it’s doing well and Nintendo surviving so many years in the game console industry all boils down to a good experience. It’s not about the progression of technology, but the progression of experience. So would it come as a surprise that all 5 participants rated Portal as far more superior than other games?

So what makes a game a good experience?

Fun. Naturally, fun comes in different forms and shapes. Some people have fun by whacking the shit outta people (I used to, but I outgrew GTA), some people have fun by jumping around at blocks and eating flowers that make them grow large. There is no general concept of fun, which is why the gaming market is segmented by genre - RPG, FPS, RTS, plt-frm-ers, chs… etc - different people prefer different kinds of fun. And when people have fun, they have a good experience.

Games are always a means to escape from the daily humdrum of life. You can’t go around killing people randomly on the street like Carl Johnson (unless your name is really Carl Johnson, and you work for the ‘hood). You can’t eat a mushroom and grow to a giant size (unless your name is Mario or Alice). Just like TV, books, sports or any works of fiction, it gives an escape to daily life, it may even inspire hope. When people are immersed in games, even temporarily, they forget all concerns of the world, they have a good experience through games.

How then does a game come to have all these features? Watch out then, for Part II of the Good Game Series

(p.s: Originally this article was called Innovative Gameplays, part of my “I” Alliteration posts, but I delayed publishing it, and it was all in one big fat article about 6000-7000 words long, but I thought it’d be better to break them up)

Cool? DiggDel.icio.usTechnoratiFeedsterFurl
RSS feed for comments on this post  |  TrackBack URI for this post