Part 4: Gameplay Testing
The final keystone of Method, game testing, and its corresponding myth:
MYTH #8: THE CONSUMER
"If you want to make a hit, listen to the consumer"
Focus Testing
If you want to find out what features to put in your game, or what type of game you should make, the last thing you should do is conduct a focus test.
What You'll Learn from a Focus Test
What's Popular
Ask a focus group “what kind of game is popular?”, and the response will be whatever they played or saw the moment before they entered the room. Unless you are planning on releasing your game within the next few weeks, this is all but useless information to you. Focus testers will not tell you the future.
How not to stand out.
Humans are by nature pack animals. If you don't believe this statement, you haven't been to a focus test. The suggestions put forth at a focus test are not designed to stand out; they are designed to blend in, and to gain approval. To create a game that sells, you need something that stands out.
A feature list of games that were good.
If you want a nice list of features from games that have nothing to do with yours, transcribe your focus test.
Having said all that, a type of testing is one of the four keystones of Method.
Game Testing
Once you have a good piece of your game done, maybe a third, it's time for gameplay testing. Aside from sitting in a room and recruiting participants like a focus test, gameplay testing has almost nothing in common with focus testing.
Conduct Two to Five Tests
No game should be released without undergoing formal and extensive gameplay testing at least two points in the development process, and perhaps at most four or five times.
Watch What they do
Gameplay testing is simply putting your game in front of consumers and watching them play. I may not trust what consumers say, but I trust what they do completely. In fact, when scheduling your Production phase, you should schedule in one complete month of designer time to be spent in the room during formal gameplay testing.
Perform Quantitative Analysis
Gameplay testing should also be analyzed not subjectively, but as much as possible quantitatively. You may observe one player dying a lot on a particular challenge, but maybe that's an anomaly. You need to be able to derive statistics from your gameplay tests which allow you to tune your gameplay to a very high degree of precision.
Example: Testers of Crash Bandicoot 2 in Japan had significant difficulty issues. One of the Japanese producers recorded the play sessions of over a dozen consumers and spent a sleepless week analyzing the resulting videotapes. The result was a heatmap of the game, marked to show every location in which more than three players died. He also created statistics of how long it took each player to complete each level. Long levels, those that took more than 30 minutes to complete, were tuned to reduce the difficulty by removing enemies or obstacles at the "pinch points" where the deaths occurred.
Check Statistics against Player Reactions.
However, it's best not to get too enamored with your statistics. That's why it's so important to be in the room with the game players as they do their test. There's a lot you can learn from body language, and your compiled statistics can show how many times the players died, but they can't tell you whether the players were enjoying themselves or hating life.