“Final Testing” – Sakurai’s Famitsu Column Vol. 459

Original column published August 21, 2014

In this article, Sakurai shares his thoughts on the process of debugging games. Thanks to sourcegamingsoma for translation help!

Note: Do not repost the full translation. Please use the first two paragraphs and link to this translation. For additional information, please read this post. This translation is for fan use only, and may not accurately reflect Masahiro Sakurai. The following is a selection from Masahiro Sakurai’s book: Think About Making the Video Games 2. If you enjoyed this article, I would strongly encourage you to support Sakurai by buying his books. If you have any questions about this article, please contact the administrator.

At the time of writing this, the Nintendo 3DS version of Smash Bros is in the middle of full-blown debug testing. Debugging is a normal part of the game making process, but I’d like to formally write about it here.

Time of writing: after submitting the manuscript, it takes about 3 weeks to appear in this magazine.

Debugging, to put it simply, is the process of finding and fixing a program’s problems and malfunctions — bugs. Even the most logical people can’t come close to the rigid nature of computers, which can only do specific actions and run complicated programs until completion. They aren’t organic creatures, but when many components are added on top of each other, computers become fickle, unpredictable things, as if they were living organisms with real bodies.

Computers cannot deviate from their code, so if a program is free of mistakes, then no bugs appear! This is probably what people think, but it’s just impractical theory. Depending on combinations of elements, various things can occur. For example, at the debugging stage of a previous Smash game, if Fighter A attacked Item B, and Fighters C and D were simultaneously hit by it, the game would freeze — there were bugs like that. Besides fighters, when you include things like items, stages, color variations, rules, and modes, it makes for an astronomical number of combinations.

The reality is that it isn’t true debugging unless every potential pattern is checked, but doing so would be an impossible task that would take mankind tens or even hundreds of years. Furthermore, there are times when even if you recreate the same actions and conditions that once caused a bug to occur, it will not reappear at all. When this happens, it’s very troublesome. Luck gets involved. In these situations, various tests are carefully repeated, with the help of many people.

The basic process starts with discovering and reporting bugs. A team separate from the main development team often does a lot of the work here. Then, the development team takes their information and fixes the bugs. We have testers in Japan of course, but also overseas. For Smash, the scope involves several hundred people. Just the debugging process alone is an enormous project, to the point where a big reason that the 3DS and Wii U versions of the new Smash game have different release dates is so the debug periods are staggered. A dreadful part of debug work is that once you fix something, you have to once again redo all of the “checks” that you did previously. Even if something looks like it works correctly at a glance, with a different combination a different bug is born — this happens a lot. It’s called a “circular bug”.

**TN: In the English-speaking software development world, this rechecking is called “regression testing”

Even so, it’s not like we can keep changing things forever and ever. There’s a timeframe nicknamed “No Touch Debug,” which applies to the production version of the program, the Master App, when it’s in the completion stage. When you reach the point where you think you’ve taken out as many bugs as you can, you debug for several days without adjusting anything else. If no large problems arise, the game is complete. There are a considerable number of games going through this trial right now.

**TN: The English equivalent term is “release candidate”

Master App: The commercial quality program is checked over for problems, and becomes treated the same as a finished product. When this review finishes, mass production starts.

However, no matter how many hours are spent, no matter how many talented people help, there are times when a bug goes undetected during testing, then is later discovered at a crucial time. Games slip past their expected release dates for various reasons, but nine times out of ten it’s because a bug had to be fixed. But, that’s better than not catching the bug and releasing as is. When game developers face off with computers, they are always risks involved.

A group of people, more numerous than sellers and creators, lend their efforts so that gigantic, complicated games and game machines can continue to be released.

Mario Club: A subsidiary of Nintendo, located near their Kyoto headquarters. They find bugs in games under development, offer opinions and user perspectives to the game creators, and aim to improve the quality of Nintendo’s software titles.

“Bug” and “Debug”: There are various types of bugs, such as ones that prevent the story from advancing, ones that make characters behave strangely, ones that create abnormal parameters, and others that would be impossible to list exhaustively. To find those bugs and test the game’s various patterns, dedicated debuggers are employed, who do nothing but play the game repeatedly. Nowadays, you can even fix bugs through data delivered over the internet.

Looking Back

Interviewer: I’ve noticed that there’s always a fuss on the internet whenever some kind of defect is found.

Sakurai: “Smash Bros. for” was the same way, but I feel that the number of bugs was relatively low, considering the number of people playing it and how many components there are in the game.

Interviewer: Developers don’t just get their information from customer support, but also from discussions on the internet, and they have to verify these claims, don’t they?

Sakurai: That’s right. Although honestly, I can leave that work to someone else…

Interviewer: I was really surprised that Smash 4 got patched. I thought, this is a sign of the times, too.

Sakurai: For a game like Smash, where you can keep playing it for many years, for it to stay in the same state the whole time is probably impossible when you consider the level of complexity of today’s games. So from now on, I’d like fix the parts that I can, little by little.

Interviewer: No matter what you do, it isn’t really a cheerful subject, is it…

Sakurai: When you heard we used several hundred people just for debugging, what did you think?

Interviewer: When I thought about the number of hours each person spent, I was shocked.

Sakurai: Well, that depends. And eventually, when you involve the workers overseas, it gets to be an amazing number. And then from there, things unfortunately got leaked…

Interviewer: Also a gloomy story.

Sakurai: Debugging on this scale, at least for big titles, I think, is probably commonplace. The number of combinations to test becomes monstrous, so not much can be done about it.

Interviewer: Hmm, right. Testing every combination in “Smash for” is physically impossible.