JustinLiew.Com

At a base level, the debugger and print statement are essential tools for debugging features and fixing bugs.  They have been around since the beginning and are well understood by programmers from every discipline.  In games, however, there are many useful tools and utilities at a higher level that make common cases easier to work with.  In no particular order, here are some I’ve found useful throughout the years:

Macro

I’m grouping together a series of tools that provide have multi-purpose use, and do not provide in and of themselves any specific functionality.

Live Tuning

Live tuning is the ability to connect a tool to the game while it is running and utilize bi-directional data to aid in development or debugging.

Examples of data flowing from the tool to the game could be, but are not limited to:

• editing world object properties in the level editor and seeing changes in game immediately.
• sending commands to the game to carry out tasks such as loading a level or jumping to a location in the world.
• telling the game to display certain classes of objects and their properties in game.

Examples of data flowing from the game to the tool could be, but are not limited to:

• displaying a list of all NPCs alive in the world in an external viewer.
•  the state of global objects such as quest states or time of day.
• for visual scripting systems, displaying which nodes are currently active and what links have been followed.
• debugging script.

Having a common system for connecting which makes it easy for a new tool to add live tuning functionality to tools and the game is key.  Having this up and running early in a game’s life cycle enables many more complex tasks to be carried out fairly easily.

Telemetry

Telemetry is the ability for a game to log info discretely as it runs.  The first requirement is that telemetry data shouldn’t increase the memory footprint or reduce performance in any substantial way.  Another ideal requirement is that it log info to a central database without user intervention, so data can be gathered in the background as the game is developed, and analyzed using the large suite of database tools available.  There are two main classes of telemetry data: development data and playtest data:

Development data is useful for fixing functional aspects of the game, and might include things such as: where the game has crashed the most, which asserts have fired the most, which level loads are the longest, where the player got stuck in level geometry, and the like.  They are data that can be captured irrespective of the kinds of things developers are doing.  Playtest data is useful for experience balancing and tuning, and can include how long quests took, where players stood still the longest, and which dialogue options were chosen.  The main point to note is that you don’t want to capture playtest data during normal development, as when testing a bug most developers tend to repeat the same steps over and over again, which skews playtest data.

Logging, including synched logs

Logging is a subset of telemetry.  Usually logs are written out to file, and at a minimum should have the ability to write to different channels at different verbosity levels, so gameplay engineers, for example, don’t have to see rendering logs.  As well, having a way to merge logs from different consoles when doing multiplayer testing is useful, as a lot of your debugging will likely involve matching up when data was sent from one machine and when it reached another.  This merge should take into account some sort of synched time, which is not perfect, but is usually good enough.  Another alternative would be to have a tool that can connect to all running consoles and capture data as they run, which gives you the merge for free.   

Ini Files, In Game Debug Menu, and Command Console

The ability to change properties of the game is important for debugging.  You might want to set the aggressiveness of your monsters, or disable lighting effects to increase framerate, or turn on an unfinished feature for testing.  .Ini files let you do this at startup, while an in-game debug menu and a command console provide different ways to change properties or run functions at runtime.  Ideally any combination of these systems you implement should use the same underlying system, so you can get access to the same functionality in a consistent manner.  Ideally the debug menu and command console are configurable externally, so engineers don’t always have to edit code to add items.  

Replay

The ability to rewind and fast forward your game is vital to fixing sequencing bugs – animation state changes, for example.  Anything with states can really benefit from replay, alongside having debug output of the state for each character involved.  That way, when playing through a game, you can rewind and replay a boss battle in slow motion to see what conditions caused changes in state.  This is also good when getting a bug from QA, as you can see the last few moves that were performed before a crash or anomaly.  The main consideration for replay is balancing the size of the replay buffer and quality of replay (what is saved).  The actual technical details of replay are an article unto themselves, so I will not go into it further.

Crash Dumps and Auto-Reporting

This is a no brainer.  When the game crashes on a QA or content creator’s machine, having the callstack of the crash is of vital importance.  Ideally, this hooks into the telemetry system and/or the bug tracking system and automatically logs the crash, along with the crash dump, callstack, and info about what level the player was in and other game-specific data.  Ideally, your telemetry system or auto-reporting handler can filter on duplicate callstacks and group them together.  Perhaps in your bug database the “repro rate” or “occurrence count” can be updated so if a crash is happening frequently these numbers will go up and the priority can potentially be automatically raised.

Soak Tests

When finaling a game, it is always those pesky soak test crashes that only happen after 8 hours of gameplay that threaten the patience of the SE team.  Having a soak test system in place that makes it easy for everyone on the team to run them on their machines every night, as well as gather pertinent data about the run, is of vital importance.  Having some of the above tools, specifically telemetry, logging, replay and crash dumps, will enable SEs to see what happened throughout the run and work backwards to figure out the problem.  Logging info such as memory highwater marks, fragmentation and the like can be useful for out of memory conditions.  The type of data you save during soak tests is endless!  Soak tests can run pre-recorded replays, or they can randomly press buttons either in-game or in the UI.  These button presses can be configured for frequency so that more common buttons (A/Cross) can be pressed more often, while directions can be held longer to simulate more common human control.

Micro

The functions below are more specific, and usually utilize a feature from the macro section above.

Skip X

The ability to skip various portions of your game is key in saving time.  Skipping the front end, for example, and directly loading into a level can make it faster to run tests or reproduce bugs.  Skipping the bootup screens can let you test the UI quicker.  Just make sure everyone tests features within the context of the entire game before deeming them fixed or finished.  

Auto connect for online

When developing a multiplayer title, a lot of time is spent navigating the matchmaking or invite flow, getting players into game, only to test a bug for 30 seconds.  Added up over the project, this is a huge time hit.  Using various macro tools, the ability to configure the game to launch as a host and start up a multiplayer game, and launch on another console as a client and connect to the host, can be a huge help.  Just ensure that you’re doing final testing through the proper UI.

Teleporting Players/NPCs

I’ve seen this useful in a couple of instances.  When doing multiplayer testing, especially for coop games, it’s sometimes useful to be able to bring the other coop player to the current player’s location.  When testing locally, it can help avoid having to move both players manually to locations in the world.  Instead, just move one player, then use a debug key combo to teleport the other player(s) to where you are.  The second instance is when you want to interact with an NPC, but their spawn location isn’t fixed in the world.  It might be easier to teleport them nearer to the player, if applicable.  Obviously for a boss who uses its environment in some meaningful way this isn’t cool, but for a generic battle or conversation it can speed up testing.

That’s all I have for now!   I’m sure readers have other great ideas, so let’s hear them in the comments!