You won't find bugs you aren't testing for

I bought a game recently called X Rebirth, a long-awaited sequel to a series of “X” games that have been released by Egosoft over the last decade. Fans of the “X” series have really been psyched about what the screenshots and videos promised from X Rebirth. A lot of people pre-ordered the game. I bought the collector’s edition because it came with the game soundtrack as well as a few other goodies. Unfortunately the game launch in mid-November was a complete disaster. The game is so riddled with bugs that it has garnered a 34/100 rating on metacritic (that’s bad), and reviewers have not been kind. The devs have been working hard, releasing patches almost weekly. For the most recent update, this paragraph in the release notes really caught my eye (emphasis mine):

It turned out that the issues many of you are seeing with trade ships not working correctly is not one bug but a whole series of connected problems… I do not want to give silly excuses here. Of course you deserve a bug free game, but you may ask why these trade problems were not uncovered during testing: We had a group of testers dedicated to testing our economy over a long period of time. These guys used a special version of the game which allowed them to run the entire economy at a higher speed. It seems that most of these problems we now see did not happen with this time acceleration. Meanwhile these problems also do not happen early on in the game or if you (like most testers do) have to restart the game frequently.

Note that they made two fundamental testing errors that cost them big time:

  1. Their testers use a “special” version of the game, meaning a version that the end-user will not actually be playing. In their case it was necessary but still dangerous as it does not simulate the conditions under which a normal user will play.
  2. They did not take into account prolonged periods of play, where the state of the game can be compromised by small bugs compounding over time.

I wish I could say that I haven’t made these same mistakes in my own career, but that would be a lie. I’ve learned the hard way, as the Egosoft devs have, that the best way to test software is to use it myself, as my customer will use it, under conditions that approximate, as close as possible, my customer’s environment and usage habits. It is especially easy as a web developer to “bounce the server”, or “clear browser cache”, or “fake a dependency” while running my application, but in production, when my app is serving my customers and their needs, they (likely) won’t be able to do these things. And even if they can they shouldn’t have to. So even though Egosoft still owes me USD $60.00 worth of gaming, at least I have learned a good lesson: always wait for reviews before I buy, and deliver the software I actually did test to the customer.