The question for day 21 of #AprilTTRPGMaker is “How many playtests?” This is a big question, and as I've said before the topic of playtesting is near and dear to my heart.
More than zero
My position is that it's unethical for a game designer to release an unplaytested game to the public. This is just a specific case of my general view that nobody should release untested products. Obviously that's a much bigger deal in a field like pharmaceuticals where problems in your product could actually kill people, but I think an ethical game designer should test their product before release to be sure that it does what reasonable people generally expect a game to do.
Holding this view has not made me popular
The fact is that many people have released unplaytested games. My view contains the implicit criticism that people shouldn't have done that, which is something those who have done it don't like to hear. I believe I've pissed off several people who have worked as freelance designers for mainstream publishers and some prominent indies by holding this position in social media conversations. Nevertheless, I think it's more important to hold views that are right than views that are popular. Obviously it's possible to be lucky and have your untested product work fine, but I still think that's a bad practice. I've also experienced people assuming that I hold this view in hypocritical bad faith and that I only express it to “gatekeep” people from getting started. Nope, as already mentioned I hold this view even for people who have been “successful”, and more importantly my views are obviously not binding on anyone except me, so no one is more “held back” by my view than I am. There's not a huge amount of harm caused by releasing untested games, so even if I had the power to do something like “drum people out of the industry” for doing it I probably wouldn't, but I think the world would be a better place if proper playtesting was a norm that everyone followed. I think it's probably in everyone's best interests to engage in good pre-release testing, anyway – the games will be better for it which is good for designers and good for players.
Less than infinity
There's a potential trap with testing that it can always seem like more is better, so you can get yourself into a pattern where you always want to do more. If you do that it pushes your release off to infinity. I don't think that's wise. If your playtesting practice also includes an element of social engagement, such as being able to show off your game at game conventions, it might be even easier to fall into this trap. I think the key to remember is that playtesting is done for a purpose, which is generally to get all the problems in a game identified so they can be ironed out and it can be released (or, occasionally, to discover that a game is so fundamentally flawed that it should be abandoned).
So how many is “just right”, Goldilocks?
I think this is hard question to answer, and “how many” may not be the ideal metric, even though that's the most easily measured. First of all, if you start counting from zero you're not taking into account how much the game needs to change based on information gained via testing. If you need to completely revamp large systems you're probably going to need more subsequent tests than games that were lucky enough to have no major problems. In my experience in testing microprocessors we tended to assume that there would be diminishing returns on our bug-finding efforts – you tend to find the easy-to-find stuff first (in an RPG context, maybe something like “the rules said to roll for X, but you didn't specify what kind of dice to roll”) and then more subtle and hard-to-find stuff later.
To some degree you can use that as a basis for determining when you're done: if you've been doing good testing and haven't needed to make any revisions to the game for a while it's probably ready for release. Because I haven't gotten to a stage where I feel I've had adequate external testing with my games I can't say with certainty, but I think that if two different independent groups were able to play one of my games to completion without problems I'd probably say it's ready for release.
The testing needs to be good
In my microprocessor validation job we had a concept of “coverage” to track how well our testing was covering the important interactions in the device-under-test. Basically you need to apply some engineering judgment to the thing you're testing to understand what sort of factors need to be varied. For example, most games let players choose whatever names they want for their characters, but usually this doesn't have any meaningful impact on play, so you probably don't need to worry that you've tested a broad variety of character names. On the other hand, if you've got a class-based system where the players need to operate as a combat team then you probably do want to make sure that you've tested with parties that have several different combinations of character classes because that often has a big impact on how a game plays at the table. If you run a million tests but each one only exercises 5% of your design then you've still got 95% of the design left to test. If your game is straightforward enough that there's not a lot of variance needed to exercise the full scope of possibilities then you may not need many tests.
The #AprilTTRPGMaker Questions
(From Kira Magrann's twitter)
I will admit that occasionally the temptation to publish stuff without playtesting is pretty large (I've often released not fully playtested things with the caveat that they're still being playtested), but I wish it weren't controversial to say people should playtest.
That's not to say that you need a broad and far-reaching playtest for every game: unless you're WoTC or Paizo that's impractical, but you need to have played the game to understand whether or not it's something you can put out in good faith.
I suspect that several A-list publishers have been skimping on playtesting (or at least, at one point, were, during the end of my time as a game reviewer), and it shows when something just hasn't gotten enough time in development and playtesting.
The most unethical game ever!
I definitely fall into the infinity trap, myself, but that's mostly because editing/polish is much less fun than design iteration.
Based on your writings here and elsewhere, I have a suspicion this belief about playtest ethics is undermining you, and you'd be better off untangling and discarding it. But I'm also pretty sure you see some higher value to it even where its utility is questionable, so carry on!