How long do you betatest?

[cat]

Yesterday I played a very buggy demo and I recently played a commercial game with obvious quirks.

So my question is for the game developers and testers here, commercial or non-commercial:

How much time do you plan for beta testing? When you have a release date, how long before it do you start with testing? Do you also do alpha testing? Do you have different testers for alpha and beta testing? What is your development/testing time ratio?
A software is never free of bugs, so when do you draw the line and release the game anyway?

I'm curious, because I get the feeling that some people here take testing very seriously while others seem to be happy by just releasing a buggy product and wait for the players to do the testing.

[CaptainD]

Good question, Cat.

I think unless there were other factors - such as entering a time-limited competition - I think the minimum test period I would be happy with would tend to be 4 weeks.  I like to have 3 testers.  With a much bigger game, I would maybe want to double to testing time and possibly introduce a fourth tester. 

A slight caveat to that - this is in addition to the continual process of testing the game ourselves (as in, whatever dev team I'm working with at the time), as development and testing tend to go hand in hand.  But I would definitely want a dedicated test team in addition to that.  Also, if it's a big game, I'd want each segment of the game to be thoroughly beta tested individually before the game as a whole was tested.

[miguel]

My first projects were buggy, I just wanted something out!
This days I gather volunteers to test the game and let them have all the time they need. And when you think it's all tested and they think they've test it 100% sure, I get 1 or 2 more testers for a final look.

But I understand why a game maker is so eager to release his/her game. You feel the need to "let it go".

[Andail]

Test all the time, with as many people as possible. That's what I've done and intend to always do.
God knows how many alpha and beta tests I've run so far. I don't even know if there are any potential buyers left out there that haven't already beta tested my game.

Anyone wants to beta test my game, by the way?

[EliasFrost]

I haven't started testing yet, but I plan to start taking volunteers shortly to test my project, and I'm only around 25% finished with the game. I've alpha and beta tested several games and from that experience I realize the importance of testing. So, test early, test often.

[Ghost]

I like to start testing as soon as the game is playable- not even finished, just with the GUI and some first locations in place. If that's not possible I like to take the old-fashioned "release when no bugs are found" approach.
For a MAGS game I reserve a little under a week for testing since that are small-scope games, though.

I should mention that I can rely on at least three very competent testers who always do their job to the fullest. Not just testing, also suggesting and bouncing back ideas, and that is a great plus.

[Ali]

I found it interesting to read a blog post by Dave Gilbert where he justified not chasing up every bug after a certain point: http://nygamedev.blogspot.co.uk/2012/11/so-our-latest-game-primordia-has-just.html

Playing the latest Broken Sword, which is extremely polished but still has a few quirks, I wondered if they followed a similar principle to Dave - "they aren't worth potentially breaking the game over in order to fix."

[David Ostman]

I've yet to bring a game of my own personal doing to a beta testing stage, but from my experience working on bigger commercial games I'd say what Ali mentions pretty much holds true. At some point you get to the stage where you have a build that is stable and you can play it all the way through with only some minor quirks reported. QA is going to keep reporting those minor bugs, it's their job, but when you're close to the final submission build you don't want to mess around too much with the source and risk introducing any major game-breaking bugs that you don't have time to find. So you just close those bug reports as "won't fix" and move on.

I mean, of course there's the issue of craftmanship involved. You want to put out the best, most polished product there is out there, but very few have the resources and luxury to polish and polish until you actually feel your baby is actually READY to take those first steps on its own out in the big scary world.

[m0ds]

To the point where you start losing money/patience..

That said, it's really great when the actual developer does a lot of beforehand testing themselves. You'd think they would do some at least, but I've seen games here where it seems they didn't even press the "Run" button before they released it.

Oh, and another apparent fact, most mainstream games, certainly of past, would be released in buggy state or not. If a date is set, it gets released then. Hence why we've all had to patch games at some point or not. Money dictates release. Indies may have more time to work on that side of things, but they're still going to have to toss up earning over extra months in testing.

Bottom line for me is, so long as we're positive we've got all the major game crashes fixed, and various people have made it through to the end without major problems, I'd be satisfied. Would never release if a known crash exists. But by that point typically many other smaller bugs would've been found/fixed so it's in a good state IMO. If a game does go out buggy, well, it's not that different to mainstream

That said, people are just happy for an update/patch. Yet to hear a grumble about having to update The Cat Lady for example, everyone was more than happy to apply the update and get the extra stuff. So if you can add extras into an update you may avoid some of the "oh FFS" nature of players by giving them something new, for free, whilst fixing things in the background

PPS. There are always those errors that are computer/hardware specific. Person A's hardware will never be exactly the same as person B's. Sometimes people B will suffer in an area of the game where nothing extraordinary happened for person A at all. In recent weeks, we've found it's been AGS causing a lot of crashes, not the game. So there's a lot to weigh up. Game crashing bugs being the biggest one to sort IMO, and from there, use your best judgement (and don't get too stressed if you can't replicate someone's bug/issue - it may just be machine dependent).

PPPS. If you do an official testing period and it takes longer than a month, you want to be asking questions. If it's taken more than 2 months, something has really gone wrong somewhere along the line. Either you're trying to push a game that clearly was never ready in the first place or you're using a pretty slack beta-test team. What you SHOULD do, is hire Arj0n

[Stupot]

Playing the latest Broken Sword, which is extremely polished but still has a few quirks, I wondered if they followed a similar principle to Dave - "they aren't worth potentially breaking the game over in order to fix."

If the Revolution forums and Tony's blog are anything to go by, the game is pretty damn buggy.  Some higher backers were given 'beta' access, but this was a token gesture. What that really meant was that they were given a broken version of the game four days before release... lucky them.  They were never seriously regarded as 'beta testers' but that didn't stop them reporting bugs in their dozens (hundreds?) with nowhere near enough time left to fix half of them.  Rev presumably have their 'own' beta testers of course, and I'm sure they did a cracking job.

I haven't played it yet myself as it won't work on my graphics card (not a bug, just a shit graphics card), but I have recently played the Director's Cuts of BS1 and BS2, and I must say they were both riddled with bugs and glitches. Especially BS1.  I really think they could have done with another round of testing and bug-fixing before release, but I suppose as you say, they have to draw the line somewhere.

[qptain Nemo]

I generally try to never show my stuff if it has any known show-stopping bugs or issues that would seriously get in the way of what I'm showing. And I wouldn't publicly release anything that has any issues that could be classified as bugs that I'm aware of until I fix them, simple as that. So to answer your actual questions: as long as it takes and I'm bad at planning time-wise anyway?