Beta-Testing

[MoodyBlues]

I'm new to the whole beta testing thing, and I was wondering - what kinds of things do you ask your testers to report, aside from bugs?  Do you ask them about atmosphere, story, gameplay, and puzzle design?  Do you have a standardized list of things you want all your beta testers to look for? 

[TheJBurger]

In the past, I've kind of broken it down to three categories for testers to comment on.

Technical Stuff:
- Bugs, Typos, Errors, Crashes, etc

Specific Stuff:
- Any pressing questions I have on the gameplay: these range from music, pacing, atmosphere, puzzle feedback, etc.

General Stuff:
- As you said, atmosphere, story, anything that the player wants to comment on. If they have complaints, suggestions, or feedback, anything is welcome.

Ever since I read a post-mortem on Portal, I've really been trying to integrate more vigorous play-testing into my future projects. In essence, you need to playtest your game over, and over, and over, to find out what doesn't work and how to fix it. If testers are having trouble remembering the story, or they always get stuck at a certain puzzle, or they're having trouble figuring out what their goals are, you need to go back and redesign those areas until you iron out all the bumps. Only when your playtesters can go through the game and receive the experience that you want, should it be considered complete.

[Dualnames]

It depends on the beta tester you have. Leon for example is testing everything gamewise. FSi, does the same as well. Bicilotti, just plays the game

[MrColossal]

I feel most beta testers just want to play a game before it's released.

In order to get the most out of your beta tester they have to actively try and break the game. That takes a lot of imagination but AGS is pretty solid as far as collision and things like that go so you can supply your testers with specific places that the game might not be solid, basically anything you coded.

They should basically be clicking on everything constantly until it breaks. Using items on things, trying to pick up things multiple times, talking to characters out of sequence, things like that. Beat the game once and then beat it again doing things differently, beat it by not talking to any characters unless you have to...etc anything that breaks the typical way of playing the game.

It may seem like a lot to ask a tester to play your game 10 times but if they don't want to test and just want an early peek at the game, they shouldn't have volunteered!

[Tuomas]

I can imagine a guy beta-testing a game like Quake. Runnig into all the walls from every possible angle. At least it seems that real fanboys do this after these games are released, just to be the first one to discover a bug in them.

[zabnat]

In my opinion beta-tester and play-tester are not the same job. Beta-tester is the one who runs in to the walls and clicks on everything untils it break. Then he describes the sequence and the result maybe even trying to recreate the problem and give instructions how to recreate the problem. Play-tester is the on who plays the game and gives feedback on the experience. Is it too hard, getting stuck on certain something etc.

[arj0n]

In my opinion there are 3 versions/stages of playtesting:

solo playtesting = alpha testing - done with internal people from tour team
local playtesting = beta testing first fase - done with external people
blind playtesting = betatesting last fase - done with external people, with use of a (nearly complete) rulebook/set of restrictions)

So a betatester can run into walls and try to break the thing and also give feedback on stuff like: is it too hard, getting stuck on certain something etc.
It just depends on what the gamecreator, the one who recruits a testteam, excatly want to be tested.

But that's my opinion while my real-live job is tester

[Leon]

When I'm asked to do a betatest I break it down into several stages. Each time the producer comes with a new version, this whole process is started over again.

Before starting the betatest sequence I agree with the producer on what he wants tested (the whole game or just a part of it), how he wants the errors reported (email or PM) and which frequency (every error, daily or once).

- First I do a complete run through to see if I can finish the game (the playtest). Keep in mind that I don't (like to) use a walkthrough. The walkthrough would direct your way of thinking and I wouldn't want to be influenced. Besides, if you need a walkthrough it means that the game lacks something, either the continuity in your story, the hints or lack of it or the puzzles being illogical. This will give me an impression of the story, the puzzles and graphics. Whenever there are crashes I stop playing until the next version. During this stage I also comment on my general feel about the game, what I like, what I don't like, where I see problems can arise when others will be playing, what parts are logical or not.

- Then I do a stress test. This means I try everything on everything. I try to outsmart the producer in finding things he wouldn't have thought of. Use every object on every object and hotspot. And then I mean every object and hotspot on every hotspot and object. You never know if the producer has missed one.  I want to create impossible situations that look very illogical but can be done by the user. If you present a keypad and highlight the 5 on it, tell him to press the 5, expect the user to press the 2. Not logical or desired, but it can happen. And anything that can happen, will happen at some stage. So make sure these events are taken care of. Also check the interaction inside your inventory. This is often overlooked.

- Then I make the walkthrough (I know the details of the game by now) and see if I can come up with a perfect score. It sometimes happens that you can get higher scores than maximum or can't get the maximum at all due to calculation errors. Also at this stage I'm being more critical at graphic glitches and conitinuity-errors.

- Finally I do the proofreading, spelling and grammar check. Sometimes the translation if needed/requested. At this stage the chance of anything changing in dialogs is minimal.

When all this is done, I request the 'final version' before release. The producer sometimes makes minor changes without notifying and I want to see the version as the end-user gets to see it (after all, if my name is in the credit list it means I'm responsible for the job and I like to deliver quality). 

I keep track of the versions I receive so I can always see if I missed something in previous versions or if it's something newly introduced. I like to learn from my mistakes as well. This gives the producer often a good indication of where to find the source of the problem. Also I'd like to see a change-log from the producer to see what has changed in the most recent version and I can concentrate extra on those areas.

As you can see, it's not just seeing if I can play the game. Like with the Hitchhikers Guide, I've played the game 68 times up till now (and Dual released another beta-version today  :o so expect at least another 4 runs to add... ) With Murran Chronicles 2 I'm still at the first stage but have played the game 8 times already and haven't even reached the finish yet. So the real work for that game hasn't even begun.

To conclude I try to suggest posibilities to the producers. When situations or objects seems strange to me, I try to give suggestions on how to change or what to adjust. Sometimes these are only minor changes but make a huge difference. And sometimes the producer just doesn't listen and goes his own way. Even better because I always keep in mind that it's not my game.... I just love to play them.

[Radiant]

I feel most beta testers just want to play a game before it's released.

That's true. That's also the difference between a good tester and a poor one.

A poor tester will want to play your game and ask for your help for pretty much every puzzle. A good tester will find a dozen bugs in the first room.