Question regarding beta testing and testers...

[Snake]

I hope this isn't a dumb question

This question is for everyone that has had their game tested before releasing.

How do you go about your beta testers - don't know if I'm saying that right.

Do you give them specific things that you want them to look for as well as just giving the game to them? Or do you just hand it to them and let them loose?

How does everyone go about it?
Thanks in advance,


--Snake

[Radiant]

I would advise not telling them anything whatsoever, and letting them figure everything out by themselves (if they can't, your interface is not clear enough, and needs to be fixed). That includes letting them figure out the puzzles (if they can't, your puzzles are probably too obtuse, and need to be fixed).

You know that actual players aren't going to follow your schedule either, so neither should your testers. Failure to adhere to this principle is what caused the (later) Sierra games to be so buggy if you strayed from the intended path.

Include as many options as you can think of to facilitate testing; the standard teleport and stuff pockets are highly useful; a key that pops up an InputBox() and dumps the result to a file (with the room number prefixed) is also useful.

Also, note that good testers are worth their weight in gold, and conversely, mediocre testers are a complete waste of your time. If a tester isn't able to produce a page full of bugs, tweaks and nitpicks within the first hour of gameplay, then either you are an amazing programmer, or it is the latter kind of tester. Okay, so that was slightly exaggerated, but the point should be clear. You don't need 20 or 30 testers - you need 2 or 3 good testers.

[Dualnames]

Mostly the problem with beta testers is that they want to add themselves on the team of the game and don't care if they beta test it or not. I'll agree with Radiant..

[MrColossal]

A beta tester isn't someone who plays through the game once and tells you all the problems they had when doing so. This is what most people who want to help test a game think. As Dualnames says, they think it's a chance to play a game before it comes out.

Their job is to try and break the game. Repeatedly. For days.

A proper better tester would click every button on every object in the room, do puzzles in every order you can do them, stop in the middle of a puzzle and click on everything again to see if anything breaks or a message is spoken wrong given the current context and stuff like that.

Usually I rely on friends for design changes. I'll have friends play it and if they get stuck somewhere I'll take a note and revisit that area. Friends playing is mostly to see the game played by various people and on different computers.

I would rely on my actual testers for show stopping bugs and everything under that involved with breaking the game or breaking flow of the game. Since most people aren't trained in QA I'd recommend giving people a guideline to go from or a template for filling out bugs so they don't get intimidated from a blank email they want to send you and so you don't get bugs back like:

"The guy can't pick up the box. He walks but can't."

Instead give them a template like:

Name:                                           Room Number:
Description of Bug:
Steps to Reproduce:
Priority Level of Bug: 1 - 5

this way all bugs will be formatted and people can easily tackle describing what happened.

[Dualnames]

Eric loved that template thingy.. Excellent idea..

[Erpy]

We've had both kinds of testers Radiant mentioned at AGDI and Himalaya. Most testers probably fall somewhere in between...they play through the game a couple of times and report bugs, but then get bored with it. Some will play through the game again if you specifically ask them to, others will stop responding. Even some applicants who come up with impressive credentials sometimes fall into the latter category. The problem that some people simply want to be "on the team" or want to play the game prior to release (especially if they've been anticipating it for some time) is nearly impossible to avoid...even with lectures and horror stories about the tedium, responsibility and such. You can try damage control by being having a critical eye to the applications people send in, but the issue remains that quality assurance and writing resumes/applications are two different skills that aren't always related. Some people may make good testers, but suck at interviews/writing applications. Others may have the gift for the gab and appearing knowledgable, but have a sucky attention span.

I agree with Radiant on the debug tools. Quest for Glory 2 VGA we're working on right now has tons of those, from the simple cheats to alter stats/money/items and warp around to another room/time block/day to more exotic stuff. A few things we've been using:

- A debug option to manually set global ints.
- A GlobalInt monitor that displays the value of the selected GlobalInt on the status bar at all times.
- Codes to jump to certain scenes/events while correctly setting all variables/conditions as they should be (as opposed to warping to the room and potentially messing up) and summoning/blocking random encounters.
- Code that allows you to save up to 99 screenshots per game session. (pressing F12 again doesn't overwrite the previous screenshot but creates a new one)
- Quicksave/Quickload hotkey that allows saving/loading in scenes where the player has control, but official saving is blocked.
- Hotkey that displays the value of some of the most relevant variables in most rooms.
- Toggling of logged saving. (autosaves when you enter a new room, but alternates between two save slots, so if you quit the game right after a bug, you'll always have a savegame from that room and a savegame from the previous room you were in, allowing for easier replication. Auto-disables when restoring a game.)

We're still looking into what options would be a good idea to give to the testers and what options are better left hidden...we've already spent some good time trying to fix phantom bugs that only pop up if you use warp/globalint/item-codes to get into a situation that wouldn't be possible when playing the game the usual way. Especially as tedium sets in, testers will be more tempted to use shortcuts and backdoors to speed things up.

What you definitely want is testers telling you:

- what kind of bug (crash, mask-issue, showstopper etc..)
- what happens (and what they think is supposed to happen)
- where it happens (room number can't be beaten)
- how it's replicated (sometimes screenshots are useful with mask-issues, savegames with most other issues)


A private beta testing forum has the disadvantage that people can spoil each other, but it has the advantage that people can confirm or elaborate on each other's bugs.

If the project to test is large, you may try out multiple "waves" of testers, replacing each wave when they start losing steam and sending fresh people in. For a short or medium game, that's probably overkill though.

[bicilotti]

Wonderftul topic with super-useful information. Out of curiosity, how many beta tester and how many days of testing took a monster project like ATOTK?

[Radiant]

Wonderftul topic with super-useful information. Out of curiosity, how many beta tester and how many days of testing took a monster project like ATOTK?

We had five testers, with somewhat varying levels of activity, but I should point out that a few of our artists also helped with the testing.

It took us two months to get from the first complete build to the version that was finally released. Although to be fair we also spent some time testing the game before that.

[SSH]

- A debug option to manually set global ints.
- A GlobalInt monitor that displays the value of the selected GlobalInt on the status bar at all times.

You use hoary old Global Ints? No wonder the project has taken so long   :o

[Erpy]

If you're implying we rely only on GlobalInts and use no self-declared ints (exported and imported for global use), we actually use plenty of both.

[Snake]

I think SSH is a little Silly Shit Head

Wow, thanks everyone for your replies A ton of good stuff has been mentioned.
Erpy, I loved your plan of debugging methods. For my game I definately need to make a room number GUI - that would definately help me out since almost every room looks the same.


--Snake