Members for rating panel, apply here!

[Andail]

Fellow members.
The games database will soon offer more features. Along with more classifications and genres, there will also be new ratings, carried out by an "expert" panel. The old users' ratings will still be there.

We now need people for this panel. They will be handpicked from the applicants by CJ and his moderators and work anonymously on a special board, where they will rate all AGS games according to guidelines in official documents.
The members of the panel will rotate each month, kicking out the oldest member and inviting a new one. After half a year, the entire panel will have been changed.

In order to apply for this very special position, you need to meet the following criteria:

1. You must be at least 18 years old.
2. You must have been an active member of the community for 2 years or longer.
3. You must not be known to misbehave or break forum policies.
4. You must have played most games in the database, and preferably all the canonical titles.
5. You must be prepared to spend quite a few hours per week playing and rating games. Lots of spare time is essential.

If you meet the criteria and wish to apply, PM member Application (below). Applying does not mean you automatically get picked. Only apply once. If you get selected, someone will PM you. This procedure will begin after a while, to allow plenty of applications.

Thank you.

[Application]

Please PM this "member" with your application.

[Baron]

As an amateur game maker in an amateur game-making community I have to say I'm a little leery of "experts".

[m0ds]

Basically, its been decided that there's too much crap in the games database, all the experts will do is realise & filter the effort worthy games from the rubbish some developers put in there. Also, this IS going ahead. If people from the forums don't want to be involved then you'll just have to put up with whatever the moderators do to filter the games in the DB. So this is basically your chance to be on that panel, or, simply leave it be

[Pumaman]

Yeah, basically at the moment if you're a newbie to the site, and you go to the Games page wanting to find some games to play, it's very difficult to filter the ones worth playing from the rest -- especially in the Short Games category, where there are a lot of good games mixed up with "my first game" ones that don't really deserve to be there.

We're not just adding a rating panel by the way, we'll also be changing the games page to allow you to browse by genre rather than length, but this will obviously require a mammoth effort to categorize all the existing games into genres.

[LimpingFish]

Would we simply be filtering on a creative level? Or would we have powers of dismissal over games that some would perceive as offensive?

Either way, I'd give it a go.

I've seen similar things on other game community websites, but usually it's borne out of necessity since the games are hosted on a central server. Somebody would upload a game in lieu of it being passed for inclusion in the database. If it was deemed worthy, on whatever grounds, it would be added. Otherwise it would be deleted.

Still, it's definately something I would be in favour of.

[SSH]

Will pick of the month still exist? That seems to be used by some to determine good games, since the current pick always features up in the top 5 downloads of the week...

[Vel]

It's good to know that the games database will be upgraded, I always thought further genre and/or theme classification was necessary in order to find games you'd be interested in. I'd apply for that position, but I don't really have as much time as is needed, and long ago, I was known to misbehave. Anyway, good luck to those who will fill these positions!

[m0ds]

I think the third rule has a little leyweigh as to how you act around the community now

And of course pick of the month will remain!!!!

I hear they're also going to include CJ's Dish of the Month on the front page too.

[CodeJunkie]

I agree that the games database needs more order.  Also, Pick of the Month and additional ratings are things to strive for when making a game, which is good.

[Shane 'ProgZmax' Stevens]

As an amateur game maker in an amateur game-making community I have to say I'm a little leery of "experts".

Aren't we all?  Many, many people have direct, often negative comments about things for which they have absolutely no practical experience themselves.  It would certainly be nice for the panel to be made up of people who have actually made games (relatively substantial ones, mind) and know about the effort and challenges involved firsthand rather than a bunch of Roger Eberts.  Game designers will tend to recognize things that most people will overlook-- clever scripting, for instance, and things that operate under the hood.  In spite of any argument, people who have no experience making games really can't understand the effort involved or identify clever uses of the engine if they haven't used it themselves.

[Vince Twelve]

Could we get some more information about the types of ratings that this anonymous panel is going to be handing out?  Is it just organizing them into "worthwhile" and "not worthwhile" piles?  Are they assigning a score to each game?  What exactly is the panel doing?  How much subjectivity is involved?

I'm glad that some cleaning up is being done on the games database and that it's being done by experienced members of the community (for reasons that ProgZ stated), and I can see the reason for anonymity, I just hope that there's some sort of transparency with regards to the procedures that the panel follows.

I would also love to be a member of this panel at some point, though I don't envy the job that the panel will have digging through the existing backlog of games!

[Baron]

I'm entirely in favour of the librarian role of an expert panel, and self-selected volunteers of two or more years standing in the community would certainly be qualified to sort the games by theme(s).

Rating by "experts"....

Since this is going ahead anyway I will refrain from questioning the decision itself and limit my curiosity to the details of the new system.  Will there be transparency (i.e. evaluation for each component of the game: story, graphics, interface, music, etc.)?  Will there be an overall rating or ranking, and how will the component criteria be weighted?  Will there be room for dissenting opinions, such as in a judicial panels (the American/Canadian supreme courts can have a divided opinion 5-4 or 7-2, for example, with dissenters publishing their opinions).  Who will watch the watchers (or evaluate the evaluators....)?  How will a panel drawn from a comparatively small pool of candidates - many of whom know each other - remain impartial when evaluating projects that their acquaintances and friends have produced?  Won't there at least be the temptation to rank such a game just a little better than maybe it deserves?  Are there procedures in place for declaring conflict of interest?  Will the specific criteria used to evaluate the games be published so that game makers eager for a better rating can conform to the systemic values, and game makers who seek to make a revolutionary name for themselves can do the exact opposite?  Will experimental material that is initially ranked poorly be re-evaluated in hindsight?  As this is a community of learners who generally improve over time, can "bad" games not simply be categorized as "training" or "early work" so that they will be seen for what they really are?  Will the panel be able to "farm out" their evaluation powers where they are not qualified to make an evaluation, such as in a foreign language game or a game that caters to a unique cultural context (in the interests of being fair).  Since "Newbies" are the source of this community's future, shouldn't one seat on the panel be open to them as a group so we can take advantage of their fresh perspective (and make them feel that their opinions are valid)?  Same thing for sub-18 year olds: don't they have the most time to commit to a project like this anyway?

I don't mean to be an agitator, but this thread was unlocked to promote discussion on the subject.

[auriond]

We're not just adding a rating panel by the way, we'll also be changing the games page to allow you to browse by genre rather than length, but this will obviously require a mammoth effort to categorize all the existing games into genres.

Not sure if this is the right time and place to bring it up, or even if it's been suggested before, but why not use tags instead? Some games are difficult to categorise (WLBSWHEAC comes to mind), and descriptive keywords might be more helpful.

As for the panel - I see the need for it, but I don't feel qualified, so I'm staying out of it.

[MashPotato]

Glad to hear the database is being reorganized, it will certainly make things more welcoming to new visitors   Just one thing, though: 2 years as a member of the forum seems a bit long for a minimum requirement.  Unfortunately I don't have the time to commit to this job, but even if I did, I'm almost a year away from being considered qualified... and it feels like I've been around here for a while (to me, at least--and I don't mean that in a bad way! )

[Fee]

Im still a noobie around here, however id like the large games database, im SLOWLY working my way through them. If this system goes ahead, maybe you guys could have the other games in a second catagory thats listed at the end of the first list.

[Andail]

The documents which will guide the panel's ratings will be public. However, there will always be a subjective factor involved, since we're dealing with humans.

This is preliminary outline to the rating document.

Of course, there will be as many opinions on this document as there are members in the community.

[Hudders]

So, foreign language games are going to suffer a large handicap, as they can't achieve more than one cup?

Just so we're clear.

[Anym]

Regarding the document, does a game have to fulfill all of the bullet points of a category to be rated accordingly or just a majority or all except one or something? If it's the former, I'm hard pressed to name even one game that would qualify for four cups. Innovative interface? Innovative graphics?

Could any InstaGame/RON/Ben Jordan title get more than one cup, as they all use some graphics that weren't originally made for the game in question?

Even for three or four cups you can't require a game to be more than "virtually bug-free". Testing can only show the presence, but not the absence of bugs and all non-trivial pieces of software (other than TeX) are going to have bugs (even AGS).

[Hudders]

Regarding the document, does a game have to fulfill all of the bullet points of a category to be rated accordingly or just a majority or all except one or something? If it's the former, I'm hard pressed to name even one game that would qualify for four cups. Innovative interface? Innovative graphics?

I think "innovative" is the wrong word in this instance. I'd say they were going more for "fresh".

Nelly Cootalot for example, uses a fresh style of graphics and interface. I wouldn't necessarily call it "innovative" but it is at least distinguishable from most other AGS games.