Members for rating panel, apply here!

[SSH]

Many of Yahtzee's games can only get 2 cups because they use RPGMaker music...

I certainly agree that while the very top mark should only go to games that are excellent everywhere, there are plenty of games that entirely neglect something like music which are otherwise excellent. As I usually play with the sound off anyway, its not a big factor for me. Also, if a game gets 4 cups and then someone finds a subtle bug in it, what happens?

There should perhaps be an "unrated" classification for games that plain just do not fit... such as foreign-language-only games

[Andail]

Please try to regard this objectively.

Sure, SSH doesn't play with the sound on, but lots of people find fitting music and sound effects extremely important. I know of people who can't play games with bad soundtracks. Therefore, sound must be accounted for when it comes to a general rating.

Sure, a Russian game with poor translation and grammatical inconsistencies can be GREAT in all other aspects, but the large mainstream audience prefers a correct language.

Sure, InstaGames can be rewarding and fun, but if they would make their own graphics it would move up the overall impression one notch.

You can keep bringing up personal examples of great games until the cows come home, but sooner or later we'd have to settle for some criteria.

My personal favourites (games like Mom's Quest and Aaron's Epic Journey) wouldn't get many cups according to this document (nor would my very own games...), but I'm trying to think outside my own black box here.

[Vel]

So there won't be further theme classification? Like, cartoon, surreal, historical...? I'm sure many people would find it immensely useful while looking for games to play.

[Andail]

Yes, as we said, there will be more genres and classifications as well.
Both regarding settings and genre and some other elements, all browsable.

[m0ds]

Something along these lines, but don't think these are set in stone...

[SSH]

You can keep bringing up personal examples of great games until the cows come home, but sooner or later we'd have to settle for some criteria.

My personal favourites (games like Mom's Quest and Aaron's Epic Journey) wouldn't get many cups according to this document (nor would my very own games...), but I'm trying to think outside my own black box here.

5 Days a Stranger qualifies for 2 cups at best, which will be perceived as not being best. Many people think that this is the best AGS game out there. That is not a personal feeling. Just because we have to draw the line somewhere is not a valid argument for drawing it in the wrong place. I don't think you can put Mom's Quest and 5 Days in the same league.

[scotch]

On the whole, I do hope the cups system comes into effect soon, even if it isn't quite as I'd like. If possible I'd like to avoid the situation where lots of games that people like will be ignored simply because they don't fit a definition of what a game should be, whereas some games that few actually enjoyed playing would be included, just because they do.

My own view is that there should be one criteria: "we liked it". With a short elaboration attached to each award.
The main problem people have with this is that some games that are good, but not to the taste of the small panel, won't get awarded as they deserve. That's definitely true, but I can see many more good games being ignored under the proposed systems than this one.

Because I don't think most people will agree with me on that, I'll suggest something else. If the goal is inclusiveness, and we don't mind a few false positives, then how about a combination of these approaches? If a game has well rendered backgrounds, competant writing, original music, 60 backgrounds, etc, then this will qualify for at least the level of cups described in the rating document. Aside from that, any game the panel happens to like for whatever reasons will be awarded more cups at their discretion. An instagame game that the panel loves wouldn't be disqualified from getting full cups if they want to award it that.

Seems like a decent compromise to me.

[Hudders]

I think you should divide the score into categories, i.e. so many cups for graphics, so many for sound, etc etc. And then a final score, (which might, or might not be an average).

Similar to how a game review might be presented in other media.

[Andail]

I don't think you can put Mom's Quest and 5 Days in the same league.

...which I didn't do. I said I personally prefer Mom's Quest, while aware how little mainstream interest there is in such a game. Simply because I prefer slightly obscure, odd games.
5 days a stranger is good, but many casual gamers would probably say the graphics are worse than many other adventure games out there. Thus, the graphics can't be considered top notch, or as good as for instance Apprentice.
So, bringing up individual games doesn't help, because for a fact, not every game can be rated fairly. It's impossible. They will always lack in some compartment.
Rather make new suggestion how the rating should be carried out.

[SSH]

I like scotch's idea: guidelines with common-sense override. Perhaps an idea of the target distributions should also be kept in mind: Only allow 2% of games (out of say 500 existing games, that makes 10) to get 5 cups,  a further 8% can have 4 cups and 20% can have 3 cups.

Subdivided ratings is not a good idea for this purpose, we want one simple number with only a few divisions. I hope we still keep the user ratings, but perhaps reset them since they have been abused in the past.

[Andail]

Yeah, such a distribution seems reasonable.

The reason why the panel should adhere as much as possible to an official document is that people will demand a certain transparency. If they sense the panel is picking personal favourites, they might feel wrongly treated.
Also, the more subjective feelings you allow for the rating, the more will the decision have to be discussed and agreed on by the entire panel. Otherwise there's no guarantee the rating will be "fair".

[Nikolas]

Not that I have enough time to be in the panel, or that I played a lot of games, but I just realised that I'm younger than 2 years in this forum! 5th of July!

Now, ok to be a member for some time, but as Mash suggested 2 years are a tad a lot!

Now, just one question. Members of the panel, will be choosen amongst those who apply, or you will take all? I would imagine that a panel would need to have some "expertise" somehow, not necessarily in creating, but certainly in playing...

Other than that I would also imagine that a 60-40% weight on public vote vs panel vote, would be the fairer system. As if the public says 5 cups, and the panel says 1 cup there is something fishy going on... The public should still have some power, even if it has been abused before. Needless to say that the end voice should be on the panel, but again, public/audience needs to be heard!

[Andail]

Nik:

The old users' ratings will still be there.

The percentage and the blue cups will co-exist. Making a new average seems reduntant, as does sub-divided ratings.

Also, since the panel will be dynamic, we need a constant flow of new applicants. The first panel will soon be set up, but already in one month one has to go and a new one will be invited. 

[Nikolas]

Next time I'll read the previous posts better (although I actually did read and follow everything. Somehow the above parts missed me... :p) sorry... 

[Baron]

I like the idea of the bell curve for the cup ratings -at least it puts the games in the context of their peers.

The word "original" comes up a lot in your rating criteria but it disadvantages games that intentionally borrow work from other sources (i.e. for the purpose of parody, etc.).  The music in the movie Fantasia was not original, for example, but the main point was that it suited the movie well.  And artists shouldn't be punished for recycling graphics from their own previous works, especially for a sequel which would seem entirely appropriate.  Qualifiers like "appropriate" or "suitable" should be added, so that a game with just one non-original component isn't locked two cups below where it would otherwise deserve.

And why not one seat open for people of 6mo. to 2 years standing?

[m0ds]

Being on this panel will be quite a privaledge, as you can tell - what with all the human element involved. Therefore 2 years is a good compromise, a lot of people have been here that long and a fair few of them are still active. Plus there are people who've been here that long but barely speak up. They are still perfectly eligible. That length of time (2yrs) kinda shows that people are dedicated to being a part of this community. Those who haven't been here 2 years may be just as dedicated, they've just gotta wait it out a bit longer like the rest of us have done

For this first run of the panel, I seriously doubt Andail is going to back down on the 2 year thing.

[Erenan]

BaRoN, I think the common-sense override that scotch and SSH are recommending would deal with things like Fantasia and what-not. I think it sounds like a good idea, and I'm interested to see the new games database layout. EDIT: I mean I'm interested to see how it turns out.

I have just barely been here for two years. But I don't really have any free time any longer. :'(

[Pumaman]

I think you should divide the score into categories, i.e. so many cups for graphics, so many for sound, etc etc. And then a final score, (which might, or might not be an average).

The existing user ratings will remain for graphics, sound, etc. These already do a good job of rating the sub-categories, and I don't see any point in giving the panel the ability to give each game an "official" graphics or sound rating.

As Andail says, the existing rating system will co-exist with the Panel-awarded Cups; the Games page will display both (eg. "Panel Rating: 3/5 Cups; User Rating: 74%"). Therefore, if you wanted to you could completely ignore the Cup-rating of a game, and just rely on the user rating to find a decent game to play.

My own view is that there should be one criteria: "we liked it".

This sounds like a nice idea; but I think that one of the main objectives here is to filter out the poorly made test games that people upload to the games page. They have every right to do so, but by ensuring that these games all get given the lowest cup rating, it enables people to easily ignore them and would motivate people to improve their games before posting them.

Therefore, I think at least a 3-stage rating is required (ie. Bad / Ok / Good); and once you're doing that, you'd may as well split it slightly further to give better feedback. But I certainly don't think that more than 5 cups would be necessary.

I'm interested to see the new games database layout

The reason that the new layout has not gone live yet is because no games actually have any genres set or cup ratings, so it would be really confusing for people searching it to get no results.

Therefore, the panel's first task will be to award cup ratings to some of the existing games in order to get us started.

[Andail]

More than 5 steps is not feasible, since the nuances will be too subtle for a human mind to distinguish between, in a fair and reasonable way.

Just three steps is not inherently wrong...but then again, while we're doing this enormous backlog of games, we might just as well implement some more steps just to add feedback. When you have only 2 steps (in an extreme case) the pressure will be rather high to make the right decision, since classifying a game as "not approved" is quite harsh. Or in other words: The mistake to rate a game that deserves 3 cups out of 5 with only 2 cups our of 5 is not as big as rating a game that deserves "ok" with "not ok".

About the strict requirements: Naturally, there are heaps of people perfectly competent out there who's only been with the community for 3 months, 7 months, a year etc, but this is a matter of trust. People will have their own creations evaluated and graded by an anonymous panel. They at least deserve to know that the members of that panel have met some minimum criteria, and those limits must be drawn somewhere.

Baron: Again, taking the movie Fantasia is the kind of isolated (and rather weird, truthfully) examples I wish that we could avoid.
Instead of thinking that we're "punishing" people, think of it like this:

Game A and game B are equally good in graphics, plot, scripting etc. However, the creators of game A has gone the lengths to actually produce their own soundtrack, developed to specially suit their game. Game B has just nicked some midi-versions of famous pop-classics. Given that A's soundtrack actually fits their game, it should help push that game towards a higher rating.

So instead of trying to find certain games that happen to be terrific in one aspect while still awful in another, try to look at the various parameters and decide what the general audience (the random players out there who wish to try on our games) will appreciate. 

[scotch]

I didn't mean there should be only one rating level, CJ (3 or 5 is fine), just that in my opinion this would work better if it's openly subjective, based on how much the panel personally like the games as they play them, and how much they estimate the gaming public would enjoy them. Rather than based on a checklist list of things that are meant to be indicative of a good game. The list of what makes a worthy game is subjective in any case, and not well agreed upon, so it seems better to leave it.

It's either "We think X, Y, and Z = good" and going to test all the games against X, Y and Z, or simply going and testing all the games on if we think they are good or not. The latter seems less likely to get weird results, to me. Especially if more than one person is testing each game.

I also think it would make the review process better for whoever volunteers. Reviewing when you aren't meant to influence the score seems like a pretty mundane task.