Rating system

[IndieBoy]

While 2 cups are certainly intended to express encouragement, they do put most players off.

I don't think it's the 2 cups themselves that put most players off. 2 cup games can be off putting only in the context that so many other games you can play that have received 3 cups or more. It's usually all about having to make choices because of a limited time.

The whole "Let's look at what you could of won" thing from Bullseye?

My previous post expresses how I feel on this whole matter, but it seems like this thread is turning into a broken record in terms of points and discussion. Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description. Surely it would be more attractive to players to see the time and effort put into a game in terms of cups rather than the level of enjoyment you will get out of the game in terms of cups.

And babar my game is better than yours 

[ThreeOhFour]

Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description.

If you lot saw how many cans of soda I drink while making each game you'd stage an intervention.

Coca cola: It may shave 3 weeks off my life with each can I drink, but it is delicious, refreshing, and bypasses the need for that stupid sleep thing.

[Babar]

Only after you complete it!


I abuse no substance while making my games. No wonder it is so damned hard .



PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems". Although I get the feeling that very very very few games have user ratings below 50%.

[Calin Leafshade]

sorting by user rating would be a good idea. Since they have quite a large sample size in comparison to the panel rating.

That way it would be useless to inflate the downloads/user rating of your game since the panel rating acts as a normaliser... if theres a game rated 98% with a billion downloads but only has a 1 cup rating its pretty clear something fishy is going on.

I fully support Babars bum.. i mean his idea

[bicilotti]

PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems".

Seconded.

[kaputtnik]

PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems".

Seconded.

That's the spirit! Thirded! 100% = 100%!

[LimpingFish]

One reason the panel was initially formed was as a solution to the inconsistency and abuse that can plague a user-vote rating system.

Now we have calls to give the user ratings more emphasis.

The 2 cup rating for The McCarthy Chronicles upset some people, fair enough. But cries of unfair treatment are counterproductive. The rating was based on a panelist's experience playing through the game, and judging the game's strengths and weaknesses. How can that be unfair? We don't rate what a creator may have intended to do, we rate the finished product. Personally, I don't support the change in rating, but I support that panelist's right to change their mind.

Let me make on thing clear, for future reference. Any ratings bullying, passive-aggressive or otherwise, will not be tolerated. If you have a problem with a rating, either leave a comment on the game's database page, in the game's official forum thread, or, if you must, PM myself or ProgZmax. No effective rating system is going to please everybody, and the panel is comfortable with this fact. Opinions vary. The world turns.

The ratings are not there to promote your game, nor are they there to dissuade people from playing it. They are just one, of many, opinions.

As to those who wonder what it takes to join the panel, potential panel members might initially be contacted due to the amount of levelheadedness, diplomacy, and respect to others that they show on these very forums; so if you come across as suitable for the job, it may eventually be offered to you.

[Calin Leafshade]

I think we've all agreed that the panel is essentially a good system.

I dont particularly think McCarthys rating should have been changed either. It sort of diminishes the 3 cup rating some how.. in a "oh well if you insist" kind of way.

Babar merely suggested that an alternative way of searching through the games would be a good idea in tandem with the panel ratings.

Either system has weaknesses but perhaps a combination would be stronger... its not really a change in the system, just a change in the capabilities of the search engine.

Open discussion never hurt anyone...

[poc301]

Open discussion never hurt anyone...

In Communist Russia...

Ratings are ratings.  I don't even look at number of cups.  I play games that sound interesting to me.  Plain and simple.

-Bill

[LimpingFish]

Open discussion never hurt anyone...

Quite right.

But flogging a dead horse is only fun for so long, and the panel has already stated it's position. We're not ignoring people's opinions on the matter, but we have a job to do. Any changes to the search capabilities of the database are here for CJ to consider, as he is the one who will have to modify the site to accommodate such requests.

We're all here because we enjoy making and playing adventure games, and the database is only one, of many ways, to showcase games. It's also only one of many opinions that may be formed about those games. Everybody who makes a game with AGS will find support in this community (which is bigger than you, me, the panel, the database, the ratings, etc), and nobody will be denied an opportunity to voice their opinion.

Feel the love.

[Calin Leafshade]

which is bigger than you,

NEVAR! *crushes tokyo beneath his feet* RAWWR!

[Helme]

Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description.

I love that idea. That would be about 50 cups for a MAGS-entry 

[MillsJROSS]

I completely agree with Babar's idea. Simply allow users to search by the user given percentage ratings, in addition to the blue cup rating, and this whole mess is sorted out. It should be a fairly straightforward task to code, so as to not take too much of CJs time. Neither the blue cup rating, nor the percentage rating, takes away from each other.

In addition, CJ, I'd also like you to do the following...
1. Make a system that shows me what other people downloaded after this game (like Amazon's feature)
2. Make AGS in 4d
3. My laundery

Not necessarily in that order.

-MillsJROSS

[Snarky]

People seem to be forgetting that 2 cups is still a good rating. I don't know. My game is 2 cups and I'm thrilled with that.

I think that's probably because 2 cups out of 5 certainly doesn't seem like a good rating, especially when 1 cup is reserved for the dregs of the games database. There are four grades left to be assigned to any game that makes an effort to entertain players and that functions well enough, technically, to be evaluated. 2 is the worst one. Even if the description of the grade in the guidelines is written a glass-half-full sort of thing, I think 90% of people who see it are going to take it as a bad rating. (No offense to your game, Mods.)

First and foremost, nobody is ever given preference. I've rated games by people I am friendly with, and games by those whom I may have had "heated debates" with in the past. I don't let personal feelings influence a rating, and I feel no other panel member would. You can take that at face value, or you can dismiss it. So it goes.

You'd think that Mitteneers were a really close-knit group like that but I've never seen evidence of preferential treatment on this forum.  Also, I've never been to Mittens and have not spoken to CJ at length and yet I've been a moderator for almost 6 years.  While I can see where talk like this might occur I've never seen any validity to it!

To be clear, I didn't suggest that panel members actually give preference to people they know (at least not consciously), just that the secrecy means no one can know for sure, and can easily think "they must have something to hide".

There is a well-established mechanism to deal with the problem of how to comment on work by people you know: full disclosure.

A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

I don't agree.  The more likely scenario is one in which an author with several games feels targeted by a reviewer and demands someone else review their work, or demands one of the less-strict reviewers do so.  Anonymity prevents the authors from feeling singled out and for this reason it works better than it would otherwise.

I was talking about players browsing the games db, not the creator.

You guys seem very concerned about the problem of creators complaining about their rating. Again, I think this is an issue that can be solved with more openness, not less. If every complaint had to be submitted openly on the games page, instead of as a PM or something, that could deter "grade-grubbing", and players could see for themselves if the arguments had any substance.

Look, you are apparently very determined to maintain the anonymity of the panel. Fine. I don't agree, but it doesn't seem like there's any chance of changing your minds. But that doesn't mean you can't still have more transparency. How about some of the other suggestions in this thread? Mods mentioned having a single point of contact for the panel, like an AGS OFCOM, for example.

If you're saying you like a game the panel did not review highly that's fine, but if you say there's a problem with the rating because you like a game that isn't rated highly that's another story.  This thread would not exist if some people did not think the latter were true.

For the record, I for one was not wildly enthusiastic about McCarthy Chronicles. I did think the original rating wasn't in line with the (very sparse) public blue-cup guidelines, as I explained in a post on the game thread, but for me the issue was more that the current system doesn't deal very well with cases when the rating appears unreasonable (which will happen now and again in any ratings system). There's little or no explanation of the grade, the criteria are secret, the reviewer is anonymous, there's no official appeal, and you can't even leave another comment on the game page if you already user-rated it.

The documents which will guide the panel's ratings will be public.

I believe in the earlier threads many of the rules were laid out and discussed but the final set of guidelines weren't posted, I'm guessing because the panel was so busy wading through hundreds of games that it just became forgotten.  It is ultimately up to CJ whether or not those guidelines are posted, though.

The panel try to be un-biassed and just follow a standard system, again maybe there needs to be an explanation of it on the page somewhere - but we can't expect everyone to play every game ever made in order to get a fair comment, and so in a lot of cases the reviewer will have to make some judgements by themselves. That person doesn't always feel its fair but I know everyone that does the panel is just following a system well before any personal preferences come in - but alas, you can't escape the fact those preferences will in cases effect the score because 2 panel members doesn't really justify 2000 players. But that's the way it is. It's not hiding anything.

I think I can safely say that no explanation on the subject of what goes into rating a game will be forthcoming, since it would benefit no one except those who would wish to pick it apart.

Wait, what? Most of those (publicly) involved with the panel have said that the criteria will/were meant to/should be made public, and now a categorical refusal all of a sudden? How did that happen?

As to those who wonder what it takes to join the panel, potential panel members might initially be contacted due to the amount of levelheadedness, diplomacy, and respect to others that they show on these very forums; so if you come across as suitable for the job, it may eventually be offered to you.

So how come you're on there? 

[LimpingFish]

So how come you're on there?  

Ah ha ha! Snarky, you wag.

To address three of your points:

An official method of contact is something that's actually been proposed a number of times throughout the history of the panel, and discussions have taken place. Such knowledge wouldn't of course have been known to those without access to the panel forum, which is where this appearance of panel inactivity on the matter may have first arisen from. We simple never came up with a satisfactory answer. Or we got sidelined with fulfilling the actual purpose of the panel; rating games. You may still see something put in place.

The five cup rating system wasn't just pulled out of the air, and it took a significant amount of time to fine tune a system the panel (and CJ) were comfortable with. The idea of half cups or a 10 cup system were also investigated, but abandoned. This is called weighing up your options, but knowing when to make a final decision. We could have tinkered with it until we were blue in the face, making nothing but largely arbitrary changes, but we had over 1000 games to get to. It was also felt that half cups or extra cups would dilute the situation, while adding little. It's also not the panel's duty to worry about how a 2 cup rating appears to potential users. The ratings are designed to give a clear indication of the panel's assessment (one opinion) of the game, and to clearly denote the shifts in opinion between each rating.

There seems to be a lot of paranoia regarding panel anonymity, peoples faith in the ability of an anonymous panel to function fairly, and the underlying fear that we must have something to hide. All three assumptions are unfounded. The ability to search the database by panelist offers two potential outcomes, both largely worthless. You could make your gaming choices based on the opinions of a certain panelist, blindly following that panelists tastes. Or you could automatically dismiss the rating history of a panelist whose views may have caused you to form a negative opinion about them in the past. Both are short-sighted, and detrimental to the goal of the panel; ensuring as wide a mix of opinions as possible.

Despite the nature of the internet, you'd be surprised to learn that a lot of people aren't inherently crooked, don't form grudges, and find little satisfaction in marking a game down simply to win some sort of imaginary one-up on a random internet denizen.

Wait, what? Most of those (publicly) involved with the panel have said that the criteria will/were meant to/should be made public, and now a categorical refusal all of a sudden? How did that happen?

My reading of the situation, and my personal opinion of how the panel should proceed. My safety may be miscalculated.

As I said in a previous post, what purpose would knowing how we rate games serve? If we said we rate games that feature flying chickens higher than others, would everybody suddenly fill their games with airborne poultry?

Everybody needs to take a step back, and concentrate on making their games to the best of their abilities. Not fulfilling a checklist.

[auriond]

As I said in a previous post, what purpose would knowing how we rate games serve? If we said we rate games that feature flying chickens higher than others, would everybody suddenly fill their games with airborne poultry?

I think it would be more like when a player happens upon a lowly-rated game, he could say "oh, that's because it has no flying chickens". He can then personally decided whether he likes flying chickens in a game or not, and then decide whether he wants to download a game or not, as opposed to where he sees a low rating in a game and jumps to the conclusion that the rater's criteria for a good game coincides with his own. A game maker could also take his low rating with more equanimity: "I didn't set out to make a game with flying chickens, so." A mature game maker could even take his HIGH rating with a pinch of salt: "I know that's just because I centred the game around flying chickens. I'll have to be careful if I make a game without those damn fowl."

Sure, it could result in a sudden explosion of games with flying chickens, but if flying chickens were really such a good indicator of a good game, would this really be a bad thing?

As someone who gets paid to rate people's work, I really do think having a list of freely available criteria is important for game makers who really want to improve. Otherwise they're just groping around in the dark.

What harm could come of making a standard list of criteria public? Cookie-cutter games? I doubt so. People are still going to do what they like. Just look at society. I doubt any one list of commandments could homogenise a whole community, especially one as diverse and creative-minded (and stubborn? ) as this one.

What good could come of making the criteria public? Game-makers could try to improve according to the list of criteria. The casual gameplayer could look at the criteria influencing the decision and decide whether he still wants to download the game. Yes, it's debatable that all this will happen, but there's the possibility.

[Shane 'ProgZmax' Stevens]

Initially the cup ratings had a more horrible-bad-fair-good-excellent progression to them, but after discussion the panel agreed that it seemed too harsh, particularly in light of all our new members eager to crank out their first games (which typically fall in the 1-2 cup category).  This is why 1 cup tends to be reserved for broken or just poorly made efforts while 2 is a 'nice try' category that is meant to encourage people to do better.  As mentioned before, 3 is essentially the standard score which is fine since most people will produce a few average but interesting games before they reach something better.  It also allows us to make 5 cups a really privileged category which I think is a good thing.  That's reasoning behind the system, anyway.

As far as the ratings criteria go, again, I think it must really be up to CJ since this is his website and having panel ratings was his decision.  I'm sure as LimpingFish points out some people will indeed pick it apart, but some people also seem intent on picking apart the panel regardless so I fail to see what harm it could do.  The one thing that I will say is the guidelines don't have any 'special' criteria you wouldn't see used for any game rating system and it goes by a set of very common sense rules 90% of you could guess in your sleep.

[Snarky]

What good could come of making the criteria public? Game-makers could try to improve according to the list of criteria. The casual gameplayer could look at the criteria influencing the decision and decide whether he still wants to download the game. Yes, it's debatable that all this will happen, but there's the possibility.

auriond articulated all the reasons why I think open criteria are a good thing better than I could. Well put!

An official method of contact is something that's actually been proposed a number of times throughout the history of the panel, and discussions have taken place. Such knowledge wouldn't of course have been known to those without access to the panel forum, which is where this appearance of panel inactivity on the matter may have first arisen from.

Witness the problems caused by lack of transparency! 

The five cup rating system wasn't just pulled out of the air, and it took a significant amount of time to fine tune a system the panel (and CJ) were comfortable with. The idea of half cups or a 10 cup system were also investigated, but abandoned.

There seems to be some miscommunication... I don't think I (or anyone) have said that we need more fine-grained ratings?

There seems to be a lot of paranoia regarding panel anonymity, peoples faith in the ability of an anonymous panel to function fairly, and the underlying fear that we must have something to hide. All three assumptions are unfounded.
...
Despite the nature of the internet, you'd be surprised to learn that a lot of people aren't inherently crooked, don't form grudges, and find little satisfaction in marking a game down simply to win some sort of imaginary one-up on a random internet denizen.

I must have been making my points badly, because this was never the gist of my argument. The reason I think non-anonymity would be better is that it would make the scores easier to interpret. Even if you're all working from a common set of criteria, there is always a subjective element, and different people have different tastes. To give a concrete example of how this works, on Adventure Gamers, I know that Evan Dickens likes neo-LucasArts-style adventures like Runaway and the Telltale games significantly more than I do. So in my mind, I correct for this difference in bias by adjusting down his scores for those titles. I don't dismiss his opinion (I'm broadly on board with his Top 20 list, for example), I just... adjust.

This whole thing about anonymous members giving preference to some people was just in response to claims that anonymity was necessary to let people review games by people they knew, since I think it doesn't really solve that issue. It's a sidetrack to the main discussion.

It's also not the panel's duty to worry about how a 2 cup rating appears to potential users. The ratings are designed to give a clear indication of the panel's assessment (one opinion) of the game, and to clearly denote the shifts in opinion between each rating.

The ratings are presumably meant to communicate something. In that case, it's worth asking whether they communicate what they're meant to, right?

Now, ProgZmax's post seems to indicate that they mean pretty much what I personally took them to mean, particularly the point that 3 is the standard score, 4 and 5 are reserved for games that excel, and 2 is... well, those games that don't quite measure up to the "average" game.

The one thing that I will say is the guidelines don't have any 'special' criteria you wouldn't see used for any game rating system and it goes by a set of very common sense rules 90% of you could guess in your sleep.

Well, maybe some criteria seem common-sense to you, but not to everyone else (or even just 90% of us):

Difficulty and length has nothing to do with a game's quality. EVER.

Don't you agree that a game shouldn't be so hard that the player can't progress at a reasonable pace, nor easy enough to let him win without trying or thinking?

Also don't you agree that a game should have at least a modicum of content before it ends, to let the player get into the plot and the characters?

You could go both ways on those question, I suppose. (Though personally I tend to agree with Dualnames on this point: just because a game is too hard or too easy for me doesn't mean that it's not perfect for someone else, and if a game is ten or even just five minutes of enjoyment and ends... well, I didn't pay for it, and no one complains that a 3-minute pop song is too short. Besides, the length is indicated separately.) However the panel decided to do it, it would be useful to know about it, again for all the reasons auriond said; because it's actually not something I could have guessed in my sleep.

Let me end this by summarizing some suggestions for ways I think the current ratings system could be improved, which should be fairly simple to implement:

• Make the ratings criteria/guidelines public, link to them from the bluecup explanations (like the "Why is my game rated 1 cup" link).
• Ask reviewers to always fill in the panel review with a short description (a sentence or two) of why the game got the rating it did (what was good enough to earn the grade it got, and what kept it from getting a higher grade, or some other format you decide on).
• Allow people to post multiple comments on the games page, maybe? (Currently there's no way to respond to a rating after you've left a user-rating.)
• Provide a contact address for the panel, for questions, update notifications (like ProgZmax described) and, yes, complaints.
• Make it clear that if any change is made to a rating or classification, the message will be posted to the games page so people can see what happened. This to discourage grade-grubbing and frivolous complaints.
• End panel anonymity. (I know you won't do this.) Even if you don't sign each review individually, you could still let people know who sits on the panel.

I hope you see that this is not about picking apart the panel, just offering some fairly minor tweaks to the system to make it more effective. It's up to you and CJ, of course, but I hope you'll consider it seriously, at least.

[Andail]

Thank you, Snarky, I think you express many valid points.

As one of the initiators of this current system, I ask for respite now to deliberate with the panel.
I would also prefer if panel members would refrain from posting until we have discussed a future direction.

As for now, it seems likely that a document with guidelines and criteria will be made public. Whether ratings will be anonymous or not is another question.

[Khris]

Since the panel is heatedly discussed in a game thread, I thought I'd bump this thread by posting my experience with the Maniac Mansion Mania games:

Pretty much from the start it became common to attach a vote to the game thread, 1 - 5.
In retrospect, many crappy games have high, undeserved ratings, especially after some very high quality episodes were published late in the series. (Also, people didn't want to down-vote newbies' games in order to not put them off of publishing episodes.)
Yet still, the votes reflect nicely what people's first gaming experience was on a scale from shit to superb.

I don't regard two neighboring ratings as always being equidistant from each other; I'd guess a rating of 1 is way worse compared to a 2 cup rating than a 4 cup rating is from getting 5 cups.

My impression of the panel's comment about Cosmos Quest III was that the author put too much work in fixing unnecessary shortcomings of the gameplay and UI and due to that lost sight a bit of the other important parts.
But, like I mentioned in the game's thread: whether the two cups are deserved or not, removing a game from the DB after a bad review is sorta childish.