Rating system

[Calin Leafshade]

Yeah I didn't protest :p and I still don't protest.

It has only helped to spur me on to improve the next episodes and get more cups for my collection.

How about getting started with that  :

I'm working on it smart arse

[Leon]

Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it.

I fail to see the point. I'm not on the panel but can't really see what the problem could be. What's to hide or protect?

I do understand that it's an honour to even be on the AGS games page and to have its accompanying audience.

You, as game maker add your own game to the db, so what's the honor? Even I, if I'd make something crappy (which would be very much the case regarding my abilities) I can add it and am being rated.

But you're right Mods, being on the panel is a scary thought if you're the one coming under fire. It's a good thing that you discussed the rating with other panel members though. I think that's the kind of reassurance we were looking for. We just weren't sure if that was the case with McCarthy.

I don't agree. Why would it be a problem to come under fire? Scared of doing your job right? If the rules are clear (as you mention) and the game is discussed within the panel, you have nothing to worry about.

[Helme]

Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it.

I fail to see the point. I'm not on the panel but can't really see what the problem could be. What's to hide or protect?

Well, some people tend to react really dickish if you critisize their art.

[Leon]

But that's just the point. If your work is being criticized through a checklist that everyone can see and follow, I don't mind if it's done anonymously. If it's a matter of interpretation and personal taste, I'd like to know who's interpretation or taste it was.

Bottom line: it should be open and clear. That prevents discussions like these which surfaces from time to time.

[Ghost]

I am still dead sure that the community rating is the one that counts. The panel rating is a pointer.
Let's see, there are, what, 200 downloads of a game, and about 28 people rated it. They all seem to like the game, some left a nice comment or useful crits, and the ratings are about 60%-ish. Now why should I even bother about the sole blue cup (or let it be two) that I got from a panel (that surely doesn't consist of 200 people)?
It's a pointer.
It's a suggestion. I admit, it is a suggestion that comes from an "official" blue-cup-backed-up panel, and it is on top of the reviews, but still. Without any intention to sound harsh, a game-maker needs some confidence in what he made. If it takes just a few blue pixels to shatter that confidence, hey.

In addition to that, if I remember correctly the rating panel was created because some people thought that community ratings were too easy to screw up and abuse, and that there should be a panel. So as far as I'm concerned, an anonymous panel is the best way to go, and if I am rated by one or three people- I couldn't care less, because all they do is making a suggestion.

[Andail]

We've noted your feelings, Leon.

At the end of the day, this is Chris Jones' site, and you add games to his database. The current system was developed as a compromise between several parties, and it has worked out suprisingly well given the nature of the task.

The rating panel takes into account specific criteria that have been agreed upon beforehand. Before the first games of the enormous backlog were rated, there were example games set up to help get a consistent rating.
The panel works under the supervision of each other and moderators. Any rating can be challenged and discussed. Panel members don't rate their own games.

Many panel members have spent literally hundreds of hours playing through the database. If they ask not to be exposed to all the complaints and pm's and "suggestions" that would follow a signed rating, let's just accept that. The database and the rating may seem all fun and game, but it's a remarkably dirty business.

Just saying, cause you come across as a tad demanding right now.  

[Ali]

We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to. Did the reviewer suffer some kind of repetetive strain injury from typing the word 'derivative'?

On the whole I think the review system serves a useful function and the ratings and comments are normally a very reliable guide.

I do think they got it wrong with a 2 cup rating for McCarthy. 1 - 2 cups discourages me from playing a game, because it suggests a lack of skill and care in the making. I don't think that's the case with McCarthy, in spite of the valid criticisms raised in the AGS review and the game's thread.

I'm not sure there needs to be a right of appeal, but I think the panel should be receptive to public opinion in this case and reconsider the rating.

[Calin Leafshade]

I'm not sure there needs to be a right of appeal, but I think the panel should be receptive to public opinion in this case and reconsider the rating.

I dont think the author should be able to appeal since they simply arent objective. No one can seperate themselves from the amount of hard work they put into a game. Everyone wants 5 cups.

However, having said that, I see no problem with the community at large suggesting a rethink since it is the community that the ratings are for.

[Helme]

We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to.

I've seen several games with a panel rating but no panel comment.

[Ryan Timothy B]

I do agree that it would be super nice if the blue cup rating could be averaged overall from all the panel members who played and rated.  So if one panel member rated a game 2 cups, and the other rated it 4 cups, it would end up at 3 cups. 
But show the blue cup rating even if only one panel member had rated the game, that way you don't have to wait 2 years for a game to be rated by all members.

It would even be cool if there could be multiple reviews from the panel.

Names would be nice, but I agree with Calin that having the name withheld it can preserve honesty.  The panel members are people we know and people we respect, and vice versa.  To rate your friend's game can be a difficult thing to do. 

It's all a matter of opinion.  For instance, Heed (by ben304) was his weakest game, in my opinion (his review even matched that of McCarthy but his rating didn't).  It wasn't my style of game, graphics were very nice, and music was great to listen to, but the puzzles were very basic and almost non existent.  Yet Heed was still rated the same as his other games (except Shifters Box--which has 4 cups) and I personally don't think Heed deserved the same 3 cup rating.
Whoever rated Heed obviously liked it better than I did, or the rating was impacted by Ben's games overall. 
Same with McCarthy Chronicles, it only deserved 2 cups in the eyes of the rater. 
Ratings will always differ from one person to another.

although the game lacks substance when you remove all the dialogue, which lowers its rating.

Obviously if you remove the dialog and nice voice acting from McCarthy it would be a very basic game which wouldn't deserve much of a rating.  But that goes for almost every game if you remove a single element.  It's a game as a whole.


I also agree with Calin that whoever rated his game, should keep their rating.  It's what they feel the game deserves, who is to argue that?

[Shane 'ProgZmax' Stevens]

Even from the start of the panel ratings there have been two types of people:

1.  Sour grapes posters who don't like their rating or ratings on games they like.

2.  People who think the ratings are generally fair even if they don't always agree with them.


I take #2 seriously and no so much #1.


As far as actual ratings go, while they are done by an individual they are often mulled over by the entire panel before the rating is applied, and yes, there are a rather lot of guidelines to go by when rating games.

Really, the system in place isn't going to change because it's already a combination of what works best and fairest.  There's no way games would be rated in a reasonable time frame if all of the panel members had to play every game to completion and meet and meditate over the minutiae, and since the very people who rate these games come from a cross-section of community members your games are all judged by your peers.  

Also, if you don't like a rating you can:

1.  Ignore it.  It's subjective anyway!

2.  If the rating comment includes out of date information (maybe you updated your game and fixed lots of things) you could re-apply for an updated rating if it's important to you by posting in your game thread, leaving a comment on your game page, or by contacting a moderator.


I think I've pretty much said all there is to say about the panel rating system, so if anyone is still bothered about it they can pm me.


Edit:  Oh, about some games not having comments:  sometimes panel members feel a game does not need feedback -- that the rating speaks for itself.  Other times they think a comment might clarify things a bit.  Again, it's a subjective decision and shouldn't be taken to heart.

[GarageGothic]

Perhaps it could be encouraged that the game developer provide links to (and scores or even brief quotes from) external reviews at the bottom of the game's description?

Of course most developers would only select favorable reviews, but that's no different from blurbs on book or DVD covers. While the ideal solution would be for review links to be added to the database without any bias, I think the developer would make more of an effort to track down reviews and keep the information up to date than the panel could do with a one time run-through of available reviews (plus that would only add additional work to an already backlogged staff).

[Calin Leafshade]

I dont think anyone has particularly demonstrated that the ratings ARE flawed. They seem pretty robust to me. Even IF the rating for mccarthy is slightly lower than expected its only 1 out of hundreds. the vast vast majority of ratings are more or less spot on.

[Leon]

Just saying, cause you come across as a tad demanding right now.  

That has never been my intention.

I know the system as it is works but that doesn't mean you have to agree with everything. Maybe I understand more than anyone else what it is to play hundreds of games. And I respect the fact that people like to stay anonymous. But since the rating isn't explained anywhere (other than in discussions like these) I just wondered why the game got the rating it did with the comments that didn't match and seen the reactions, I wasn't alone.

I know too well that things won't change. That's not my intention nor my perception. Respect to the panel (whoever they are)...

[Snarky]

I think the actual ratings are usually very reasonable, and whatever system you went with anyone would be able to find cases where they disagreed with the rating. In my opinion the user comments are more or less worthless (and the weird scales used for the user ratings don't help), while the "official" reviews are actually helpful. Having a selected panel of reliable judges has definitely worked.

However, I don't like how opaque the process is. An anonymous rating just appears, either without comment or with just a brief note. Or sometimes a game doesn't get rated at all. It's not broken down into various factors (like the user ratings, or even competitions like the background blitz). The criteria the panel uses aren't really even described anywhere: this whole notion that gameplay is the decisive factor (and not just in the sense of "it has to be fun to play", but "it has to involve some challenging, substantial gameplay") is certainly one way to do the evaluation, but it's a philosophy that favors certain types of games over others, and how are visitors browsing the database supposed to know that? The descriptions of the various ratings don't mention anything of the kind.

A couple of people have said that if a game creator feels the rating is unfair, it can be appealed. Again, this isn't mentioned anywhere, so it's really a kind of inside secret for people who happen to know about the possibility.

Finally, this whole point about anonymity meaning that the panel members can be honest about games made by people they know. I think that's nonsense. Unless it were a double-blind test (where the panel member doing the rating wouldn't know who made the game), you're not going to eliminate relationship bias. Granted, in a community this small, where everybody knows everybody (more or less), some of these incestuous occurrences are probably unavoidable, but then at least you should have full disclosure.

The result of the anonymity is simply that there's no way for an outsider to know whether the panel are giving preferential treatment to their forum or Mittens friends. Now, I'm sure the panel members are being as honest as they can, but there's a widespread perception that the AGS community is clique-ish, and I remember instances with a number of different game creators who thought they were being discriminated against. So this kind of thing can only lead to more paranoia and conspiracy theories.

I personally think that if you're going to - in effect - publicly slate or hype specific games, particularly in an official capacity, you should be willing to stand by your rating with at least your forum identity. And if you are going to criticize and judge, you should be ready for your judgment to be criticized, too. But that's more of a personal opinion. (I've been pretty harsh on games made by people I like, and people I'm friendly with. Actually, in several cases criticism I've offered has led to a positive dialogue and a closer friendship.)

A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

In conclusion, if the process was more transparent, it wouldn't seem so capricious. Anomalous reviews wouldn't appear to be so arbitrary, because readers would be able to tell why it got the score it did.

I actually just went back and looked at the original thread from when the panel got created. Andail wrote "The documents which will guide the panel's ratings will be public." Did that happen? If it did, where is it?

[auriond]

1.  Sour grapes posters who don't like their rating or ratings on games they like.

2.  People who think the ratings are generally fair even if they don't always agree with them.

Correct me if I'm wrong, but I don't see much indication that we're dealing with #1 here. If we as a community can't speak up for games we like, that's a sad situation. It's not like we're some kind of clique who fawn over each other's games within this community like high school girls. If anything, our feedback is more valuable because we're game developers ourselves. If you look at the McCarthy thread that started all this you'll see an equal measure of praise and criticism.

Basically what happened was this: a game expected to get at least higher than everage ratings, didn't, because of a criteria we didn't know existed (substantial gameplay). If this had been made clear to us from the start, we would have shrugged our shoulders and said "hey, it WAS a short game with not much challenge, fair enough". But we had to have three threads for the fact to emerge that substantial gameplay is a deciding factor.

You, as game maker add your own game to the db, so what's the honor?

The honour to even have the privilege to freely add your game to a database that has a huge audience outside of AGS. It's certainly not a God-given right.

[LimpingFish]

This thread seemed to rapidly gain legs, so bear with me if I repeat anything that's been said before.

First and foremost, nobody is ever given preference. I've rated games by people I am friendly with, and games by those whom I may have had "heated debates" with in the past. I don't let personal feelings influence a rating, and I feel no other panel member would. You can take that at face value, or you can dismiss it. So it goes.

I, nor ProgZmax, have ever actively hidden our membership of the panel (we've both been part of it since the beginning), but we also support other panel member's right not to reveal themselves. The whole anonymous aspect to the situation was simply to avoid a lot of PMs from disgruntled creators. Not adding our names to each rating is part of this strategy. Frankly, I really don't care to enter into discussions with creators, simply because I don't think they can say anything that would change my mind. If you have to explain why your game deserves a higher rating, you're already wasting your time. And mine. Also, it avoids animosity between developers and panelists. It helps to avoid any possible cases of "I don't want so and so reviewing my games anymore!" that may arise.

Ratings are discussed in the panel forum, away from prying eyes (and fragile egos).

We try to rate games on a fairly regular basis, but I admit that we sometimes fall behind. We are not automatons. Finding time to rate everything (and bear in mind, it is everything) that's added to the database can be a job in itself. We could provide daily (shit, hourly!) ratings if people didn't mind their games being played for five minutes and having a rating handed down on the basis of that.

But the majority of games aren't rated unless they've been played to completion (or as close to as we can get). And we use a simple system of merit to calculate that rating. It has proven successful up until now, and I see no reason to complicate it. And no game, to my knowledge, has unfairly suffered through it's use. It may result in longer rating updates, but it works.

As for communication problems, CJ gets the odd PM, and passes it along to us. The amount of complaints we've had doesn't seem to warrant any drastic action, really, but I second ProgZmax's sentiment: I'll happily accept PMs. I'll also happily reserve the right to ignore them if they talk absolute arse.

If you like a game, but disagree with it's rating, leave a comment in the database. That's why it's there. Your opinion is valid, but no more or less than most, including the panel's.

There is no alchemy. There is no formula. There is no inner sanctum you need to enter to get your opinion across, and no rule against speaking up for a game you admire. This is a forum, after all. Just don't automatically expect your opinion to make a difference. We rate a game on it's strengths and weaknesses, not on how loud the support for it shouts.

There is nothing a developer can do beforehand to guarantee a high rating.

Except make the best game they can.

[Shane 'ProgZmax' Stevens]

The result of the anonymity is simply that there's no way for an outsider to know whether the panel are giving preferential treatment to their forum or Mittens friends.

You'd think that Mitteneers were a really close-knit group like that but I've never seen evidence of preferential treatment on this forum.  Also, I've never been to Mittens and have not spoken to CJ at length and yet I've been a moderator for almost 6 years.  While I can see where talk like this might occur I've never seen any validity to it!

A couple of people have said that if a game creator feels the rating is unfair, it can be appealed. Again, this isn't mentioned anywhere, so it's really a kind of inside secret for people who happen to know about the possibility.

I'm not sure who mentioned 'unfair' ratings, but I did state that if an author corrects various problems with their game (severe grammar issues, glitches that break the game, wonky gameplay) they can request a replay.  This is not the same as an author thinking they did not get a fair shake, which isn't something I would necessarily consider to be grounds for a review. 

A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

I don't agree.  The more likely scenario is one in which an author with several games feels targeted by a reviewer and demands someone else review their work, or demands one of the less-strict reviewers do so.  Anonymity prevents the authors from feeling singled out and for this reason it works better than it would otherwise.

I actually just went back and looked at the original thread from when the panel got created. Andail wrote "The documents which will guide the panel's ratings will be public." Did that happen? If it did, where is it?

I believe in the earlier threads many of the rules were laid out and discussed but the final set of guidelines weren't posted, I'm guessing because the panel was so busy wading through hundreds of games that it just became forgotten.  It is ultimately up to CJ whether or not those guidelines are posted, though.

Correct me if I'm wrong, but I don't see much indication that we're dealing with #1 here. If we as a community can't speak up for games we like, that's a sad situation. It's not like we're some kind of clique who fawn over each other's games within this community like high school girls

If you're saying you like a game the panel did not review highly that's fine, but if you say there's a problem with the rating because you like a game that isn't rated highly that's another story.  This thread would not exist if some people did not think the latter were true.

[Ilyich]

I think right now the matter is discussed from the wrong angle. And the right one, in my opinion, is - What's it good for?

Sure, the panel rating is a nice validation for aspiring developers, it shows them that someone actually cares and watches over them mere mortals.

And it's always good to know, that some sort of sophisticated panel of experienced gamers has a similar opinion of a game you liked/disliked.

But who really need these blue cups, are the newcomers to AGS. The first step for them is obvious - award winners, picks of the months and 4-5 cups-rated games. And though it may seem like a lot, most of these great titles will still be filtered by the personal taste of the player. Then what? You either go to the forums for advice, or plow through a field of 3-cup games, hoping to find that special one, that will make your life day complete.
Surely, you won't bother checking the lower rated ones, and that's why I think this argument started in the first place - in the eyes of the player 2 cups means "Unworthy".

And when something a lot of people liked and some even admired is deemed unworthy (in this case - The McCarthy Chronicles 1, which just now got it's third cup, viva la relovution ), it usually serves as a ground to call the whole rating system "bad".

And the plain truth is - AGS-panel works perfectly well as it is, and changing anything in it's structure won't change a thing (well, maybe the arguments will be a little different ). No matter what you do, there will always be a slight margin of error, after all, we are only human. Beep. So thanks to the panel for making the common user's life a tad easier, you're doing  a great job!

[SSH]

We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to. Did the reviewer suffer some kind of repetetive strain injury from typing the word 'derivative'?

I think that right at the beginning of the review panel when they were working through the 900-odd game backlog that was already in the DB, there was no Panel Comment field and that this was added later. That's my recollection from my own ancient-history brief time on the panel. So no comment is perhaps an indication that the reviewer was keen to review that game early... or not.

And as for worthy 3-cuppers that missed out on Awards, if you look at the picks of the month, that's what many of them ARE, particularly in Feb/March each year. As I've said many times before, I'm very happy to recieve suggestions for Pick of the Month (especially from people who aren't the author!) but nearly no-one ever does. I'm sure that if McCarthy doesn't win any awards then it has a good chance of becoming a PoTM next year.