Adventure Game Studio

To prevent other topics becoming 'off-topic' I separated this from the other thread (http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39438.0) into this one to continue the discussion here.

Quote from: Ryan Timothy on Wed 16/12/2009 02:16:33
I didn't expect to see a 2 cup rating. 
After seeing this I actually looked at all the screenshots of every 2 cup games, and out of the few 2 cup games I actually played, I think there were only 2 games I didn't expect to see in there.  This game, and another one.

And to reply to Leon's suggestion.  I have to disagree.  I believe the ratings panel does an exceptional job doing what they do.  I may not agree with this particular rating, but the majority of the ratings, I do agree with.
One reason why I prefer the rating panel over the 'public' rating, is mostly because it's consistent and reliable.  I also like reading the rating panel reviews as well.
I think the rating system should remain as is, with the rating panel deciding the big rating.  You can always appeal if you don't believe your game deserves the rating--perhaps it'll change, or it may actually remain.  No harm in trying.

I know that this has raised questions before and there's a valid point in all opinions. I agree with Ryan that the current rating would result in the most consistent rating if you can see a person, not a group, but the problem is that you don't know who rated your game. It's a panel but different members, different opinions. Is there a list somewhere where you can see who's on board? Why not display who rated your game instead of "a panel". Looks like they want to be protected from their opinion.

My problem with the current system is that it looks like an elite group that judges your game. You can't see who, your game is just being judged. You can't discuss the rating or comment on it. The comments on the games page are for the game, not the judgment.

Quote from: LimpingFish on Tue 18/08/2009 19:18:14
No big mystery, really. :)

Some games, short ones in particular, might get rated faster simply because a panel member can play and rate them fairly quickly. Or, since we also play these games for fun, a panel member might choose something that especially appeals to them, even though other unrated games may have been in the database for a longer period of time.

To be frank, if we happen to be playing a full-length game that's proving to be a bit of a slog, it may take us longer to get around to forming a complete and fair opinion of that game.

Regardless, we have a finite amount of free time to devote to rating these games, but we try to maintain a fair balance.

When a game is rated, its correct categories, as determined by the panel member, will be set if needed.

Plus, you always have the user ratings to keep you going.

But we're always happy to hear people's opinions about the database, and we're always open to new ideas.

That all sounds very informal to me. It's just whoever feels like it and whenever he feels like it. When using a rating system like this every game should be judged, no matter how long the game is or if it's attractive or not. And it should be judged as soon as possible after release.

In fact, AFAIK the panel is meant to be anonymous (I know who they're though, as with any of the moderators here) and is composed of chosen forum members who would cycle once a while to ensure that it's not always the same people. Though each game is actually mainly reviewed by one single member, I think there're some general rules that each of them agreed on and would discuss among themselves whenever a game is rated, so the ratings should be quite objective.

And no, I'm not a member of the panel and probably never will be, as I don't have the time for games.

I agree with the anonymity to some degree.. although it does remove accountability it possibly preserves honesty.

I dont think the games should be reviewed by a single person though. The strength of a 'panel' is multiple opinions, that is kind of the purpose of a panel.

So perhaps the games can be reviewed by more than one person and an amalgamation of their opinions used?

QuoteYou can't discuss the rating or comment on it. The comments on the games page are for the game, not the judgment.

And that is good.

Creating a game is hard work for everyone. You invest a lot of your free time, you do everything you can to please the gamers... but who am I telling ;)

There are a lot of games that aren't much entertaining but still interesting to see how time consuming it must have been.
Even if a game's terrible, it's still a great thing that it's creator made it.

But the rating panel's job is to forget all this. It has to rate the game from a casual gamer's position.. well, and the casual gamer has different demands.
When I show games to my friends, they usually prever the 4-cup rated ones (even though they don't know the rating). They don't know how much work it has been to finish a game, they just want to play a nice game. And the panel does great work finding these games.

Also, why's nobody complaining about not being the pick of the month? 


For getting your ego messaged, we have the game threads.  :=

Why should it be anonymous? If you're rated by a group, maybe. In this case you're rated by an individual... who's hiding in a group (that's how I see it). So why not mention your name? Afraid people disagree?

@abstauber:

Your friends would certainly miss out on the McCarthy Chronicles, amongst other great games.

Like I said, the individuals in the panel are right most of the time. But if they're not, who to address? Why the anonimity, the secrecy around it? It feels to me like an elite group (although they don't see themselves as such). Why not discuss the games more openly and then decide the number of cups.

If you only want to play the 'good' games, wait for the awards. Then you'll know what was liked by the public and play those.

Quote from: abstauber on Wed 16/12/2009 10:43:22

Also, why's nobody complaining about not being the pick of the month? 

I would very much like to be pick of the month.. damn that Marionette!

PM me and I'll send you my paypal address  :=

Ultimately, its done by volunteers, and there's a backlog as it is so any attempts to force more work or double-reviews, etc. on the volunteers will fall flat. However, perhaps there should be a particular effort to review any games that are over 6 months old and not yet reviewed. (e.g. Soviet Unterzoegersdorf Sector II).

I think, though, that there's a huge range of 3 cup games. Some Ben Jordan and Ben304 games are in there and are great and then you've got things like Rock Rock Rock which astounded me by getting 3 cups (no offence to the authors! :) )

QuoteSo why not mention your name? Afraid people disagree?

Anonymity saves friendships ;)

The pick of the month doesn't have to be anonymous, since it only praises. The panel however also slates.

@Calin
;D

Quote from: Leon on Wed 16/12/2009 10:43:34In this case you're rated by an individual...

Wait, wait, so the rating ISN'T by a panel of people? It's just one opinion? I was under the impression that there was some system of rating by a committee. Although looking back at the application thread for the panel, it never said the games would be rated by the whole panel, and I suppose it would be unrealistic for so many people to play the same games and pick a rating. Guess that's my naivety showing.

Quote from: Calin Leafshade on Wed 16/12/2009 10:46:13.. damn that Marionette!

What are you complaining about? You ARE the Marionette.  ;D

Since I don't know how many people are in the panel, why not let more people from the panel play (minimum of 3?) and take the average as a score? With every 'judged' event, you always see individual scores that makes up your end total. And I fail to see the reason why it should be anonymous? Afraid to make the wrong decisions? If I give my opinion about a game, my name should be there. It's my opinion.

If there's a rating by a group, you shouldn't have to put a name under it but I'd still like to know who that group is. And what it was that made them decide the way they did.  Why for instance not show the initials of the person from the panel who rated it, if it's only one person.

I understand that not everyone can play the game, that it's voluntary and takes time. I also see the purpose of the rating and it being done by a 'fixed group of people'. But I fail to see why it all has to be done anonymously.

I was under the impression the games were reviewed by the panel as a whole or at least a portion of the panel. If only a single person rates it how is it any more authoritative than any other review?

I thought the purpose of the panel was to act as a kind of 'normaliser' which is not able to be influenced by the author in any way but still reflects a good estimate of the quality. I mean what if my game was rated by someone who appreciates the puzzles and jokes found in adventures games but nothing else? they might as well have just rated it "boring"

If however you have a group of people is gets rid of that bias.

Quote from: auriond on Wed 16/12/2009 11:00:30

Quote from: Calin Leafshade on Wed 16/12/2009 10:46:13.. damn that Marionette!

What are you complaining about? You ARE the Marionette.  ;D

Very true :p I really need to get round to recording those lines... tonight i promise :p

With respect to everyone here aren't we taking this a little bit too seriously?

I know that a lot of time and effort goes into most finished games (even mine, although that may appear hard to believe! ;D) but at the end of the day I haven't paid anyone anything to rate my game and I can't see that the panel has a duty to review any or all of the games released. If this site held itself out as a review site and offered some kind of promise or indication that all games made would be reviewed then that would be one thing, but speaking personally, I'm just glad of the exposure games can get here rather than just 20 or so of my family and friends!

I don't know who the panel members are, but my guess is that they are just a bunch of guys (and guyettes of course) who work and have other commitments and are doing the community a favour by the ratings they do give. To be honest, if I was on the panel and saw complaints and people trying to hold me to account I don't know how much longer I would stay on it!

Basically this whole thing is really just a hobby to nearly all of us (although seriously draining and hard work at times) and I doubt very much whether those who have produced commercial games are particularly concerned with their blue cup scores!

Okay, since it's all about the score McCarthy got, I've finally downloaded and played it a bit.

I guess you must have slipped in another league :) In the league of awesome games, McCarthy is more like an interactive theatre with not too much focus on gameplay.
I suppose the guy who rated your game was disappointed that your game could have been so much more.
To be honest, if McCarthy would have no sound, cheesy paint graphics and cEgo in it, 2 cups would have been fair in my oppinion.

But since it's not, I think McCarthy is really underrated.


@Leon:

QuoteYour friends would certainly miss out on the McCarthy Chronicles, amongst other great games.

Since they are not addicted to the genre, I'm afraid they live with it. They don't like my game(s) either :D

Quote from: Intense Degree on Wed 16/12/2009 12:37:58
With respect to everyone here aren't we taking this a little bit too seriously?

Well it all started when a game we loved got shortchanged... or so we feel.

You're right of course, that the people on the panel have lives outside of AGS. But that doesn't mean we can't work towards fairness here. If even two people play a game and give a rating, and an average of said rating was taken as the Blue Cup rating, I doubt we would be kicking up such a fuss. The problem is that the panel was supposed to be established in order to provide an unbiased viewpoint, and the question is whether the current way it's being done is achieving that aim of an unbiased view.

so are we all agreed? 5 cups for McCarthy? ok cool.

In all seriousness, I'm a musician so i am *no* stranger to bad reviews.. seriously some reviewers can be feral. Especially if youve spent the evening unknowingly chatting up their girlfriend..

And McCarthy has attracted a fair amount of criticism on both the thread in completed games and on the games page so it is by no means perfect but I feel its a little arbitrary to have a single person decide a rating which is supposed to denote 'quality' in some kind of objective manner. Just because a game doesnt deliver what the reviewer wanted (in this case a challenge) it doesnt mean the game should be penalised for that.

Now this is obviously not the fault of the reviewer. They didnt like it so they marked it down which is entirely reasonable but the rating they gave doesnt seem to be representative of the general community which is exactly what the cup rating should be. If the game was rated by a 'panel' of reviewers it would be more fair.

I could quote the player approval rating but that seems to be a little premature since i have no doubt it will likely drop dramatically as more people vote. My game currently out-rates practically every 4 cup game but its obviously inferior to those.

I will say however that I actually agree with practically every panel rating I see. They are usually perfectly in line with the player approval rating. Sometimes its almost scary how accurate they are in terms of reading the opinion of the community. I also do not want the rating of McCarthy to be changed. It would be arrogant of me to simply assert that the reviewer is wrong and should change their opinion. The rating has actually helped me to see just how weak McCarthy is in a great number of areas and that it does suffer from "style over substance". It has only helped to spur me on to improve the next episodes and get more cups for my collection.

Well, one game on the database has a panel rating & review by me!

And guess what, IT AER TEH MAC-HARTY BARNACLES!!!!!

No, no its not. But for the record the game I reviewed was discussed with other panel members and not just written & judged straight off by me. Or Sly Stallone. Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it. I don't know who wrote the cup-review thing for my game but I still appreciate it & always the criticism. And low cup ratings can seem like a criticism, but yeah - just one of those things in the developer world to man-up about :) What seems like negativity around game production ratings and stuff can actually become a very good motivator for work on future titles. Unless of course you're a moany kid...I dunno...

We have to remember also that games are rated by the panel mostly on gameplay first and then all the rest. If there isn't any gameplay in your game then it simply isn't going to score. 2 cups is actually quoted as being "a game worth checking out". If you've got gameplay you probably get a 2, if you've got good gameplay you've worked up to a 3, then if you have good atmosphere you get your 4 or 5. So I'm assuming, though I haven't played - McArthy Chronicals fell short of 3 or 4 cups because it's a "short game" which apparently has "minimal gameplay". That to me sounds like 1 interactive cutscene. :P

Still, I don't think anyone has actively given up AGS because they've had low cup ratings, or their game just wasn't picked up or recieved well. Nor do I think it's ever affected the right praises being said for certain games. Now you've made a game you'll get the fever for it and just keep going with new ones, twos, trilogies, regardless of cup ratings. Helme asked about what gets a game downloaded, and I just wanted to mention that its generally putting your game other places than just the AGS db that will get it noticed. And in the outside world bluecup ratings don't mean a whole lot! AGS db comments are quaint - and in themselves a way for the people to be anonymous. The reviews that really matter will be sent to you personally!

Anyone is already pretty elite themselves to have the privelledge to post games to the AGS db and know they will be picked up on by a wide community, and more as time goes on and AGS and its games continue to appear in the media and stuff. AGS db and its cup rating is pretty weak in terms of criticism you could really be facing from outside sources, review websites and such. And it seems like you know that :)

Quote from: Calin Leafshade on Wed 16/12/2009 14:33:23
My game currently out-rates practically every 4 cup game but its obviously inferior to those.

Cult fame (http://www.mytopdozen.com/Best_Cult_Films.html) baby!  ;D

Clockwork orange only at 12? Thats bullshit.. which idiot rated that...

QuoteIt has only helped to spur me on to improve the next episodes and get more cups for my collection.

How about getting started with that  ::)

I do understand that it's an honour to even be on the AGS games page and to have its accompanying audience. And seriously, criticism comes with the territory. I think anyone who's made a game knows that.

But then why not make the process transparent? It's not clear, if you click on the "What is this?" next to the bluecup ratings, that the rating is based mostly on gameplay, or that it was implemented by one person or many. Again it all comes down to what is the bluecup rating for? If it's all that unimportant then why have it in the first place? I think we agree that it was needed, and therefore it is important, if not in the big scheme of things then at least to the AGS community.

In the end we just want to know how the ratings are implemented. I don't think that's a great deal to ask. The application thread for the panel mentioned a set of guidelines that every reviewer would have to follow. If we know what this set of guidelines is, we'd have a clearer idea of how to take the bluecup ratings. Notice that it wasn't Calin who started protesting his rating, but the rest of us who played his game. So it's not a matter of game makers being whiny over low ratings on their own games, but rather other members of the community who went "Why did this game get such low ratings? Hey, how are the ratings decided anyway? Who does them?"

When you get right down to it we made our games with AGS, so we care what AGS people think of them and how they are rated on the AGS games page. I don't think we can help it unless we never invested anything in this community in the first place.

But you're right Mods, being on the panel is a scary thought if you're the one coming under fire. It's a good thing that you discussed the rating with other panel members though. I think that's the kind of reassurance we were looking for. We just weren't sure if that was the case with McCarthy.

Yeah I didn't protest :p and I still don't protest.

Quote from: abstauber on Wed 16/12/2009 15:31:27

QuoteIt has only helped to spur me on to improve the next episodes and get more cups for my collection.

How about getting started with that  ::)

I'm working on it smart arse :P

Quote from: Mods on Wed 16/12/2009 15:11:02
Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it.

I fail to see the point. I'm not on the panel but can't really see what the problem could be. What's to hide or protect?

Quote from: auriond on Wed 16/12/2009 15:31:58
I do understand that it's an honour to even be on the AGS games page and to have its accompanying audience.

You, as game maker add your own game to the db, so what's the honor? Even I, if I'd make something crappy (which would be very much the case regarding my abilities) I can add it and am being rated.

Quote from: auriond on Wed 16/12/2009 15:31:58
But you're right Mods, being on the panel is a scary thought if you're the one coming under fire. It's a good thing that you discussed the rating with other panel members though. I think that's the kind of reassurance we were looking for. We just weren't sure if that was the case with McCarthy.

I don't agree. Why would it be a problem to come under fire? Scared of doing your job right? If the rules are clear (as you mention) and the game is discussed within the panel, you have nothing to worry about.

Quote from: Leon on Wed 16/12/2009 16:20:52

Quote from: Mods on Wed 16/12/2009 15:11:02
Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it.

I fail to see the point. I'm not on the panel but can't really see what the problem could be. What's to hide or protect?

Well, some people tend to react really dickish if you critisize their art.

But that's just the point. If your work is being criticized through a checklist that everyone can see and follow, I don't mind if it's done anonymously. If it's a matter of interpretation and personal taste, I'd like to know who's interpretation or taste it was.

Bottom line: it should be open and clear. That prevents discussions like these which surfaces from time to time.

I am still dead sure that the community rating is the one that counts. The panel rating is a pointer.
Let's see, there are, what, 200 downloads of a game, and about 28 people rated it. They all seem to like the game, some left a nice comment or useful crits, and the ratings are about 60%-ish. Now why should I even bother about the sole blue cup (or let it be two) that I got from a panel (that surely doesn't consist of 200 people)?
It's a pointer.
It's a suggestion. I admit, it is a suggestion that comes from an "official" blue-cup-backed-up panel, and it is on top of the reviews, but still. Without any intention to sound harsh, a game-maker needs some confidence in what he made. If it takes just a few blue pixels to shatter that confidence, hey.

In addition to that, if I remember correctly the rating panel was created because some people thought that community ratings were too easy to screw up and abuse, and that there should be a panel. So as far as I'm concerned, an anonymous panel is the best way to go, and if I am rated by one or three people- I couldn't care less, because all they do is making a suggestion.

We've noted your feelings, Leon.

At the end of the day, this is Chris Jones' site, and you add games to his database. The current system was developed as a compromise between several parties, and it has worked out suprisingly well given the nature of the task.

The rating panel takes into account specific criteria that have been agreed upon beforehand. Before the first games of the enormous backlog were rated, there were example games set up to help get a consistent rating.
The panel works under the supervision of each other and moderators. Any rating can be challenged and discussed. Panel members don't rate their own games.

Many panel members have spent literally hundreds of hours playing through the database. If they ask not to be exposed to all the complaints and pm's and "suggestions" that would follow a signed rating, let's just accept that. The database and the rating may seem all fun and game, but it's a remarkably dirty business.

Just saying, cause you come across as a tad demanding right now.  

We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to. Did the reviewer suffer some kind of repetetive strain injury from typing the word 'derivative'?

On the whole I think the review system serves a useful function and the ratings and comments are normally a very reliable guide.

I do think they got it wrong with a 2 cup rating for McCarthy. 1 - 2 cups discourages me from playing a game, because it suggests a lack of skill and care in the making. I don't think that's the case with McCarthy, in spite of the valid criticisms raised in the AGS review and the game's thread.

I'm not sure there needs to be a right of appeal, but I think the panel should be receptive to public opinion in this case and reconsider the rating.

Quote from: Ali on Wed 16/12/2009 18:50:50
I'm not sure there needs to be a right of appeal, but I think the panel should be receptive to public opinion in this case and reconsider the rating.

I dont think the author should be able to appeal since they simply arent objective. No one can seperate themselves from the amount of hard work they put into a game. Everyone wants 5 cups.

However, having said that, I see no problem with the community at large suggesting a rethink since it is the community that the ratings are for.

Quote from: Ali on Wed 16/12/2009 18:50:50
We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to.

I've seen several games with a panel rating but no panel comment.

I do agree that it would be super nice if the blue cup rating could be averaged overall from all the panel members who played and rated.  So if one panel member rated a game 2 cups, and the other rated it 4 cups, it would end up at 3 cups. 
But show the blue cup rating even if only one panel member had rated the game, that way you don't have to wait 2 years for a game to be rated by all members.

It would even be cool if there could be multiple reviews from the panel.

Names would be nice, but I agree with Calin that having the name withheld it can preserve honesty.  The panel members are people we know and people we respect, and vice versa.  To rate your friend's game can be a difficult thing to do. 

It's all a matter of opinion.  For instance, Heed (by ben304) was his weakest game, in my opinion (his review even matched that of McCarthy but his rating didn't).  It wasn't my style of game, graphics were very nice, and music was great to listen to, but the puzzles were very basic and almost non existent.  Yet Heed was still rated the same as his other games (except Shifters Box--which has 4 cups) and I personally don't think Heed deserved the same 3 cup rating.
Whoever rated Heed obviously liked it better than I did, or the rating was impacted by Ben's games overall. 
Same with McCarthy Chronicles, it only deserved 2 cups in the eyes of the rater. 
Ratings will always differ from one person to another.

Quotealthough the game lacks substance when you remove all the dialogue, which lowers its rating.

Obviously if you remove the dialog and nice voice acting from McCarthy it would be a very basic game which wouldn't deserve much of a rating.  But that goes for almost every game if you remove a single element.  It's a game as a whole.


I also agree with Calin that whoever rated his game, should keep their rating.  It's what they feel the game deserves, who is to argue that?

Even from the start of the panel ratings there have been two types of people:

1.  Sour grapes posters who don't like their rating or ratings on games they like.

2.  People who think the ratings are generally fair even if they don't always agree with them.


I take #2 seriously and no so much #1.


As far as actual ratings go, while they are done by an individual they are often mulled over by the entire panel before the rating is applied, and yes, there are a rather lot of guidelines to go by when rating games.

Really, the system in place isn't going to change because it's already a combination of what works best and fairest.  There's no way games would be rated in a reasonable time frame if all of the panel members had to play every game to completion and meet and meditate over the minutiae, and since the very people who rate these games come from a cross-section of community members your games are all judged by your peers.  

Also, if you don't like a rating you can:

1.  Ignore it.  It's subjective anyway!

2.  If the rating comment includes out of date information (maybe you updated your game and fixed lots of things) you could re-apply for an updated rating if it's important to you by posting in your game thread, leaving a comment on your game page, or by contacting a moderator.


I think I've pretty much said all there is to say about the panel rating system, so if anyone is still bothered about it they can pm me.


Edit:  Oh, about some games not having comments:  sometimes panel members feel a game does not need feedback -- that the rating speaks for itself.  Other times they think a comment might clarify things a bit.  Again, it's a subjective decision and shouldn't be taken to heart.

Perhaps it could be encouraged that the game developer provide links to (and scores or even brief quotes from) external reviews at the bottom of the game's description?

Of course most developers would only select favorable reviews, but that's no different from blurbs on book or DVD covers. While the ideal solution would be for review links to be added to the database without any bias, I think the developer would make more of an effort to track down reviews and keep the information up to date than the panel could do with a one time run-through of available reviews (plus that would only add additional work to an already backlogged staff).

I dont think anyone has particularly demonstrated that the ratings ARE flawed. They seem pretty robust to me. Even IF the rating for mccarthy is slightly lower than expected its only 1 out of hundreds. the vast vast majority of ratings are more or less spot on.

Quote from: Andail on Wed 16/12/2009 18:42:15
Just saying, cause you come across as a tad demanding right now.  

That has never been my intention.

I know the system as it is works but that doesn't mean you have to agree with everything. Maybe I understand more than anyone else what it is to play hundreds of games. And I respect the fact that people like to stay anonymous. But since the rating isn't explained anywhere (other than in discussions like these) I just wondered why the game got the rating it did with the comments that didn't match and seen the reactions, I wasn't alone.

I know too well that things won't change. That's not my intention nor my perception. Respect to the panel (whoever they are)...

I think the actual ratings are usually very reasonable, and whatever system you went with anyone would be able to find cases where they disagreed with the rating. In my opinion the user comments are more or less worthless (and the weird scales used for the user ratings don't help), while the "official" reviews are actually helpful. Having a selected panel of reliable judges has definitely worked.

However, I don't like how opaque the process is. An anonymous rating just appears, either without comment or with just a brief note. Or sometimes a game doesn't get rated at all. It's not broken down into various factors (like the user ratings, or even competitions like the background blitz). The criteria the panel uses aren't really even described anywhere: this whole notion that gameplay is the decisive factor (and not just in the sense of "it has to be fun to play", but "it has to involve some challenging, substantial gameplay") is certainly one way to do the evaluation, but it's a philosophy that favors certain types of games over others, and how are visitors browsing the database supposed to know that? The descriptions of the various ratings don't mention anything of the kind.

A couple of people have said that if a game creator feels the rating is unfair, it can be appealed. Again, this isn't mentioned anywhere, so it's really a kind of inside secret for people who happen to know about the possibility.

Finally, this whole point about anonymity meaning that the panel members can be honest about games made by people they know. I think that's nonsense. Unless it were a double-blind test (where the panel member doing the rating wouldn't know who made the game), you're not going to eliminate relationship bias. Granted, in a community this small, where everybody knows everybody (more or less), some of these incestuous occurrences are probably unavoidable, but then at least you should have full disclosure.

The result of the anonymity is simply that there's no way for an outsider to know whether the panel are giving preferential treatment to their forum or Mittens friends. Now, I'm sure the panel members are being as honest as they can, but there's a widespread perception that the AGS community is clique-ish, and I remember instances with a number of different game creators who thought they were being discriminated against. So this kind of thing can only lead to more paranoia and conspiracy theories.

I personally think that if you're going to - in effect - publicly slate or hype specific games, particularly in an official capacity, you should be willing to stand by your rating with at least your forum identity. And if you are going to criticize and judge, you should be ready for your judgment to be criticized, too. But that's more of a personal opinion. (I've been pretty harsh on games made by people I like, and people I'm friendly with. Actually, in several cases criticism I've offered has led to a positive dialogue and a closer friendship.)

A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

In conclusion, if the process was more transparent, it wouldn't seem so capricious. Anomalous reviews wouldn't appear to be so arbitrary, because readers would be able to tell why it got the score it did.

I actually just went back and looked at the original thread from when the panel got created. Andail wrote "The documents which will guide the panel's ratings will be public." Did that happen? If it did, where is it?

Quote from: ProgZmax on Wed 16/12/2009 19:24:41
1.  Sour grapes posters who don't like their rating or ratings on games they like.

2.  People who think the ratings are generally fair even if they don't always agree with them.

Correct me if I'm wrong, but I don't see much indication that we're dealing with #1 here. If we as a community can't speak up for games we like, that's a sad situation. It's not like we're some kind of clique who fawn over each other's games within this community like high school girls. If anything, our feedback is more valuable because we're game developers ourselves. If you look at the McCarthy thread that started all this you'll see an equal measure of praise and criticism.

Basically what happened was this: a game expected to get at least higher than everage ratings, didn't, because of a criteria we didn't know existed (substantial gameplay). If this had been made clear to us from the start, we would have shrugged our shoulders and said "hey, it WAS a short game with not much challenge, fair enough". But we had to have three threads for the fact to emerge that substantial gameplay is a deciding factor.

Quote from: Leon on Wed 16/12/2009 16:20:52
You, as game maker add your own game to the db, so what's the honor?

The honour to even have the privilege to freely add your game to a database that has a huge audience outside of AGS. It's certainly not a God-given right.

This thread seemed to rapidly gain legs, so bear with me if I repeat anything that's been said before.

First and foremost, nobody is ever given preference. I've rated games by people I am friendly with, and games by those whom I may have had "heated debates" with in the past. I don't let personal feelings influence a rating, and I feel no other panel member would. You can take that at face value, or you can dismiss it. So it goes.

I, nor ProgZmax, have ever actively hidden our membership of the panel (we've both been part of it since the beginning), but we also support other panel member's right not to reveal themselves. The whole anonymous aspect to the situation was simply to avoid a lot of PMs from disgruntled creators. Not adding our names to each rating is part of this strategy. Frankly, I really don't care to enter into discussions with creators, simply because I don't think they can say anything that would change my mind. If you have to explain why your game deserves a higher rating, you're already wasting your time. And mine. Also, it avoids animosity between developers and panelists. It helps to avoid any possible cases of "I don't want so and so reviewing my games anymore!" that may arise.

Ratings are discussed in the panel forum, away from prying eyes (and fragile egos).

We try to rate games on a fairly regular basis, but I admit that we sometimes fall behind. We are not automatons. Finding time to rate everything (and bear in mind, it is everything) that's added to the database can be a job in itself. We could provide daily (shit, hourly!) ratings if people didn't mind their games being played for five minutes and having a rating handed down on the basis of that.

But the majority of games aren't rated unless they've been played to completion (or as close to as we can get). And we use a simple system of merit to calculate that rating. It has proven successful up until now, and I see no reason to complicate it. And no game, to my knowledge, has unfairly suffered through it's use. It may result in longer rating updates, but it works.

As for communication problems, CJ gets the odd PM, and passes it along to us. The amount of complaints we've had doesn't seem to warrant any drastic action, really, but I second ProgZmax's sentiment: I'll happily accept PMs. I'll also happily reserve the right to ignore them if they talk absolute arse.

If you like a game, but disagree with it's rating, leave a comment in the database. That's why it's there. Your opinion is valid, but no more or less than most, including the panel's.

There is no alchemy. There is no formula. There is no inner sanctum you need to enter to get your opinion across, and no rule against speaking up for a game you admire. This is a forum, after all. Just don't automatically expect your opinion to make a difference. We rate a game on it's strengths and weaknesses, not on how loud the support for it shouts.

There is nothing a developer can do beforehand to guarantee a high rating.

Except make the best game they can.

Quote
The result of the anonymity is simply that there's no way for an outsider to know whether the panel are giving preferential treatment to their forum or Mittens friends.

You'd think that Mitteneers were a really close-knit group like that but I've never seen evidence of preferential treatment on this forum.  Also, I've never been to Mittens and have not spoken to CJ at length and yet I've been a moderator for almost 6 years.  While I can see where talk like this might occur I've never seen any validity to it!

Quote
A couple of people have said that if a game creator feels the rating is unfair, it can be appealed. Again, this isn't mentioned anywhere, so it's really a kind of inside secret for people who happen to know about the possibility.

I'm not sure who mentioned 'unfair' ratings, but I did state that if an author corrects various problems with their game (severe grammar issues, glitches that break the game, wonky gameplay) they can request a replay.  This is not the same as an author thinking they did not get a fair shake, which isn't something I would necessarily consider to be grounds for a review. 

Quote
A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

I don't agree.  The more likely scenario is one in which an author with several games feels targeted by a reviewer and demands someone else review their work, or demands one of the less-strict reviewers do so.  Anonymity prevents the authors from feeling singled out and for this reason it works better than it would otherwise.

Quote
I actually just went back and looked at the original thread from when the panel got created. Andail wrote "The documents which will guide the panel's ratings will be public." Did that happen? If it did, where is it?

I believe in the earlier threads many of the rules were laid out and discussed but the final set of guidelines weren't posted, I'm guessing because the panel was so busy wading through hundreds of games that it just became forgotten.  It is ultimately up to CJ whether or not those guidelines are posted, though.

Quote
Correct me if I'm wrong, but I don't see much indication that we're dealing with #1 here. If we as a community can't speak up for games we like, that's a sad situation. It's not like we're some kind of clique who fawn over each other's games within this community like high school girls

If you're saying you like a game the panel did not review highly that's fine, but if you say there's a problem with the rating because you like a game that isn't rated highly that's another story.  This thread would not exist if some people did not think the latter were true.

I think right now the matter is discussed from the wrong angle. And the right one, in my opinion, is - What's it good for?

Sure, the panel rating is a nice validation for aspiring developers, it shows them that someone actually cares and watches over them mere mortals.

And it's always good to know, that some sort of sophisticated panel of experienced gamers has a similar opinion of a game you liked/disliked.

But who really need these blue cups, are the newcomers to AGS. The first step for them is obvious - award winners, picks of the months and 4-5 cups-rated games. And though it may seem like a lot, most of these great titles will still be filtered by the personal taste of the player. Then what? You either go to the forums for advice, or plow through a field of 3-cup games, hoping to find that special one, that will make your life day complete.
Surely, you won't bother checking the lower rated ones, and that's why I think this argument started in the first place - in the eyes of the player 2 cups means "Unworthy".

And when something a lot of people liked and some even admired is deemed unworthy (in this case - The McCarthy Chronicles 1, which just now got it's third cup, viva la relovution :)), it usually serves as a ground to call the whole rating system "bad".

And the plain truth is - AGS-panel works perfectly well as it is, and changing anything in it's structure won't change a thing (well, maybe the arguments will be a little different :)). No matter what you do, there will always be a slight margin of error, after all, we are only human. Beep. So thanks to the panel for making the common user's life a tad easier, you're doing  a great job! :)

Quote from: Ali on Wed 16/12/2009 18:50:50
We're all missing the big issue here, which is: Nelly Cootalot doesn't have a AGS Review Panel comment! All the other rated games seem to. Did the reviewer suffer some kind of repetetive strain injury from typing the word 'derivative'?

I think that right at the beginning of the review panel when they were working through the 900-odd game backlog that was already in the DB, there was no Panel Comment field and that this was added later. That's my recollection from my own ancient-history brief time on the panel. So no comment is perhaps an indication that the reviewer was keen to review that game early... or not.

And as for worthy 3-cuppers that missed out on Awards, if you look at the picks of the month, that's what many of them ARE, particularly in Feb/March each year. As I've said many times before, I'm very happy to recieve suggestions for Pick of the Month (especially from people who aren't the author!) but nearly no-one ever does. I'm sure that if McCarthy doesn't win any awards then it has a good chance of becoming a PoTM next year.

People should be able to disable AGS rating comments and panels or contact the guy who rated them and ask the reason. I find that a 3 cup rating on a game that is praised by 90% of the people to be 10/10, not to receive a 4 cup (defines the game as a must-play), to be unjust. As unjust is that H2G2 towel day got rated 3 cups though it can be enjoyable and breakdown by one dollar was rated with two stars because it had 1 puzzle that was a little hard. Difficulty and length has nothing to do with a game's quality. EVER. Shifter's Box is like a two hour game MAX, though it's a really enjoyable experience. While other games last more, and offer nothing. I've never really bothered about the rating system, but certainly seems unfair for this game.

if you can disable it whats the point? everyone who got less than 3 cups would disable it.

Dualnames, I wasn't able to understand half of your post.

Quote
As unjust is that H2G2 towel day got rated 3 cups though it can be enjoyable and breakdown by one dollar was rated with two stars because it had 1 puzzle that was a little hard

Are you saying it's "unjust" that your game got 3 cups...because it's enjoyable? Don't get it. Is it worth more? Or less?
And is Breakdown's rating unfair because of...what? The panel left a comment to explain the rating. Apparently the panel member found that "Shows a lot of promise but the gameplay consists of more than a few tedious pixel hunts or illogical/strange solutions."

Quote
Difficulty and length has nothing to do with a game's quality. EVER.

Don't you agree that a game shouldn't be so hard that the player can't progress at a reasonable pace, nor easy enough to let him win without trying or thinking?

Also don't you agree that a game should have at least a modicum of content before it ends, to let the player get into the plot and the characters?

I think everyone needs to remember, before you jump on the complaint bandwagon, that the descripition for 2 cups is that the game is reasonable and worth a try.

No wonder Andail.

I just find that H2G2 Towel Day got a three cup rating, and I considered it short of fair. I however seem to find unfair to have a game (MCC), rated only 3 cups. Perhaps you are right though. I find reason in your points, and to be fair you are right.

I never did find a rating to be unjust concerning to the comment it had. However it's like focusing on the cons of the game. It's either wrong to focus on the pros and the cons of the game and rate it based on that.

As for difficulty, indeed the puzzles should be REASONABLE.People nowadays forget that. I detest pixel hunt except if it's done right.

Gah! I never get involved with these random forum rants:

I always thought the ratings were down to organising the database and not a specific, 100% accurate and official rating made by "AGS elite". Such an organisation is only to really benefit non-ags community members and non-active community members, which will tend to only play the 4/5 cup rated games (generally). Any other game with whatever rating is still played by the community and will get the same exposure as any rated game on the database (excluding the honour and promotion of pick of the month and the awards). I mean people aren't going to be put off by a rating, I wouldn't be and this is also mentioned by a few members in this thread. Evidence is mags games, hourgames, orow games, currently unrated games, currently popular game threads etc.

Now disabling the rating after over a thousand games have been individually played and rated over the last few years, with the same system is silly and is rather insulting to the panel who have taken the time to do a service to the community which is un-paid.

Although saying this nothing is perfect, and it's going to be near impossible to make a 100% consistent system, due to the nature of the community and the medium that is games which is all rather subjective. I am totally honoured, with my demo (which was originally a short game rated 1 cup) and the 3 cup rated mags game I was involved with, to be in the database and was individually rated. However I am not taking it as a level of stature which I come think people come to desire, i.e I'm not screaming my ratings for my games in the street.

Anyway the database is controlled by a panel, let them go do what they want with their database. The awards, the threads in forums, personal blogs and countless other review type sites is down to everyone else. And with this whole thing been started by Calin's game I'm sure, by the recent publicity he will be a strong contender in these up and coming awards, which the community decide.

Games are mean't to be fun, why can't we be all light hearted.. and fun?

I've never said anything about my games because they deserved every rating they've got. And I find it more or less fair, so it's up to Calin, to say if he feels that the rating he got is unfair. I'm lighthearted alright. Hell, for all I care, I'm mostly proud of them, even those that got a 1 cup rating.

I didnt feel the 2 cup rating was that unfair.. it seemed reasonable enough after the initial "thats bullshit" moment.

The 3 cup rating seems perfectly justified to me and I don't feel the game has merit to be a 4 cup game by any criteria.

People seem to be forgetting that 2 cups is still a good rating. I don't know. My game is 2 cups and I'm thrilled with that.

As for why the panel should remain anynomous etc, simply cos none of the panel members should be asked to justify what they're doing nor should their personal lives be interupted because of it. A system is in place, it has guidelines, that is that! You really don't have to put your games on the db if you think your cup rating is going to affect downloads or attention.

QuoteI fail to see the point. I'm not on the panel but can't really see what the problem could be. What's to hide or protect?

Nothing to hide or protect, it just saves time. Perhaps there does need to be one point of contact, a bit like an AGS OFCOM so that where people feel the rating isnt right, be it about cups or where say violence hasn't been correctly rated - to vent it directly to the panel - but that would be a system like an email form, because again I don't think anyone really wants to spend the extra time discussing all the rating and panel stuff.

The panel try to be un-biassed and just follow a standard system, again maybe there needs to be an explanation of it on the page somewhere - but we can't expect everyone to play every game ever made in order to get a fair comment, and so in a lot of cases the reviewer will have to make some judgements by themselves. That person doesn't always feel its fair but I know everyone that does the panel is just following a system well before any personal preferences come in - but alas, you can't escape the fact those preferences will in cases effect the score because 2 panel members doesn't really justify 2000 players. But that's the way it is. It's not hiding anything.

The only thing it protects is privacy. I don't want you bugging me about my affiliation with the panel, and I'm sure many of the others that help on it don't want to be pestered about it either. It's a bit of fun and we all knew from the offset that some people would not like the ratings in some cases - and lo & behold it does happen occasionally, people have removed all their games because of one bad rating, pulled their downloads and undoubtedly done other things I don't know about. But the serious game developers just keep going, or just publish games to their own websites.

The reason why people forget a two cup rating is still good is that it's only one step up from "Not serious entertainment". Also that 2/5 = 40%. :P "Sorry kid... you missed the halfway mark by that much. You nearly made it though. For serious. Welp, better luck next time."

Not much else more to say at this point except to wait for official word on whether those mysterious criteria will be made public.

Quote from: auriond on Fri 18/12/2009 00:49:42
Not much else more to say at this point except to wait for official word on whether those mysterious criteria will be made public.

There is no mystery.

And some games do indeed miss the 3 cup mark by that much. In some cases it goes to a discussion. Sometimes a majority supports the 3 cup, sometimes the 2 cup.

We always knew the 2 cup rating was going to be the biggest opinion divider, but we accept that.

It may indeed only be one step up from a 1 cup, but it's also only one step down from a 3 cup. Cup half empty, cup half full.

I think I can safely say that no explanation on the subject of what goes into rating a game will be forthcoming, since it would benefit no one except those who would wish to pick it apart. Although I can make up a secret formula, if people need one:

Effort + Talent * Excellence = 5 cups! \o/

Seriously, if you enter into making a game trying to second guess someone else's opinion, you not going to enjoy the experience.

Which is one reason why I believe:
1/5 cups should be for joke games, poorly made games, or games that aren't really even worthwhile or broken.  
2/5 cups should be for somewhat entertaining games with little content (in some areas perhaps lacking dialog, interactions, animations, a story or other important elements)
and obviously 3/5 being for the average game

These are free games, so obviously we shouldn't be too strict on the rating but also not too lenient either since honest ratings will help promote the games that deserve the higher rating, and in the end they'll promote AGS; giving us more games!! :)

I must have missed the post where they were made public then... *goes back to look*

Half-empty, half-full indeed. Well, we can't all be optimists. :D

I agree with Ryan Timothy, that it is intuitive to think of 3/5 as average.  The bell-curve of games in the database is heavily skewed in that there are very few 4/5 and 5/5 games.  Perhaps those categories should have been merged into a "really good" 4/4 or 5/5 category....  But it's all pointless to debate it now, since you'll never muster the enthusiasm to go back and re-rate the 900+ games already in the database.  For consistency's sake we are more or less stuck with the status quo, so us game makers are just going to have to roll with the punches.  If a game has been perceived to be slighted, recent evidence suggests that raising a ruckus will correct the problem -that's good enough for me.

I think a section where you could provide links to reviews in the games database is a wonderful idea, but it will have to be author driven.  Any serious potential player would rely more on in-depth reviews than some ill-explained rating system when deciding whether or not to download, and the sorry fact of the matter is that the non-serious player is just going to judge the game on its screen-shot anyway. 

Quote from: Ryan Timothy on Fri 18/12/2009 02:44:21
These are free games, so obviously we shouldn't be too strict on the rating but also not too lenient either since honest ratings will help promote the games that deserve the higher rating, and in the end they'll promote AGS; giving us more games!! :)

By saying that, you're demeaning free games. Quite a few of them are much better than the majority of commercial ones. I say, be strict with the ratings, and make the best games stand out as much as possible.

Btw When I'm browsing the database I read the number of cups like this:

1/5Cup - probably a really badly put together game
2/5Cup - there are some serious problems with this game
3/5Cup - this game could be anything in terms of gameplay quality, but at least you can be sure it works
4/5Cup - a stand-out game
5/5Cup - a stand-out game Type 2
Pick of The Month - a stand-out game Type SSH

Usually I have a hard time taking notice of the User Ratings. Too many percents, so I forget them all in 2 seconds. I take note of how many people voted and commented though.

And in the end I don't pay much attention to the ratings, just download the games which screenshots appeal to me (unless the game info is off-putting). And then usually I don't have time to play them all, so I pick out some of them randomly from my AGS games folder.

QuoteBy saying that, you're demeaning free games. Quite a few of them are much better than the majority of commercial ones.

By commercial games, you mean AGS ones right?  Because I haven't played an AGS game yet that is better than a big title Adventure game (Sierra, Lucasarts, etc).

And yes, I am demeaning free games slightly, but you can't honestly tell me that a 5 cup AGS game means: It's as good as Freddy Pharkas, Monkey Island, DOTT, etc.  It just means it's a damn good 'AGS' game, or at least it's how I view it.  Although the Apprentice game comes pretty close, and I haven't played the KQII+ VGA yet.

In terms of production values AGS games cant compete since commercial games are developed on a full time basis.

However in terms of enjoyment and a compelling tale I think I prefer trilby's notes to DOTT.

To say that a 'big budget' game cant compete in terms of the narrative is baseless as far as i can see.

Quote from: Ascovel on Fri 18/12/2009 04:18:32
Quite a few of them are much better than the majority of commercial ones.

Seconded! AGS-games can't compete in terms in graphics and voice acting, but in terms of puzzles and story-telling.

For example sometime ago I played forced myself to spend time with Runaway 2 after I payed money for it. I can't remember much AGS-games with such crappy puzzle design.

Useful intervention!

Hasn't anybody ever considered that the fine line between crap and genius can't be expressed within a system of five (5) different grades? It just can't!

You read a review in some games magazine that says:

"72%. Great motocycle physics, but holes in the track and lacking spraypaint mode subtract somewhat from the overall enjoyment."

Then you read the next one.

"81%. Motocycle tuning mode works really well, racing is lots of fun for some hours. Only the sometimes uninspired career mode dialogs prevent a higher rating."

Which one would you choose? Do 9 points on a 100 point scale make a difference? I do not have any suggestion to solve this problem, of course, but I think any rating system with less than ten different grades is necessarily inaccurate. While 2 cups are certainly intended to express encouragement, they do put most players off.

Quote from: kaputtnik on Fri 18/12/2009 12:03:00
While 2 cups are certainly intended to express encouragement, they do put most players off.

I don't think it's the 2 cups themselves that put most players off. 2 cup games can be off putting only in the context that so many other games you can play that have received 3 cups or more. It's usually all about having to make choices because of a limited time.

It is my understanding that the ratings system is first and foremost an attempt to organise the games database, more than to make judgement calls on individual games. It is true that a 1-5 rating cannot completely encompass everything a game has to offer, but realistically, neither does a 1-10 rating. You might as well go to the absurd extreme: from a cursory glance, it seems there are about 1100 entries in the games database. Should there be 1100 gradations in the games ratings? Then you'd be getting even more arguments about how "This game is not better than that game!" and it'd all become a competition.

Quote from: Ascovel on Fri 18/12/2009 12:31:35

Quote from: kaputtnik on Fri 18/12/2009 12:03:00
While 2 cups are certainly intended to express encouragement, they do put most players off.

I don't think it's the 2 cups themselves that put most players off. 2 cup games can be off putting only in the context that so many other games you can play that have received 3 cups or more. It's usually all about having to make choices because of a limited time.

The whole "Let's look at what you could of won" thing from Bullseye?

My previous post expresses how I feel on this whole matter, but it seems like this thread is turning into a broken record in terms of points and discussion. Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description. Surely it would be more attractive to players to see the time and effort put into a game in terms of cups rather than the level of enjoyment you will get out of the game in terms of cups.

And babar my game is better than yours  :=

Quote from: IndieBoy on Fri 18/12/2009 13:13:20
Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description.

If you lot saw how many cans of soda I drink while making each game you'd stage an intervention.

Coca cola: It may shave 3 weeks off my life with each can I drink, but it is delicious, refreshing, and bypasses the need for that stupid sleep thing.

Only after you complete it! :P


I abuse no substance while making my games. No wonder it is so damned hard :(.



PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems". Although I get the feeling that very very very few games have user ratings below 50%.

sorting by user rating would be a good idea. Since they have quite a large sample size in comparison to the panel rating.

That way it would be useless to inflate the downloads/user rating of your game since the panel rating acts as a normaliser... if theres a game rated 98% with a billion downloads but only has a 1 cup rating its pretty clear something fishy is going on.

I fully support Babars bum.. i mean his idea

Quote from: Babar on Fri 18/12/2009 13:18:38
PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems".

Seconded.

Quote from: bicilotti on Fri 18/12/2009 14:59:40

Quote from: Babar on Fri 18/12/2009 13:18:38
PS: Maybe an option to sort games by user ratings would be helpful to bring out some of the "unpanelly gems".

Seconded.

That's the spirit! Thirded! 100% = 100%!

One reason the panel was initially formed was as a solution to the inconsistency and abuse that can plague a user-vote rating system.

Now we have calls to give the user ratings more emphasis.

The 2 cup rating for The McCarthy Chronicles upset some people, fair enough. But cries of unfair treatment are counterproductive. The rating was based on a panelist's experience playing through the game, and judging the game's strengths and weaknesses. How can that be unfair? We don't rate what a creator may have intended to do, we rate the finished product. Personally, I don't support the change in rating, but I support that panelist's right to change their mind.

Let me make on thing clear, for future reference. Any ratings bullying, passive-aggressive or otherwise, will not be tolerated. If you have a problem with a rating, either leave a comment on the game's database page, in the game's official forum thread, or, if you must, PM myself or ProgZmax. No effective rating system is going to please everybody, and the panel is comfortable with this fact. Opinions vary. The world turns.

The ratings are not there to promote your game, nor are they there to dissuade people from playing it. They are just one, of many, opinions.

As to those who wonder what it takes to join the panel, potential panel members might initially be contacted due to the amount of levelheadedness, diplomacy, and respect to others that they show on these very forums; so if you come across as suitable for the job, it may eventually be offered to you.

I think we've all agreed that the panel is essentially a good system.

I dont particularly think McCarthys rating should have been changed either. It sort of diminishes the 3 cup rating some how.. in a "oh well if you insist" kind of way.

Babar merely suggested that an alternative way of searching through the games would be a good idea in tandem with the panel ratings.

Either system has weaknesses but perhaps a combination would be stronger... its not really a change in the system, just a change in the capabilities of the search engine.

Open discussion never hurt anyone...

Quote from: Calin Leafshade on Fri 18/12/2009 19:42:01
Open discussion never hurt anyone...

In Communist Russia...

Ratings are ratings.  I don't even look at number of cups.  I play games that sound interesting to me.  Plain and simple.

-Bill

Quote from: Calin Leafshade on Fri 18/12/2009 19:42:01
Open discussion never hurt anyone...

Quite right.

But flogging a dead horse is only fun for so long, and the panel has already stated it's position. We're not ignoring people's opinions on the matter, but we have a job to do. Any changes to the search capabilities of the database are here for CJ to consider, as he is the one who will have to modify the site to accommodate such requests.

We're all here because we enjoy making and playing adventure games, and the database is only one, of many ways, to showcase games. It's also only one of many opinions that may be formed about those games. Everybody who makes a game with AGS will find support in this community (which is bigger than you, me, the panel, the database, the ratings, etc), and nobody will be denied an opportunity to voice their opinion.

Feel the love. :)

Quote from: LimpingFish on Fri 18/12/2009 19:59:24
which is bigger than you,

NEVAR! *crushes tokyo beneath his feet* RAWWR!

Quote from: IndieBoy on Fri 18/12/2009 13:13:20
Why don't you personally rate your own game by the amount of cups of coffee you drank while making it and add that to the games description.

I love that idea. That would be about 50 cups for a MAGS-entry  ;D

I completely agree with Babar's idea. Simply allow users to search by the user given percentage ratings, in addition to the blue cup rating, and this whole mess is sorted out. It should be a fairly straightforward task to code, so as to not take too much of CJs time. Neither the blue cup rating, nor the percentage rating, takes away from each other.

In addition, CJ, I'd also like you to do the following...
1. Make a system that shows me what other people downloaded after this game (like Amazon's feature)
2. Make AGS in 4d
3. My laundery

Not necessarily in that order.

-MillsJROSS

Quote from: Mods on Thu 17/12/2009 23:37:32
People seem to be forgetting that 2 cups is still a good rating. I don't know. My game is 2 cups and I'm thrilled with that.

I think that's probably because 2 cups out of 5 certainly doesn't seem like a good rating, especially when 1 cup is reserved for the dregs of the games database. There are four grades left to be assigned to any game that makes an effort to entertain players and that functions well enough, technically, to be evaluated. 2 is the worst one. Even if the description of the grade in the guidelines is written a glass-half-full sort of thing, I think 90% of people who see it are going to take it as a bad rating. (No offense to your game, Mods.)

Quote from: LimpingFish on Thu 17/12/2009 01:44:08
First and foremost, nobody is ever given preference. I've rated games by people I am friendly with, and games by those whom I may have had "heated debates" with in the past. I don't let personal feelings influence a rating, and I feel no other panel member would. You can take that at face value, or you can dismiss it. So it goes.

Quote from: ProgZmax on Thu 17/12/2009 03:21:45
You'd think that Mitteneers were a really close-knit group like that but I've never seen evidence of preferential treatment on this forum.  Also, I've never been to Mittens and have not spoken to CJ at length and yet I've been a moderator for almost 6 years.  While I can see where talk like this might occur I've never seen any validity to it!

To be clear, I didn't suggest that panel members actually give preference to people they know (at least not consciously), just that the secrecy means no one can know for sure, and can easily think "they must have something to hide".

There is a well-established mechanism to deal with the problem of how to comment on work by people you know: full disclosure.

Quote from: ProgZmax on Thu 17/12/2009 03:21:45

Quote
A final point against anonymity is that a reader/player gets to know the reviewer's tastes, which helps interpret the ratings. That also leads, I think, to a more constructive attitude towards the ratings. Instead of going "wtf was the reviewer thinking?!", someone who disagrees will be more likely to go "this guy just doesn't like this type of game".

I don't agree.  The more likely scenario is one in which an author with several games feels targeted by a reviewer and demands someone else review their work, or demands one of the less-strict reviewers do so.  Anonymity prevents the authors from feeling singled out and for this reason it works better than it would otherwise.

I was talking about players browsing the games db, not the creator.

You guys seem very concerned about the problem of creators complaining about their rating. Again, I think this is an issue that can be solved with more openness, not less. If every complaint had to be submitted openly on the games page, instead of as a PM or something, that could deter "grade-grubbing", and players could see for themselves if the arguments had any substance.

Look, you are apparently very determined to maintain the anonymity of the panel. Fine. I don't agree, but it doesn't seem like there's any chance of changing your minds. But that doesn't mean you can't still have more transparency. How about some of the other suggestions in this thread? Mods mentioned having a single point of contact for the panel, like an AGS OFCOM, for example.

Quote from: ProgZmax on Thu 17/12/2009 03:21:45If you're saying you like a game the panel did not review highly that's fine, but if you say there's a problem with the rating because you like a game that isn't rated highly that's another story.  This thread would not exist if some people did not think the latter were true.

For the record, I for one was not wildly enthusiastic about McCarthy Chronicles. I did think the original rating wasn't in line with the (very sparse) public blue-cup guidelines, as I explained in a post on the game thread, but for me the issue was more that the current system doesn't deal very well with cases when the rating appears unreasonable (which will happen now and again in any ratings system). There's little or no explanation of the grade, the criteria are secret, the reviewer is anonymous, there's no official appeal, and you can't even leave another comment on the game page if you already user-rated it.

Quote from: Andail on Wed 30/05/2007 09:50:02
The documents which will guide the panel's ratings will be public.

Quote from: ProgZmax on Thu 17/12/2009 03:21:45I believe in the earlier threads many of the rules were laid out and discussed but the final set of guidelines weren't posted, I'm guessing because the panel was so busy wading through hundreds of games that it just became forgotten.  It is ultimately up to CJ whether or not those guidelines are posted, though.

Quote from: Mods on Thu 17/12/2009 23:37:32The panel try to be un-biassed and just follow a standard system, again maybe there needs to be an explanation of it on the page somewhere - but we can't expect everyone to play every game ever made in order to get a fair comment, and so in a lot of cases the reviewer will have to make some judgements by themselves. That person doesn't always feel its fair but I know everyone that does the panel is just following a system well before any personal preferences come in - but alas, you can't escape the fact those preferences will in cases effect the score because 2 panel members doesn't really justify 2000 players. But that's the way it is. It's not hiding anything.

Quote from: LimpingFish on Fri 18/12/2009 02:36:36I think I can safely say that no explanation on the subject of what goes into rating a game will be forthcoming, since it would benefit no one except those who would wish to pick it apart.

Wait, what? Most of those (publicly) involved with the panel have said that the criteria will/were meant to/should be made public, and now a categorical refusal all of a sudden? How did that happen?

Quote from: LimpingFish on Fri 18/12/2009 19:33:48As to those who wonder what it takes to join the panel, potential panel members might initially be contacted due to the amount of levelheadedness, diplomacy, and respect to others that they show on these very forums; so if you come across as suitable for the job, it may eventually be offered to you.

So how come you're on there?  :P

Quote from: Snarky on Sat 19/12/2009 07:47:27
So how come you're on there?  :P

Ah ha ha! Snarky, you wag.

To address three of your points:

An official method of contact is something that's actually been proposed a number of times throughout the history of the panel, and discussions have taken place. Such knowledge wouldn't of course have been known to those without access to the panel forum, which is where this appearance of panel inactivity on the matter may have first arisen from. We simple never came up with a satisfactory answer. Or we got sidelined with fulfilling the actual purpose of the panel; rating games. You may still see something put in place.

The five cup rating system wasn't just pulled out of the air, and it took a significant amount of time to fine tune a system the panel (and CJ) were comfortable with. The idea of half cups or a 10 cup system were also investigated, but abandoned. This is called weighing up your options, but knowing when to make a final decision. We could have tinkered with it until we were blue in the face, making nothing but largely arbitrary changes, but we had over 1000 games to get to. It was also felt that half cups or extra cups would dilute the situation, while adding little. It's also not the panel's duty to worry about how a 2 cup rating appears to potential users. The ratings are designed to give a clear indication of the panel's assessment (one opinion) of the game, and to clearly denote the shifts in opinion between each rating.

There seems to be a lot of paranoia regarding panel anonymity, peoples faith in the ability of an anonymous panel to function fairly, and the underlying fear that we must have something to hide. All three assumptions are unfounded. The ability to search the database by panelist offers two potential outcomes, both largely worthless. You could make your gaming choices based on the opinions of a certain panelist, blindly following that panelists tastes. Or you could automatically dismiss the rating history of a panelist whose views may have caused you to form a negative opinion about them in the past. Both are short-sighted, and detrimental to the goal of the panel; ensuring as wide a mix of opinions as possible.

Despite the nature of the internet, you'd be surprised to learn that a lot of people aren't inherently crooked, don't form grudges, and find little satisfaction in marking a game down simply to win some sort of imaginary one-up on a random internet denizen.

QuoteWait, what? Most of those (publicly) involved with the panel have said that the criteria will/were meant to/should be made public, and now a categorical refusal all of a sudden? How did that happen?

My reading of the situation, and my personal opinion of how the panel should proceed. My safety may be miscalculated.

As I said in a previous post, what purpose would knowing how we rate games serve? If we said we rate games that feature flying chickens higher than others, would everybody suddenly fill their games with airborne poultry?

Everybody needs to take a step back, and concentrate on making their games to the best of their abilities. Not fulfilling a checklist.

Quote from: LimpingFish on Sat 19/12/2009 23:00:38
As I said in a previous post, what purpose would knowing how we rate games serve? If we said we rate games that feature flying chickens higher than others, would everybody suddenly fill their games with airborne poultry?

I think it would be more like when a player happens upon a lowly-rated game, he could say "oh, that's because it has no flying chickens". He can then personally decided whether he likes flying chickens in a game or not, and then decide whether he wants to download a game or not, as opposed to where he sees a low rating in a game and jumps to the conclusion that the rater's criteria for a good game coincides with his own. A game maker could also take his low rating with more equanimity: "I didn't set out to make a game with flying chickens, so." A mature game maker could even take his HIGH rating with a pinch of salt: "I know that's just because I centred the game around flying chickens. I'll have to be careful if I make a game without those damn fowl."

Sure, it could result in a sudden explosion of games with flying chickens, but if flying chickens were really such a good indicator of a good game, would this really be a bad thing?

As someone who gets paid to rate people's work, I really do think having a list of freely available criteria is important for game makers who really want to improve. Otherwise they're just groping around in the dark.

What harm could come of making a standard list of criteria public? Cookie-cutter games? I doubt so. People are still going to do what they like. Just look at society. I doubt any one list of commandments could homogenise a whole community, especially one as diverse and creative-minded (and stubborn? ;) ) as this one.

What good could come of making the criteria public? Game-makers could try to improve according to the list of criteria. The casual gameplayer could look at the criteria influencing the decision and decide whether he still wants to download the game. Yes, it's debatable that all this will happen, but there's the possibility.

Initially the cup ratings had a more horrible-bad-fair-good-excellent progression to them, but after discussion the panel agreed that it seemed too harsh, particularly in light of all our new members eager to crank out their first games (which typically fall in the 1-2 cup category).  This is why 1 cup tends to be reserved for broken or just poorly made efforts while 2 is a 'nice try' category that is meant to encourage people to do better.  As mentioned before, 3 is essentially the standard score which is fine since most people will produce a few average but interesting games before they reach something better.  It also allows us to make 5 cups a really privileged category which I think is a good thing.  That's reasoning behind the system, anyway.

As far as the ratings criteria go, again, I think it must really be up to CJ since this is his website and having panel ratings was his decision.  I'm sure as LimpingFish points out some people will indeed pick it apart, but some people also seem intent on picking apart the panel regardless so I fail to see what harm it could do.  The one thing that I will say is the guidelines don't have any 'special' criteria you wouldn't see used for any game rating system and it goes by a set of very common sense rules 90% of you could guess in your sleep.

Quote from: auriond on Sun 20/12/2009 03:37:35
What good could come of making the criteria public? Game-makers could try to improve according to the list of criteria. The casual gameplayer could look at the criteria influencing the decision and decide whether he still wants to download the game. Yes, it's debatable that all this will happen, but there's the possibility.

auriond articulated all the reasons why I think open criteria are a good thing better than I could. Well put!

Quote from: LimpingFish on Sat 19/12/2009 23:00:38
An official method of contact is something that's actually been proposed a number of times throughout the history of the panel, and discussions have taken place. Such knowledge wouldn't of course have been known to those without access to the panel forum, which is where this appearance of panel inactivity on the matter may have first arisen from.

Witness the problems caused by lack of transparency!  ;)

Quote from: LimpingFish on Sat 19/12/2009 23:00:38
The five cup rating system wasn't just pulled out of the air, and it took a significant amount of time to fine tune a system the panel (and CJ) were comfortable with. The idea of half cups or a 10 cup system were also investigated, but abandoned.

There seems to be some miscommunication... I don't think I (or anyone) have said that we need more fine-grained ratings?

Quote from: LimpingFish on Sat 19/12/2009 23:00:38
There seems to be a lot of paranoia regarding panel anonymity, peoples faith in the ability of an anonymous panel to function fairly, and the underlying fear that we must have something to hide. All three assumptions are unfounded.
...
Despite the nature of the internet, you'd be surprised to learn that a lot of people aren't inherently crooked, don't form grudges, and find little satisfaction in marking a game down simply to win some sort of imaginary one-up on a random internet denizen.

I must have been making my points badly, because this was never the gist of my argument. The reason I think non-anonymity would be better is that it would make the scores easier to interpret. Even if you're all working from a common set of criteria, there is always a subjective element, and different people have different tastes. To give a concrete example of how this works, on Adventure Gamers, I know that Evan Dickens likes neo-LucasArts-style adventures like Runaway and the Telltale games significantly more than I do. So in my mind, I correct for this difference in bias by adjusting down his scores for those titles. I don't dismiss his opinion (I'm broadly on board with his Top 20 list (http://www.adventuregamers.com/article/id,186), for example), I just... adjust.

This whole thing about anonymous members giving preference to some people was just in response to claims that anonymity was necessary to let people review games by people they knew, since I think it doesn't really solve that issue. It's a sidetrack to the main discussion.

Quote from: LimpingFish on Sat 19/12/2009 23:00:38
It's also not the panel's duty to worry about how a 2 cup rating appears to potential users. The ratings are designed to give a clear indication of the panel's assessment (one opinion) of the game, and to clearly denote the shifts in opinion between each rating.

The ratings are presumably meant to communicate something. In that case, it's worth asking whether they communicate what they're meant to, right?

Now, ProgZmax's post seems to indicate that they mean pretty much what I personally took them to mean, particularly the point that 3 is the standard score, 4 and 5 are reserved for games that excel, and 2 is... well, those games that don't quite measure up to the "average" game.

Quote from: ProgZmax on Sun 20/12/2009 05:39:44The one thing that I will say is the guidelines don't have any 'special' criteria you wouldn't see used for any game rating system and it goes by a set of very common sense rules 90% of you could guess in your sleep.

Well, maybe some criteria seem common-sense to you, but not to everyone else (or even just 90% of us):

Quote from: Andail on Thu 17/12/2009 17:19:20

Quote from: Dualnames
Difficulty and length has nothing to do with a game's quality. EVER.

Don't you agree that a game shouldn't be so hard that the player can't progress at a reasonable pace, nor easy enough to let him win without trying or thinking?

Also don't you agree that a game should have at least a modicum of content before it ends, to let the player get into the plot and the characters?

You could go both ways on those question, I suppose. (Though personally I tend to agree with Dualnames on this point: just because a game is too hard or too easy for me doesn't mean that it's not perfect for someone else, and if a game is ten or even just five minutes of enjoyment and ends... well, I didn't pay for it, and no one complains that a 3-minute pop song is too short. Besides, the length is indicated separately.) However the panel decided to do it, it would be useful to know about it, again for all the reasons auriond said; because it's actually not something I could have guessed in my sleep.

Let me end this by summarizing some suggestions for ways I think the current ratings system could be improved, which should be fairly simple to implement:

• Make the ratings criteria/guidelines public, link to them from the bluecup explanations (like the "Why is my game rated 1 cup" link).
• Ask reviewers to always fill in the panel review with a short description (a sentence or two) of why the game got the rating it did (what was good enough to earn the grade it got, and what kept it from getting a higher grade, or some other format you decide on).
• Allow people to post multiple comments on the games page, maybe? (Currently there's no way to respond to a rating after you've left a user-rating.)
• Provide a contact address for the panel, for questions, update notifications (like ProgZmax described) and, yes, complaints.
• Make it clear that if any change is made to a rating or classification, the message will be posted to the games page so people can see what happened. This to discourage grade-grubbing and frivolous complaints.
• End panel anonymity. (I know you won't do this.) Even if you don't sign each review individually, you could still let people know who sits on the panel.

I hope you see that this is not about picking apart the panel, just offering some fairly minor tweaks to the system to make it more effective. It's up to you and CJ, of course, but I hope you'll consider it seriously, at least.

Thank you, Snarky, I think you express many valid points.

As one of the initiators of this current system, I ask for respite now to deliberate with the panel.
I would also prefer if panel members would refrain from posting until we have discussed a future direction.

As for now, it seems likely that a document with guidelines and criteria will be made public. Whether ratings will be anonymous or not is another question.

Since the panel is heatedly discussed in a game thread, I thought I'd bump this thread by posting my experience with the Maniac Mansion Mania games:

Pretty much from the start it became common to attach a vote to the game thread, 1 - 5.
In retrospect, many crappy games have high, undeserved ratings, especially after some very high quality episodes were published late in the series. (Also, people didn't want to down-vote newbies' games in order to not put them off of publishing episodes.)
Yet still, the votes reflect nicely what people's first gaming experience was on a scale from shit to superb.

I don't regard two neighboring ratings as always being equidistant from each other; I'd guess a rating of 1 is way worse compared to a 2 cup rating than a 4 cup rating is from getting 5 cups.

My impression of the panel's comment about Cosmos Quest III was that the author put too much work in fixing unnecessary shortcomings of the gameplay and UI and due to that lost sight a bit of the other important parts.
But, like I mentioned in the game's thread: whether the two cups are deserved or not, removing a game from the DB after a bad review is sorta childish.

"I don't have a dog in this fight.."  as the saying goes but a couple of idea and observations come to mind that may be helpful so I thought I'd share my thoughts.

Criteria
I think there may be some confusion as to what is meant by "criteria".   Is it really necessary to know all the nitty gritty details of how the rating was given?   The problem with revealing this is that the final judgement of the panel is largely subjective so revelation of objective guidelines the panel has made for itself, IMHO, is not very helpful and likely to be counter-productive.   I fully appreciate that using objective criteria helps achieve a measure of consistency between panel members and between games but in the end it comes down to subjective opinion(s).  I think the reluctance the criteria, rightly so,  stems from the realization that the relationships between objective criteria and ratings are necessarily nebulus. 

Perhaps the need for transparency would be better served by a more general description of the process.   Something like the following for example, illustrates the tone and level of detail I am suggesting; the specifics would obviously be different.

Quote
The ratings panel is composed of xx members of the Ags community and experienced game developers.  Panel members play each game from beginning to end and note such factors as playability, enjoyability, dialogue, story, graphics, sound, music, etc... to form an opinion.  At the end of an evaluation period the panel members discuss their individual opinions and reach a consensus.    One of the panel members is charged with writing a review consistent with the consensus of the panel.

Review
My first suggestion for the review is that the the cup ratting be clickable and linked to the rating. 

My second suggestion is that the review be written for the benefit of the game author rather than the game players.   How so?  I was a member of Toastmasters, for a number of years, where members learn speaking and leadership skills.  The key ingredient to the success of their program is the way members' practice speeches are critiqued, which I will share with you. 

Someone is assigned to give an evaluation speech immediately after the main speech.   The evaluation is usually structured into three parts consisting of no more than three points each.   The evaluation speech begins by pointing out the best parts of the speech.   Next the evaluator points out what could  be improved.  There is no point in mentioning more than three things because

1) The speaker may become disillusioned/discouraged
2) It's difficult for people to remember more than three things at a time

The evaluation ends on a positive note by mentioning what things the evaluator would like to see more of in the future.   This essentially presents negatives in a positive light (i.e. "half full" rather than "half empty").  This is also an opportunity to encourage the speaker  to continue making speeches and improving.

I think all of this or something similar could be applied to the panel's reviews.   Game authors wouldn't be in the dark about why they got the rating they got.  They would have clear guidance and specific suggestions to improve on their next effort.    Game players would get the same or more benefit from this sort of review as before.

Author Communication
I think the author should be given advance notice of the rating and a copy of the review before publication as a matter of professional courtesy.   It may be a good idea to invite the author to respond if he is so inclined and the panel could take the author's response into account before publication.   It may even be desirable to include the author's response to the end of the review;  this would perhaps keep responses civil and mature.   Anyway a process something like this would cut short any debate or controversy about overly generous or overly harsh panel ratings.

Anonymity
I don't really want to know who the panel members are.  I don't think any benefit comes of it.  The panel ought to create a virtual forum member named MrBlueCup, MisterBlue, BC or some such name so that they can communicate with the community.   If someone has a gripe or a sincere question they could pm  MisterBlue.

Good points, Rick.
I'm afraid it's a bit fruitless to debate this more. There is simply no way we can satisfy everyone's demands. There will soon be a thread informing about the panel. In the mean time, please don't start more threads about the rating.