Rating system

[Leon]

To prevent other topics becoming 'off-topic' I separated this from the other thread into this one to continue the discussion here.

I didn't expect to see a 2 cup rating. 
After seeing this I actually looked at all the screenshots of every 2 cup games, and out of the few 2 cup games I actually played, I think there were only 2 games I didn't expect to see in there.  This game, and another one.

And to reply to Leon's suggestion.  I have to disagree.  I believe the ratings panel does an exceptional job doing what they do.  I may not agree with this particular rating, but the majority of the ratings, I do agree with.
One reason why I prefer the rating panel over the 'public' rating, is mostly because it's consistent and reliable.  I also like reading the rating panel reviews as well.
I think the rating system should remain as is, with the rating panel deciding the big rating.  You can always appeal if you don't believe your game deserves the rating--perhaps it'll change, or it may actually remain.  No harm in trying.

I know that this has raised questions before and there's a valid point in all opinions. I agree with Ryan that the current rating would result in the most consistent rating if you can see a person, not a group, but the problem is that you don't know who rated your game. It's a panel but different members, different opinions. Is there a list somewhere where you can see who's on board? Why not display who rated your game instead of "a panel". Looks like they want to be protected from their opinion.

My problem with the current system is that it looks like an elite group that judges your game. You can't see who, your game is just being judged. You can't discuss the rating or comment on it. The comments on the games page are for the game, not the judgment.

No big mystery, really.

Some games, short ones in particular, might get rated faster simply because a panel member can play and rate them fairly quickly. Or, since we also play these games for fun, a panel member might choose something that especially appeals to them, even though other unrated games may have been in the database for a longer period of time.

To be frank, if we happen to be playing a full-length game that's proving to be a bit of a slog, it may take us longer to get around to forming a complete and fair opinion of that game.

Regardless, we have a finite amount of free time to devote to rating these games, but we try to maintain a fair balance.

When a game is rated, its correct categories, as determined by the panel member, will be set if needed.

Plus, you always have the user ratings to keep you going.

But we're always happy to hear people's opinions about the database, and we're always open to new ideas.

That all sounds very informal to me. It's just whoever feels like it and whenever he feels like it. When using a rating system like this every game should be judged, no matter how long the game is or if it's attractive or not. And it should be judged as soon as possible after release.

[Gilbert]

In fact, AFAIK the panel is meant to be anonymous (I know who they're though, as with any of the moderators here) and is composed of chosen forum members who would cycle once a while to ensure that it's not always the same people. Though each game is actually mainly reviewed by one single member, I think there're some general rules that each of them agreed on and would discuss among themselves whenever a game is rated, so the ratings should be quite objective.

And no, I'm not a member of the panel and probably never will be, as I don't have the time for games.

[Calin Leafshade]

I agree with the anonymity to some degree.. although it does remove accountability it possibly preserves honesty.

I dont think the games should be reviewed by a single person though. The strength of a 'panel' is multiple opinions, that is kind of the purpose of a panel.

So perhaps the games can be reviewed by more than one person and an amalgamation of their opinions used?

[abstauber]

You can't discuss the rating or comment on it. The comments on the games page are for the game, not the judgment.

And that is good.

Creating a game is hard work for everyone. You invest a lot of your free time, you do everything you can to please the gamers... but who am I telling

There are a lot of games that aren't much entertaining but still interesting to see how time consuming it must have been.
Even if a game's terrible, it's still a great thing that it's creator made it.

But the rating panel's job is to forget all this. It has to rate the game from a casual gamer's position.. well, and the casual gamer has different demands.
When I show games to my friends, they usually prever the 4-cup rated ones (even though they don't know the rating). They don't know how much work it has been to finish a game, they just want to play a nice game. And the panel does great work finding these games.

Also, why's nobody complaining about not being the pick of the month? 


For getting your ego messaged, we have the game threads. 

[Leon]

Why should it be anonymous? If you're rated by a group, maybe. In this case you're rated by an individual... who's hiding in a group (that's how I see it). So why not mention your name? Afraid people disagree?

@abstauber:

Your friends would certainly miss out on the McCarthy Chronicles, amongst other great games.

Like I said, the individuals in the panel are right most of the time. But if they're not, who to address? Why the anonimity, the secrecy around it? It feels to me like an elite group (although they don't see themselves as such). Why not discuss the games more openly and then decide the number of cups.

If you only want to play the 'good' games, wait for the awards. Then you'll know what was liked by the public and play those.

[Calin Leafshade]

Also, why's nobody complaining about not being the pick of the month? 

I would very much like to be pick of the month.. damn that Marionette!

[SSH]

PM me and I'll send you my paypal address  

Ultimately, its done by volunteers, and there's a backlog as it is so any attempts to force more work or double-reviews, etc. on the volunteers will fall flat. However, perhaps there should be a particular effort to review any games that are over 6 months old and not yet reviewed. (e.g. Soviet Unterzoegersdorf Sector II).

I think, though, that there's a huge range of 3 cup games. Some Ben Jordan and Ben304 games are in there and are great and then you've got things like Rock Rock Rock which astounded me by getting 3 cups (no offence to the authors! )

[abstauber]

So why not mention your name? Afraid people disagree?

Anonymity saves friendships

The pick of the month doesn't have to be anonymous, since it only praises. The panel however also slates.

@Calin

[auriond]

In this case you're rated by an individual...

Wait, wait, so the rating ISN'T by a panel of people? It's just one opinion? I was under the impression that there was some system of rating by a committee. Although looking back at the application thread for the panel, it never said the games would be rated by the whole panel, and I suppose it would be unrealistic for so many people to play the same games and pick a rating. Guess that's my naivety showing.

.. damn that Marionette!

What are you complaining about? You ARE the Marionette.  

[Leon]

Since I don't know how many people are in the panel, why not let more people from the panel play (minimum of 3?) and take the average as a score? With every 'judged' event, you always see individual scores that makes up your end total. And I fail to see the reason why it should be anonymous? Afraid to make the wrong decisions? If I give my opinion about a game, my name should be there. It's my opinion.

If there's a rating by a group, you shouldn't have to put a name under it but I'd still like to know who that group is. And what it was that made them decide the way they did.  Why for instance not show the initials of the person from the panel who rated it, if it's only one person.

I understand that not everyone can play the game, that it's voluntary and takes time. I also see the purpose of the rating and it being done by a 'fixed group of people'. But I fail to see why it all has to be done anonymously.

[Calin Leafshade]

I was under the impression the games were reviewed by the panel as a whole or at least a portion of the panel. If only a single person rates it how is it any more authoritative than any other review?

I thought the purpose of the panel was to act as a kind of 'normaliser' which is not able to be influenced by the author in any way but still reflects a good estimate of the quality. I mean what if my game was rated by someone who appreciates the puzzles and jokes found in adventures games but nothing else? they might as well have just rated it "boring"

If however you have a group of people is gets rid of that bias.

.. damn that Marionette!

What are you complaining about? You ARE the Marionette. 

Very true :p I really need to get round to recording those lines... tonight i promise :p

[Intense Degree]

With respect to everyone here aren't we taking this a little bit too seriously?

I know that a lot of time and effort goes into most finished games (even mine, although that may appear hard to believe! ) but at the end of the day I haven't paid anyone anything to rate my game and I can't see that the panel has a duty to review any or all of the games released. If this site held itself out as a review site and offered some kind of promise or indication that all games made would be reviewed then that would be one thing, but speaking personally, I'm just glad of the exposure games can get here rather than just 20 or so of my family and friends!

I don't know who the panel members are, but my guess is that they are just a bunch of guys (and guyettes of course) who work and have other commitments and are doing the community a favour by the ratings they do give. To be honest, if I was on the panel and saw complaints and people trying to hold me to account I don't know how much longer I would stay on it!

Basically this whole thing is really just a hobby to nearly all of us (although seriously draining and hard work at times) and I doubt very much whether those who have produced commercial games are particularly concerned with their blue cup scores!

[abstauber]

Okay, since it's all about the score McCarthy got, I've finally downloaded and played it a bit.

I guess you must have slipped in another league In the league of awesome games, McCarthy is more like an interactive theatre with not too much focus on gameplay.
I suppose the guy who rated your game was disappointed that your game could have been so much more.
To be honest, if McCarthy would have no sound, cheesy paint graphics and cEgo in it, 2 cups would have been fair in my oppinion.

But since it's not, I think McCarthy is really underrated.


@Leon:

Your friends would certainly miss out on the McCarthy Chronicles, amongst other great games.

Since they are not addicted to the genre, I'm afraid they live with it. They don't like my game(s) either

[auriond]

With respect to everyone here aren't we taking this a little bit too seriously?

Well it all started when a game we loved got shortchanged... or so we feel.

You're right of course, that the people on the panel have lives outside of AGS. But that doesn't mean we can't work towards fairness here. If even two people play a game and give a rating, and an average of said rating was taken as the Blue Cup rating, I doubt we would be kicking up such a fuss. The problem is that the panel was supposed to be established in order to provide an unbiased viewpoint, and the question is whether the current way it's being done is achieving that aim of an unbiased view.

[Calin Leafshade]

so are we all agreed? 5 cups for McCarthy? ok cool.

In all seriousness, I'm a musician so i am *no* stranger to bad reviews.. seriously some reviewers can be feral. Especially if youve spent the evening unknowingly chatting up their girlfriend..

And McCarthy has attracted a fair amount of criticism on both the thread in completed games and on the games page so it is by no means perfect but I feel its a little arbitrary to have a single person decide a rating which is supposed to denote 'quality' in some kind of objective manner. Just because a game doesnt deliver what the reviewer wanted (in this case a challenge) it doesnt mean the game should be penalised for that.

Now this is obviously not the fault of the reviewer. They didnt like it so they marked it down which is entirely reasonable but the rating they gave doesnt seem to be representative of the general community which is exactly what the cup rating should be. If the game was rated by a 'panel' of reviewers it would be more fair.

I could quote the player approval rating but that seems to be a little premature since i have no doubt it will likely drop dramatically as more people vote. My game currently out-rates practically every 4 cup game but its obviously inferior to those.

I will say however that I actually agree with practically every panel rating I see. They are usually perfectly in line with the player approval rating. Sometimes its almost scary how accurate they are in terms of reading the opinion of the community. I also do not want the rating of McCarthy to be changed. It would be arrogant of me to simply assert that the reviewer is wrong and should change their opinion. The rating has actually helped me to see just how weak McCarthy is in a great number of areas and that it does suffer from "style over substance". It has only helped to spur me on to improve the next episodes and get more cups for my collection.

[m0ds]

Well, one game on the database has a panel rating & review by me!

And guess what, IT AER TEH MAC-HARTY BARNACLES!!!!!

No, no its not. But for the record the game I reviewed was discussed with other panel members and not just written & judged straight off by me. Or Sly Stallone. Seriously though, the panel cycles, and you never know - you may be on it one day. And then you'll come to realise why it's best to just be anonymous about it. I don't know who wrote the cup-review thing for my game but I still appreciate it & always the criticism. And low cup ratings can seem like a criticism, but yeah - just one of those things in the developer world to man-up about What seems like negativity around game production ratings and stuff can actually become a very good motivator for work on future titles. Unless of course you're a moany kid...I dunno...

We have to remember also that games are rated by the panel mostly on gameplay first and then all the rest. If there isn't any gameplay in your game then it simply isn't going to score. 2 cups is actually quoted as being "a game worth checking out". If you've got gameplay you probably get a 2, if you've got good gameplay you've worked up to a 3, then if you have good atmosphere you get your 4 or 5. So I'm assuming, though I haven't played - McArthy Chronicals fell short of 3 or 4 cups because it's a "short game" which apparently has "minimal gameplay". That to me sounds like 1 interactive cutscene.

Still, I don't think anyone has actively given up AGS because they've had low cup ratings, or their game just wasn't picked up or recieved well. Nor do I think it's ever affected the right praises being said for certain games. Now you've made a game you'll get the fever for it and just keep going with new ones, twos, trilogies, regardless of cup ratings. Helme asked about what gets a game downloaded, and I just wanted to mention that its generally putting your game other places than just the AGS db that will get it noticed. And in the outside world bluecup ratings don't mean a whole lot! AGS db comments are quaint - and in themselves a way for the people to be anonymous. The reviews that really matter will be sent to you personally!

Anyone is already pretty elite themselves to have the privelledge to post games to the AGS db and know they will be picked up on by a wide community, and more as time goes on and AGS and its games continue to appear in the media and stuff. AGS db and its cup rating is pretty weak in terms of criticism you could really be facing from outside sources, review websites and such. And it seems like you know that

[Scarab]

My game currently out-rates practically every 4 cup game but its obviously inferior to those.

Cult fame baby! 

[Calin Leafshade]

Clockwork orange only at 12? Thats bullshit.. which idiot rated that...

[abstauber]

It has only helped to spur me on to improve the next episodes and get more cups for my collection.

How about getting started with that  :

[auriond]

I do understand that it's an honour to even be on the AGS games page and to have its accompanying audience. And seriously, criticism comes with the territory. I think anyone who's made a game knows that.

But then why not make the process transparent? It's not clear, if you click on the "What is this?" next to the bluecup ratings, that the rating is based mostly on gameplay, or that it was implemented by one person or many. Again it all comes down to what is the bluecup rating for? If it's all that unimportant then why have it in the first place? I think we agree that it was needed, and therefore it is important, if not in the big scheme of things then at least to the AGS community.

In the end we just want to know how the ratings are implemented. I don't think that's a great deal to ask. The application thread for the panel mentioned a set of guidelines that every reviewer would have to follow. If we know what this set of guidelines is, we'd have a clearer idea of how to take the bluecup ratings. Notice that it wasn't Calin who started protesting his rating, but the rest of us who played his game. So it's not a matter of game makers being whiny over low ratings on their own games, but rather other members of the community who went "Why did this game get such low ratings? Hey, how are the ratings decided anyway? Who does them?"

When you get right down to it we made our games with AGS, so we care what AGS people think of them and how they are rated on the AGS games page. I don't think we can help it unless we never invested anything in this community in the first place.

But you're right Mods, being on the panel is a scary thought if you're the one coming under fire. It's a good thing that you discussed the rating with other panel members though. I think that's the kind of reassurance we were looking for. We just weren't sure if that was the case with McCarthy.