Adventure Game Studio

Maybe it's the result of the two cup rating of Cosmos Quest 3...?
The review panel is getting harder and harder...
If I would read this comment I guess I would have a reason to be turned a bit down.

"A lot of effort has gone into this project, with original graphics and audio throughout, and largely solid technical execution.

But a leaden pace, bland characters, little indication on where the screen exits are located until you pixel-hunt your way over them, and only the barest bones of an engaging plot, reduce playing the third entry in this ongoing sci-fi series to a chore. The graphics are consistent, and the music is pleasant, but the lack of direction and ponderous gameplay are a severe hindrance.

Some puzzles seem arbitrary and, in a number of cases, confusing."

Quote from: Le Woltaire on Thu 07/01/2010 20:35:25
Maybe it's the result of the two cup rating of Cosmos Quest 3...?
The review panel is getting harder and harder...
If I would read this comment I guess I would have a reason to be turned a bit down.

"A lot of effort has gone into this project, with original graphics and audio throughout, and largely solid technical execution.

But a leaden pace, bland characters, little indication on where the screen exits are located until you pixel-hunt your way over them, and only the barest bones of an engaging plot, reduce playing the third entry in this ongoing sci-fi series to a chore. The graphics are consistent, and the music is pleasant, but the lack of direction and ponderous gameplay are a severe hindrance.

Some puzzles seem arbitrary and, in a number of cases, confusing."

It's one thin being hard, and one thing not being so hard. A game downloaded and praised by so many people to be the best ever to get a two cup rating is the epitomy of not being objective. A game though good, but not really that good, should be fine with a 2 cup. "Worth a try"

If you view that one comment as reason enough to remove a game that many people seem to have enjoyed, then exposing your games to public consumption may not be for you.

Removing games from the database is an insult to the people who took the time to come back and comment on it, and to the panel member who took the time to play through it and rate it. It's the equivalent of a child taking their ball and going home because their team is losing.

Le Woltaire, you quote the panel comment, but you don't point out just what part of it you disagree with; as though the injustice lies in it's very existence?

The panel is not unfairly "hard" and the panel is not swayed by public opinion; positive or negative.

If people can't accept the possibility that their game may experience some negative feedback, they shouldn't add it to the database.

Otherwise, I really don't know why the fuck we bother.

EDIT: Apologies for the language. Dualnames, on reading your post again, I'm not sure if you're condemning the rating...or supporting it?

I just think it's a pitty to loose an active developer...
We already lost Alkis this year and since Harg was already developing Cosmos Quest IV I believe that this is now going to be cancelled or similar. I think every independent developer is needed in the scene.

I am not complaining about the panels opinion and maybe the panel is right (I just played the demo of CQIII).
I am just pointing out a possible reason for the developers action.

Sure it is not really the action of an adult to delete all the games from a database.
On the other side I can understand the disappointment and try to put myself in the position of the author.
I just wrote him an email and asked if everything is OK...

It might also be possible that he got problems with the commercial release.
Maybe copyright issues or tax related stuff with the collection office.
This is something that many developers forget when they quickly put a low cost single game into the market...

Hell, who knows... maybe he even has a gigantic crisis and is commiting suicide in this moment...

Fair enough, but an explanation beforehand from the author wouldn't be asking too much. He took the time to delete his games and alter his contributions to this thread, so why not leave a short note?

I too can understand an author's disappointment when a game doesn't perform as they may have hoped, but I also understand that not everybody is going to be satisfied with a game and that negative critiques are part and parcel of releasing your game to public opinion. We take the good with the bad, and we, hopefully, respond to it in a constructive way.

If Harg pulled his games because of the panel comment, then he's sending the message that the panel is at fault for voicing it's opinion. Therefore, by default, anybody who doesn't like his games is in the wrong.

That can't be a healthy mindset when developing games for other people to play, can it?

Losing any developer is sad, I agree. But a developer who voluntarily removes themselves is a different matter...

Even though I haven't played the games, I have faith that Harg had good and sensible reasons for removing the games. Once you post a game in the database it starts to belong to the community in some form and it is very sad when games are removed for whatever reason. I have faith that maybe even better versions of the games or completely different more awesome games are going to appear in the future from him.

I really hope Harg will explain himself or at least show himself in the forums again. If not I'm sure he appreciates the time people have invested into playing his games.

@ Le Woltaire
Why don't you start a thread about how new and "inexperienced" developers and what they must go through to post their first game. I'm sure it would be of great help to others and I would be most willing to discuss the topic, but not really in this thread.
(I'm trying my best to not to sound like a moderator hehe)

Quote from: IndieBoy on Fri 08/01/2010 01:22:01
Why don't you start a thread about how new and "inexperienced" developers and what they must go through to post their first game. I'm sure it would be of great help to others and I would be most willing to discuss the topic, but not really in this thread.
(I'm trying my best to not to sound like a moderator hehe)

Not a bad idea. I would have a thing or two to say about it and I'd be more than happy to share my experience of dealing with publishers, critics and everything else involved:)

I'm really perplexed by the whole situation here... Escpecially that someone has mentioned tax problems and I just imagined something like this happening to me ( if this is what actually happened)...  That sent a shiver down my spine...
Then someone mentioned suicide...
  I think we should assume that Harg, a character deeply rooted in science fiction genre, has been abducted by aliens, then forced to remove his games from the database, as he got too close to the uncomfortable truth they tried to hide. This is the only explanation fair to him... :)

Sour grapes.  I'll comment further if/when Harg bothers to explain himself.  He's 33 and certainly old enough not to be throwing tantrums over a difference of opinion, that's for sure.

Quote from: IndieBoy on Fri 08/01/2010 01:22:01
@ Le Woltaire
Why don't you start a thread about how new and "inexperienced" developers and what they must go through to post their first game. I'm sure it would be of great help to others and I would be most willing to discuss the topic, but not really in this thread.
(I'm trying my best to not to sound like a moderator hehe)

I am not really the right guy to give advice to unexperienced commercial developers, because I am not commercial.
However, if you need advice, then this is a very intelligent manifesto for the small developer:
http://www.gamasutra.com/blogs/SorenAndersen/20100106/4025/Hobbyist_Game_Developer_Manifesto.php

Quote from: Grim Reaper on Fri 08/01/2010 03:04:05
I'm really perplexed by the whole situation here... Escpecially that someone has mentioned tax problems and I just imagined something like this happening to me ( if this is what actually happened)...  That sent a shiver down my spine...
Then someone mentioned suicide...

About taxes: In my country that has one of the most complex tax systems of the world and the highest tax rates in Europe becoming an independent commercial game developer means suicide because he will have to pay more than he earns.

They quickly make estimations about your possible incomings and ask you 20.000 â,¬ without reason in advance. At the end of the year they might give you something back if you can proof that you don't earn so much, but still you had to pay a lot of money during the year...  This destroys most of commercial newbees in any sector.



Ok, I got a part of an explanation from Harg:

"If (AGS_Panel_Review_Number_Of_Persons < 2) {
    I_Dont_Want_To_Be_Part_Of_This_Community = true;
    Delete_The_Games();
} else {
    My_Games_Are_Really_Bad = true;
    Delete_The_Games();

}"



So the questions that come out of this:

How many persons are in the AGS Panel?
How many are needed to show up the blue cups?
Is it just one opinion that is needed?
Who is the AGS panel?

Should we make a thread about this?

QuoteHow many persons are in the AGS Panel?

It's not a relevant question to this issue (and his response was extremely childish, anyway) but there are considerably more than 2 members on the panel.

QuoteHow many are needed to show up the blue cups?

1, just like most review sites.  This will not change.

QuoteIs it just one opinion that is needed?

While the judges are apt to discuss and explain their ratings to each other, ultimately it is one opinion and that is all that's necessary for a subjective rating.  The important word you listed yourself: opinion.

QuoteWho is the AGS panel?

For the sake of argument I am on the panel.  Other members are anonymous at their discretion and ratings will always be anonymous.

QuoteShould we make a thread about this?

Not another thread, please no.  There have been a few already.

Didn't the panel want to make their work more transperent with some sort of "This is how the panel rates and reviews games"-thread?
If some questions are frequently asked a F.A.Q. would be a good idea.

Helme, looks like they're discussing it (http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39595.msg522017#msg522017).

QuoteMy_Games_Are_Really_Bad = true;
     Delete_The_Games();

As for this, well...

Quote
2 Cups   A reasonable game, worth a try
Feel free to disagree with the panel's rating, and please don't get offended if you think your game has been under-rated!

Honestly, after the level headed way in which the McCarthy rating was dealt with in the end, it disappoints me to see something like this. It's obvious that the panel ratings are open to discussion, and it's a bit saddening to see a knee jerk reaction like this. (No offence intended towards Harg)

The structure of the things should be:

                 AGS
         GAME  GAME
REVIEW REVIEW REVIEW

Now the things are like this:

                 AGS
         REVIEW REVIEW
GAME GAME GAME GAME GAME

Or with other words: AGS PANEL is trying to tell us how we have to write the games. No way! The indie game creator is an artist not an engenier!

Quote from: Harg on Fri 08/01/2010 12:44:55
Or with other words: AGS PANEL is trying to tell us how we have to write the games. No way! The indie game creator is an artist not an engenier!

I guess opinions like this are why the panel puts the "Feel free to disagree with the panel's rating, and please don't get offended if you think your game has been under-rated!" statement that I just quoted on the ratings breakdown.

The panel are not the police. They don't make laws and say "You must make your game like this!". You're entitled to disagree with them, and surely it's more appropriate to send a message to one of the members of the panel (some are anonymous, granted, but ProgZ and LimpingFish have always been quite open about the fact that they're on the panel) and discuss it with them, instead of deleting your hard work?

I can understand being upset at getting such a rating after spending your free time making a game you truly believe in. I can't understand favoring the course of action you took over discussing the issue.

It's clear that you have some fans that really believe in what you're doing - focus on the positives. I hate to see developers getting discouraged, especially when it's from something like this.

Quote from: Ben304 on Fri 08/01/2010 12:42:55
Helme, looks like they're discussing it (http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39595.msg522017#msg522017).

Thanks. I couldn't keep track where the discussion ended.

Quote from: Harg on Fri 08/01/2010 12:44:55
Or with other words: AGS PANEL is trying to tell us how we have to write the games. No way! The indie game creator is an artist not an engenier!

I think I overrated the influcence of the panel reviews a few weeks ago. Since 2 of my games have a panel review by now, there hasn't changed much about the dls or the feedback I get for them even with one review that is not very flattering.

I think CQ III is totally underrated (but it's my opinion). The progression since the previous game and the current one is evident but the rating is worse. Ok, I can understand that the reviewer don't like the game but  I can't find the logic of the poor rating.

The AGS panel rating is  is one of the most important elements for the majority of the people to select a game. I think there's a lot of work in CQ III and the results are very good.

There're enough critiques and suggestions recently to the panel but I don't see that the members of the AGS Panel are receptive to them. It seems that we are the last link of the chain (personal opinion, maybe not real).

Well I always liked the idea of a panel. And in most cases I agreed with it. But I just was confused by the fact, that panel ratings changed.

I mean: Earl Bobby 2 -which is one of my favourite games- got 5 cups.
One year later it had 4 cups.
Same with other games.

Why do they change the rating?
That is not a serious method.

It is like you say: "I think my steak tastes great.
Oh I remember that steak it tasted terrible."

"I want a glass of milk.
Stop I wanted a coffee..."

Quote from: LimpingFish on Thu 07/01/2010 23:20:46
Dualnames, on reading your post again, I'm not sure if you're condemning the rating...or supporting it?

I'm supporting it. I have to say, I did not see the comment if there was any by the AGS Panel. So I don't have an opinion on this. But on my side of this community, I have made several games. And they all got 2 cups and 2 of them got 1 cup. And the result was reasonable and I completely agree with it. Hell, even the posts of the people who played my games said that they lacked something. And they were right. I've removed games from the database, but not because I didn't like the comments, but because I lost the working links of the download. I feel the panel should keep to their ratings at all costs, and if the author is minded by all this he should contact a moderator, and question the reasons of the bad rating/comment.

Quote from: Le Woltaire on Fri 08/01/2010 10:51:32
1.How many persons are in the AGS Panel?
2.How many are needed to show up the blue cups?
3.Is it just one opinion that is needed?
4.Who is the AGS panel?
5.Should we make a thread about this?

1,2,3,4,5: No/No need to know.

I think if we know who are the guys posting in the panel ratings, then we lose the grip of this. Then we lose this community, and if we reach into a point where we only appreciate those who are praising us,

for if they gave you a 4 cup rating and it was unjust, you wouldn't remove the games from the database would you Harg?!

then I fear we'll have a whole community of shitheads.
As for Calin, indeed it's the only part of the panel I find unjust. But people it's just one occasion! Doesn't mean because one game got rated low, we should question all games.

Quote from: Nergal on Fri 08/01/2010 13:20:37
I think CQ III is totally underrated (but it's my opinion). The progression since the previous game and the current one is evident but the rating is worse. Ok, I can understand that the reviewer don't like the game but  I can't find the logic of the poor rating.

The AGS panel rating is  is one of the most important elements for the majority of the people to select a game. I think there's a lot of work in CQ III and the results are very good.

They're enough critiques and suggestions recently to the panel but I don't see that the members of the AGS Panel are receptive to them. It seems that we are the last link of the chain (personal opinion, maybe not real).

Progress has nothing to do with a game ending up good. Nothing. Nothing at all. The question everyone should be asking is this:

Is the game enjoyable?

If:
Yes, totally, I could replay it: 5 cups
Yes, it is really worth it: 4 cups
Yes, it can be enjoyable: 3 cups
Yes, if you get past its problems: 2 cups
No, not really: 1 cup

Doesn't matter if the graphics are great, if the atmosphere is great, if the music is great (I've had this once), but the gameplay is bad. The interface is a problem. The dialog makes no sense.

Progress is important because it  turns the rating into something illogical. If the last game (CQ2) has 3 cups and the next game is better, the 2 cups rating isn't logic.

And well, for me the game is more enjoyable than the other Cosmos Quest. Ok, it's my opinion and the AGS reviewer has other opinion but with this system, the opinion of ONE person is more important than the opinion of a lot of people.

Why the user rating needs five votes to be showed? Why the AGS review is made for a one person and has more importance than the user ratings? That's the key, for me. ;)

One man review is a madness. Try to watch 3 great moves one after another. The first will be great, the other 2 - NOT so great.

How 1 man can decide which game is good and which is bad when he has a list of games to review and he has to do it quickly (because this website needs content to be more popular). In this case, for me, every game will be boring...

But like Nergal said - the games and authors here are the last peace of the chain.

Quote from: Nergal on Fri 08/01/2010 14:11:41
Progress is important because it  turns the rating into something illogical. If the last game (CQ2) has 3 cups and the next game is better, the 2 cups rating isn't logic.

And well, for me the game is more enjoyable than the other Cosmos Quest. Ok, it's my opinion and the AGS reviewer has other opinion but with this system, the opinion of ONE person is more important than the opinion of a lot of people.

Why the user rating needs five votes to be showed? Why the AGS review is made for a one person and has more importance than the user ratings? That's the key, for me. ;)

Kapputnik said that game reviews differ. For example DITR. It was reviewed to be 7/10 in some sites, and others 10/10. It depends on what YOU like on a game. And seriously people, if you judge games by screenshots/ratings/views/comments you're just getting a pic of the game, and doesn't mean you'll agree to that point. Hell, check the comments on this:

http://www.adventuregamestudio.co.uk/games.php?action=detail&id=820

If it wasn't for either the rating or the panel's rating, people would consider this to be really great game. But it's not. If Harg values his effort, then he should remember that sometimes it's not the goal but the journey.

2 cup game means it's reasonable! For the love of god. It's not bad! Lots of good games are rated 2 cups.

Quote from: Nergal on Fri 08/01/2010 14:11:41
Why the AGS review is made for a one person and has more importance than the user ratings?

For my decision if I want to give a game a try or not the user ratings are more important.
But I think this is a minority opinion.

@Dual
Wow! You created the "best AGS game ever" that's "not recommended to anyone"  ;)

QuoteProgress is important because it  turns the rating into something illogical. If the last game (CQ2) has 3 cups and the next game is better, the 2 cups rating isn't logic.

Whether or not the next game (or any game) is better is subjective and entirely dependent on an individual's response to the game.  The rating system is no different than a player rating in that respect; the only difference is that it's one 'critic' vs many.  ALL ratings like this are subjective opinions and should never ever be taken as a personal attack, nor should they be taken so seriously that you have a hissy fit and remove your games from the database.  If you don't like the rating then either ignore it or try to improve on some of the criticisms offered in the comment (both by the judge and by other players).  Some people are better equipped to deal with criticism than others.  It's clear that you are not one of these people, Harg, and therefore I wonder why you place your games where they are judged by anyone at all?  You could just as easily host them on your own webspace and offer a link in the Completed Games forum and no one suffers as a result.  When you open yourself up to criticism by placing a product out there in a place where it is compared and judged then you should be man or woman enough to handle the consequences, period.

QuoteAs for Calin, indeed it's the only part of the panel I find unjust.

Fair, unfair is a matter of pure 100% opinion when it comes to game ratings.  The person who played the game and rated it had a very specific experience unique from your own or anyone elses and that's how it should be.  Please get over this whole 'it's unfair because I don't like the rating' rant, all of you.

That's all I have to say on this for now.

Quote from: ProgZmax on Fri 08/01/2010 15:26:02
When you open yourself up to criticism by placing a product out there in a place where it is compared and judged then you should be man or woman enough to handle the consequences, period.

ProgZmax, I was open to criticism - almost 4 or 5 years, but not anymore. I agreed with all critics about Cosmos Quest 1 and Cosmos Quest 2. I fixed everything for the Cosmos Quest 3 (1 style graphics, main character sprite was changed, and of course pixel hunt that you will never stop complaining).
For the pixel hunt:
- I tried to use bigger objects, hotspots and characters.
- I placed label (gui) when the mouse is over hotspot, character or object
- when the mouse is over exit of the current room - a mouse cursor is changing to "GO" and a label is appearing also to tell the player that there is a exit. Also the double click - so the player don't have to wait until the character reach the edge of the screen.

What do you want more to stop complain about pixel hunt? Do you want me to place an arrows in every room where the player can walk? NOW WAY. It's a game. It's a challenge! It's not a movie!

And all these pixel-hunt improvements were added/applied to the previous 2 games and I make special posts here on the forum. Of course there was no reaction for these improvements from AGS PANEL. Did someone remake the review after these improvements ? No. Why? Because no one told you? But when I deleted the games, you immediately came with the reaction.

I don't like the way you are treating the games. The games are the valuable content here in AGS website, not the reviews (if someone can call that a review - just a few lines). Support the game makers!

Now don't ge me wrong, anyone who puts stuff out there for other people to play/read etc, should be prepared to accept that not everyone is going to like their work. This is VERY important if you're going to release stuff to the public. However...

QuoteFair, unfair is a matter of pure 100% opinion when it comes to game ratings.  The person who played the game and rated it had a very specific experience unique from your own or anyone elses and that's how it should be.

Exactly! So why should that person's opinion be displayed more prominently than the opinions of others? I can certainly see why people would be opposed to it. If it was a bunch of people who came to the same conclusion then that's different. Hence why the User Rating's system is far more informative and apropriate in my opinion.

QuotePlease get over this whole 'it's unfair because I don't like the rating' rant, all of you.

I was under the impression this was supposed to be a community... ???

Quote from: IndieBoy on Fri 08/01/2010 01:22:01
I would be most willing to discuss the topic, but not really in this thread.

Well I see you guys have been busy then  ;D
To be honest I'm disappointed in the reason behind all of this game removal.
I think I'm going to start a little side project and give all you AGSers the glory cups you want, and you may deserve.
My view on the whole panel thing is clear. It's for the database, ie computing a subjective medium into a number or "score" so that it is easily organised and indexed. Not to be an official review of the game. Increasing the rating because they tried a little harder this time would corrupt the system and would in-turn affect the whole community. The comment is just to justify the rating, to avoid confrontation and suggest weaker areas to be improved.

Man. I wish space pirates got 2cups... before I turned it into a Demo of course  :=

As for Cosmos Quest, I'm sad I don't have the opportunity to play these games, and now I know I will definitely never play them.

ProgZ, big fail there with all the respect.

I think the community is friendly, though I find two things on this matter wrong.

1. Harg's decision to remove your games.

You could have pmed Andail and complain to him in private and not make this whole thing as public as you did. That shows you respect the community and the guys on the panel that are working hard, little.

2. Why do you care that one guy's opinion will make your game look bad?

What about the BAD comments you can't remove?
Also, I totally agree with IndieBoy.

Harg:

You point to having fixed many 'problems' that plagued your previous games, but I think you fail to see that bugs/gameplay issues are only a part of a game.  There's also whether or not a person enjoyed the visuals, the music, the game itself, and enjoyment is a big part of whether or not a person is going to like your game.  Sometimes really buggy games draw attention for just being fun while highly polished games with nothing interesting going on do not draw attention.  I'm not saying this applies to you as I have not played your latest game but it's something to consider.  I'm also not sure what you mean by games being 'valuable content' either, exactly.  They're only as valuable, as worthwhile as people find them to be, again a highly subjective thing.  One man's trash is another man's treasure.

And believe me, I do know what it's like to have a game you've spent considerable time on slapped around by people because their idea of what was good and fun differed from mine, but at the end of the day you have to let that go and realize that YOU, Harg, are not put on this Earth to please anyone.  If someone rates your game poorly or doesn't like it, who cares?  Just keep doing what you're doing if it makes you happy.

QuoteExactly! So why should that person's opinion be displayed more prominently than the opinions of others?

Quick, go and email Gamepro and Gamespy and IGN and Gamespot and Adventuregamers and all the other sites that have single person ratings (oh and don't forget reviews!) for games that they have no place displaying them!  Back in the real world, though, sites do this sort of thing all the time and frankly I'm mystified why, when Chris Jones decides to implement a similar system on his own fucking website for games made with his own engine people are appalled?

QuoteI was under the impression this was supposed to be a community.

And I'm under the impression that there's an age limit for being part of this community along with an expected level of maturity that includes being able to take game ratings with a grain of salt, whether made by one person or 20.

QuoteProgZ, big fail there with all the respect.

I really have no idea what you just typed.

Quote from: Dualnames on Fri 08/01/2010 16:56:17
ProgZ, big fail there with all the respect.
I tt 1:hink the community is friendly, though I find two things on this matter wrong.
1. Harg's decision to remove your games.
You could have pmed Andail and complain to him in private and not make this whole thing as public as you did. That shows you respect the community and the guys on the panel that are working hard, little.
2. Why do you care that one guy's opinion will make your game look bad?
What about the BAD comments you can't remove?
Also, I totally agree with IndieBoy.

About 1: The decision to remove the games is not wrong. Actualy I thing its positive for the game and for the AGS website. After the removing the games I received many emails from other game developers that supports me (i'll keep theirs nicknames). Many of them gave very useful information about websites where I can publish the game. I'm telling you guys from AGS - I'm not the only one who's complaining from AGS PANEL.

About 2: Every AGS game developer cares about the AGS PANEL opinion. That's for sure!

1. I'm supportive enough if that's what you mean. I just suggested there was a better way to do this, than create a rant about it. You could actually replace the link here with the one of your site, without creating this off-topic.

We know. Perhaps if there are other complains it's up to the panel to decide it's actions.

2. To be honest, i'd love to take anyone's side on the matter, but I can't. I tried to stand up for the panel , but i see reasoning in your posts too.

As someone who has dealt with a 2 cup panel rating I have one thing to say,

Get a grip.

What possible motive could you have for removing your game?

Its either protest or simply to avoid a bad review.
One of those is stupid and the other is dishonest.

If your game was given an undeserved 5 cups would you have removed it out of protest? I fail to see the difference.

Someone reviewed your game and gave it a less than favourable review and so you remove it.
That is essentially silencing the press.

It's not like the panel rating strives to be an objective, perfect rating. It's by one person and highly subjective. Therefore, it's perfectly fine to not give two shits about it, especially if it contradicts the consensus of the user ratings.

The only problem I see is that a casual player looking for recommended games might mistake the rating for an average of several judges or even all votes by a huge community. The explanatory text doesn't exactly convey the opposite either.

I didn't remove my short, crappy games although they got 1 and 2 cups, respectively. I'd feel more embarrassed if I did actually.

The more we heckle the panel, the less inclined they will be to rate games.  I do not want to see the rating panel disappearing, or them voting all under average games with 3 cups (even if they honestly felt it deserved a 1 or 2 cup).  I think they are doing a great job, let's not cease their involvement. 

The last thread (http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39595.0) about the rating system has already expressed everyone's feelings.

We should leave Harg with his decision to remove his games from the database, and he can sort it out with the panel via pm.

As far as I see it, unfortunately, this isn't a completed game thread anymore.  It should be locked.

QuoteAnd I'm under the impression that there's an age limit for being part of this community along with an expected level of maturity that includes being able to take game ratings with a grain of salt, whether made by one person or 20.

Yeah of course everyone should be mature - but your remark earlier stank of someone throwing his toys out of the pram...exactly what your accusing Harg of doing. So I'm sorry but if that's classed as being mature, then I'm Hulk Hogan (which I'm not by the way, in case anyone was perhaps unsure.)  :)

On a far more constructive note, would it perhaps be better to make it clear which member of the AGS Panel has reviewed each game? After all Progz, is is not true that all the publications you list give the name of the person judging a game? Makes it much clearer that it's just the opinion of one person...

If it's just an opinion (which it is) then why be so mysterious about whose opinion it is? To use myself as an example - if people find their opinions on games differ greatly from the views I express on my reviews blog, then they can choose not put any value in my thoughts. Knowing there opinions are often different from my own, they can choose to avoid my reviews if they wish.

The whole mysterious AGS Panel thing doesn't allow people to perhaps follow a certain critic whom they know has a similar taste to them.

I don't think it's a huge issue but it would be a shame to lose game developers because of such a problem.

The whole problem with this community is that some people go High-and-Mighty, and that's about as f4^^&$& as i can take. Can someone go rate Harg's game 5 cups? Please? And in fact go rate all under-average games with 4 cups at least?

Find Therma, that's wrong of you to ask. Imagine if I was rating games for instant and I rate a game unfairly according to its author, none would want me to rate games anymore, and Andail would have to either find another or assign games to some other guy. And if a guy overrated games, everyone would want him to rate their games.

So how about this? Rate your own games.
I'd hate to lose intergrity as a community, because somebody can't take life less seriously. You know what I've been making LC 3 for 2 years nearly. And although I improved everything and added voice acting it got rated the same cups. And you know why? Because the damn game wasn't one f^*(% enjoyable than LC 2 was. And two cups means: Worth a try! FFS it's not bad.

I'm in favor of the privacy.

Would you demand to get a lower rating if CQIII was given a 4 cup which it didn't deserve?

Don't complain because the panel has standards. If you want universal approval for your game, show it to your mother. Just don't be childish and remove your game out of spite.

Probably the amount of cups isn´t the problem here. My game got a 1-cup rating, but the  panel comment was really nice

Quote
"Great look and feel, and awesome music. Although the game is buggy and rather not clear. The game showed a lot of potential."

, so I didn´t care.

Probably Harg did feel more offended by the comment, than by the 2-cup rating itself.
That would be the only thing I´d like to see concerning the panel reviews: that they should be written in a nice way, as this would also fit the niceness of this community.

Quote from: DrWhite on Fri 08/01/2010 20:17:14
Probably Harg did feel more offended by the comment, than by the 2-cup rating itself.
That would be the only thing I´d like to see concerning the panel reviews: that they should be written in a nice way, as this would also fit the niceness of this community.

No, that compromises the journalistic integrity of the panel.

Like any publication, the writers should be able to write what they like providing it is either true or opinion.

The only people who should have control are the panel leader (if one exists) and CJ since its his site.

QuoteThe AGS panel rating is  is one of the most important elements for the majority of the people to select a game.

QuoteAbout 2: Every AGS game developer cares about the AGS PANEL opinion. That's for sure!

But not every players do.

Players care for screenshots, hooking story synopsis, the game's features, the way the game is presented, the persons who made the game, the gameplay, the price tags... The panel's opinion is just one variable that will be ignored everytime the screenshots or plot overview or every other variables will have the players say "Man, this game looks great, I'll give it a try!". I've played poorly rated games that were a whole lot of fun and critically acclaimed games that bored the crap out of me.

So, if you want my opinion, take the panel's opinion with a grain of salt, put the games back for download/in the database/on the forum, take into account the constructive criticism and ignore the unconstructive one, keep on making games and never forget that it's impossible to please everyone and just like there will be people who won't like your games, there will be people who will like them, who will want to play more of your games, and stopping to submit your games here or in the database will only make it harder for them to find and play your latest games.

It's your call, buddy. What do you honestly think will hurt your games more? A panel review a lot of people don't give a shit about, or that from now on your current and future games will be even harder to find?

Quote from: Nergal on Fri 08/01/2010 13:20:37

The AGS panel rating is  is one of the most important elements for the majority of the people to select a game.

No... really not.
Most people that come from outside don't even know what a blue cup is.
I've never seen one in real life.
And I don't know what a blue cup has to do with point and click adventures...

Maybe this is going to be the reason why other engines will beat AGS in the future?
Shouldn't we change the logo?
Bad marketing concept to use an insider object as front page sign...

Quote from: Le Woltaire on Fri 08/01/2010 21:55:33
Maybe this is going to be the reason why other engines will beat AGS in the future?
Shouldn't we change the logo?
Bad marketing concept to use an insider object as front page sign...

Other engines will "beat" ags because of a logo? I think not.

User friendliness, features and the actual games that get made with it are far more important than a mere symbol.

Quote from: Le Woltaire on Fri 08/01/2010 21:55:33
Maybe this is going to be the reason why other engines will beat AGS in the future?
Shouldn't we change the logo?
Bad marketing concept to use an insider object as front page sign...

Thats possibly the strangest thing ive ever heard.

Quote from: Le Woltaire on Fri 08/01/2010 21:55:33
Most people that come from outside don't even know what a blue cup is.
I've never seen one in real life.

They dont exist.

If you bring blueness and cupness together they annhilate like anti-matter.

I was joking, man...

This has just gotten silly.

Quote from: HargHow 1 man can decide which game is good and which is bad when he has a list of games to review and he has to do it quickly (because this website needs content to be more popular).

I can't fathom how you've reached this opinion of the rating process. Pick a gaming website at random (IGN, Gamespot, Adventure Gamers, etc), and you'll find reviews are usually the work of a single author. A single author with a single opinion. Most, like the AGS database, also have a player rating, because, as anyone with a crumb of common sense knows, no one opinion is ultimate.

And the panel tries to play games through to completion, barring any fatal bugs that prevent us from doing so. There is no rush for "content".

Harg, if you feel that your work is too good to face the same critiques every other game has is the database, then that's your decision. You won't find a better reception anywhere else if you continue in this manner. You've managed to insult most of us here.

Quote from: Harg
ProgZmax, I was open to criticism - almost 4 or 5 years, but not anymore.

So you've reached the pinnacle of game design? This is as ludicrous a statement as it is a pretentious one. The is no ultimate point we reach were we've learned all we can. Commercial developers with 20+ years under their belt still strive to improve their games and are usually their own harshest critics.

But to basically say "I'm an artist, and you'll either like what I do, or shut your mouth!" is an astonishing response to a community that has been supportive. From giving you a forum to advertise your games, to providing technical help should you require it, the AGS community has been nothing but supportive. Your behavior not only in removing your games, but also in the manner you've conducted yourself in this thread is quite baffling.

All this could have been avoided with a short pm:

"I do not agree with the rating comments for my game Cosmos Quest III. Due to this, I have decided to remove my game from the database."

And you would have probably received a short pm in reply, asking you to reconsider, but saying the final decision was up to you.

And we all could have gotten on with our lives.

But you pulled all of your games, and made it a point to advertise this fact in the CQIII thread. Which smacks of an attention-seeking "That's SO unfair!" hissy fit; something that belongs in the playground.

This whole situation has depressed me.

If a game receives a bad critic, the developer must accept it. If the users criticize things of a system that they don't like, should not members do the same thing?  ::)

Hell, I see no point being such an ass, because you think so highly of yourself, when those poor guys that worked so hard on King's Quest got only 3 cups. What should they say? But see that's what you've misunderstood. 3 cups mean must play! That's a really good rating for that game. It's reasonable!

But the whole point is during the making of this game, you thought how you'd get those 4 cups you really wanted. And you lost that loving feeling! That's all have to say.
And that is explained by how the world received your games. The topics on the other cosmo quests have 16000 views each and over 35 replies. This one has 4691 and like 30 replies and it's all not for the game and wayyyyy before the game got rated for the panel. That says it. And that's the people opinions!

I might be harsh, but isn't so the truth?

Quote from: LimpingFish on Fri 08/01/2010 23:43:22
But you pulled all of your games, and made it a point to advertise this fact in the CQIII thread. Which smacks of an attention-seeking "That's SO unfair!" hissy fit; something that belongs in the playground.

Well, it seems that Harg just left a brief note that he is removing his Cosmos Quest games without elaborating on it. It hardly "smacks of an attention-seeking "That's SO unfair!" hissy fit" and while it is unfortunate I don't understand how it is insulting to others.

What I wanted to add to the discussion is: No one can force anyone else to like something (or even play something), even if one has perfect arguments. The only solution is to create primarily for different reasons than expected reception by other people. A cold reception might take out some joy of your efforts, but no enough to stop you.

Split this for obvious reasons.

I propose people on both sides try and calm down a touch.

To all the Panel Members: Don't take it personally when someone removes their games. Let them do what they want and let it be. There are hundreds of games not removed after being rated, focus on those and pat yourselves on the back if you want. Also accept that you are in a community and people have the right in this community to criticize functions of it. If people are upset and you can't think of a way to calmly help, don't post. Especially because you are a involved in a function these forums and therefor a voice for them.

To all the others: The panel members aren't evil. They are members of this community and volunteered to do something that was established to try and help sort out the games database. It's awesome that you all are so passionate about this community that you even care about this subject. There are so many forums that are just dead dead dead because people don't have a passion for the community. The only thing I ask is to try and remain civil and not get personal. If someone is getting personal with you, try and ignore them.

To all everyone: I would recommend against telling each other to not talk about a certain topic because you are particularly sick of hearing about it. If you are sick of it, stay out of the thread.

To all the single ladies: 'sup, 28/M/NY/20 feet tall, pix pls?

Quote from: LimpingFish on Fri 08/01/2010 23:43:22

Quote from: Harg
ProgZmax, I was open to criticism - almost 4 or 5 years, but not anymore.

So you've reached the pinnacle of game design? This is as ludicrous a statement as it is a pretentious one.

I think the thread is suffering a bit from the fact that several of the people posting don't seem to be 100% fluent in English. (I can only understand what the heck Dualnames is trying to say about 50% of the time, for example.) I'm guessing that Harg doesn't mean that his games are now above criticism, just that he's been hearing the same complaints for 4 or 5 years, that he has (in his view) addressed them without getting any credit for it, and that he no longer has patience for criticisms he doesn't consider valid. But hey, that's just my interpretation!

Along the same lines, I'm pretty sure "How many persons are in the AGS Panel?" meant how many people are involved in reviewing/rating each individual game.

While I think that the panel review process is far from perfect, any review system worth a damn is going to hand out negative reviews from time to time, and some people are going to be unhappy. Harg, your grievances suggest fixes like providing longer reviews, or have several panel members offer their individual perspectives, but how does that fit in with your assertion that the games, not the reviews, are the core content of the Games DB? (Which I agree with.)

I think deleting the games is an unfortunate overreaction. At the same time, if Harg doesn't like this community or agree with the way important functions are being run, he's well within his right to withdraw. He's hardly the first or only creator to be oversensitive about criticism, anyway!

What annoys me a bit is the attitude of some responses of "it's just one opinion, deal with it!"--like the rating featured prominently on the AGS site doesn't matter, and it's wrong to care. Of course most developers are going to care! Essentially taunting people for "being a baby" when they don't deal as well as they probably should is not very classy, and when done by panel members, it doesn't make the panel--or this community--look very good. On that I agree with Find Therma. (Bet you wish you'd stayed anonymous now, you unwitting spokesmen for the panel! ;D )

Not that I don't understand ProgZmax or LimpingFish's impulse--I've done pretty much the same thing (http://www.adventuregamestudio.co.uk/yabb/index.php?topic=25316.msg318035#msg318035) myself; it can be really annoying when someone takes back something they had previously shared with the community, especially if you invested your time responding to it.

Quote from: MrColossal on Sat 09/01/2010 01:11:55To all the single ladies: 'sup, 28/M/NY/20 feet tall, pix pls?

20 FEET TALL WHAT

I've come up with a solution.

It's so simple, I can't belive no one thought of it.

You ready..... It'l blow your mind.....


Give every game a 3 cup rating, then everyone's happy.  Except for the people who had a higher rating, but they're in the minority so that's ok.    ;D

Problem solved, fools.

Quote from: Green Boy on Sat 09/01/2010 01:51:18
Give every game a 3 cup rating, then everyone's happy.  Except for the people who had a higher rating, but they're in the minority so that's ok.    ;D

I honestly have a feeling it'll eventually turn into something similar to that, with all these reactions lately.



Wasn't that Cosmo game a commercial game?  I can understand the panel being tougher on commercial games, because cost vs enjoyment falls into the formula.

Everyone needs to get a grip. The rating disclaimer says itself that it's okay to disagree, but don't feel offended if the rating isn't high enough in your opinion. I can understand feeling hurt that someone didn't like "your baby," but the system is in place to highlight the really astounding games. The last thing I'd want is for someone to rate my game anything other than what they thought it deserved.

Quote from: Green Boy on Sat 09/01/2010 01:51:18
I've come up with a solution.

It's so simple, I can't belive no one thought of it.

You ready..... It'l blow your mind.....


Give every game a 3 cup rating, then everyone's happy.  Except for the people who had a higher rating, but they're in the minority so that's ok.    ;D

There's more subtle solution.
Make, say, 20 cups minimal grade and 25 cups maximal grade. In this case you'll keep general rating concept and yet make lower grades less unpleasant.

Quote from: Ascovel on Sat 09/01/2010 00:51:48
Well, it seems that Harg just left a brief note that he is removing his Cosmos Quest games without elaborating on it. It hardly "smacks of an attention-seeking "That's SO unfair!" hissy fit" and while it is unfortunate I don't understand how it is insulting to others.

I don't think the one line assertion "Cosmos Quest I-III - The games are removed by the author." qualifies as a note, but I see your point. Removing the games because he doesn't agree with the rating is insulting to the panel member who took the time to play his game (it's not a particularly short game) and critique what he felt were it's flaws. If we waste time rating those games that are going to end up being removed from the database for no other reason than sour grapes, why should we bother at all? It's also insulting to suggest we rate games haphazardly to meet some nonexistent deadline, when the author in question didn't bother to enter into any sort of civil communication with any facet of the community before pulling his games. If he had he might have discovered that this isn't the case. I can't really find any other way to describe it, other that a hissy fit. A tantrum.

Quote from: Snarky
I think the thread is suffering a bit from the fact that several of the people posting don't seem to be 100% fluent in English. (I can only understand what the heck Dualnames is trying to say about 50% of the time, for example.) I'm guessing that Harg doesn't mean that his games are now above criticism, just that he's been hearing the same complaints for 4 or 5 years, that he has (in his view) addressed them without getting any credit for it, and that he no longer has patience for criticisms he doesn't consider valid. But hey, that's just my interpretation!

That's a valid point, but any such mis-communication might possibly have been avoided, if the author hadn't chosen such a counterproductive and knee-jerk approach. Personally, it says to me that he's tired of people pointing out the faults in his games. Period.

Quote from: Harg
AGS PANEL is trying to tell us how we have to write the games. No way! The indie game creator is an artist not an engenier!

Imperfect English or not, that seems a pretty strong statement.

But, has anyone here actually played Cosmos Quest III? I'd like to hear some opinions from outside this debate; opinions that may or may not dispute the validity of the comment left by the panel.

EDIT: I'm using the lesser "insult", rather than "INSULT!". "Insulting" like a insolent french waiter, not "INSULTING!" like somebody pissing in your cornflakes, while shitting on your dog. Just a little clarification.

Don't I just love this community. When something is f$^&% up, someone comes and blaims my communication skills :D.

I think we should take this matter more lighthearted, and I apologize for probably sounding like an ASS and taking this too damn seriously.

Personally I blame AGA. ;)

PS: I think I made some valid points back there so I'll sum up:
1)I think Harg should have taken this into pms with the panel instead of removing his games.

2)The lack of posts and interest of Cosmos Quest III probably makes the choice of two cups just. If someone can post the AGS panel rating, so people who HAVE  played this game justify on it, would be perfect.

3)I think the panel should remain as it is, and it's as most people have said, guidelines.

4)On no account should people consider that because the panel is giving a low rating, they don't count effort. They are of course game makers themselves!

[On second thought, not sure how much point there was to the first bit of the post, and I actually don't want to stir up trouble just for the sake of it, so snip it goes. Hope no one was working on rating it!]

Quote from: LimpingFish on Sat 09/01/2010 03:11:30It's also insulting to suggest we rate games haphazardly to meet some nonexistent deadline, when the author in question didn't bother to enter into any sort of civil communication with any facet of the community before pulling his games. If he had he might have discovered that this isn't the case. I can't really find any other way to describe it, other that a hissy fit. A tantrum.

Someone else (Dualnames?) suggested he should have PMed you or ProgZ. OK, maybe that would have been a better way to handle it--but it's worth pointing out that the site offers no point of contact, that the info-text with the rating suggests that game makers should not entertain complaints, and that in forum comments, both you and ProgZmax (as the highest-profile panel members) have been quite hostile to complaints about panel ratings. It does not, over all, come across as an invitation to "enter into any sort of civil communication".

I think you confuse hostile with assertive and to the point.  No, LimpingFish and I haven't been hostile.  Hostile would be essentially telling anyone with a complaint to fuck off and not continuing to discuss the issue.  Now, while we may get annoyed by recent frequent complaints and extreme actions taken by certain members we've never been hostile, only restated how the system works and that the ratings are subjective, and most importantly, not to be taken seriously.  Certainly not to the point where you feel your work is being insulted and/or discredited. 

Now, I could see someone taking that sort of view with someone of influence and power like John Carmack or Jane Jensen, but generic, faceless forum members?  We're really not important enough to place on such a pedestal, nor do any of our ratings (panel or otherwise) amount to a hill of beans in the greater scheme of things.  They are merely there as suggestions and opinions by people who have played the game.  If anyone is going to be swayed by one person's opinion then I can honestly say you don't want that person playing your game anyway, because if they are interested in what you've got they'll play it regardless and draw their own conclusions.

And on the topic of being annoyed, one of the key reasons is the accusatory tone in posts like yours, Snarky.  You go from accusing the panel of potentially hiding something by not offering 100% disclosure to accusing the two panel members willing to actually discuss the ratings and their significance of being hostile.  None of your accusations are either helpful or constructive and none of them help to make any of these discussions go any smoother.  Stop trying to dredge up conflict with charged statements (this applies to everyone) and come up with something that actually benefits the discussion so that it can move forward rather than grind its wheels in 'you're this' or 'the panel's that'.

If any of you have some sober questions about the panel, by all means private message me.  There are certain things I cannot discuss, like why a rating was given, but I can explain some of the rationale that goes into the ratings.  Also, the list of guidelines will be made available soon so you can see what the panel uses as a reference when playing each game in the database.  I stress reference because people tend to take lists as a literal interpretation of what's going on and personal reactions to a game play a large role in whether or not it will be well-received.  Hopefully that makes sense!

Finally, my primary interest right now is to try and help Harg see that the rating isn't a personal attack on him or his game, nor is it of any real value.  The panel member who played it was not bowled over by the game and listed his/her reasons in the comment, and that's all.  Please don't take these ratings seriously, guys.  Trust me when I say there is no malice in any of the ratings offered, and if that time ever comes it will be dealt with severely.  We're not doing this to discourage anyone, and by and large I think we've succeeded in that.  There's a clear vocal minority who are displeased with their ratings, but there's very little that can be done to make them happy except give them a different rating, which is disingenuous.

Would it help if a person upset by their game rating was given a more detailed account of the rationale behind it, delivered to them by myself (or LimpingFish) via private message?  Would that be something people might appreciate in the future?  Granted, it's not going to help everyone but I suppose it might help some people?

I'm sure the panel judge who rated Cosmo Quest III would be more than willing to write up a couple of paragraphs on what he/she liked and disliked about the game and how he/she came to a final rating if you're interested, Harg.

What do you think?

Ratings are, and always have been a flawed system of telling people whether a game/film/whatever is good. Good, fun and entertaining is too subjective an opinion to be rated in any way. I guess a more appropriate way would be having 3 cups being the AVERAGE since it is in the MIDDLE. Shit games and awful games (like my masterpiece Gorthor) deserve 1 and two cup ratings. But ratings themselves can be confusing, which I think is why people are having an issue with it. Some ratings consider anything around 50% to be average. Some others consider 70% to be average. It's not an accurate way of telling whether a game is good or not. Do away with the whole thing and perhaps include a small review only. You can still have a list of highly recommended games.

Why else would a highly flawed game like Limey Lizard get such a high rating? In my opinion, the radical user interface detracted from the playing experience, and a walking dead made me rage quit. But that game is rated 4 cups. The rating system is flawed.

I don't really think it is appropriate for such a highly established and recognised community member to be saying 'fuck' in every sentence either, especially when you are trying to argue your point.

Quote from: ProgZmax on Sat 09/01/2010 06:40:02
I think you confuse hostile with assertive and to the point.  No, LimpingFish and I haven't been hostile.

Let me illustrate. I think comments like these indicate that you are hostile to having people contact you with complaints:

Quote from: ProgZmax on Wed 16/12/2009 19:24:41
Even from the start of the panel ratings there have been two types of people:

1.  Sour grapes posters who don't like their rating or ratings on games they like.

Quote from: LimpingFish on Thu 17/12/2009 01:44:08
Frankly, I really don't care to enter into discussions with creators, simply because I don't think they can say anything that would change my mind. If you have to explain why your game deserves a higher rating, you're already wasting your time. And mine.

Quote from: LimpingFish on Fri 18/12/2009 19:33:48
Let me make on thing clear, for future reference. Any ratings bullying, passive-aggressive or otherwise, will not be tolerated. If you have a problem with a rating, either leave a comment on the game's database page, in the game's official forum thread, or, if you must, PM myself or ProgZmax. [Snarky's emphasis]

And wrt hostility to criticism in general:

Quote from: ProgZmax on Sat 09/01/2010 06:40:02
And on the topic of being annoyed, one of the key reasons is the accusatory tone in posts like yours, Snarky.

Those and other comments along those lines sound to me like a hostile attitude.

Quote from: ProgZmax on Sat 09/01/2010 06:40:02
Hostile would be essentially telling anyone with a complaint to fuck off and not continuing to discuss the issue.  Now, while we may get annoyed by recent frequent complaints and extreme actions taken by certain members we've never been hostile, only restated how the system works and that the ratings are subjective, and most importantly, not to be taken seriously.  Certainly not to the point where you feel your work is being insulted and/or discredited.

By "hostile to" I just mean that you are opposed to it, you do not welcome it, and you give every indication that it annoys you.

Quote from: ProgZmax on Sat 09/01/2010 06:40:02
And on the topic of being annoyed, one of the key reasons is the accusatory tone in posts like yours, Snarky.  You go from accusing the panel of potentially hiding something by not offering 100% disclosure to accusing the two panel members willing to actually discuss the ratings and their significance of being hostile.  None of your accusations are either helpful or constructive and none of them help to make any of these discussions go any smoother.  Stop trying to dredge up conflict with charged statements (this applies to everyone) and come up with something that actually benefits the discussion so that it can move forward rather than grind its wheels in 'you're this' or 'the panel's that'.

The point I was trying to make in my latest post was that there's a disconnect between what people are saying disgruntled game makers should do and the instructions and signals the game makers actually get. Want the implicit "helpful and constructive" suggestion made explicit? CHANGE THAT!

I also ended my involvement in the last round of argument around this with an explicit list of constructive suggestions, so I take exception to your characterization. We could go round and round on who's been most confrontational [examples deleted for the sake of diplomacy], but instead let me say that you seem to have taken offense at the word "hostile", so would you prefer "unreceptive" instead?

As for what should be done about situations like this, I pretty much think: nothing. People were deleting or threatening to delete their games from the db and their posts from the forum following criticism or dispute before the panel existed, and will no doubt continue to do so whatever system is in place. The best way to deal with it, I think, is to respond with equanimity, and gently try to convince the person to reconsider, and if that fails just let them do what they want and not worry too much about it.

But sure, opening lines of communication so people can blow off steam before it comes to such radical measures is certainly worth a try.

I had nothing to do with the ratings of any of my games, Ryan, and of course the rating system (like all subjective ratings systems) is flawed.  You believe you are right in that a game doesn't deserve 4 cups because that's what you believe.  Whoever rated it believed differently, and that really is that.

As far as using language, I use it when I think it's appropriate to stress a point, and that's my prerogative.  I don't consider myself to be an 'important' ags member; I moderate a single board and rate a few games, big deal.  Grundislav is a far more important cog in this community than I am and he is neither a moderator nor a panel member.


Snarky:

In that case I apologize.  While unreceptive and hostile do not in any way mean the same thing, if that's what you meant than I could see where you might get that idea, though that's really not the case.  Most of my statements have been to reflect that there are in fact guidelines we go by in the ratings and that we aren't going to change ratings just because a few people are unhappy about them because it's disingenuous and taints future objectivity.  If that's being unreceptive than I can't really do anything about it, but there have been suggestions made that we have publicly acknowledged with potential, your request for the guidelines being one of them.  I don't think you're being exactly fair in your assessment overall, but there's no point in going around about it.

I'm still curious about what people think of being given extended feedback about their rating upon request, though.  This could extend to a good or bad rating, theoretically, but since we're doing this on our free time I'd like to limit it to people who really have an issue.  Would anyone find this to be a helpful way for them to come to terms with their rating?

Oh jeepers, forum drama. I'll get the flame shields...
I think that anytime an argument breaks out on the internet to the point that the posts are multiple paragraphs long, all dissenting parties should be required to talk to each other face to face, IRL.

Yay, sort it out the Jay & Silent Bob way (http://www.youtube.com/watch?v=MjWFZPJZTxU)!

Quote from: ProgZmax on Sat 09/01/2010 08:14:11
I had nothing to do with the ratings of any of my games, Ryan, and of course the rating system (like all subjective ratings systems) is flawed.  You believe you are right in that a game doesn't deserve 4 cups because that's what you believe.  Whoever rated it believed differently, and that really is that.

Aye, I agree. I was pointing out how subjective the rating system is. In addition, I never explicitly stated that I thought you would rate your own game. I know this would never happen.

User rating and panel rating must be seen as equal (or even user rating as higher), I don't think this is the case currently. I know user rating can be jimmied with, which is why we have the panel rating in the first place. Limey Lizard, for example, has a user rating that is pretty much on par with the panel rating. This means it was probably rated correctly. Some people may not have liked it for whatever reason, but the majority did. But if there were the case where a game was rated 2 by the panel and got a user rating of 75%, then the panel rating should at least be looked at, to ensure the ratings are 'correct' (bad word to use with such a subjective matter, but oh well...)

I don't honestly disagree with your points because I have no opinion of the ratings either way.  The cup rating system by a panel was CJ's idea and the motivation and reasoning for it is his to provide at his leisure.  I would hazard a guess that he wanted to mimic the system 90% of sites similar to this one do with an 'editorial' rating that appears at the top of each game rated and then people decide whether or not it's of value.  Perhaps the reason there's so much controversy in this case is because there are so many games (and so many RATED games) in the database vs the average site that will do one or two reviews a month?  Either way I don't object in any way to the user ratings being more prominent if that will suddenly make everyone happy since both sets of ratings are based on personal experiences (group vs single).

QuoteI can't fathom how you've reached this opinion of the rating process. Pick a gaming website at random (IGN, Gamespot, Adventure Gamers, etc), and you'll find reviews are usually the work of a single author. A single author with a single opinion.

You can't use this example. Because while all review sites use a single reviewer for a single game, there's thousands of review sites and blogs out there that will review the same game, so if you don't agree with one's opinion, you can just check another site, because there will be plenty of alternative reviews. For many AGS games, the AGS panel is the only review they'll ever get.

But don't get me wrong, as much as I wanted games to be reviewed by more than one person, I don't want you to change anything to the review panel. I've long accepted that the current system is as perfect as it will ever be. There's just too many games in the database and too much work done already, changing just one little thing could and would demoralize everyone who's been involved in the panel in the last couple of years, and in the end, there would still be people who'd complain about unfair reviews so it would be completly pointless.

QuoteThere's more subtle solution.
Make, say, 20 cups minimal grade and 25 cups maximal grade. In this case you'll keep general rating concept and yet make lower grades less unpleasant.

Haha! It's funny because I had exactly the same idea yesterday. Have the second cup worth 5 cups. So we'd have games rated 1 cup, 6 cups, 7 cups, 8 cups and 9 cups, no more problems with the second cup not being obvious that it means "Worth a try", and no need to overhaul the whole system. ;D

-edit-
Oh, as for the idea of giving more importance to the user ratings, how about ditching (or converting) the current 0-100% user rating system completly and replacing it with a 0-5 cups rating? Not unlike the system on Abandonia.com. Or would that be too much trouble for nothing?

My two cents: I think the panel reviewers are doing a good job all in all. Yeah, I didn't agree with the original rating for the McCarthy Chronicles. I didn't play Cosmos Quest 3, so I cannot say if the rating was fair. But "fair" is subjective anyway, so I agree that the reviewers should stick to their decisions, as long as they can argue for them.

Quote from: Layabout on Sat 09/01/2010 07:12:32
Ratings are, and always have been a flawed system of telling people whether a game/film/whatever is good. Good, fun and entertaining is too subjective an opinion to be rated in any way. I guess a more appropriate way would be having 3 cups being the AVERAGE since it is in the MIDDLE. Shit games and awful games (like my masterpiece Gorthor) deserve 1 and two cup ratings. But ratings themselves can be confusing, which I think is why people are having an issue with it. Some ratings consider anything around 50% to be average. Some others consider 70% to be average. It's not an accurate way of telling whether a game is good or not. Do away with the whole thing and perhaps include a small review only. You can still have a list of highly recommended games.

I agree that one possible problem is the perception of the 5-cup-scale. No matter what the formal definition for each cup-rating is, it is a simple fact that most people see anything below 3 cups as not worth their time. A reason might be that in school and academic circles everything below average is deemed as not good enough. When I write a scientific paper that is peer-reviewed, anything below 3 means the paper is rejected by the reviewer, 3 means neutral, and better than 3 are various degrees of acceptance. Similarily, even with PC game reviews which are often done on a scale from 0 - 100%, anything below 50% is usually interpreted as rotten and definitely not recommended. An average but in no way exceptional game would typically be rated around 75%.

A solution? I don't really know. It would probably be best to redefine the meaning of the cups, so that 3 means average. As it is, of 952 games in the database 47 games have 4 or 5 cups (only 4 games have 5 cups). This is heavily skewed towards the bottom end of the spectrum. But changing the meaning of the cups would mean re-rating all the games, which is not feasible either.

In the end the system should probably stay as it is. It is my subjective feeling that the panel reviews have become stricter recently, with polished games without major faults getting 2/5 cups. But I choose to see this as a good sign - it doesn't necessarily mean that the reviewers have become unfair or that the quality of AGS games is declining. On the contrary, AGS games have become so high-quality that simply being good is not enough to stand out anymore. It just means the community as a whole gets better all the time.

And for me, personally, the panel rating is only one of several criteria when I decide whether to download a game. The screenshot and user-rating are at least as important to me for the first impression.

All right, listen now. There will shortly be a nice comprehensive thread informing about the panel.

As Eric said, try to calm down a bit. Take the whole rating thing with a pinch of salt.

It's absurd to believe that a system like this can satisfy everyone. There are hundreds and hundreds of game authors in the database, every single one of them will never be pleased.

It's impossible to have a huge debate every time someone disagrees with the rating. Try to remember that this is ultimately a project sanctioned by CJ.

Try to understand that a public discussion is not viable. There are as many opinions as there are members in this community. Count the amount of suggestions given in just this thread. No matter how eloquently expressed, an opinion is just opinion. There is no absolute, god-given true path to walk here.

Quote from: Wesray on Sat 09/01/2010 12:26:05
No matter what the formal definition for each cup-rating is, it is a simple fact that most people see anything below 3 cups as not worth their time. A reason might be that in school and academic circles everything below average is deemed as not good enough.

More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

Quote from: LimpingFish on Sat 09/01/2010 03:11:30
Removing the games because he doesn't agree with the rating is insulting to the panel member who took the time to play his game (it's not a particularly short game) and critique what he felt were it's flaws. If we waste time rating those games that are going to end up being removed from the database for no other reason than sour grapes, why should we bother at all?

To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

It would be best if both the panel members as well as the game creators would be able to find some sort of satisfaction in the work they do, and not take for granted the amount of sympathy and respect they'll receive for their efforts.

I'm going to try and respond to some of the points raised so far:

How is it fair that only 1 person on the panel rates each game?

In an ideal world, we would have a panel of 10 people, who all played every game and came to a consensus for the panel rating. This would be possible if the AGS Website was a company with lots of money that could employ full-time reviewers to do this.

In reality, all the panel members are volunteers who have other things to do, and if we had to get several people to play each game through before rating it, they would never have the time to rate all the games that get released.

If you don't like the rating, just get over it, it's not that important

I don't think it's that simple. One major difference on this website is that we have the game developers and the game reviewers as part of the same community. Normally, you have several websites that review games (eg. Adventure Gamers), and each of them give their own opinion on a new game.

The AGS Panel is different because it is seen to be the "official" opinion of the AGS community, and is featured on the website of the development tool that the game author was using. So in that respect, the implication is that it is more important than ratings given elsewhere.

For that reason, I think it's important that the Panel always tries to be as polite as possible with their comments, bearing in mind that they are peer reviewing other developers work, and the last thing we want is to put people off making games, or drive them to leave the community.

One other thing to bear in mind is that the AGS Panel is much harder to please than many review sites -- for example, the panel have so far rated 827 games, and only four of them got a 5-cup rating. You can see the spread of ratings here:

(http://www.adventuregamestudio.co.uk/temp/cupratings.jpg)

The 2-cup rating is by far the most common, and 2/5 is not a "bad" rating as it would be on some review sites.

Quote from: Andail on Sat 09/01/2010 13:39:59
It's absurd to believe that a system like this can satisfy everyone. There are hundreds and hundreds of game authors in the database, every single one of them will never be pleased.

Yeah, it's obvious that no rating system can please everyone. As I said, I fully support the current one. With that in mind, I still think some brainstorming to address minor issues doesn't hurt. Some of the community's ideas might be considered sometime in the future, when a new rating system might be deviced.

Quote from: Ascovel on Sat 09/01/2010 15:36:58
More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

In that regard its even good to have only a small pack of really special games (4-5 cups), that are a great advertisement for AGS. People who don't have much time can just play those and ignore the rest, or play them once they have more time. The only annoying issue with the current system is perhaps the fuzzy differentiation between games with the lower grades. But still, an opinion is an opinion and even with more detailed grading many authors would certainly feel their games have been misjudged.

So, as I said, yay for the current system, and I'm sure increased transparency of the rating process via the new thread won't hurt! :)

Quote from: Pumaman on Sat 09/01/2010 17:04:41
One other thing to bear in mind is that the AGS Panel is much harder to please than many review sites -- for example, the panel have so far rated 827 games, and only four of them got a 5-cup rating. You can see the spread of ratings here:

The 2-cup rating is by far the most common, and 2/5 is not a "bad" rating as it would be on some review sites.

Thank you! That was the point I tried to make earlier. I think knowing the way this panel works and rates games (and seeing this graph!) might ease the depression of some authors when they see their hard work rated rather low. Getting a 2-cup rating is not bad, in fact it means you have developed an AGS game of average quality, and that's surely a compliment.

Maybe it would help if some people visualized the 1 cup rating to mean a value between 0 and 1 (inclusive), thereby creating a 6-cup spread where 2 appears more clearly as 'below average' rather than 'bad'?  Just a thought. 

As CJ said, the panel ratings tend to be harder to please -- but then again -- haven't all of us gotten that way over time?  If you want to know what I think the real reason is, it's the sheer quantity of content on the site that ramps up the challenge.  There are many many games in the database, and each year there's a marked improvement in both the engine and the quality of the output so people are getting more and more demanding and/or picky about what's out there, raising the bar for the next game in all fields.  There were far simpler games made in 2000-2003 that people absolutely loved that would completely tank compared to the games submitted today, both by panel and public judgment.

On many sites the reviewers are also active posters so I'm not sure that's a factor here, though.

Finally, I'm not sure the AGS Panel rating should be seen as an official community rating at all; the user rating does just that already.  I've always seen the AGS Panel rating as one of those editorial ratings that appear along with the community rating (like Gamespot) that people weigh against each other to arrive at a conclusion about the game.

Quote from: Ascovel on Sat 09/01/2010 15:36:58
More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

It would be best if both the panel members as well as the game creators would be able to find some sort of satisfaction in the work they do, and not take for granted the amount of sympathy and respect they'll receive for their efforts.

Well said on both points, Ascovel!

Great to hear from CJ on this. To me it seem like what he and Andail say (and some of the things ProgZ mentioned earlier) offers a very positive way forward.

I'm not sure if this is the right place to raise this, but if we're thinking of making the user ratings more prominent, would it make sense to have another look at their ratings scales? Maybe this is just me, but when I rate a game I don't really feel like the scales capture what I like/don't like about it, so that the ratings don't seem to closely reflect what I think of the game. Main points: "Immersion" groups together a bunch of different things that (to me) don't really go together--like story and music--under a confusing label, and the "Puzzles" scale seems to conflate level of difficulty with originality, balance and good puzzle design.

Maybe if the panel guidelines are made public, they could form the basis for new user rating scales as well? (I haven't seen them yet, so I don't know whether that would be a good idea.) Just as a forinstance, maybe part of why a 2-cup panel rating seems bad is that on the user-rating scale, it matches the more negative "Play it if you're bored, not much fun to be had here." That "problem" would go away if both ratings used similar scales.

Or perhaps use the AGS Award categories (a reduced set of them at least) as the basis for the ratings categories? We argue over them every year, so I think they've been honed a little more than at least the user ratings categories.

I'd also suggest adding a difficulty and length rating. Since these will vary from player to player, it's the sort of thing it's useful to get multiple people's input on.

Now if everyone else thinks the way things are now is working fine, this is clearly not worth doing. Just thought I would put it out there. The downside would obviously be what to do with the old ratings. I would expect that "Visual" and "Enjoyment" could be transferred over to the new set; for the others it would depend--maybe abandon them, maybe leave them as no-longer-active "legacy ratings" on the games page, maybe use them to initialize vaguely related scales in the new system, but with a low weighting relative to future input.

Wasn't it this game (http://www.adventuregamestudio.co.uk/games.php?action=detail&id=1119) that was the first to be removed from the database by its author due to being unsatisfied with the rating? Either way, it looks like the author realised it was worth having it in the database because it's back, whatever the rating, whatever the comments. I wouldn't be surprised if Harg returns his games to the DB in the near future.

Yes, it's easy to be offended by the smallest things personal to us. Just last night I wrote a long rant on Facebook about some folks that have really annoyed me lately, but at the last minute I was able to delete it and walk away from it. I realise it can be hard to stop yourself from doing the most drastic actions, ie removing games, posting that rant, etc when it really does hit you on a personal level - but it really is that stopping yourself part that shows you've got maturity enough to handle these kind of things.

And no matter how sympathetic others will be, I know that I will always look at removals of games because of a rating or comment the author feels is vindicative, to be childish. If I did it even if I felt just, I would still know that the action of removing a game because of some outside comment I don't agree with - would be childish. And plenty of adults do things just as childish than children! Doing it isn't going to make everyone jump up and support you. It's just going to create threads where there are 5 pages of the same three views being spoken by 50 different people, and makes for an extremely dull read.

Some of the games I've worked on have average cup ratings and yes I feel some of the reviwer comments feel a little bit harsh...but as it's stated if you're putting your game up on a public DB then what do you expect? And as said before it's a privallege to be on the AGS db, just as much it is that CJ even bothered to make an engine for us to do stuff with :) You can't be a creative person and put your creations out there without feeling the brunt of unsatisfied customers occasionally, perhaps even frequently. And if those comments really get you down, perhaps game making, or creative persuits on the whole - just aren't really your destiny? ;) You need to be able to take the bad with the good in this game, and because of the way it appeals to lots of other people, you need to be able to take the even worse with the bad and not go flying off the rails along the way.

After reading the entire thread, I have to say Harg's actions seem like a knee-jerk reaction and I'm sure he/she will come round to it and realise just being on the db is a good thing. Same as Calin realised eventually the system is actually doing him justice, and the same way Yrolg returned to the database. It's like a DO LOOP we can't escape but someone forgot to remove the IF WHEN for knee-jerk reaction capabilities. Please note that last sentance was inspired by Dualnames ;)

Alright, there is now an official information thread over at completed games.
http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39821.0

This information as been approved by CJ and the Rating Panel.

Individual members of the rating panel will not partake in more debates regarding specific ratings. If you have comments or questions regarding the rating of a game, PM me or Progz.

Thanks for finally posting the guidelines, Andail.  Hopefully it will resolve some questions people have had about the panel rationale.

I haven't really been following this but I was wondering if there'd be any way to notify people when their games get rated? From what I can tell the judges rate the games whenever they get around to it rather than by any release schedule or whatever, which is understandable, but it also means you don't know if they've gotten around to your game unless you accidentally stumble across it. A lot of times I don't realise the panel has been there until someone else leaves a comment on the game later on. I guess what I'm saying is that it'd be nice to have a 'recently judged' box in the game database functioning about the same way as the 'latest user comments' section. It'd be easier to check than the gamepages themselves and could also act as a way of getting more attention to games which get a high panel rating etc.
Just a thought!

I agree that entering into these tit-for-tat arguments is a waste of time, and I'm pissed that I allowed myself to become ensnared. Just as I became annoyed with a perceived lack of support and appreciation for what the panel does, I also understand that a developer can become just as annoyed and exasperated when faced with what appears to be a cold-hearted rating.

I still believe Harg could've handled the situation better, and with less effort than it took to handle it the way he did, and I still stand by the panel's right to offer their honest opinion. My stance on these facts isn't likely to change. If that troubles you, I can't really offer any advice.

I seem to have cultivated this image of an "Obey me!" overlord in the minds of some people, which is disheartening, as it's far from the truth; my time spent on the panel since it's inception would hopefully attest to this. I am passionate about the time I dedicate to this community, and it's easy to become confrontational when that dedication is questioned in an accusatory manner.

With the guidelines now public, we should now have some factual basis to future discussions.

Quote from: Ascovel
To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

Quite right. It's just hard to remain optimistic in the face of blustering discordance. My time on the panel is limited, and I try to rate as wide a range of games as I can; just as every panel member does. If I was to dedicate the hours a relatively full-length game requires to appreciate it fully, only to have the reason that time was spent rendered void, I'd have a right to be a bit pissed. Fun can also be subjective, as I'm not going to enjoy every game I play. No reviewer enjoys everything they review. This aspect of their time spent reviewing can be thought of as "work". I wouldn't like to spend time working, only to have that work evaporate. If you haven't gleaned this fact from what I've being saying throughout this thread, then I can't really explain it further.

ProgZ, Andail, and I, as the most public panel members, have always been open to discussion about ratings. Maybe not so open regarding demands and ultimatums, but I'll always happily entertain a civil query regarding just what it is that I contribute to the panel. I may sometimes make my feelings a bit too clear on whatever the matter at hand happens to be at the time, but since we're all adults here, we shouldn't require our opinions to be delivered wrapped in cotton wool.

I'd like to thank everybody who voiced their opinion in this latest debate, and I now consider my role in it to be at an end. :)

Mods: Ah..thanks?

Andail: Congrats I personally believe your action will make things less dramatic and easier for the panel as well!

I will try to explain why I removed the games from the database.

Some of the developers here are trying to promote their games on other websites. Some are trying to get a review for their games from other good websites for adventure games. And with this AGS panel review there is no way someone to take you seriously.

My goal wasn't to insult the AGS community. As you can see on Cosmos Quest website, I still support AGS as great tool for creating a games. I just don't agree with AGS Panel. I tried to avoid definitions like "leaden pace", "bland characters", "lack of direction", "ponderous gameplay" for my game that came from one person. Actually, the definitions were the things that irritated me most, not so the cups.

Why I didn't send a PM to AGS panel before the deletion? I did't know who is in the AGS Panel. Now, due to the deletion, everybody knows who to contact. And I'm glad that there is a movement in that subject (the official information thread about reviewing the games).

Quote from: LimpingFish on Sat 09/01/2010 21:44:04
It's just hard to remain optimistic in the face of blustering discordance. My time on the panel is limited, and I try to rate as wide a range of games as I can; just as every panel member does. If I was to dedicate the hours a relatively full-length game requires to appreciate it fully, only to have the reason that time was spent rendered void, I'd have a right to be a bit pissed. Fun can also be subjective, as I'm not going to enjoy every game I play. No reviewer enjoys everything they review. This aspect of their time spent reviewing can be thought of as "work". I wouldn't like to spend time working, only to have that work evaporate. If you haven't gleaned this fact from what I've being saying throughout this thread, then I can't really explain it further.

I can very well understand your disappointment. I'm sure there's a lot of work involved in preparing the panel reviews and I respect and appreciate what you do, as well as the serious approach to the reviews. But the database is primarily a place for authors to showcase their games. I think they have full moral right to decide they're better off promoting their works in a different way. It's not like they delete someone's full-length written review by removing a game.

QuoteNow, due to the deletion, everybody knows who to contact. And I'm glad that there is a movement in that subject (the official information thread about reviewing the games).

This just isn't true, Harg.  If you'd have read the ongoing rating panel discussion you would have known that I was on the panel and willing to take PM's (as well as LimpingFish), something that was only restated here to remind some of you that we aren't a shadow operation :).  Also, we'd been planning to release the guidelines for awhile now but holidays got in the way of a few panel members being able to offer their input on the subject.  I think you did manage to give the guidelines a kick to get posted in a timelier fashion but I wouldn't say your actions were constructive or beneficial, either to yourself or to the community.  I hope at some point in the future you will look back on this and see your response as a dramatic one and re-upload your games for other people to play and form their own opinions.

Again, if you'd like a more detailed workup of why your game received its rating just PM me your request and I'll ask the judge responsible to provide one for you.

I'm not a panel member, but something that disappoints me is the fact that, while we do sometimes see people thanking the review panel, the only time that people remember them in any big way is when somebody complains (I'm guilty of this too). The rest of the time we seem to expect them to keep rating games without any recognition for their efforts, which is a pretty crappy attitude considering the friendly place this is.

If was was doing something and the only time anyone commented on what I was doing was to say "Hey! That's not how you do it! Don't do it like that anymore!" I'd say "Screw this".

And I'm not saying this so my games all get 27 cup ratings, it really is something to consider. We waited for ages to get this panel thing going on, it'd be a shame to lose it now that we've got it!

^^^^ Agreed.

I demand one more cup on my game Where did Sam go? !!
"Joke" aside, I agree with Ben304 here..

The fact is, the panel cant just give any games good ratings to support the developers!
I mean, then the whole point of the system would be ruined.
Also, if we get the new website up now with the similar looking user rating I dont think 2 cups will look that bad if the user rating is 3 or more cups!

Also I can't really say if the panel was fair or not about the rating of SQIII as I haven't played it..

Quote from: Peder Johnsen on Sun 10/01/2010 21:43:02
The fact is, the panel cant just give any games good ratings to support the developers!

Peder, you are missing the point. No one wants better rating. I just don't want the games to be reviewed by a single person. If there is no way one game to be reviewed by many people, then only technical areas should be commented. Otherwise, the review will be very subjective.

Harg, he's missing your point because your point is very strange.
A review is always written by only one person. At least at all the major magazines or websites I ever visit. They may have a staff of reviewers, but ultimately, the published text will have been penned by only one person.
I hope you understand that a review or preview or commentary or what have you is always subjective. It's in the nature of reviews. However, by having a panel of very experienced and trusted reviewers, we at AGS have ensured that the reviews are as balanced and neutral and qualified as you'll ever get them.

It's fine that you don't want to expose your game to other opinions, but try to think through what it is you're asking here.

Thanks for the Rating Panel Information thread. It adds a lot of transperency to the work of the panel.

Here's an idea. Perhaps when a new game is added to the DB there can be two submit buttons: One for "I agree for my game to be reviewed by a single member of the review panel and their rating to be available for people to see and sort by and  I accept that I may not like their opinion because all opinions are subjective" and another with "I do not want my game to be reviewed by the review panel and accept that I will therefore not get a cup rating". Although people should presumably be allowed to change their minds later within reason.

Id say thats a good idea SSH,

Yeah, I agree. It would also save the Panel reviewers a lot of time and having to deal with bitter game authors.

If that were the case, when I go to sort games by cup rating and that one persons game isn't shown because it doesn't have a cup rating.  I guess it would be their own fault that I'll never play their game, due to not seeing it in the list.  Unless the game strikes my interest from the completed games thread.

There are two people I can see selecting the 'do not rate my game' option. 
One, someone who released a joke game, or deep down knows their game isn't worth anything at all (which I can say I'm ok with number one). 
Two, someone who just can't handle the truth and wants to hide the true rating their game deserves.  They also deep down know their game isn't worth a higher rating, but when they see the lower rating attached to their game, they can't handle it.

If I made a game and it received a rating I wasn't too pleased with, or even a review commenting on bad/broken areas which I had believed were perfectly done.  I wouldn't freak out.  Of course, I would probably drop my jaw in disbelief at first.  Then I'd play my game over again from beginning to end and see if there was any truth to the review, have I been blind to my own creation? 
If I ended up noticing the flawed areas of my game that the lesser review and rating was indicating, I'd probably engrave that information into my brain for my next game, so that I wouldn't do it again.
But, If I still couldn't see why it was rated so much lower than I anticipated, I probably would have pm'd progz because I've always known he's been on the panel.  And ask him to get the person who reviewed and rated it to explain it more in depth, or even possibly re review and rate the game.


Anyway, out of all honestly.  If we as a community of game players, have the option to vote yes or no to this idea of skipping out on the panel ratings and reviews... I say No.  I do not want some games to be able to avoid what every game must go through.  Otherwise everything the panel has done here in the past few years, would be useless.

Keep doing what you're doing guys (and it looks so good now with my new blue cups on the new template page.. mmm lol).

Quote from: Ryan Timothy on Mon 11/01/2010 18:53:52
There are two people I can see selecting the 'do not rate my game' option.  

• The third type would be people who, for whatever reason, do not have confidence in the panel/process, or who expect to receive an unfair rating.
• The fourth type would be people who are opposed to ratings in principle, and think that people should just try it for themselves without preconceptions.
• The fifth type would be people who think boiling an experience down to a single score is useless, and the only way to assess a game is through a detailed review.

I don't agree with any of these opinions, but I've heard all of them expressed by various people, and making the panel rating voluntary would be a simple way to address their concerns. I bet you'd hear a lot fewer complaints as well.

I wasn't sure about the idea at first (after all, the ratings are mainly for the benefit of prospective players, not for creators), but the more I think about it the more I think it makes sense. It probably wouldn't affect too many entries, and like you said, it'd be their loss. And after all, it's better to have the games in the database without a rating than to have people not post their game there in the first place, or delete it from the db in protest, right?

Instead of a rating, you could display a message stating "At the request of the creator, this game has not been rated".

QuoteAnyway, out of all honestly.  If we as a community of game players, have the option to vote yes or no to this idea of skipping out on the panel ratings and reviews... I say No.  I do not want some games to be able to avoid what every game must go through.  Otherwise everything the panel has done here in the past few years, would be useless.

Why would it be useless? You think that if we don't rate 100% of the games, 0% or 95% makes no difference?

After Snarkys post an option to choose wether a game will be rated makes more sense to me. But what about the user ratings? Should there be an option for no rating at all or no panel rating?

Oh, and I can think of two more reasons I think are valid:

• The sixth type would be commercial game makers who agree that the ratings are fair, but know that they're graded on a tough curve, and worry that potential buyers won't realize that, say, a 3 cup-rating is quite good.
• The seventh type would be people who want a review, but not right now. For example, they might want to wait until after the AGS Awards, or if they're making an episodic game they might want to hold off until a couple of episodes are out. (Sure, they could probably just PM the panel contact and ask them to wait a while, but that might not occur to creators who aren't active on the forums.)

Because I believe it won't be a small percentage that choose the 'don't rate my game' option.  I can foresee it being 25% or more.

I've pressed that Lucky Dip feature in the new website layout more than 100 times now, and to tell you the truth, I haven't seen a game with a lesser player rating than blue cup rating.  Which is why I completely trust the panel, because the average gamer is just like "meh, 10 out of 10, I loved it" -- you see that comment all the time.  Every AGS game that I've played, I can agree more with the panel rating than I do with the player rating.  Which is why the panel was there in the first place.

I would almost rather the games not be in the database, then to see no ratings at all.  The keyword there being: almost.

Well, you probably don't want to make it so that asking to be excluded from the panel rating is just as easy as being included, or you might get too many people who just don't feel like it for no particular reason. If it's just a little bit of effort, you'll filter out these low-commitment rejectionists while still giving people a real choice.

Maybe those asking not to be rated should have to fill in a text box explaining their decision (to "help the panel understand" why people choose not to participate). Or maybe there shouldn't be an option on the form, just a message saying that if they don't want a rating, email or PM this address.

This is a bit silly. If you choose not to display a rating, it's because you consider it a bad rating. I predict that 99% of those who opt out will first have accepted to receive a rating, then changed their mind when it turned out lower than expected. Fine way of wasting people's time.
Moreover, the database will have a flawed balance towards higher rated games, with a huge bunch of unrated games. I don't see how this helps anyone, even the developers.

You're suggesting that humans are more complicated than they are, Snarky. The world of a developer is very small - good ratings are good, bad ratings are bad. Whatever philosophical theories or sentiments you have about reviewing stand pale to this simple fact - we all want good feedback.

Also, some of your reasons are even better used as arguments for compulsary rating. So a person who wants to sell his game doesn't want bad publicity? Well, that's life. Are we going to support that developers who can't make games worth buying still manages to trick people into buying them, by erasing mediocre reviews?

Quote from: Snarky on Mon 11/01/2010 21:19:43
Maybe those asking not to be rated should have to fill in a text box explaining their decision (to "help the panel understand" why people choose not to participate).

Maybe they should be presented with a form containg an emotional stability test. It wouldn't be widely known, but you would have to actually fail the test for your request to be listened to. ;)

Quote from: Andail on Mon 11/01/2010 21:56:06
This is a bit silly. If you choose not to display a rating, it's because you consider it a bad rating. I predict that 99% of those who opt out will first have accepted to receive a rating, then changed their mind when it turned out lower than expected. Fine way of wasting people's time.

It shouldn't be possible to opt out once the rating is posted.

Like Ascovel says, I was assuming that you couldn't ask to hide the rating after it had been assigned (though you should probably be able to request a rating even if you initially opted out), because otherwise the selection bias would obviously be enormous.

Andail, maybe you should have another look at what I wrote in my posts, since you seem to have missed a couple of details. (For example, I didn't talk about developers wanting to avoid bad ratings, but being afraid that the ratings would be misunderstood.)

How about adding a disclaimer that by posting your game in the database, then you accept that there will be a rating of your game by a panel for organisational purposes only and if you don't agree then why don't you just make a post in the completed games forum instead.

Or how about giving the option to the game makers to hide the panel comment once it has been made? Although the comment usually justifies the rating, disgruntled game makers might prefer not to remind the potential players of the games' pros and cons. Which seems to be the thing a lot of people are biting on to with these rating rants.

However purposely unrated games in the database makes the previous ratings pointless and makes the search tool utterly useless.

I don't want to correct you Snarky, but commercial games aren't rated and either are demos, so I doubt it will put potential buyers off. Unless you are commenting on the possible previous free and rated games of that same developer, which surely shouldn't be given special treatment rating wise because they are trying to sell something...

Also same thing for the awards, I understand that the game needs to be in the database for it to be eligible to vote for it, but hiding/delaying the rating/comment to win an award seems a little wrong to me.

As of the episodic games point, I understand, but I think the first game should be rated then if the author wants to challenge that rating they can with using the other instalments as evidence. Seems like the fairest way to me.

Quote from: IndieBoy on Mon 11/01/2010 22:37:43
How about adding a disclaimer that by posting your game in the database, then you accept that there will be a rating of your game by a panel for organisational purposes only and if you don't agree then why don't you just make a post in the completed games forum instead.

Sure. I guess the question then becomes if we'd rather have games that the creators don't want rated by the panel in the database, or not. I always thought the database was meant to be an as-complete-as-possible repository for AGS games, so players would have a single go-to location; it doesn't exist for the sake of the panel ratings. They're there "for organisational purposes only", as you say.

Also, submission to the database is currently a requirement to be eligible for the AGS Awards.

QuoteHowever purposely unrated games in the database makes the previous ratings pointless and makes the search tool utterly useless.

Again, I don't follow this argument. Missing ratings for a few games invalidate the whole system? If that was the case, shouldn't submitted games be kept off the games db until the panel rates them?

QuoteI don't want to correct you Snarky, but commercial games aren't rated and either are demos, so I doubt it will put potential buyers off. Unless you are commenting on the possible previous free and rated games of that same developer, which surely shouldn't be given special treatment rating wise because they are trying to sell something...

It's a good point about them already not being rated. In any case, I wasn't suggesting giving commercial games special treatment, and certainly not to manipulate the ratings to bolster sales. I was just arguing that there are many reasons beyond "I don't want people to know how bad my game is" for why game makers might want to opt out of having their game rated by the panel. Specifics would vary from case to case, depending on the situation, and I gave a few hypothetical examples.

QuoteAlso same thing for the awards, I understand that the game needs to be in the database for it to be eligible to vote for it, but hiding/delaying the rating/comment to win an award seems a little wrong to me.

As of the episodic games point, I understand, but I think the first game should be rated then if the author wants to challenge that rating they can with using the other instalments as evidence. Seems like the fairest way to me.

I don't see anything underhanded in wanting AGS voters to come into your game with as few preconceptions as possible. It's probably not the decision I'd make if I had a game that was eligible for awards, but different people take different approaches.

And while challenging or appealing the rating at a later date is one way to deal with cases like the last example, wouldn't it be better to have a way of achieving the same result in a less confrontational way?

Ok Snarky I'll explain my view/points a little more clearer:

From my understanding: The database is a collection of ags games that are user submitted, which are indexed in genre, length, type and cups. My point was, for example having half of the database entries without any of these fields completed would make them unsearchable, therefore it wouldn't be a complete database. I know it wouldn't be as drastic as this in reality, but there would be a difference if a potential 5 cup game was chosen not to be rated by the author, therefore it would be least likely to be found and therefore would be a loss to the community/potential players. To prevent this I'm suggesting to add the disclaimer. It would mean no drastic change to how the database is built and structured, and also I think it would be fair for the game designers who don't want their work to be "judged" and to the ags panel/community, that is all. It's a thing that the majority of people who submit their games already expect this so why shouldn't it be as a rule? As such as the rule for the game must be submitted in the database for it to be able to voted for the awards. 

The point of the ratings and the adding of the new genres and things a couple of years back was to make it easier to search for games. So my point was: by having entries in the database purposely not being rated would be taking a step back in the database's evolution. Also there is a difference of between a waiting-to-be-rated game and a game that has been purposely unrated. I just don't think that the community who vote and nominate games for awards are going to be swayed with the game's rating in the database. As the recent few threads have shown, the community can/will either ignore the ratings, challenge them or accept them. In addition if any such disagreement takes place about a game I'm sure it would be resolved before the awards, hence no damage to the chances of that game's award success.   

As of the preconceptions of the new player I agree, that is why I suggested the option to hide the panel's comment.


I just don't understand why however more pages/threads about this kind of thing we never seem to have came up with the ultimate solution and just gone with it. It seems like at least one person will never agree with an idea. Also I promised myself I wouldn't post in one of these threads again.. whoops!

Well, you'll never get 100% consensus on anything. But by the same token, no system is so perfect that it can't be improved.  :-\

I understand your argument that not having a rating for some games makes them a bit harder to find, and that this reduces the value of the db. But it's better than not having them in the db at all, which is the alternative offered by your proposal, isn't it?

Technically, classifying something as "unrated" is also a kind of categorization, so I don't think you have to see it as a "step back" for the database.

As far as I can tell, people have suggested 3 options so far:

1. Panel rating will be non-optional for games submitted to the database. Submitters are given a disclaimer to this effect. People who don't want their games rated will not add them to the db (or they'll do so without realizing/noticing that they're signing up for a review, and be unhappy).
2. Submitted games are rated by default, unless the submitter specifically opts out through some documented process. A new classification of "Not rated by request of the creator" is added (and could even be made searchable as a "0 cups" rating if desired).
3. Panel rating is generally automatic, but game makers can prevent it by taking extraordinary action (PMing Andail, for example). This option will not be mentioned on the submission page.

I'm assuming that 3. is the system currently in place, at least for as long as the game submission page doesn't mention anything about the panel.

All the options seem more or less viable, though no. 2 is my clear favorite, personally. However, I think if CJ (or whoever) decides to go with option 1., the rule that AGS Award nominees have to be in the Games DB should be changed. It doesn't seem right to me that submitting to be rated by the panel should be compulsory if one wants to be eligible for the awards.

As for your idea of letting game makers hide the panel comments (but not the rating), I think it runs into the same objection Andail raised earlier: it's just a way for game makers to suppress bad reviews, and thereby compromises the integrity of the panel.

I still don't really understand the theory behind the five cup system. Did anyone ever think what two cups out of five would look attractive to the average browser? I also get confused as people have been saying contradictory things.

CJ has said "The 2-cup rating is by far the most common, and 2/5 is not a "bad" rating as it would be on some review sites."

Progzmax said " where 2 appears more clearly as 'below average' "
and on other occasions, "while 2 is a 'nice try' category that is meant to encourage people to do better.  As mentioned before, 3 is essentially the standard score which is fine since most people will produce a few average but interesting games before they reach something better."

On the 'what is this' page "2 Cups   A reasonable game, worth a try"

Andail "I think everyone needs to remember, before you jump on the complaint bandwagon, that the descripition for 2 cups is that the game is reasonable and worth a try."

So guys... what is it? Average usually means the medium. The middle of the road... Even progzmax, the supposed head of the ags panel, seems to be confused about the meaning of the 2 cup score.

A game that offers multiple endings, has improved graphics over the authors previous 4 starred titles, and an interesting plot gets a two cup rating. Did the reviewer only play it once? This game if you care: http://www.adventuregamestudio.co.uk/games.php?action=detail&id=1194 (http://www.adventuregamestudio.co.uk/games.php?action=detail&id=1194)
This is one of those situations where the panel score is at odds with the user rating.

There seem to be a lot of inconsistencies with the scoring system that make it an unreliable way of judging whether one would enjoy a game.

To make things even worse, you have decided to publish the means in which you rate games. People will still complain, saying 'ohh but I did all of that'.

BTW that bar chart makes the community look terrible. Everyones making shit games, and only a few make games worth playing. That is what it looks like to an outsider seeing all the two cup games. NO ONE sees two cups as being in any way GOOD. It is a BAD score, to players, to creator, to anyone except the review panel that isn't Progz. Who also seems to see it as a bad score.

Quote from: Ryan Timothy on Mon 11/01/2010 20:52:38
I've pressed that Lucky Dip feature in the new website layout more than 100 times now, and to tell you the truth, I haven't seen a game with a lesser player rating than blue cup rating.  

It is true that the panel is more unforgiving than the players rating, but what you have experienced is probably also caused by a different scale. To me:

It was ok, play it if you got some some spare time (the middle option in the "rate this game" overall enjoyment)
and
A reasonable game, worth a try (2 cups panel rating)

are pretty similar, but they will lead to a 3 orange cups (player) vs. 2 blue cups (panel), and those two judgments agree.

This could be conveniently addressed changing the options in "rate this game - overall enjoyment", bearing in mind that effects will take place only in the long round (as more votes are collected means will slowly change).

Maybe we should just make the scale from 2-6 cups instead of 1-5 and then everyone's egos will be artificially inflated? This suggestion sounds facetious, but if it reduces complaints, why not? 1 cup can be reserved for games that are so bad they don't even run, etc.

I think we should remember that the 5 cup rating has to span a very broad spectrum.

AGS has created games which are barely playable and masterpieces of fiction.

In my eyes 3 cups can only be considered 'good for an amateur game' and things dont really start to be 'good' until the 4 and 5 cup mark.

I have only released one game and only been here around 5-6 months so my experience of AGS games is limited but if the scale is linear and the benchmark is trilbys notes (which got 5 cups) and shifters box (which got 4) then i would rate the vast majority of games I play as 2 cups or even lower. And that includes my own.

The resolution of the cup rating simply isnt high enough to be as useful as you guys seem to want it to be.

You should use the cup rating more like a basic technical benchmark to answer a few simple questions.

- Is this game fun
- Is this game buggy
- Does this game make basic logical sense.

beyond that I fail to see what you want from the rating system.

Completely support Layabout opinion. And Back Door Man is a great game - nice polished graphics, multiply endings - its a different game! Who rated this game? We want to know names? :)

Quote from: Calin Leafshade on Tue 12/01/2010 07:46:59
I think we should remember that the 5 cup rating has to span a very broad spectrum.

AGS has created games which are barely playable and masterpieces of fiction.

In my eyes 3 cups can only be considered 'good for an amateur game' and things dont really start to be 'good' until the 4 and 5 cup mark.

I have only released one game and only been here around 5-6 months so my experience of AGS games is limited but if the scale is linear and the benchmark is trilbys notes (which got 5 cups) and shifters box (which got 4) then i would rate the vast majority of games I play as 2 cups or even lower. And that includes my own.

The resolution of the cup rating simply isnt high enough to be as useful as you guys seem to want it to be.

You should use the cup rating more like a basic technical benchmark to answer a few simple questions.

- Is this game fun
- Is this game buggy
- Does this game make basic logical sense.

beyond that I fail to see what you want from the rating system.

I am responsible for a 1 cup game. It was playable, but very short, very easy and by god was it ugly. I did it as a test and a challenge to create the first 'game' with the new windows roomedit. I don't even think it should deserve even a 1 cup rating. It was horrible. Truly awful. Yet we get a 2 cup rated game with higher production values, in engrossing story with multiple endings (here I refer to Back Door Man). That doesn't make a lot of sense does it.

Neither does giving a game like Ben There, Dan That!, a highly enjoyable game with a very high user rating 3 cups with comments along the lines of 'lots of rooms, not much to do', and giving a game like shifters box, which has the same thing (lots of rooms, usually only 1 puzzle per room) 4 cups. Or Limey Lizard, which has about 20 useless rooms in a maze puzzle, which may lead to a walking dead if you aren't careful... no, scrap that, it's easy to come across just from exploring. (also 4 cups) Another game by the same author isn't even complete and gets 3 cups.

There seems to be a lack of consistency with the panel rating. This is what is troubling me.

You say 3 cups can only be considered 'good for an amateur game'... well what else would it be? Despite there being a handful of commercial AGS games, 99% of content created with AGS are amateur games. A clearer and more consistent panel is needed.

*edit*

And oh yay, an example of a game that doesn't really have any game play. 3 cups http://www.adventuregamestudio.co.uk/games.php?action=detail&id=1266 (http://www.adventuregamestudio.co.uk/games.php?action=detail&id=1266) Yes the game has consistent graphics, animation, etc (all of which are lovely), but for anyone who has never experienced Scid and the original Red Flagg, they would be disappointed and wonder why this game was rated 3 cups, when other games, which have a far greater degree of story and game play get 2 cups and a similar consistency of graphics.

Yes, Harg and Layabout, let's fill this thread with specific titles whose rating we don't agree with.

Then let's invent a system so perfect that among the thousand-something games in the database, there wouldn't be even one or two ratings that any single one of the thousands of members wouldn't disagree on.

This suggestion is probably a bit too much information but might not be hard to implement. For any given reviewer on the panel who has reviewed a statistically significant number of games (e.g. 30 or more based on six sigma principles) then the reviewer is kept secret, but the cup rating says : 3 cups... is better or equal to X% of the other games reviewed by this reviewer. Then at least you'd be able to see if it was reviewed by a hardass or a softie.

As usual I'm too lazy busy to read all the pages in the thread. I'm not sure whether my idea is good or valid, but it may be useful for brainstorming to finding a better system.

I'll just ignore the matter of inconsistencies about the rating standard, and whether the games get appropriate ratings at the moment, as I never devote the time into this I'm obviously not eligible to comment on these in details.

What I see is, as 2-cup games are the most common, they're sort of having an average quality among all the games. If I am say, a teacher, who rates students' works I'll say I feel (so don't take me seriously) this is sort of equivalent to a 'C grade'.

So, under this criterion, 3-cup games are most likely 'good to quite good', which are considered 'B grade'. Then, obviously, 4-cup games are 'quite good to very good', which are considered to get an 'A'.

What do the 1-cup and 5-cup ratings really mean? This is my view:

5-cup games are those that are 'excellent' and are highly recommended (as suggested by the sheer number of 5-cup games at the moment), so they're something like getting an 'A+ grade'.

If you agree to the above, it's easy to say that 1-cup games are actually 'D grade or below'. Now, the problem lies here. 'D grade or below' can mean many things. It can be a game that some effort was actually put into it but the overall result isn't as enjoyable because of various reasons (say, uninteresting stories, bugs, language problems, disturbing graphics, etc.) which could be considered to belong in the 'D or a better E' department. These games are actually quite different from effortless attempts that are merely stuff messily put together (such deserve 'the lower part of E, or F') and in fact, some people are still willing to try out those 'D or better E grade' games but under the current system all of these are 1-cup games, which makes it difficult for them to filter out the games they want to try.

Because of this, my idea is like this:
F to lower E - 1 cup (worthless to try)
better E to D - 2 cups (try if you really want)
C - 3 cups (same as current 2-cup games)
B - 4 cups (same as current 3-cup games)
A - 5 cups (same as current 4-cup games)
A+ - still 5 cups, but with the cup graphics changed to gold or shiny or something like that (they can be just categorised as 'highly recommended' in the search filter)

which is actually a 6-cup system disguised as a 5-cup one (unlike the joking 2-6 cup system someone mentioned). In this way we don't have to change much and many people may be happy as apparently most games seem to be rated 1 cup higher.

Note that this applies only to the panel scale. I don't see the public system need to change. Since the public ratings are determined by votes from various people I don't see it need a special treatment to include an 'A+ grade'.

Quote from: Andail on Tue 12/01/2010 08:46:18
Yes, Harg and Layabout, let's fill this thread with specific titles whose rating we don't agree with.

Then let's invent a system so perfect that among the thousand-something games in the database, there wouldn't be even one or two ratings that any single one of the thousands of members wouldn't disagree on.

I'm actually pointing out how inconsistent the rating system seems to be due to the subjective nature of such a system. I personally loved shifters box. I think it deserves a 4 cup rating, which is consistent with the guidelines you posted. I loved Red Flagg, but it has a major flaw in the gameplay department and would fit more in the 2 cup rating as per the guidelines. Yes SpacePirateCaine by far exceeded the expectation of those who had played the original Red Flagg, but it felt more like an interactive cut-scene than a game.

Sure I do single out Back Door Man as a game that has been given an incorrect rating, but I challenge people to play it going through the Panel's guidelines and see how many crosses you can put against it.

The Panel system was put into place to moderate the games against being artificially jiggered with, and I totally agree with the sentiment behind the whole system. To me it seems there is too much subjectivity to the ratings. I'd like to think if one day I make a game of substance that I might get an objective panel rating... although with all the fuss I'm causing right now I'm doomed to be subjected to a 1 cup rating for everything I ever create.

I also agree with what Gilbert Cheung says. Always. It would bring it more in line with the proposed yellow cup system, whilst also being able to clearly see inconsistencies, which could then be flagged as aforementioned.

User ratings should also be bought in line with the panel rating, as this would provide a more accurate picture of what the general public thought of the game. Same guidelines, same overall 'cup' system. This way if the user rating was seen to differentiate too much from the panel rating, the panel could be flagged to check if the user rating was subjected to jiggery, or if the original panel rating was fair and objective.

I think Gilbert makes some good points.  His suggestion addresses the concern that the uninitiated will interpret 2 cup games as below average crap. 

Following up on his theme allow me to suggest that perhaps the rating process should begin with the presumption that the game under review  is average (i.e. a rating of 3).  Then the reviewer would then score it higher or lower according to the process  laid out by Andail and the other panel members.   The object of doing this is to calibrate the rating process so that 3 cups == average as Gilbert suggests.

I would also like to suggest that one possibility for depicting Giiberts "A+" rating would to be have  the cups with steam coming off them. 

---

As I recall the original problem with the user rating is that there were many games with just a few  votes and undeserved high ratings.   Perhaps it would be better just have user ratings that also included a weighted panel rating.   

For example the panel rating could have a 30 vote weight.  So if the someone make a crappy  game and then he and a couple of his fiends gave the game an undeserved  5 cup rating and the panel gave a 1 cup rating the end result would be ((30*1)+(3*5))/(30+3) = 45/33 = 1.36, which would still be a 1 cup rating.  There is noting magic about the panel weight of 30;  we or CJ could decide what value is appropriate.

---

I would also like to repeat one of my earlier suggestions I made in the other thread as it seems that thread was already dead and some may not have seen it .

In this thread I suggested that the review should be written for the benefit of the game author rather than the game players.   How so?  I was a member of Toastmasters, for a number of years, where members learn speaking and leadership skills.  The key ingredient to the success of their program is the way members' practice speeches are critiqued, which I will share with you.

Someone is assigned to give an evaluation speech immediately after the main speech.   The evaluation is usually structured into three parts consisting of no more than three points each.   The evaluation speech begins by pointing out the best parts of the speech.   Next the evaluator points out what could  be improved.  There is no point in mentioning more than three things because

1) The speaker may become disillusioned/discouraged
2) It's difficult for people to remember more than three things at a time

The evaluation ends on a positive note by mentioning what things the evaluator would like to see more of in the future.   This essentially presents negatives in a positive light (i.e. "half full" rather than "half empty").  This is also an opportunity to encourage the speaker  to continue making speeches and improving.

I think all of this or something similar could be applied to the panel's reviews.   Game authors wouldn't be in the dark about why they got the rating they got.  They would have clear guidance and specific suggestions to improve on their next effort.    Game players would get the same or more benefit from this sort of review as before.

I also suggested that authors be given advance notice of the rating and review so that they may give the reviewer some feedback before publication.  This is a professional courtesy rather than an invitation to whine.  Given the panel's work load as described in these discussion it wouldn't be surprising if a reviewer occasionaly missed something an author felt strongly about.   

Quote from: RickJ on Tue 12/01/2010 11:45:30
For example the panel rating could have a 30 vote weight.  So if the someone make a crappy  game and then he and a couple of his fiends gave the game an undeserved  5 cup rating and the panel gave a 1 cup rating the end result would be ((30*1)+(3*5))/(30+3) = 45/33 = 1.36, which would still be a 1 cup rating.  There is noting magic about the panel weight of 30;  we or CJ could decide what value is appropriate.

That implies that most of user that actually rate games are unqualified to do so, whereas the panel is full of heavenly wisdom and fairness. I think this method is quite undemocratic (in the sense that every vote should count the same).

As a member of this great community and maker of crappy games I want to say that a 6 cup ranking would fit better.

All the considered demands and skills would be given a percentage by the board, like:

Art : 30%
Story : 50%
Tech skill: 20%
...and so on...

If the board shows those numbers or not it's a different issue,

the result of the combination of this percentages would be automatically turned into cups:
0 - 20 % - 1cup
20% - 40% - 2 cups
40% 55% - 3 cups
55% - 70% - 4 cups
70 - 90% - 5 cups
90 - 100% - 6 cups

I'll take the panel's side on this topic for once. Layabout, miguel and others, it sounds like you're trying to solve the "problem" that the reviews are a subjective opinion. Someone said on SSH's blog:

Quote from: RadoI support the AGS panel because their work is free, and the panel can not hire more people to make the review more objective.

I support the developer because I think that is not fair AGS panel (which in the case is a single person) to comment the creative work. In this case I think it's better AGS panel to review only technical areas of the game.

But what players are interested in is "should I play this game?" And that mainly comes down to intangible qualities like "fun," which are inherently subjective. So miguel, I don't think your system would produce ratings that are any more helpful than the current guidelines. You can't just feed in numbers to a formula and get out a rating, particularly since games are so different. To play along with your idea for a second, in some games the story might indeed be 50% of the enjoyment, while in others the puzzles are 50% and the story only 10%.

The original idea Andail had for how the ratings would be set also strove for perfect objectivity, by using a checklist. There was a specific list of requirements for getting a particular rating, and if you failed any of them, you didn't qualify. When he proposed it, it got a lot of criticism because it would assign counter-intuitive ratings to a lot of games (some of the most popular games would only have got one or two cups), and so the panel modified it to get the system you see today, where the guidelines are much more flexible. I think that was absolutely the right decision.

And as long as panel members have to use their own judgment, and have to quantify inherently subjective experiences like "fun", there will always be room for disagreement. And with different members of the panel having different tastes and using the scales slightly differently, there will be some inconsistency between ratings. (And I should point out that IMO the average user ratings of the games are much more inconsistent than the panel ratings.)

No, the system isn't perfect, but do you really think that there is a perfect way to turn the experience that each and every individual player will have with a particular game into a number from one to five? (Or six?)

That's why I think the most important thing is not the number of cups by itself, but to always use the panel comments to explain what factors led to the rating. Rick, your idea for how these comments might be structured sounds like one good approach; it's similar to the "Pros/Cons/Verdict" capsule-review that Adventure Gamers uses.

Quote from: Gilbet V7000a on Tue 12/01/2010 09:27:24
Because of this, my idea is like this:
F to lower E - 1 cup (worthless to try)
better E to D - 2 cups (try if you really want)
C - 3 cups (same as current 2-cup games)
B - 4 cups (same as current 3-cup games)
A - 5 cups (same as current 4-cup games)
A+ - still 5 cups, but with the cup graphics changed to gold or shiny or something like that (they can be just categorised as 'highly recommended' in the search filter)

I like this idea. It reminds me a bit of the rating system in the "home of the underdogs" where there is a picture of a dog next to the highly recommended games.

As far as I understood for the original 'Panel Rating' discussion, it was open to change after the first couple of years to hopefully implement a 'decent' system.

Well I can see your point of view and understand what you're saying.

But even if things like "fun" cannot be measured easily, it's still the opinion of a judge that counts in the end, and that takes me back to "Uncut" magazine. When criticising a song or an album you can see on the same number/month that the same album has different ratings because it was analysed by different critics.
What happens is that I got to know how they think music is or should be and so I know that if that critic says "it's not bad" than I know I should buy it because it's good to my standarts, and if another one says "Top quality release" then I know there's no point buying it because we aren't in the same "wave".

What I mean and think is that we should have something of the two worlds:
    - yes to %'s and numeric ratings translated to cups;
    - yes to a final critic by someone that does sign what he wrote;

This way someone searching the database may download a game by me with a poor critic but in the end discover that it had some nice moments and he enjoyed it.
The next game he'll download criticised by the same critic he will know that, that particular critic, is someone that values high level completed results and therefore the downloader knows it's fine for his standarts.
Eventually he'll find a huge positive critic to a game by the same person, then he'll know for sure that the game in question should be top quality.

Anyway, whatever you decide is ok to me, good games allways come to surface, maybe they need time. Look at sites like abandonia and related for an example. It was crap back then but now people want to play them.

Quote from: Layabout on Wed 13/01/2010 10:48:32
As far as I understood for the original 'Panel Rating' discussion, it was open to change after the first couple of years to hopefully implement a 'decent' system.

Yes indeed, and so all the comments in this thread are being taken into consideration by the panel. But from here on, I think everyone's had a chance to say what they want to say on the subject for now. There may be some slight changes when the new web design becomes active but I think the majority still feel the panel rating is fair and working fine. And soon, when player rating sits neatly side by side - there will be little issue. I wonder if MobyGames get the same flack for MobyScore ;)

We can all discuss this system that most people seem to be fine with again when the next member blows their lid  := There has been progress this time round so do not feel "nothing" has been achieved folks! :) You know who to contact if you have any problems with the panel ratings.