More Panel Ratings Discussion

[Shane 'ProgZmax' Stevens]

I had nothing to do with the ratings of any of my games, Ryan, and of course the rating system (like all subjective ratings systems) is flawed.  You believe you are right in that a game doesn't deserve 4 cups because that's what you believe.  Whoever rated it believed differently, and that really is that.

As far as using language, I use it when I think it's appropriate to stress a point, and that's my prerogative.  I don't consider myself to be an 'important' ags member; I moderate a single board and rate a few games, big deal.  Grundislav is a far more important cog in this community than I am and he is neither a moderator nor a panel member.


Snarky:

In that case I apologize.  While unreceptive and hostile do not in any way mean the same thing, if that's what you meant than I could see where you might get that idea, though that's really not the case.  Most of my statements have been to reflect that there are in fact guidelines we go by in the ratings and that we aren't going to change ratings just because a few people are unhappy about them because it's disingenuous and taints future objectivity.  If that's being unreceptive than I can't really do anything about it, but there have been suggestions made that we have publicly acknowledged with potential, your request for the guidelines being one of them.  I don't think you're being exactly fair in your assessment overall, but there's no point in going around about it.

I'm still curious about what people think of being given extended feedback about their rating upon request, though.  This could extend to a good or bad rating, theoretically, but since we're doing this on our free time I'd like to limit it to people who really have an issue.  Would anyone find this to be a helpful way for them to come to terms with their rating?

[ddq]

Oh jeepers, forum drama. I'll get the flame shields...
I think that anytime an argument breaks out on the internet to the point that the posts are multiple paragraphs long, all dissenting parties should be required to talk to each other face to face, IRL.

[GarageGothic]

Yay, sort it out the Jay & Silent Bob way!

[Layabout]

I had nothing to do with the ratings of any of my games, Ryan, and of course the rating system (like all subjective ratings systems) is flawed.  You believe you are right in that a game doesn't deserve 4 cups because that's what you believe.  Whoever rated it believed differently, and that really is that.

Aye, I agree. I was pointing out how subjective the rating system is. In addition, I never explicitly stated that I thought you would rate your own game. I know this would never happen.

User rating and panel rating must be seen as equal (or even user rating as higher), I don't think this is the case currently. I know user rating can be jimmied with, which is why we have the panel rating in the first place. Limey Lizard, for example, has a user rating that is pretty much on par with the panel rating. This means it was probably rated correctly. Some people may not have liked it for whatever reason, but the majority did. But if there were the case where a game was rated 2 by the panel and got a user rating of 75%, then the panel rating should at least be looked at, to ensure the ratings are 'correct' (bad word to use with such a subjective matter, but oh well...)

[Shane 'ProgZmax' Stevens]

I don't honestly disagree with your points because I have no opinion of the ratings either way.  The cup rating system by a panel was CJ's idea and the motivation and reasoning for it is his to provide at his leisure.  I would hazard a guess that he wanted to mimic the system 90% of sites similar to this one do with an 'editorial' rating that appears at the top of each game rated and then people decide whether or not it's of value.  Perhaps the reason there's so much controversy in this case is because there are so many games (and so many RATED games) in the database vs the average site that will do one or two reviews a month?  Either way I don't object in any way to the user ratings being more prominent if that will suddenly make everyone happy since both sets of ratings are based on personal experiences (group vs single).

[blueskirt]

I can't fathom how you've reached this opinion of the rating process. Pick a gaming website at random (IGN, Gamespot, Adventure Gamers, etc), and you'll find reviews are usually the work of a single author. A single author with a single opinion.

You can't use this example. Because while all review sites use a single reviewer for a single game, there's thousands of review sites and blogs out there that will review the same game, so if you don't agree with one's opinion, you can just check another site, because there will be plenty of alternative reviews. For many AGS games, the AGS panel is the only review they'll ever get.

But don't get me wrong, as much as I wanted games to be reviewed by more than one person, I don't want you to change anything to the review panel. I've long accepted that the current system is as perfect as it will ever be. There's just too many games in the database and too much work done already, changing just one little thing could and would demoralize everyone who's been involved in the panel in the last couple of years, and in the end, there would still be people who'd complain about unfair reviews so it would be completly pointless.

There's more subtle solution.
Make, say, 20 cups minimal grade and 25 cups maximal grade. In this case you'll keep general rating concept and yet make lower grades less unpleasant.

Haha! It's funny because I had exactly the same idea yesterday. Have the second cup worth 5 cups. So we'd have games rated 1 cup, 6 cups, 7 cups, 8 cups and 9 cups, no more problems with the second cup not being obvious that it means "Worth a try", and no need to overhaul the whole system.

-edit-
Oh, as for the idea of giving more importance to the user ratings, how about ditching (or converting) the current 0-100% user rating system completly and replacing it with a 0-5 cups rating? Not unlike the system on Abandonia.com. Or would that be too much trouble for nothing?

[Wesray]

My two cents: I think the panel reviewers are doing a good job all in all. Yeah, I didn't agree with the original rating for the McCarthy Chronicles. I didn't play Cosmos Quest 3, so I cannot say if the rating was fair. But "fair" is subjective anyway, so I agree that the reviewers should stick to their decisions, as long as they can argue for them.

Ratings are, and always have been a flawed system of telling people whether a game/film/whatever is good. Good, fun and entertaining is too subjective an opinion to be rated in any way. I guess a more appropriate way would be having 3 cups being the AVERAGE since it is in the MIDDLE. Shit games and awful games (like my masterpiece Gorthor) deserve 1 and two cup ratings. But ratings themselves can be confusing, which I think is why people are having an issue with it. Some ratings consider anything around 50% to be average. Some others consider 70% to be average. It's not an accurate way of telling whether a game is good or not. Do away with the whole thing and perhaps include a small review only. You can still have a list of highly recommended games.

I agree that one possible problem is the perception of the 5-cup-scale. No matter what the formal definition for each cup-rating is, it is a simple fact that most people see anything below 3 cups as not worth their time. A reason might be that in school and academic circles everything below average is deemed as not good enough. When I write a scientific paper that is peer-reviewed, anything below 3 means the paper is rejected by the reviewer, 3 means neutral, and better than 3 are various degrees of acceptance. Similarily, even with PC game reviews which are often done on a scale from 0 - 100%, anything below 50% is usually interpreted as rotten and definitely not recommended. An average but in no way exceptional game would typically be rated around 75%.

A solution? I don't really know. It would probably be best to redefine the meaning of the cups, so that 3 means average. As it is, of 952 games in the database 47 games have 4 or 5 cups (only 4 games have 5 cups). This is heavily skewed towards the bottom end of the spectrum. But changing the meaning of the cups would mean re-rating all the games, which is not feasible either.

In the end the system should probably stay as it is. It is my subjective feeling that the panel reviews have become stricter recently, with polished games without major faults getting 2/5 cups. But I choose to see this as a good sign - it doesn't necessarily mean that the reviewers have become unfair or that the quality of AGS games is declining. On the contrary, AGS games have become so high-quality that simply being good is not enough to stand out anymore. It just means the community as a whole gets better all the time.

And for me, personally, the panel rating is only one of several criteria when I decide whether to download a game. The screenshot and user-rating are at least as important to me for the first impression.

[Andail]

All right, listen now. There will shortly be a nice comprehensive thread informing about the panel.

As Eric said, try to calm down a bit. Take the whole rating thing with a pinch of salt.

It's absurd to believe that a system like this can satisfy everyone. There are hundreds and hundreds of game authors in the database, every single one of them will never be pleased.

It's impossible to have a huge debate every time someone disagrees with the rating. Try to remember that this is ultimately a project sanctioned by CJ.

Try to understand that a public discussion is not viable. There are as many opinions as there are members in this community. Count the amount of suggestions given in just this thread. No matter how eloquently expressed, an opinion is just opinion. There is no absolute, god-given true path to walk here.

[Igor Hardy]

No matter what the formal definition for each cup-rating is, it is a simple fact that most people see anything below 3 cups as not worth their time. A reason might be that in school and academic circles everything below average is deemed as not good enough.

More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

Removing the games because he doesn't agree with the rating is insulting to the panel member who took the time to play his game (it's not a particularly short game) and critique what he felt were it's flaws. If we waste time rating those games that are going to end up being removed from the database for no other reason than sour grapes, why should we bother at all?

To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

It would be best if both the panel members as well as the game creators would be able to find some sort of satisfaction in the work they do, and not take for granted the amount of sympathy and respect they'll receive for their efforts.

[Pumaman]

I'm going to try and respond to some of the points raised so far:

How is it fair that only 1 person on the panel rates each game?

In an ideal world, we would have a panel of 10 people, who all played every game and came to a consensus for the panel rating. This would be possible if the AGS Website was a company with lots of money that could employ full-time reviewers to do this.

In reality, all the panel members are volunteers who have other things to do, and if we had to get several people to play each game through before rating it, they would never have the time to rate all the games that get released.

If you don't like the rating, just get over it, it's not that important

I don't think it's that simple. One major difference on this website is that we have the game developers and the game reviewers as part of the same community. Normally, you have several websites that review games (eg. Adventure Gamers), and each of them give their own opinion on a new game.

The AGS Panel is different because it is seen to be the "official" opinion of the AGS community, and is featured on the website of the development tool that the game author was using. So in that respect, the implication is that it is more important than ratings given elsewhere.

For that reason, I think it's important that the Panel always tries to be as polite as possible with their comments, bearing in mind that they are peer reviewing other developers work, and the last thing we want is to put people off making games, or drive them to leave the community.

One other thing to bear in mind is that the AGS Panel is much harder to please than many review sites -- for example, the panel have so far rated 827 games, and only four of them got a 5-cup rating. You can see the spread of ratings here:



The 2-cup rating is by far the most common, and 2/5 is not a "bad" rating as it would be on some review sites.

[Wesray]

It's absurd to believe that a system like this can satisfy everyone. There are hundreds and hundreds of game authors in the database, every single one of them will never be pleased.

Yeah, it's obvious that no rating system can please everyone. As I said, I fully support the current one. With that in mind, I still think some brainstorming to address minor issues doesn't hurt. Some of the community's ideas might be considered sometime in the future, when a new rating system might be deviced.

More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

In that regard its even good to have only a small pack of really special games (4-5 cups), that are a great advertisement for AGS. People who don't have much time can just play those and ignore the rest, or play them once they have more time. The only annoying issue with the current system is perhaps the fuzzy differentiation between games with the lower grades. But still, an opinion is an opinion and even with more detailed grading many authors would certainly feel their games have been misjudged.

So, as I said, yay for the current system, and I'm sure increased transparency of the rating process via the new thread won't hurt!

One other thing to bear in mind is that the AGS Panel is much harder to please than many review sites -- for example, the panel have so far rated 827 games, and only four of them got a 5-cup rating. You can see the spread of ratings here:

The 2-cup rating is by far the most common, and 2/5 is not a "bad" rating as it would be on some review sites.

Thank you! That was the point I tried to make earlier. I think knowing the way this panel works and rates games (and seeing this graph!) might ease the depression of some authors when they see their hard work rated rather low. Getting a 2-cup rating is not bad, in fact it means you have developed an AGS game of average quality, and that's surely a compliment.

[Shane 'ProgZmax' Stevens]

Maybe it would help if some people visualized the 1 cup rating to mean a value between 0 and 1 (inclusive), thereby creating a 6-cup spread where 2 appears more clearly as 'below average' rather than 'bad'?  Just a thought. 

As CJ said, the panel ratings tend to be harder to please -- but then again -- haven't all of us gotten that way over time?  If you want to know what I think the real reason is, it's the sheer quantity of content on the site that ramps up the challenge.  There are many many games in the database, and each year there's a marked improvement in both the engine and the quality of the output so people are getting more and more demanding and/or picky about what's out there, raising the bar for the next game in all fields.  There were far simpler games made in 2000-2003 that people absolutely loved that would completely tank compared to the games submitted today, both by panel and public judgment.

On many sites the reviewers are also active posters so I'm not sure that's a factor here, though.

Finally, I'm not sure the AGS Panel rating should be seen as an official community rating at all; the user rating does just that already.  I've always seen the AGS Panel rating as one of those editorial ratings that appear along with the community rating (like Gamespot) that people weigh against each other to arrive at a conclusion about the game.

[Snarky]

More likely it's simply the situation of limited time for games one has vs. the number of games to choose from. As a gamer browsing the AGS database you really wish that the AGS panel suggestions are correct, so that if you listen to them you neither waste time nor miss a gem.

To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

It would be best if both the panel members as well as the game creators would be able to find some sort of satisfaction in the work they do, and not take for granted the amount of sympathy and respect they'll receive for their efforts.

Well said on both points, Ascovel!

Great to hear from CJ on this. To me it seem like what he and Andail say (and some of the things ProgZ mentioned earlier) offers a very positive way forward.

I'm not sure if this is the right place to raise this, but if we're thinking of making the user ratings more prominent, would it make sense to have another look at their ratings scales? Maybe this is just me, but when I rate a game I don't really feel like the scales capture what I like/don't like about it, so that the ratings don't seem to closely reflect what I think of the game. Main points: "Immersion" groups together a bunch of different things that (to me) don't really go together--like story and music--under a confusing label, and the "Puzzles" scale seems to conflate level of difficulty with originality, balance and good puzzle design.

Maybe if the panel guidelines are made public, they could form the basis for new user rating scales as well? (I haven't seen them yet, so I don't know whether that would be a good idea.) Just as a forinstance, maybe part of why a 2-cup panel rating seems bad is that on the user-rating scale, it matches the more negative "Play it if you're bored, not much fun to be had here." That "problem" would go away if both ratings used similar scales.

Or perhaps use the AGS Award categories (a reduced set of them at least) as the basis for the ratings categories? We argue over them every year, so I think they've been honed a little more than at least the user ratings categories.

I'd also suggest adding a difficulty and length rating. Since these will vary from player to player, it's the sort of thing it's useful to get multiple people's input on.

Now if everyone else thinks the way things are now is working fine, this is clearly not worth doing. Just thought I would put it out there. The downside would obviously be what to do with the old ratings. I would expect that "Visual" and "Enjoyment" could be transferred over to the new set; for the others it would depend--maybe abandon them, maybe leave them as no-longer-active "legacy ratings" on the games page, maybe use them to initialize vaguely related scales in the new system, but with a low weighting relative to future input.

[m0ds]

Wasn't it this game that was the first to be removed from the database by its author due to being unsatisfied with the rating? Either way, it looks like the author realised it was worth having it in the database because it's back, whatever the rating, whatever the comments. I wouldn't be surprised if Harg returns his games to the DB in the near future.

Yes, it's easy to be offended by the smallest things personal to us. Just last night I wrote a long rant on Facebook about some folks that have really annoyed me lately, but at the last minute I was able to delete it and walk away from it. I realise it can be hard to stop yourself from doing the most drastic actions, ie removing games, posting that rant, etc when it really does hit you on a personal level - but it really is that stopping yourself part that shows you've got maturity enough to handle these kind of things.

And no matter how sympathetic others will be, I know that I will always look at removals of games because of a rating or comment the author feels is vindicative, to be childish. If I did it even if I felt just, I would still know that the action of removing a game because of some outside comment I don't agree with - would be childish. And plenty of adults do things just as childish than children! Doing it isn't going to make everyone jump up and support you. It's just going to create threads where there are 5 pages of the same three views being spoken by 50 different people, and makes for an extremely dull read.

Some of the games I've worked on have average cup ratings and yes I feel some of the reviwer comments feel a little bit harsh...but as it's stated if you're putting your game up on a public DB then what do you expect? And as said before it's a privallege to be on the AGS db, just as much it is that CJ even bothered to make an engine for us to do stuff with You can't be a creative person and put your creations out there without feeling the brunt of unsatisfied customers occasionally, perhaps even frequently. And if those comments really get you down, perhaps game making, or creative persuits on the whole - just aren't really your destiny? You need to be able to take the bad with the good in this game, and because of the way it appeals to lots of other people, you need to be able to take the even worse with the bad and not go flying off the rails along the way.

After reading the entire thread, I have to say Harg's actions seem like a knee-jerk reaction and I'm sure he/she will come round to it and realise just being on the db is a good thing. Same as Calin realised eventually the system is actually doing him justice, and the same way Yrolg returned to the database. It's like a DO LOOP we can't escape but someone forgot to remove the IF WHEN for knee-jerk reaction capabilities. Please note that last sentance was inspired by Dualnames

[Andail]

Alright, there is now an official information thread over at completed games.
http://www.adventuregamestudio.co.uk/yabb/index.php?topic=39821.0

This information as been approved by CJ and the Rating Panel.

Individual members of the rating panel will not partake in more debates regarding specific ratings. If you have comments or questions regarding the rating of a game, PM me or Progz.

[Shane 'ProgZmax' Stevens]

Thanks for finally posting the guidelines, Andail.  Hopefully it will resolve some questions people have had about the panel rationale.

[thecatamites]

I haven't really been following this but I was wondering if there'd be any way to notify people when their games get rated? From what I can tell the judges rate the games whenever they get around to it rather than by any release schedule or whatever, which is understandable, but it also means you don't know if they've gotten around to your game unless you accidentally stumble across it. A lot of times I don't realise the panel has been there until someone else leaves a comment on the game later on. I guess what I'm saying is that it'd be nice to have a 'recently judged' box in the game database functioning about the same way as the 'latest user comments' section. It'd be easier to check than the gamepages themselves and could also act as a way of getting more attention to games which get a high panel rating etc.
Just a thought!

[LimpingFish]

I agree that entering into these tit-for-tat arguments is a waste of time, and I'm pissed that I allowed myself to become ensnared. Just as I became annoyed with a perceived lack of support and appreciation for what the panel does, I also understand that a developer can become just as annoyed and exasperated when faced with what appears to be a cold-hearted rating.

I still believe Harg could've handled the situation better, and with less effort than it took to handle it the way he did, and I still stand by the panel's right to offer their honest opinion. My stance on these facts isn't likely to change. If that troubles you, I can't really offer any advice.

I seem to have cultivated this image of an "Obey me!" overlord in the minds of some people, which is disheartening, as it's far from the truth; my time spent on the panel since it's inception would hopefully attest to this. I am passionate about the time I dedicate to this community, and it's easy to become confrontational when that dedication is questioned in an accusatory manner.

With the guidelines now public, we should now have some factual basis to future discussions.

To be sure preparing the Panel reviews must take a lot of dedication and time. However, in what you say you automatically assume that for every game that is removed from the database, the respective panel member reviewer didn't get any value out of playing it, wasted time, and it was a huge favor on their part to take up the terrible chore of reviewing this game. I feel something's wrong here. In the end it's all supposed to be done in the name of fun, isn't it?

Quite right. It's just hard to remain optimistic in the face of blustering discordance. My time on the panel is limited, and I try to rate as wide a range of games as I can; just as every panel member does. If I was to dedicate the hours a relatively full-length game requires to appreciate it fully, only to have the reason that time was spent rendered void, I'd have a right to be a bit pissed. Fun can also be subjective, as I'm not going to enjoy every game I play. No reviewer enjoys everything they review. This aspect of their time spent reviewing can be thought of as "work". I wouldn't like to spend time working, only to have that work evaporate. If you haven't gleaned this fact from what I've being saying throughout this thread, then I can't really explain it further.

ProgZ, Andail, and I, as the most public panel members, have always been open to discussion about ratings. Maybe not so open regarding demands and ultimatums, but I'll always happily entertain a civil query regarding just what it is that I contribute to the panel. I may sometimes make my feelings a bit too clear on whatever the matter at hand happens to be at the time, but since we're all adults here, we shouldn't require our opinions to be delivered wrapped in cotton wool.

I'd like to thank everybody who voiced their opinion in this latest debate, and I now consider my role in it to be at an end.

[Dualnames]

Mods: Ah..thanks?

Andail: Congrats I personally believe your action will make things less dramatic and easier for the panel as well!

[Iliya]

I will try to explain why I removed the games from the database.

Some of the developers here are trying to promote their games on other websites. Some are trying to get a review for their games from other good websites for adventure games. And with this AGS panel review there is no way someone to take you seriously.

My goal wasn't to insult the AGS community. As you can see on Cosmos Quest website, I still support AGS as great tool for creating a games. I just don't agree with AGS Panel. I tried to avoid definitions like "leaden pace", "bland characters", "lack of direction", "ponderous gameplay" for my game that came from one person. Actually, the definitions were the things that irritated me most, not so the cups.

Why I didn't send a PM to AGS panel before the deletion? I did't know who is in the AGS Panel. Now, due to the deletion, everybody knows who to contact. And I'm glad that there is a movement in that subject (the official information thread about reviewing the games).