Panel Ratings?

[Lasca]

I was just wondering what happened with the panel ratings (has nothing to do with my own game, and I resent the accusation!).
I noticed there have been almost no ratings since january. Honestly, just curious.
/Lasca

[m0ds]

The panel needs fresh blood! It wasn't met with great enthusiasm the last time I suggested it. Or the time before that or the one before that

But yeah, it sucks the ratings have dried up. No two ways about it. Would be nice to see Cat Lady get a rating after 18 months, considering TSP got one after about 1 week

[Adeel]

The panel needs fresh blood!

I've the most freshest blood you could ever find here. Just send me a copy of your upcoming games, Mods, and I'll happily rate them.

[miguel]

If a new panel is to be then I ask for a more formal approach from the people who rate and write the official statement.
If a group of people will be rating games then it would be nice if they can come up with method and form to present it, I would like games to be rated by typical adventure technical skills and then a conclusion statement if you must. But this statement should be signed by the author so that the game makers know who the fuck is writing about their games and what aptitudes do they have to do it.

[Lasca]

I would like games to be rated by typical adventure technical skills and then a conclusion statement

I'm not sure I understand this? I think it's impossible to write a review from absolute objective perspective, and what makes a good review is that it feels personal and descibes the authours impressions of the game/book/movie/whatever. But this is a much debated issue.
I do agree that a signature on the reviews would be good, although I can see the problem with some people trying to affect the authours, or getting upset by their opinions.
I do also however feel that when you publish a creative product (wheter it's a review of a game or a game itself) you must be prepared that your product will produce reactions, and in my opinion that's an important part of creative production.
I get the impression that you have a bad experience of the reviews miguel, and hope that I don't step on any toes. That's not my intention.
/Lasca

[m0ds]

Haha, well maybe a tad extreme miguel? I've been slack on rating but like to be anonymous when I do play and rate Horse Park A slowdown on ratings suggests it's not really doing what it was set up for, so perhaps this can come round about now for a new bout of rates courtesy of Adeel's blood (and others). Andail does have a stringent ratings checklist that we all do our best to abide by when games are rated, sometimes a detailed breakdown is nice and sometimes it isn't. Either way, blue cups helps people search and it's always nice to see a feature like this in a healthy mode. It's up to the panel team and Admin G Admin of course But yeah I'm in support of it. I think most panel people just got busy... myself included.

[LimpingFish]

Sorry to hear people are feeling bummed about the lack of ratings.

In general, it's quite difficult to maintain a steady steam of ratings, since the people who contribute to the panel do so without reward and during their free time. Unless they're constantly playing the latest AGS games, and willing to take the time to document their opinions, then there's probably never going to be a 1:1 release/review turnover.

But, yes, there has been a huge drop off since the beginning of this year, save (thankfully) for a few dedicated members. Things have slowed considerably for the last year or two, to be honest.

But this statement should be signed by the author so that the game makers know who the fuck is writing about their games and what aptitudes do they have to do it.

Why? The panel has always been about making sure the people who are given the ability to rate games are long-serving, respected members of the community. When you have a problem with a rating, you simply address it to the panel as if it were a single entity, since, unless panel members take issue with a particular rating, the panel as a whole stands over it's individual members opinions. This also avoids individual panel members becoming the target of over-zealous developers and/or harassing PMs.

The panel was never intended to be the definitive voice on the quality of AGS games. It was simply conceived as a way to help visitors to the site pick their way through the madness of the database, and try to deliver the best games the community has to offer in as painless a manner as possible.

I think most panel people just got busy... myself included.

Indeed. Like I said earlier, the panel will, understandably, become a low priority when its members are confronted by work and life problems. Unless we can find a team of unemployed shut-ins with a love of adventure games, there will always be periods of panel inactivity.

[Gribbler]

Well, I've been waiting almost a year and a half for the panel review of my game I just assumed true art needs contemplating, deep feelings are involved, etc. so take your time. There's no rush.

[miguel]

I would like games to be rated by typical adventure technical skills and then a conclusion statement

I'm not sure I understand this? I think it's impossible to write a review from absolute objective perspective, and what makes a good review is that it feels personal and descibes the authours impressions of the game/book/movie/whatever. But this is a much debated issue.
I do agree that a signature on the reviews would be good, although I can see the problem with some people trying to affect the authours, or getting upset by their opinions.
I do also however feel that when you publish a creative product (wheter it's a review of a game or a game itself) you must be prepared that your product will produce reactions, and in my opinion that's an important part of creative production.
I get the impression that you have a bad experience of the reviews miguel, and hope that I don't step on any toes. That's not my intention.
/Lasca

Well, the more methodical and formal you are the closer you get to being objective. Looks like you rather prefer to have a romanticized version of a review than a proper description of the subject. Yes, the personal reviewer take is important and most times is what make a good review. But that takes time and wisdom.
A signature is just common usage everywhere for obvious reasons, the academic one is that it prevents the review to be a fraud. A fraud? Yes. Gork333, Anthony Peterson and an anonymous person write about my game and they all think it's shit. I still would prefer that Anthony wrote it, I would like him to have some kind of background that I could check, what is his job, career, games he made, etc...
I am prepared to receive criticism, I just don't like it. You got that impression right, Lasca.

from Mods:

Haha, well maybe a tad extreme miguel? I've been slack on rating but like to be anonymous when I do play and rate Horse Park

Yes the F word does stir things up. Apart from that what I asked is pretty innocent, no? And why wouldn't you sign a Horse Park review?

from LimpingFish:

Why? The panel has always been about making sure the people who are given the ability to rate games are long-serving, respected members of the community. When you have a problem with a rating, you simply address it to the panel as if it were a single entity, since, unless panel members take issue with a particular rating, the panel as a whole stands over it's individual members opinions. This also avoids individual panel members becoming the target of over-zealous developers and/or harassing PMs.

The panel was never intended to be the definitive voice on the quality of AGS games. It was simply conceived as a way to help visitors to the site pick their way through the madness of the database, and try to deliver the best games the community has to offer in as painless a manner as possible.

My opinion is that long-serving respected members assigned to constitute a panel of reviews is an excellent idea. I also think that some reviews aren't really good, to be honest. Most are rushed 2 paragraphs that do nothing to the game, good or bad.
Some reviews do have the basis of a proper review where the technical aspects are mentioned and then there are some quality notes from the reviewer where he throws his personal opinion. I ask just this: why not all of them? Why not make it a standard for all of the panel reviewers?
Although you say that the panel is not the definitive quality stamp over an AGS game maybe it's time to become more serious about it.

Signing your name on a review will increase responsibility.

Yes I know, it's a free job and people don't have time.

[Snarky]

Well, we argued over this (extensively!) when the panel was set up, and I don't suppose the panel is prepared to change their minds now. I still think putting a name or pseudonym to the review would be very helpful, because it puts it into context. Not everyone has the same tastes or priorities, so it's important to know where the reviewer is coming from; if you can look at a series of reviews by the same person, you can get a sense of their likes and dislikes.

I'm sure you know some reviewers where you can rely on liking anything they hate, or where you mentally adjust their grade up or down to compensate for their and your own biases.

The thread on Maddox brought me to this article, which makes the same point somewhat more forcefully:

In order for anyone to give a shit about your opinion, we need a frame of reference so we know what makes you tick, what your personality is like and what kind of sense of humor you have. My opinion matters because I have a vast body of work, and am a known writer so when I say something "sucks" or isn't funny, people know I'm coming from a place of genius, truth and concentrated righteousness. That's why critics like Roger Ebert have opinions that matter; not because his opinion is more valid than yours, but because people know his taste in entertainment and movies, so when he pans video games for not being "art," people like me know to ignore him because he's a curmudgeonly old man who probably doesn't understand how or why people play, never spent much time with them and probably sucks at them.

[Lasca]

miguel, I think this is just a matter of personal opinion. The question of what a review actually is or should be.
However I don't think I have a romanticized view of review. I work in a profession where I get reviews ALL THE TIME, and the only way of being able of living with that, is to percieve the reviews as an opinion. This is also why I think it's impossible to "grade" a product of creativity. I never look at the cups, I only read the review and comments. Trying to measure a game by a bunch of criteria dictates that there's a right way and wrong way for a game to be, that there's some kind of universal standard and a "perfect" game. However, like I wrote in the beginning of this post, this is a personal opinion, and this is a question where it's impossible to reach consensus.
And I agree with you that signing your review would be good, but I personaly believe that the opinion is more important than the person behind it. And the opinion often tells you what you need about the person behind it, right? And if Gork333, Anthony Peterson and an anonymous wrote about your game and all loved it, would you still have a preference in who wrote it and want to do a background check?
Anyway, this is probably not the relevant discussion regarding the panel ratings.
I guess what needs to be clarified is what their purpose is and why we want them/need them. And who "we" are and if we all have a mandate to affect the decisions.
Like I said, I like them, regardless wheter the opinion is good/bad/indifferent, and regardless of how well it's formulated, because I like the thought that somebody has taken the time to look at your creation and try to express them self about it. That is infact my idea about art - action -reaction.

[miguel]

Cool links, Snarky.

I don't care about the perfect game, what I do think is that a game review should conform to some basic skills/features clearly identified by the reviewer. Those features should be commented as objective as they can be. After that, by all means, do write some gaming poetry and I'll applaud. What I refuse to understand is the lack of dedication and depth on some reviews and the opposite on others.
I also think that if Gork333 wrote a review I'd still try to check him up while right now there's no way of anybody to check who writes the reviews.
There's no need to clarify the purpose of the panel, I think it's a wonderful idea.

[NickyNyce]

I've noticed that the reviews by the AGS Panel memebers and the community are sometimes different. I noticed that the community seems to rate games higher than the Panel members. Hearing from multiple people instead of one, is a far better way to see if your game has struck a chord with people. This is not a knock against whoever is rating these games, because I think for the most part, they are pretty much spot on. The community seems to give games a slightly higher rating than maybe they deserve. The problem might be what the panel guy or gal actually says, instead of just putting a certain number of cups down that hurts peoples feelings.

As a game maker, hearing what the community thinks is what should be most important to all game makers, instead of one individuals opinion. Why is it so important to get someone from within AGS to rate your game with blue cups? 

The only time I think a panel review might be necessary is when nobody has rated a game. But than again, that should tell you something. If the community rates your game and you get enough responses, isn't that all that matters?

This puts a lot of pressure on whoever rates these games from the panel. After a bunch of responses from the community, it becomes the panel members opinion, against what the community thinks. At the moment, there seems to be two ratings for each game. What the community thinks and what the panel member thinks. Why not just let the community rate games on what they think instead of one person having to take on that responsibilty. I for one, would not want the job to rate everyones game and put my super honest stamp of approval on it.

Maybe those blue cups should go up in count by the score the game gets as it is rated by the community?

[Andail]

Alright, I have been very lazy when it comes to the rating work, so perhaps I shouldn't be very vocal about how it's run, but as the one who kind of initiated the whole thing I can at least offer my opinion.

The reason for the anonymity aspect is that we're kind of part of the community we rate, and that makes it a bit awkward. It's true that for a review to make perfect sense, it's important to know the preferences and pet peeves of the reviewer in question, but here we kind of sacrificed that luxury, to provide better integrity.
Usually, a critic probably doesn't make their own games, or hang out daily with the authors whose work they criticize.
Our way of doing it allows the panelists to be honest.

Of course, this could be solved by using psuedonyms.

When the panel works (i.e. when the panelists are active playing and rating games) it works very well. It's always hard to keep up the steam when it comes non-profit work, so if some member needs a longer break that's probably natural. 

There's a thread with a check-list over at released games if you want to see roughly what the panelists go by. Ultimately, the merits of a game are extremely hard to quantify, so there's always a vast amount of subjectivity involved. However, with the current panel, you at least have people who've played hundreds and hundreds of games and been involved in almost every aspect of AGS and the community.

We can look over the situation again and see if we end up recruiting some new blood.

[Calin Leafshade]

I really like the idea of pseudonyms.

Each panel member could be assigned an adventure game related pseudonym like Guybrush or Green Tentacle or whatever and then you could understand the broader context of the rating. Some panelistas are harsher than others and being able to see who rated it in relation to all the other ratings would be great.

[Cassiebsg]

Personally, I rarely give much credit to how many stars, or in this case blue cups, a game/movie/show has, but will rather read the why the reviewer thought so. In other words, maybe the reviewer thinks "this game is a gem because of all the mini-games and cinematic hugs cut scenes", then even if it gets the max blue cups it's probably still not the game for me, since I rather be free from (forced) mini-games and long cuts scenes tend to bore me (if I want to watch a movie, I'll seat by the sofa and turn my brain off for about 1,5 hour... lol).
So, for me, the reasons why one rates a game like they do, are more important than the rate it self. It also provides the author with some feed back/criticism, where to improve the next game (assuming the author agrees with the points, that is).

[Lasca]

miguel
Ah, I get it now.

I don't care about the perfect game, what I do think is that a game review should conform to some basic skills/features clearly identified by the reviewer. Those features should be commented as objective as they can be. After that, by all means, do write some gaming poetry and I'll applaud. What I refuse to understand is the lack of dedication and depth on some reviews and the opposite on others.
I also think that if Gork333 wrote a review I'd still try to check him up while right now there's no way of anybody to check who writes the reviews.

You're probably right. I think I was focusing more on the response from the maker of a game on a review than the effect that review might have on a possible player of the game. Thanks for some perspective on that snarky.
Pseudonyms sounds like a good idea. That way you can read other review written by the same authour.
After reading the checklist I feel like it's a very well constructed system, and I guess it's inevitable that the ratings will halt during periods. But perhaps a solution to that is to add more people to the panel.

[Baron]

The Ratings Panel will make you pull the plug on your game! 

+1 for reviewer pseudonyms.  I mean, everyone else who comments has to leave a name, so it's only fair.

[elentgirl]

As a game maker, I want to point out two important aspects of the rating system.

My first game, which I posted a few years ago, was rated by the panel.  It also received a number of reviews from players, some enjoying it for the types of puzzle I have included, other hating it for the same reason!  It seems to me that the panel tried to give me objective criticism of what worked and didn't work in the game, which I found much more helpful when it came to planning my next game than the personal preferences of players.

The second reason that the rating is important is the perceived value of blue cups over orange cups when players are looking for a game to play.  There is a filter on the search criteria for games to include only those with two or more cups which only looks for blue cups, not orange ones (I tested this by searching for a particular un-rated game that has more than two orange cups).

While I appreciate all the valid reasons why there has been a slowdown in rating games (I have been waiting for more than a year for my second game to be rated), I feel it is important that the rating process is continued.  I have no issue with how the games are currently rated, but would like to see a return to more speedy rating.