Rating system

[RickJ]

"I don't have a dog in this fight.."  as the saying goes but a couple of idea and observations come to mind that may be helpful so I thought I'd share my thoughts.

Criteria
I think there may be some confusion as to what is meant by "criteria".   Is it really necessary to know all the nitty gritty details of how the rating was given?   The problem with revealing this is that the final judgement of the panel is largely subjective so revelation of objective guidelines the panel has made for itself, IMHO, is not very helpful and likely to be counter-productive.   I fully appreciate that using objective criteria helps achieve a measure of consistency between panel members and between games but in the end it comes down to subjective opinion(s).  I think the reluctance the criteria, rightly so,  stems from the realization that the relationships between objective criteria and ratings are necessarily nebulus. 

Perhaps the need for transparency would be better served by a more general description of the process.   Something like the following for example, illustrates the tone and level of detail I am suggesting; the specifics would obviously be different.

The ratings panel is composed of xx members of the Ags community and experienced game developers.  Panel members play each game from beginning to end and note such factors as playability, enjoyability, dialogue, story, graphics, sound, music, etc... to form an opinion.  At the end of an evaluation period the panel members discuss their individual opinions and reach a consensus.    One of the panel members is charged with writing a review consistent with the consensus of the panel.

Review
My first suggestion for the review is that the the cup ratting be clickable and linked to the rating. 

My second suggestion is that the review be written for the benefit of the game author rather than the game players.   How so?  I was a member of Toastmasters, for a number of years, where members learn speaking and leadership skills.  The key ingredient to the success of their program is the way members' practice speeches are critiqued, which I will share with you. 

Someone is assigned to give an evaluation speech immediately after the main speech.   The evaluation is usually structured into three parts consisting of no more than three points each.   The evaluation speech begins by pointing out the best parts of the speech.   Next the evaluator points out what could  be improved.  There is no point in mentioning more than three things because

1) The speaker may become disillusioned/discouraged
2) It's difficult for people to remember more than three things at a time

The evaluation ends on a positive note by mentioning what things the evaluator would like to see more of in the future.   This essentially presents negatives in a positive light (i.e. "half full" rather than "half empty").  This is also an opportunity to encourage the speaker  to continue making speeches and improving.

I think all of this or something similar could be applied to the panel's reviews.   Game authors wouldn't be in the dark about why they got the rating they got.  They would have clear guidance and specific suggestions to improve on their next effort.    Game players would get the same or more benefit from this sort of review as before.

Author Communication
I think the author should be given advance notice of the rating and a copy of the review before publication as a matter of professional courtesy.   It may be a good idea to invite the author to respond if he is so inclined and the panel could take the author's response into account before publication.   It may even be desirable to include the author's response to the end of the review;  this would perhaps keep responses civil and mature.   Anyway a process something like this would cut short any debate or controversy about overly generous or overly harsh panel ratings.

Anonymity
I don't really want to know who the panel members are.  I don't think any benefit comes of it.  The panel ought to create a virtual forum member named MrBlueCup, MisterBlue, BC or some such name so that they can communicate with the community.   If someone has a gripe or a sincere question they could pm  MisterBlue.

[Andail]

Good points, Rick.
I'm afraid it's a bit fruitless to debate this more. There is simply no way we can satisfy everyone's demands. There will soon be a thread informing about the panel. In the mean time, please don't start more threads about the rating.