Members for rating panel, apply here!

[SSH]

It wisnae me that made Richard game of the month! It was that oldest of old-guards, CJ...

[Babar]

Aren't the ratings being done with the player in mind, as opposed to developers? In that case, as mentioned before, I'm not sure how originality affects anything (I'd be more on the lookout for graphical consistency), and scripting... I suppose if it makes things easier for the player. I have no worries about favourite games, but I can think of quite a few which would be rated pretty low with the current classifications, which I'd think would be worthy of attention they wouldn't get: Pleurghburg, Apprentice 2, QfG4.5 (come on, it was fun ), MMD, RON, Ben Jordan, META, Robotragedy 2, etc., etc. I'm pretty sure it'll not be just isolated cases which may suffer.

I dunno...I'm just giving my input for what it's worth. A rating is just a rating, and if something goes a bit too far off, there's always the user votes to balance things out.

[blueskirt]

Snarky, the speed issue isn't primarily because it has to be done before a certain date, it's because the project is more likely to die out if it turns into a very lengthy process.

Also, this current interest is a reaction to my initiative. There have been plenty of discussions in the past, but they never really had any fruitful outcome. Sure, people have done a great job uploading and mirroring various catalogues of games (Scotch among others), but the official AGS database has only slowly deteriorated.
To make things happen you sometimes need more decisive actions and less talk.

I'm not against input (I'm very willing to give up my initial ideas about the rating - and have evidently done so in the designated forum to give room for better ideas) but if every single member of this community should be able to bring the progress to a halt just because they envision how their favourite game will be rated low, nothing will happen. After all, this is CJ's personal site, not the UN.

All that will happen is that the games database will have more working links, better classifications and a rating. People should be very happy that there exist a group who will deal with this.

It might just be me, but this sound a tiny little bit like "Let's marry now because in a month we may not love each other anymore."

If you want to rush the project, feel free to do so, but for a project of such proportion, I too think that it would be better to take the time to debate the thing openly, propose new ideas, reach a concensus or find a system that will please nearly everyone or at least a big majority and get the thing done correctly after the first try, rather than rush everything, end up with a system that only please an handful, get people involved and in a few years from now have a similar discussion like we're having right now to replace the rating system with another one, which would only piss off all the persons who invested energies in this proposed rating system.

It is also important that we reach a system that will convince as many persons as possible because it's these persons who'll propose their help and time to play and rate each games, and like some persons pointed earlier, if they don't like the rating system they won't be interested in joining the project.

So I say, let's not skip steps and let's openly propose 2 or 3 models and throw ideas to optimize those models. Then we could find a dozen of games of all kind and all backgrounds and review them with each of these models to see if they work well, see their strength, weakness and correct them if we need to, then we'll choose or vote for one model which will be the final one and get started with it. Who know, the original model could have been the best one all along, but if we don't compare it to the others, we'll never know for sure. It ain't the UN, but let's think of it more like presidential election. In the end one system will be chosen over the other by a majority and it's this system that will be used. If there must absolutly have one system, let's at least try them all, and give every systems their chance before picking one that we'll use for years.

However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

[LimpingFish]

However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

Nobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

There's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.

I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

The formation of a rating, and cataloging, body should be arranged and instigated by one or more moderators, which I believe it has been, since we trust these people to maintain our forums, run the awards, etc.

I don't see why this should prove any different.

[Hudders]

Well said that man.

[Shane 'ProgZmax' Stevens]

Aren't the ratings being done with the player in mind, as opposed to developers?

The short answer to this is no.  The long answer is sort-of.  The idea is to be as fair as possible to the developers while taking in mind the general popularity of the game and balancing that with the game's actual strengths/weaknesses.  It's impossible to be 100% objective but it's not impossible to strive for objectivity.  The system being established is intended to be as fair as possible to the author while remaining honest.

Also Mills, I wasn't implying that you're a scared child but I stand by the statement that you're preaching doom without seeing the system in action.  I really, really dislike it when people attack something without seeing a working example from which to form a complete opinion.  Right now you (and not you alone, really) are forming an opinion based on a single document that is a WIP of some basic criteria used for rating the games.  Meanwhile, if you have some suggestions to improve the document format then by all means share them.

What I'm missing is an explanation for what it takes for a game to get a certain rating. In this thread, people seem to assume that they need to fulfill all the criteria of that category.

Don't worry, Snarky.  This and some other things you mentioned are already being ironed out.


And yeah, the moderators do understand and care about the frustration and questions you guys have, I know I do.  Just try not to overreact before the picture is complete and there are some actual examples of the ratings in action.  Can we at least agree to that?

[blueskirt]

However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

Nobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

There's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.

I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

Alright, alright.

*Blueskirt adds "concensus" to the list of word he'll never ever use again.

I did not say (or meant to say) it had to please everybody, I said it had to please a majority, 50%+1. Chances are we'll be able to see the rating system in action before it's on. If the rating system is really close to be finalized, then nevermind what I say, but if it were to take another bunch days and if in the meantime we can find another system that could be a good or better alternative, I fail to see how one additionnal day to rate the games with the second system, 2 additionnal days to do some corrections and up to 4 days to vote (depending of who between everyone who can apply to the panel, the panel or the moderators get the power to decide on which system to use), will considerably delay the process.

I too personally trust our moderators to decide what is the best for us and I don't have anything against the one proposed, even if I'd like to see it in action. I'd just rather avoid seeing this topic brought up again 7 years from now because we were too impatients to wait 7 more days when the rating system was created. Things are easier to correct right now than they will be when people will have invested years to rate all those games.

[MillsJROSS]

I appreciate this and understand where you're coming from. Basically, we've had several public discussions on how the games database should work, and we usually end up without a consensus and nothing happens. The reason this initiative was discussed behind closed doors was so that we could make a decision and get on with it.

I don't agree, because it seems one of the times we discussed things we actually got the system we have now. I mean we discussed among the forum, and things happened. Obviously, the system we have now wasn't as great as we expected, but we've learned from it, and it was a step up from our last system.

Nobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

No one ever said this was a democracy, but we've mostly been very good about discussing things before a decision is made. Yes, at some point someone, CJ, has to give the greenlight to do something. However, as a longtime member of this forum, I very much would like to express my oppinions. So I am doing just that.

There's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.


I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

Since when have we really had that much forum politics? I think what we need to do is discuss and then have action. Obtain as many oppinions as possible and move forward with that. I think there are people with good ideas, and I don't feel that we should not care about what people have to say.

The formation of a rating, and cataloging, body should be arranged and instigated by one or more moderators, which I believe it has been, since we trust these people to maintain our forums, run the awards, etc.

I don't see why this should prove any different.

Well, for one, it's not up to moderators to decide what awards win or not. Most of the categories were also attained by open discussion. A moderator is a person of established trust, and they should be a part of this, but we should have a say as well.

Also Mills, I wasn't implying that you're a scared child but I stand by the statement that you're preaching doom without seeing the system in action.  I really, really dislike it when people attack something without seeing a working example from which to form a complete opinion.  Right now you (and not you alone, really) are forming an opinion based on a single document that is a WIP of some basic criteria used for rating the games.  Meanwhile, if you have some suggestions to improve the document format then by all means share them.

Of course I don't agree with this. All I'm getting now is promises that things are being done and snippets of information as to what is happening. I'm not in the know, so to speak. Perhaps, I'd be less prone to be against this material, if all the discussions were at least open to the public eye. I don't have to see something in action to form an oppinion about it. I can concede the fact that I may be wrong, I may be crazy, but it just may be a lunatic your looking for. If this system works, great! But I just in general, don't like feeling that my oppinions on the subject seem not as weighted as people in the current secret panel.

Ideas:
One, I don't like a forum age requirement for the panel. As mentioned, newer members aren't embedded into this community have a more subjective oppinion of games. As we're appealing to people who are essentially just browsing the game section, without being in a community, it seems newer members voting habits would be more in touch with said browsers.

Two, if a panelist is a hard working individual, than he/she should not have a predestined time to leave the panel. Don't get rid of people who are doing a good job. Just don't cap off the amount of people in the panel, either.

Three, I think a good system would almost be what rotten tomatoe has. In this fashion people can  either go by the critics or the masses. With the only criteria being, "I liked it," "I didn't like it."  And if people wanted to, right up comments/reviews so maybe the author could get some vital feedback, too all the merrier.

Four, the moderators should be reponsible for getting rid of dead links on the game database, whether that's by moving it to a deadlink section that can come back alive, if the link is reestablished, or just deleting them. They should also be responsible for genre selection. Moderators should, to me, have the power to move things accordingly, and to keep things running smoothly, and that's the end of the line.

And I'm done, for now.

-MillsJROSS

[Shane 'ProgZmax' Stevens]

Okay, I'll discuss your points.  Note that I'm not the one who established the ruleset for selecting judges (Andail did) but I happen to agree with it.

The age requirement.  I think this is reasonable because while not every adult behaves responsibly, I would argue that most understand responsibility better than children/teens do and won't flake out as much.

Panel rotation.  I think some of the expectation here is to bring in new people on a regular basis to make the ratings representative of a community rather than 7 people.  I understand that if you're a hard working person you'd like to stay on the panel and this is perhaps something that could be flexible, though I think it really depends on how the community reacts to the initial ratings.  I definitely think a maximum panel limit is necessary to keep the group communicating effectively.

'I liked it/I didn't like it' doesn't really tell me anything, personally.  I'm more concerned about why something received the rating it did than what the reviewer rated it as.  There will be a comments section for judges to explain the rating if it's warranted and will provide a more detailed explanation behind the cup rating (were there serious flaws that prevented it from being completed, etc).

CJ's been working on the db system to allow judges to adjust genre, fix broken links and so on as they rate the games.  The author will be able to go back and alter the genre if they think the panel made a mistake.  As for giving moderators that same access I'm not sure it's something that will happen.  It's up to CJ, really.

If you want to form a negative opinion based on an incomplete model that's up to you, but I don't see it as very constructive.

[Pumaman]

No one ever said this was a democracy, but we've mostly been very good about discussing things before a decision is made. Yes, at some point someone, CJ, has to give the greenlight to do something. However, as a longtime member of this forum, I very much would like to express my oppinions. So I am doing just that.

As I say, I appreciate this and I concede that we could have done better with the way this was made public. I also appreciate that this smacks a bit of the nanny state saying "we know what's best for you", and that usually we do these things with a full public consultation here.

I think that one of the reasons we've gone ahead with it rather suddenly is that we got to a point where there was enthusiasm amongst the moderators to improve the database. If we had waited too long discussing it, it's only natural that people lose motivation and you end up with nobody willing to actually do the work when the time comes. So, it seemed a good idea to sieze on it and make something happen.

So in summary, I apologise if people feel left out. When we have the system working, there'll be plenty of time for us to review it and see if it works in practice; and if not, we can make changes.

I'd just rather avoid seeing this topic brought up again 7 years from now because we were too impatients to wait 7 more days when the rating system was created.

Actually, I think it's inevitable that it'll need to be changed in a few years time. For example, 4 years ago there was no need for a Review Panel because the number of games was small enough that it was practical for people to sift through themselves when looking for a game to play.

But now, times have changed and we need to devise a method to cope with that change. And I'm sure that in 5 years time, things will have changed again.

[Andail]

Quick question; Snarky and Mills, would you like to join the panel?

[MillsJROSS]

Quick answer: I'd like to help anyway I can. I graduated last month, started work, but I now have a lot of free time I never had while in school.

The age requirement.  I think this is reasonable because while not every adult behaves responsibly, I would argue that most understand responsibility better than children/teens do and won't flake out as much.

I think I used the wrong word, here. I meant forum age. I do respect there be a minimum age requirement.

-MillsJROSS

[Snarky]

Snarky, the speed issue isn't primarily because it has to be done before a certain date, it's because the project is more likely to die out if it turns into a very lengthy process.

Well, yes. I appreciate that. It does seem like it would be possible to start with things like cleaning up broken links, and maybe even playing some of the games to-be-rated while discussing the system, though. And if you're discussing it behind the scenes anyway, would more open dialogue really slow it down that much?

I'm not against input (I'm very willing to give up my initial ideas about the rating - and have evidently done so in the designated forum to give room for better ideas) but if every single member of this community should be able to bring the progress to a halt just because they envision how their favourite game will be rated low, nothing will happen. After all, this is CJ's personal site, not the UN.

I don't think anyone wants to bring the process to a halt. And I do think that to consider how the ratings of a few games generally considered good (e.g. AGS Award winners) would come out is a useful, quick-and-dirty way to get a sense for a ratings system (before actually applying it in practice).

You know, I'm not trying to complain, though. I was unhappy with a tone of hostility I sensed in the earlier posts, but I like the way things are seeming to progress. The panel system seems quite sane, I thought your draft document was a good start, and given that the people involved are reading this thread and hopefully considering the suggestions and criticisms, I am optimistic that the final system will reflect the preferences of the community. (Personally I like what I'm now hearing about augmenting the ratings with short explanatory comments, for example.)

Where I'm coming from is that philosophically, I am nearly always in favor of democracy, openness and consultation. Take how CJ runs the AGS engine. OK, it's his application, and he can choose to do whatever he likes. Generally, though, people suggest new features publicly, there may be a discussion, CJ engages with the suggestion, lets the suggester know whether or not the change or addition will be made, and explains why. If that kind of dialogue could be possible here (and I realize why it might be more difficult for this purpose), I think that would be great.

Don't worry, Snarky.  This and some other things you mentioned are already being ironed out.

I'm glad to hear that. I offered the input not because I want to decide how the thing is run, but because I thought they were good and important ideas that could help improve the ratings process. The most urgent recommendation from me is to design the system so that it attempts to evaluate games on their own terms, valuing their individual strengths, rather than according to their weakest element among some set of pre-defined categories. I feel confident that the panel and organizers see the importance of this, especially since AGS games have a pretty extraordinary range, and many of the (in my opinion) "best" and most interesting titles are pretty unique in one way or another (whether it's META or The Shivah).

Quick question; Snarky and Mills, would you like to join the panel?

Thanks for the invite, Andail. I will have to decline, though. First of all, I don't qualify by reason of not having played anything close to most games in the database. Secondly, I simply cannot make the time commitment that is required. When you start rotating panel members I might be interested in joining for a while, if I'm out of a job at the time or something like that.

[LimpingFish]

Well, for one, it's not up to moderators to decide what awards win or not. Most of the categories were also attained by open discussion. A moderator is a person of established trust, and they should be a part of this, but we should have a say as well.

I was simply comparing the process of setting up and maintaining those particular activities.
And having a say before something has even been put into effect is a little premature.

What's the alternative? We can discuss discussing a possible discussion about maybe thinking about the process of beginning to set up a discussion about setting up a panel? Are we that anally retentive about the symantics of the situation?

Or do we try to move foward with one of the possible systems of rating and cataloging and leave the subject open for input from the community, as a whole, with the possiblity of ongoing change and refinement?

Not to be crude, but we must shit or get off the pot.

In my opinion.

EDIT: If everybody with an interest applies to join the panel, thus declaring they have the time and interest to devote themselves to the process, instead of pointing out problems they have with it, then we'll all get a chance to participate and shape the outcome.

EDIT:

It wisnae me that made Richard game of the month! It was that oldest of old-guards, CJ...

My apologies, SSH

[MillsJROSS]

What's the alternative? We can discuss discussing a possible discussion about maybe thinking about the process of beginning to set up a discussion about setting up a panel? Are we that anally retentive about the symantics of the situation?

I think your perception of discussion is a bit over the top. We're discussing a plan of attack, openly, and then proceding with what we (CJ) thinks is best. However, as already mentioned, the way this thread was brought forth seemed a bit less open then how we usually run things.

It seems there is a sense of urgency to get things done immediately or something bad will happen. Yes, discussion might slow down action, but it might help refine our ideas before they are put to action. Regardless of what we do, we're going to end up fixing something down the line. 

Not to be crude, but we must shit or get off the pot.

I'm just trying to make sure we won't get anything on the seat.

-MillsJROSS

[SSH]

All those who are arguing about this, if they promise to give your games 5 cups, will you shut up?  :

[mätzyboy]

I think there is a point to what people are saying about giving some sort of over all rating of the experience of playing a game, rather that qualifying a set amount of credits for different categories. By setting a limit to how many cups you can get for different aspects of a game you are dictating what is to be considered a good game. In the long run this would lead to a cluster of similar games getting good ratings (the same, of course, goes for bad, mediocre, etc. games...). General properties of this "good game" would probably be a quite typical adventure game of old school type, with stunning graphics, home brewed music etc. and I think this can well have a serious impact on the exposure of games with a different take on the engine/genre/storytelling/graphics/*add interesting aspect of choice here*. An average over the panel's individual appreciation of a game would be more interesting, in my opinion, rather than a quantified set of points for predefined properties.

A quite interesting role of this community is the development of the adventure game genre as a whole. By restricting what is considered a good adventure game, by rewarding a certain type of games, the incentive to broaden and evolve this genre of games, that we all share a love for, is lowered.

However, this is all in good fun, now isn't it? So give it a bashing and see what comes out of it. I think it would be very interesting to see what games will acheive what rating. There are always other channels of promoting your game if you are unhappy with the exposure you get from this site.

Kudos to Andail who is putting an effort in to get something to happen!

[Radiant]

This is preliminary outline to the rating document.

With respect to this, it would seem that any game using a Sierra or Lucas GUI does not have "a fully functional, originally designed interface", thus could get no more than one cup?

I could think of several popular games that this rating system would not do justice. For instance, Larry Vales has good plot and funny dialogue, but crummy graphics. Spooks is well written but appears to lack "advanced scripting". The King's Quest remakes are popular, but lack original plot since the plot was written twenty years ago by Roberta. And META simply doesn't fit in the box period.

It would seem that getting three or four bluecups is excessively rare, and that the categories for one and two would be haphazardly large, containing many games of varying levels of interestingness.

I believe the rating system is a good idea but could stand some refinement.


Oh yeah - would it be possible for a game author to improve his game to get better ratings (e.g. get a re-rate after bugfixing/typofixing/etc)?

Also, I'd like to volunteer for a future panel. I believe the current posts are filled, but at present I don't have time anyways, and in the fall I probably would.

[Vel]

To me, any game that runs smoothly and without any bugs features advanced scripting.

[SinSin]

It looks like the rating system will have to go through various changes as time goes by just like AGS itself. Im sure that the people that rate the games will understand that changes will have to be made to the rules of rating due to things like the advancements made to ags. People should not worry about the ratings im sure that the people who are rating will do a good job and who knows if the ratings are rerevised then maybe the game will score higher who knows.
If you feel that your game should have a higher rating try looking thru your game again to see what you can improve.