Adventure Game Studio

Fellow members.
The games database will soon offer more features. Along with more classifications and genres, there will also be new ratings, carried out by an "expert" panel. The old users' ratings will still be there.

We now need people for this panel. They will be handpicked from the applicants by CJ and his moderators and work anonymously on a special board, where they will rate all AGS games according to guidelines in official documents.
The members of the panel will rotate each month, kicking out the oldest member and inviting a new one. After half a year, the entire panel will have been changed.

In order to apply for this very special position, you need to meet the following criteria:

1. You must be at least 18 years old.
2. You must have been an active member of the community for 2 years or longer.
3. You must not be known to misbehave or break forum policies.
4. You must have played most games in the database, and preferably all the canonical titles.
5. You must be prepared to spend quite a few hours per week playing and rating games. Lots of spare time is essential.

If you meet the criteria and wish to apply, PM member Application (below). Applying does not mean you automatically get picked. Only apply once. If you get selected, someone will PM you. This procedure will begin after a while, to allow plenty of applications.

Thank you.

Please PM this "member" with your application.

As an amateur game maker in an amateur game-making community I have to say I'm a little leery of "experts".

Basically, its been decided that there's too much crap in the games database, all the experts will do is realise & filter the effort worthy games from the rubbish some developers put in there. Also, this IS going ahead. If people from the forums don't want to be involved then you'll just have to put up with whatever the moderators do to filter the games in the DB. So this is basically your chance to be on that panel, or, simply leave it be :)

Yeah, basically at the moment if you're a newbie to the site, and you go to the Games page wanting to find some games to play, it's very difficult to filter the ones worth playing from the rest -- especially in the Short Games category, where there are a lot of good games mixed up with "my first game" ones that don't really deserve to be there.

We're not just adding a rating panel by the way, we'll also be changing the games page to allow you to browse by genre rather than length, but this will obviously require a mammoth effort to categorize all the existing games into genres.

Would we simply be filtering on a creative level? Or would we have powers of dismissal over games that some would perceive as offensive?

Either way, I'd give it a go.

I've seen similar things on other game community websites, but usually it's borne out of necessity since the games are hosted on a central server. Somebody would upload a game in lieu of it being passed for inclusion in the database. If it was deemed worthy, on whatever grounds, it would be added. Otherwise it would be deleted.

Still, it's definately something I would be in favour of.

Will pick of the month still exist? That seems to be used by some to determine good games, since the current pick always features up in the top 5 downloads of the week...

It's good to know that the games database will be upgraded, I always thought further genre and/or theme classification was necessary in order to find games you'd be interested in. I'd apply for that position, but I don't really have as much time as is needed, and long ago, I was known to misbehave. Anyway, good luck to those who will fill these positions!

I think the third rule has a little leyweigh as to how you act around the community now ;)

And of course pick of the month will remain!!!!

I hear they're also going to include CJ's Dish of the Month on the front page too.

I agree that the games database needs more order.  Also, Pick of the Month and additional ratings are things to strive for when making a game, which is good.

QuoteAs an amateur game maker in an amateur game-making community I have to say I'm a little leery of "experts".

Aren't we all?  Many, many people have direct, often negative comments about things for which they have absolutely no practical experience themselves.  It would certainly be nice for the panel to be made up of people who have actually made games (relatively substantial ones, mind) and know about the effort and challenges involved firsthand rather than a bunch of Roger Eberts.  Game designers will tend to recognize things that most people will overlook-- clever scripting, for instance, and things that operate under the hood.  In spite of any argument, people who have no experience making games really can't understand the effort involved or identify clever uses of the engine if they haven't used it themselves.

Could we get some more information about the types of ratings that this anonymous panel is going to be handing out?  Is it just organizing them into "worthwhile" and "not worthwhile" piles?  Are they assigning a score to each game?  What exactly is the panel doing?  How much subjectivity is involved?

I'm glad that some cleaning up is being done on the games database and that it's being done by experienced members of the community (for reasons that ProgZ stated), and I can see the reason for anonymity, I just hope that there's some sort of transparency with regards to the procedures that the panel follows.

I would also love to be a member of this panel at some point, though I don't envy the job that the panel will have digging through the existing backlog of games!

I'm entirely in favour of the librarian role of an expert panel, and self-selected volunteers of two or more years standing in the community would certainly be qualified to sort the games by theme(s).

Rating by "experts"....

Since this is going ahead anyway I will refrain from questioning the decision itself and limit my curiosity to the details of the new system.  Will there be transparency (i.e. evaluation for each component of the game: story, graphics, interface, music, etc.)?  Will there be an overall rating or ranking, and how will the component criteria be weighted?  Will there be room for dissenting opinions, such as in a judicial panels (the American/Canadian supreme courts can have a divided opinion 5-4 or 7-2, for example, with dissenters publishing their opinions).  Who will watch the watchers (or evaluate the evaluators....)?  How will a panel drawn from a comparatively small pool of candidates - many of whom know each other - remain impartial when evaluating projects that their acquaintances and friends have produced?  Won't there at least be the temptation to rank such a game just a little better than maybe it deserves?  Are there procedures in place for declaring conflict of interest?  Will the specific criteria used to evaluate the games be published so that game makers eager for a better rating can conform to the systemic values, and game makers who seek to make a revolutionary name for themselves can do the exact opposite?  Will experimental material that is initially ranked poorly be re-evaluated in hindsight?  As this is a community of learners who generally improve over time, can "bad" games not simply be categorized as "training" or "early work" so that they will be seen for what they really are?  Will the panel be able to "farm out" their evaluation powers where they are not qualified to make an evaluation, such as in a foreign language game or a game that caters to a unique cultural context (in the interests of being fair).  Since "Newbies" are the source of this community's future, shouldn't one seat on the panel be open to them as a group so we can take advantage of their fresh perspective (and make them feel that their opinions are valid)?  Same thing for sub-18 year olds: don't they have the most time to commit to a project like this anyway?

I don't mean to be an agitator, but this thread was unlocked to promote discussion on the subject.

Quote from: Pumaman on Tue 29/05/2007 22:37:00
We're not just adding a rating panel by the way, we'll also be changing the games page to allow you to browse by genre rather than length, but this will obviously require a mammoth effort to categorize all the existing games into genres.

Not sure if this is the right time and place to bring it up, or even if it's been suggested before, but why not use tags instead? Some games are difficult to categorise (WLBSWHEAC comes to mind), and descriptive keywords might be more helpful.

As for the panel - I see the need for it, but I don't feel qualified, so I'm staying out of it.

Glad to hear the database is being reorganized, it will certainly make things more welcoming to new visitors :)  Just one thing, though: 2 years as a member of the forum seems a bit long for a minimum requirement.  Unfortunately I don't have the time to commit to this job, but even if I did, I'm almost a year away from being considered qualified... and it feels like I've been around here for a while (to me, at least--and I don't mean that in a bad way! ;))

Im still a noobie around here, however id like the large games database, im SLOWLY working my way through them. If this system goes ahead, maybe you guys could have the other games in a second catagory thats listed at the end of the first list. :D

The documents which will guide the panel's ratings will be public. However, there will always be a subjective factor involved, since we're dealing with humans.

This is preliminary outline to the rating document. (http://www.andail.com/agsclass.doc)

Of course, there will be as many opinions on this document as there are members in the community.

So, foreign language games are going to suffer a large handicap, as they can't achieve more than one cup?

Just so we're clear.

Regarding the document, does a game have to fulfill all of the bullet points of a category to be rated accordingly or just a majority or all except one or something? If it's the former, I'm hard pressed to name even one game that would qualify for four cups. Innovative interface? Innovative graphics?

Could any InstaGame/RON/Ben Jordan title get more than one cup, as they all use some graphics that weren't originally made for the game in question?

Even for three or four cups you can't require a game to be more than "virtually bug-free". Testing can only show the presence, but not the absence of bugs and all non-trivial pieces of software (other than TeX) are going to have bugs (even AGS).

Quote from: Anym on Wed 30/05/2007 12:29:06
Regarding the document, does a game have to fulfill all of the bullet points of a category to be rated accordingly or just a majority or all except one or something? If it's the former, I'm hard pressed to name even one game that would qualify for four cups. Innovative interface? Innovative graphics?

I think "innovative" is the wrong word in this instance. I'd say they were going more for "fresh".

Nelly Cootalot for example, uses a fresh style of graphics and interface. I wouldn't necessarily call it "innovative" but it is at least distinguishable from most other AGS games.

Many of Yahtzee's games can only get 2 cups because they use RPGMaker music...

I certainly agree that while the very top mark should only go to games that are excellent everywhere, there are plenty of games that entirely neglect something like music which are otherwise excellent. As I usually play with the sound off anyway, its not a big factor for me. Also, if a game gets 4 cups and then someone finds a subtle bug in it, what happens?

There should perhaps be an "unrated" classification for games that plain just do not fit... such as foreign-language-only games

Please try to regard this objectively.

Sure, SSH doesn't play with the sound on, but lots of people find fitting music and sound effects extremely important. I know of people who can't play games with bad soundtracks. Therefore, sound must be accounted for when it comes to a general rating.

Sure, a Russian game with poor translation and grammatical inconsistencies can be GREAT in all other aspects, but the large mainstream audience prefers a correct language.

Sure, InstaGames can be rewarding and fun, but if they would make their own graphics it would move up the overall impression one notch.

You can keep bringing up personal examples of great games until the cows come home, but sooner or later we'd have to settle for some criteria.

My personal favourites (games like Mom's Quest and Aaron's Epic Journey) wouldn't get many cups according to this document (nor would my very own games...), but I'm trying to think outside my own black box here.

So there won't be further theme classification? Like, cartoon, surreal, historical...? I'm sure many people would find it immensely useful while looking for games to play.

Yes, as we said, there will be more genres and classifications as well.
Both regarding settings and genre and some other elements, all browsable.

Something along these lines, but don't think these are set in stone...

(http://screen7.co.uk/junc/alongtheselines.jpg)

Quote from: Andail on Wed 30/05/2007 14:38:15
You can keep bringing up personal examples of great games until the cows come home, but sooner or later we'd have to settle for some criteria.

My personal favourites (games like Mom's Quest and Aaron's Epic Journey) wouldn't get many cups according to this document (nor would my very own games...), but I'm trying to think outside my own black box here.

5 Days a Stranger qualifies for 2 cups at best, which will be perceived as not being best. Many people think that this is the best AGS game out there. That is not a personal feeling. Just because we have to draw the line somewhere is not a valid argument for drawing it in the wrong place. I don't think you can put Mom's Quest and 5 Days in the same league.

On the whole, I do hope the cups system comes into effect soon, even if it isn't quite as I'd like. If possible I'd like to avoid the situation where lots of games that people like will be ignored simply because they don't fit a definition of what a game should be, whereas some games that few actually enjoyed playing would be included, just because they do.

My own view is that there should be one criteria: "we liked it". With a short elaboration attached to each award.
The main problem people have with this is that some games that are good, but not to the taste of the small panel, won't get awarded as they deserve. That's definitely true, but I can see many more good games being ignored under the proposed systems than this one.

Because I don't think most people will agree with me on that, I'll suggest something else. If the goal is inclusiveness, and we don't mind a few false positives, then how about a combination of these approaches? If a game has well rendered backgrounds, competant writing, original music, 60 backgrounds, etc, then this will qualify for at least the level of cups described in the rating document. Aside from that, any game the panel happens to like for whatever reasons will be awarded more cups at their discretion. An instagame game that the panel loves wouldn't be disqualified from getting full cups if they want to award it that.

Seems like a decent compromise to me.

I think you should divide the score into categories, i.e. so many cups for graphics, so many for sound, etc etc. And then a final score, (which might, or might not be an average).

Similar to how a game review might be presented in other media.

Quote from: SSH on Wed 30/05/2007 16:13:33
I don't think you can put Mom's Quest and 5 Days in the same league.

...which I didn't do. I said I personally prefer Mom's Quest, while aware how little mainstream interest there is in such a game. Simply because I prefer slightly obscure, odd games.
5 days a stranger is good, but many casual gamers would probably say the graphics are worse than many other adventure games out there. Thus, the graphics can't be considered top notch, or as good as for instance Apprentice.
So, bringing up individual games doesn't help, because for a fact, not every game can be rated fairly. It's impossible. They will always lack in some compartment.
Rather make new suggestion how the rating should be carried out.

I like scotch's idea: guidelines with common-sense override. Perhaps an idea of the target distributions should also be kept in mind: Only allow 2% of games (out of say 500 existing games, that makes 10) to get 5 cups,  a further 8% can have 4 cups and 20% can have 3 cups.

Subdivided ratings is not a good idea for this purpose, we want one simple number with only a few divisions. I hope we still keep the user ratings, but perhaps reset them since they have been abused in the past.

Yeah, such a distribution seems reasonable.

The reason why the panel should adhere as much as possible to an official document is that people will demand a certain transparency. If they sense the panel is picking personal favourites, they might feel wrongly treated.
Also, the more subjective feelings you allow for the rating, the more will the decision have to be discussed and agreed on by the entire panel. Otherwise there's no guarantee the rating will be "fair".

Not that I have enough time to be in the panel, or that I played a lot of games, but I just realised that I'm younger than 2 years in this forum! 5th of July!

Now, ok to be a member for some time, but as Mash suggested 2 years are a tad a lot!

Now, just one question. Members of the panel, will be choosen amongst those who apply, or you will take all? I would imagine that a panel would need to have some "expertise" somehow, not necessarily in creating, but certainly in playing...

Other than that I would also imagine that a 60-40% weight on public vote vs panel vote, would be the fairer system. As if the public says 5 cups, and the panel says 1 cup there is something fishy going on... The public should still have some power, even if it has been abused before. Needless to say that the end voice should be on the panel, but again, public/audience needs to be heard!

Nik:

Quote from: Andail on Wed 23/05/2007 17:43:45
The old users' ratings will still be there.

The percentage and the blue cups will co-exist. Making a new average seems reduntant, as does sub-divided ratings.

Also, since the panel will be dynamic, we need a constant flow of new applicants. The first panel will soon be set up, but already in one month one has to go and a new one will be invited. 

Next time I'll read the previous posts better (although I actually did read and follow everything. Somehow the above parts missed me... :p) sorry...  :-[

I like the idea of the bell curve for the cup ratings -at least it puts the games in the context of their peers.

The word "original" comes up a lot in your rating criteria but it disadvantages games that intentionally borrow work from other sources (i.e. for the purpose of parody, etc.).  The music in the movie Fantasia was not original, for example, but the main point was that it suited the movie well.  And artists shouldn't be punished for recycling graphics from their own previous works, especially for a sequel which would seem entirely appropriate.  Qualifiers like "appropriate" or "suitable" should be added, so that a game with just one non-original component isn't locked two cups below where it would otherwise deserve.

And why not one seat open for people of 6mo. to 2 years standing?

Being on this panel will be quite a privaledge, as you can tell - what with all the human element involved. Therefore 2 years is a good compromise, a lot of people have been here that long and a fair few of them are still active. Plus there are people who've been here that long but barely speak up. They are still perfectly eligible. That length of time (2yrs) kinda shows that people are dedicated to being a part of this community. Those who haven't been here 2 years may be just as dedicated, they've just gotta wait it out a bit longer like the rest of us have done :)

For this first run of the panel, I seriously doubt Andail is going to back down on the 2 year thing.

BaRoN, I think the common-sense override that scotch and SSH are recommending would deal with things like Fantasia and what-not. I think it sounds like a good idea, and I'm interested to see the new games database layout. EDIT: I mean I'm interested to see how it turns out.

I have just barely been here for two years. ;D But I don't really have any free time any longer. :'(

Quote from: Hudders on Wed 30/05/2007 16:22:35
I think you should divide the score into categories, i.e. so many cups for graphics, so many for sound, etc etc. And then a final score, (which might, or might not be an average).

The existing user ratings will remain for graphics, sound, etc. These already do a good job of rating the sub-categories, and I don't see any point in giving the panel the ability to give each game an "official" graphics or sound rating.

As Andail says, the existing rating system will co-exist with the Panel-awarded Cups; the Games page will display both (eg. "Panel Rating: 3/5 Cups; User Rating: 74%"). Therefore, if you wanted to you could completely ignore the Cup-rating of a game, and just rely on the user rating to find a decent game to play.

QuoteMy own view is that there should be one criteria: "we liked it".

This sounds like a nice idea; but I think that one of the main objectives here is to filter out the poorly made test games that people upload to the games page. They have every right to do so, but by ensuring that these games all get given the lowest cup rating, it enables people to easily ignore them and would motivate people to improve their games before posting them.

Therefore, I think at least a 3-stage rating is required (ie. Bad / Ok / Good); and once you're doing that, you'd may as well split it slightly further to give better feedback. But I certainly don't think that more than 5 cups would be necessary.

QuoteI'm interested to see the new games database layout

The reason that the new layout has not gone live yet is because no games actually have any genres set or cup ratings, so it would be really confusing for people searching it to get no results.

Therefore, the panel's first task will be to award cup ratings to some of the existing games in order to get us started.

More than 5 steps is not feasible, since the nuances will be too subtle for a human mind to distinguish between, in a fair and reasonable way.

Just three steps is not inherently wrong...but then again, while we're doing this enormous backlog of games, we might just as well implement some more steps just to add feedback. When you have only 2 steps (in an extreme case) the pressure will be rather high to make the right decision, since classifying a game as "not approved" is quite harsh. Or in other words: The mistake to rate a game that deserves 3 cups out of 5 with only 2 cups our of 5 is not as big as rating a game that deserves "ok" with "not ok".

About the strict requirements: Naturally, there are heaps of people perfectly competent out there who's only been with the community for 3 months, 7 months, a year etc, but this is a matter of trust. People will have their own creations evaluated and graded by an anonymous panel. They at least deserve to know that the members of that panel have met some minimum criteria, and those limits must be drawn somewhere.

Baron: Again, taking the movie Fantasia is the kind of isolated (and rather weird, truthfully) examples I wish that we could avoid.
Instead of thinking that we're "punishing" people, think of it like this:

Game A and game B are equally good in graphics, plot, scripting etc. However, the creators of game A has gone the lengths to actually produce their own soundtrack, developed to specially suit their game. Game B has just nicked some midi-versions of famous pop-classics. Given that A's soundtrack actually fits their game, it should help push that game towards a higher rating.

So instead of trying to find certain games that happen to be terrific in one aspect while still awful in another, try to look at the various parameters and decide what the general audience (the random players out there who wish to try on our games) will appreciate. 

I didn't mean there should be only one rating level, CJ (3 or 5 is fine), just that in my opinion this would work better if it's openly subjective, based on how much the panel personally like the games as they play them, and how much they estimate the gaming public would enjoy them. Rather than based on a checklist list of things that are meant to be indicative of a good game. The list of what makes a worthy game is subjective in any case, and not well agreed upon, so it seems better to leave it.

It's either "We think X, Y, and Z = good" and going to test all the games against X, Y and Z, or simply going and testing all the games on if we think they are good or not. The latter seems less likely to get weird results, to me. Especially if more than one person is testing each game.

I also think it would make the review process better for whoever volunteers. Reviewing when you aren't meant to influence the score seems like a pretty mundane task.

I agree with the points scotch has raised.

I also think we may be analyzing this a little too much. From my understanding, this plan is simply to catalog games in a way that is easily accessable to a "layman" from outside the community, and expanding the search options to be more flexible. And removing those entries that, really, offer little to prospective downloaders.

And I'm all for removing the "Oh look! I made a game LOL!" entries that clog the database. I'm also aware that this can be construde as censorship, or an elitist few dictating what is and isn't worthy. I'm not interested in debating the symantics of rating Yatzhee's games over Anonymous Inept Developer no.362's games. I presume, on the whole, the distinction between "acceptable" games over those suitable for removal would be fairly clear.

If you want to piss up a "game" to show to your classmates fine, but you shouldn't automatically have the right to a listing in the game database if your game is obviously lacking any "artistic" merit (and yes, I'm aware that people's impressions of artistic merit can fluctuate wildy, but I'm sure we can all agree on the basics) or contains questionable material. The database is part of the "official" AGS homepage, and, however ridiculous it may seem, its contents may be judged indictive of the community as a whole.

I think rewarding a game for a bit of clever scripting is taking it a little too far, as it really isn't something that would be noticable (or important) to somebody just looking for a fun game to play. I certainly wouldn't be interested in examining a game to that extent.

What happen to great looking games which suffer from a handful of game crashing errors like No-Action Jackson?

What happen to games with graphics that do not age very well? In several case we're talking about the best games from 6 and 5 years old games that will be judged next to the games released in the last months. And what will happen in the next 6 years? Will the cup rating suffer from inflation as the quality of games keep on increasing?

Also regarding soundtrack, while originality is great, having a soundtrack that fit the game and the scenes' mood is probably more important than whether or not it comes from a RPG maker's sample soundtrack. Originality doesn't always mean that it's good or that it fits the game.

I too would prefer some sort of system that rate every aspects individually, then make an average or a customized average where some aspects worth a bit more or a little bit less in the balance, but also where some games can win bonus points for original soundtrack, innovation and unique features, because the classifying system that is currently proposed with its pass or crash feel can be unfair for several games, even AGS awards winners that have a negliged aspects.

Quote from: Pumaman on Wed 30/05/2007 19:12:54

Quote from: Hudders on Wed 30/05/2007 16:22:35
I think you should divide the score into categories, i.e. so many cups for graphics, so many for sound, etc etc. And then a final score, (which might, or might not be an average).

The existing user ratings will remain for graphics, sound, etc. These already do a good job of rating the sub-categories, and I don't see any point in giving the panel the ability to give each game an "official" graphics or sound rating.

As Andail says, the existing rating system will co-exist with the Panel-awarded Cups; the Games page will display both (eg. "Panel Rating: 3/5 Cups; User Rating: 74%"). Therefore, if you wanted to you could completely ignore the Cup-rating of a game, and just rely on the user rating to find a decent game to play.

Yes, but this time the rating will actually means something if it is done by knowledgable, dedicated and responsible peers instead of a legion of fanboys, even if I'm still unsure on how such system will take into account friendships and grudges between judges and creators.

Also, 2 categories that should be in the search engine:
AGS Award winners
AGS Award nominees

Oh yes, I think award nominees should be added to the database. And really there should be a way to exclude the booby-prize awards from that, too.

I might inform you that the first panel is filled, and the members in question will soon be invited.
Further applications now will fill empty spots from next month and forward.

(this is slightly off-topic, sorry about that)

Seeing that we're discussing the games database, would it be impossible/impractical to include an option to delete your own games from the database? It's kind of embarrassing to see some of my old crappy games there and I'd like to remove them from the database (what the hell was I thinking when I made those games?!). The games are really _not worth playing_ and I don't think the download links even work anymore.

Or would that mess the database up somehow? I'm just asking, because years ago I added the infamous Goldlagoon to the Adventure developers' games database and it seems to be next to impossible to remove the game from there now. I've actually contacted Erwin and some other people about removing the entry, but nothing has happened. I just don't think that the game is going to be finished in the next decade or so, so there's no point in having it in the games database (Erwin, if you read this, is there anything you can do?).

QuoteI'm still unsure on how such system will take into account friendships and grudges between judges and creators.

Well, we're expecting some sort of professionalism from the panel -- that's why the requirements to be on it are as they are. Also, it's a panel of judges, so if one particular judge was a fanboy or enemy of a game author, the others should outnumber them.

QuoteSeeing that we're discussing the games database, would it be impossible/impractical to include an option to delete your own games from the database

It would be possible, but I'm not really in favour because by removing your game you're also removing the reviews and ratings that people have taken time out to submit.

However, if anyone has a game that they added to the games page, and really really want to remove, PM me and I can sort it out.

Pity I have no time. I fit every other category quite well, espeically the "played all games" bit.

Andail: The panel is already filled you say. Does this also mean that the rating system you proposed is now final, or is it still open for debate? It might be a good idea to use a system that people can agree on, before we get stuck with one for years.

I also really dislike the proposed clinical checklist system. It may be suitable for commercial games where all aspects of a game can be argued are equally worthy since the developers have enough people working on it, but who are we kidding? We're making amateur adventure games, mostly on our own. Any game I made would only get 3 bluecups at best, or more likely a 2, simply because of the art and music criteria. In fact, that seems to apply to almost every single AGS game ever made.

We're not making commerical games. If someone makes a brilliant game with pedestrian graphics I would far prefer it got a rating of "Highly recommended - has an engaging plot, believable characters, and stimulating puzzles. The graphics may not be up to everyone's liking though." instead of "2 bluecups" or "3 bluecupes" which seems to be what the proposed system would result int.

My vote goes to a rating system of 3 levels: Not recommended, recommended, and highly recommended. Including a short description of the reasons why, except perhaps for the not recommended option.

I see Esseb's point, but the following...

QuoteI also really dislike the proposed clinical checklist system. It may be suitable for commercial games where all aspects of a game can be argued are equally worthy since the developers have enough people working on it, but who are we kidding? We're making amateur adventure games, mostly on our own. Any game I made would only get 3 bluecups at best, or more likely a 2, simply because of the art and music criteria. In fact, that seems to apply to almost every single AGS game ever made

...isn't helping anyone. If a game has "inferior" graphics it should be rated as such, regardless of the ins and outs of how the graphics were created. Plenty of games made by a single person have "good" graphics, or "good" music, etc.

A single developer who's game has "good" graphics shouldn't be rewarded because of that fact, just as a single developer whose game has "bad" graphics shouldn't have allowances made for them.

Should a game be punished for having bad music, or no music? If it makes an impact on the enjoyability of a game...then yes, imo.

Critics, and that's what any panel formed will inherently be, can only put forth an opinion, not a fact written in stone. And as we all know, opinions are like snowflakes.

Quote from: Esseb on Sat 02/06/2007 00:18:34
Any game I made would only get 3 bluecups at best, or more likely a 2, simply because of the art and music criteria. In fact, that seems to apply to almost every single AGS game ever made.

If all games are rated on the same scale, there is no problem.

I'm sure that unless your game is infinitely superior to the average, it's not going to get a top score; regardless of what the "top" is. That's as it should be. This is about rewarding the incredible efforts that some achieve and weeding out the dross that is going to put people off AGS altogether. If your game is average, it deserves an average score; whether this is 2 cups or forty-eight cups, if everything's being rated on the same criteria what does it matter?

You might feel like you're being being penalised for art and music; but if someone was to go and create something spectacular, shouldn't they be rewarded?

I really don't like this idea very much. I agree we do need some new rating system, but I'm not sure this is the way to go.

For one, I don't like the idea of getting rid of panel members. Mainly, for consistency sake. This isn't to say panel members should be set in stone. I just think that if your doing a good job of playing games and voting to some criteria, then there's no reason to get rid of you. You're being productive, and we seem to be under the assumption that there's an endless supply of people who are qualified AND willing to replace people. Get all the people on the panel you can, is really what I'm getting at. Including people under the two year mark. Seems to me, if someone can play games and rate them, it's more accurate if their not embedded within this community as much. If anything, newer people would tend to be more subjective. Also, with the backlog of games it seems you need all the help you can get.

I'm iffish about the criteria being used to judge games. It could be my lack of understanding on how this is going to work, but it just doesn't sit well with me. It seems that there would be games that I love that would fall on the negative side. Is each point getting it's own cup? To me that would make more sense. In this way people could pick and choose what is more important to them when looking at a breakdown of how many cups were given in certain areas.

I don't know how the information will be displayed, just cups, I assume. But I think there has to be some breakdown of how the rating was gotten. The bad graphics lowered the rating to something. Otherwise the cup system is really no better than the percentage system.

I agree with scotch. If this is rating is going to be used more as a filter for people playing the games, than constructive criticism for the actual author of the game, it should be more based on the subjective, "We Liked it." Then some small reasons why.

If a game is buggy just give it some sort of icon that represents how buggy, and let players choose from that.

While it's evident that a lot of thought has been put into this, it also seems like this is being rushed without really having a good discussion about it. It doesn't seem we need a panel till we've discussed all the kinks out. I also just see a panel of people slowly losing interest.

I don't think we need a "dedicated" panel. I think we need a better rating system that is open to everyone. I think we don't need to add on to the old rating system, but almost wipe it clean. I think we need a rating system that is specifically helpful to the author of the game, as well as used to filter out worse games. And I think we then need a panel of people to use the general masses oppinion and rate it according to that.

I really think this needs to be discussed further, and if it wasn't so late in the night for me, I would elaborate and suggest criteria that I think should be in there. The only gem of an idea here is breaking things up into genre's, otherwise, I'm opposed to this way of going about the voting system.

-MillsJROSS

Some quick responses.

First of all, what we have on our hands is a situation, that needs to be solved, preferably quickly.
The situation is basically that 1/3-1/2 of the database entries are dead links, and another 1/4 are made up by games so bad they're not worth downloading.

Another part of this situation is that we have a system of rating games now (the users' percentage) that is far from consistent; some games have a hundred votes, some have only one, and people most often just vote 100% for their favourite games and 0% for games they hate, or games whose authors they hate. Furthermore, people don't have any criteria to weigh their votes against.
Considering that so many members are suddenly very concerned with how the rating is carried out, I find it peculiar that this voting has never been brought to attention. Sure, the verdict of an official panel will weigh more, but the users' percentages have for very long been the only system, and bound to have made huge impact on people's downloading habits.

In order to finally deal with this situation (which nobody has really cared about, since AGS-ers themselves rarely deal with the games database) I've suggested to put together a group of people who will, besides fixing dead links and classifications, also provide a simple rating of their own, that will co-exist with the old ratings.

This group is small enough to allow for quick decisions and close-knit cooperation, but large enough to ensure some sort of validity.

This group is, as we speak, trying to design as good a rating document as possible. Trust me, they are working hard and the results are promising.
And yes, any official document the group will present will be regarded as guidelines and not a set of rules. The guidelines will cater for consistency and the integrity of the individual panel member will cater for common sense.

To allow every single member of this community to have their say is simply not viable. Everyone has their own favourite game, and everyone will be subjective, and everyone will want to argue a bit for argument's sake.

We have some 800 titles to wade through. We need to be effective and professional. We're trying a method now, and this method has been approved by CJ. If, in the end, your very own favourite little darling game ends up with one cup less than you had expected, it's not much to do about it. We can't accomodate for all. In return, we might end up with a games database that looks like somebody actually cared about it.

PS:
The document I presented first was only to have something to start out from, something to look at. The panel members are presenting their own ideas and we'll try to agree on something asap.
Not every AGS-member can have their wishes accomodated for, especially when they aren't prepared to take part in the very labour (signing up for the panel) of sorting the database themselves.

Also, if you don't like the rating you should learn to deal with it just like you deal with the scores players give you.  You can't change those, either.  In the end it's just an opinion, and if you really feel your game is better than that then more power to you.  No one on the panel is going to intentionally slam any game in the database, and some games that haven't been given much notice (older games, non-award winners) will actually benefit as a result of the ags cup system, since the panel ratings could well be higher than the average ags scores for one reason or another.  Cheer up and don't be so down on a process just because it's new or seems scary.  Many game sites have a site rating and it doesn't hurt them.  DON'T PANIC!

I don't think panic would be the word I'd use, this is something I don't like, I said so. If the system is used, I will adjust, I never said I wouldn't.

QuoteFirst of all, what we have on our hands is a situation, that needs to be solved, preferably quickly.
The situation is basically that 1/3-1/2 of the database entries are dead links, and another 1/4 are made up by games so bad they're not worth downloading.

It seems to me, that dead links aren't an enormous problem. Remove the game (Which Ace Quest has been part of this category for a while). Or put it in a section of dead links (to perhaps keep the voting system intact), and if people want to bring their game back from dead links then they can always do so. Then either contact the author through PM, and tell them if they want to readd their game, their welcome. That's a problem that could be catalogued in a day, and dealt with by a panel, but has little to do with voting.

If 1/4 of the games are really bad, then people will download them and find out for themselves. This in itself isn't a problem. The problem of our voting system isn't the unplayable games, it's the good ones that never got found.

QuoteAnother part of this situation is that we have a system of rating games now (the users' percentage) that is far from consistent; some games have a hundred votes, some have only one, and people most often just vote 100% for their favourite games and 0% for games they hate, or games whose authors they hate. Furthermore, people don't have any criteria to weigh their votes against.

That's been a problem for a while, and I agree completely that something needs to be done about it.  But to say there is no criteria would be a lie. The voting system does have criteria, it's a question of whether people are sticking to it or not.

QuoteConsidering that so many members are suddenly very concerned with how the rating is carried out, I find it peculiar that this voting has never been brought to attention. Sure, the verdict of an official panel will weigh more, but the users' percentages have for very long been the only system, and bound to have made huge impact on people's downloading habits.

Um...I find it peculiar that you don't remember this voting system being brought to attention on, I believe, two fairly big threads.

AGSers definitely care how this system is being done with.

QuoteWe have some 800 titles to wade through. We need to be effective and professional. We're trying a method now, and this method has been approved by CJ. If, in the end, your very own favourite little darling game ends up with one cup less than you had expected, it's not much to do about it. We can't accomodate for all. In return, we might end up with a games database that looks like somebody actually cared about it.

This seems like a very condescending statement to me, if it isn't I apologize. Bit it seems as if you're saying I'm arguing because my games won't get the points I think they deserve, which is far from my mind. I'm not concerned with games I've created. If I was, my games wouldn't have dead links. Now maybe you are talking generaly, but your post is, more or less, in response to mine, so I interpreted as a comment at me.

QuoteNot every AGS-member can have their wishes accomodated for, especially when they aren't prepared to take part in the very labour (signing up for the panel) of sorting the database themselves.

Once again, don't know if this is general or not, if it is, I once again apologize. The last thing I want to do is be condescending myself. I for one, am prepared to take part in the very labour, but I don't agree with the system given to me, so I don't feel inclined to do volunteer my work.

It just feels like there's a lot of discussion behind closed doors when this is something for the community. It wasn't really brought forth to the community, but it's more like, this is what we're going to do, deal with it. Well, I for one will deal with it, but I don't want to deal with it.

Once again looking and ProgZmax's post I want to repeat myself. My game has nothing to do with this discussion. I know people think that I always have AQ on my mind 100%, and perhaps two or three years ago, you might have been right. But my game has nothing to do with disliking this voting system. I just don't like this voting system on it's own merits.

QuoteCheer up and don't be so down on a process just because it's new or seems scary.  Many game sites have a site rating and it doesn't hurt them.  DON'T PANIC!

I'm not uncheery. I'm just not going to be silent when I don't particularly like something. I'm not a scared child waiting for the boogey voting sytem to come and get me, I'm an adult, who perceives this idea to need work, and it needs to be discussed on the forums, rather than in some other silly little place where the community can't offer oppinions.

-MillsJROSS

I would also argue that dead links are definately not in need of a panel, while bad games, which seem to be in need, are only a matter of common sense... At least, somehow I've never downloaded a "bad" game, only twice to actually see what it was about, but fully aware it was a game worth of the penis award all the way!

What I would suggest maybe, is that the games have necessarily 2 scrennies, and some other info of the game, in order to enter the database. That way you basically get what you get in a store while browsing through the PC games section. You don't necessarily need the review to know if a game will suck, be brilliant, or something in between. At least, not in most cases... I have noticed that a lot of games are lacking most info...

But I do prefer to see 2 screens with graphics, than listen to a panel tell me "this is good". (If that panel had a specific name, which I trust (helm, Andail, loominous), telling me that the game has amazing graphics, then I would trust them...).

Right now the panel, somehow is not filled with experts, is it?

Anyways. I feel that I have little to say really. Since I don't use the rating system anyways, I don't mind, and I'm sure that there has been thought behind the whole panel idea... Anyway you go, will be fine for me :)

Oh dear, this is turning into an argument, isn't it? Sorry to contribute to that, if you're looking for the strictly constructive parts of this post you can skip down to under the quotes. But...

It's starting to sound a bit like you don't want input. I thought the point of making the work-in-progress guidelines public was partly to solicit comments and suggestions.  ???

Quote from: Andail on Sat 02/06/2007 10:29:43
Some quick responses.

First of all, what we have on our hands is a situation, that needs to be solved, preferably quickly.
The situation is basically that 1/3-1/2 of the database entries are dead links, and another 1/4 are made up by games so bad they're not worth downloading.

Yeah, it is a problem, and it's great that it's being fixed. But it's been like that for years, and I don't see why all of a sudden it's such a great hurry that we can't take a couple of weeks to talk about it and see if we can find a solution most people will be happy with. I don't think we'll have another chance to get it right, unless we want to ask people to go through 800+ games again.

I applaud the job the panel and the organizers are doing, but some public consultation can do wonders for the general acceptance and support for the system.

QuoteAnother part of this situation is that we have a system of rating games now (the users' percentage) that is far from consistent; some games have a hundred votes, some have only one, and people most often just vote 100% for their favourite games and 0% for games they hate, or games whose authors they hate. Furthermore, people don't have any criteria to weigh their votes against.
Considering that so many members are suddenly very concerned with how the rating is carried out, I find it peculiar that this voting has never been brought to attention. Sure, the verdict of an official panel will weigh more, but the users' percentages have for very long been the only system, and bound to have made huge impact on people's downloading habits.

Well, that goes for all of us, doesn't it? You, me, the people on the panel, CJ... we have all (AFAIK) more or less ignored the games database until relatively recently. I think this current interest expresses people's belief that we can get something that is actually useful, something that works.

QuotePS:
The document I presented first was only to have something to start out from, something to look at. The panel members are presenting their own ideas and we'll try to agree on something asap.
Not every AGS-member can have their wishes accomodated for, especially when they aren't prepared to take part in the very labour (signing up for the panel) of sorting the database themselves.

Seems a bit unfair to chastise people for not signing up when several of the posters have complained that they can't, because of the restrictions (member for more than two years, played most of the games in the database), and when the work is also so demanding. (No, I can't spend "quite a few hours" each week doing this work. I don't have plenty of spare time!)

OK, so those are my objections to the tone of the discussion. Here are my comments and suggestions on the procedures:

I like the idea of a ratings panel, and I agree that having some sort of standard is useful. For comparison, here (http://www.adventuregamers.com/editorial.php) is the guide to ratings on Adventure Gamers, which in my experience work pretty well. The combination of an overall points-based judgment and a summary of pros/cons gives a pretty good idea of what a game is like. Of course, commercial games tend to have more consistency in production values than our homebrew titles, so some adaptation is probably necessary.

Looking at the draft guidelines, I think they are overall pretty reasonable. What I'm missing is an explanation for what it takes for a game to get a certain rating. In this thread, people seem to assume that they need to fulfill all the criteria of that category. In my opinion, that would be a bad idea. It means that all games would be rated based on their weakest component, and wouldn't at all reward games that excel in one, or a couple of areas. A system like that seems to not evaluate games on their own terms, either. (One game may be all about telling a good story, while another may be all about solving devious puzzles, and a third about the audiovisual experience.)

Instead, I would propose that the bullet points under each category be interpreted as possible reasons to award that rating. So a game could get three blue cups for featuring a "thoughtfully composed plot" and original, consistent graphics, and being bug-free, even though the interface is just a re-skinned version of the default Sierra cursors, and there are no elements that feature "advanced scripting".

Perhaps a rule of thumb for assigning a category is that it fulfills two of the bullet points of that category, or one of the bullet points of the category above, and more or less fulfills the bullet points of the category below (with perhaps one exception). This would seem to strike a balance between emphasizing a game's strengths, and requiring some consistent level of quality of the other aspects. I guess I see it as a formalized version of the "common sense rule" people have mentioned.

Other comments: I would list story and puzzles as separate points, since a game can have a great story with run-of-the-mill puzzles (or deliberately avoid "puzzles" in favor of an "interactive story" approach). Also, I think "gameplay" is better than puzzles, because there are other types of interactivity an adventure game can make use of. I wouldn't imply in the guidelines that only "elaborately conceived puzzles" can reach the top rating, because good gameplay is something that fits the game and is fun, not necessarily something that is twistingly complex.

I don't understand the mentions of "innovative style" with regards to the graphics. Which is to say I don't understand what it means ("innovative" relative to other AGS games? To adventure games? To all computer games? To art and animation in general?) and couldn't really point to a game that I would consider graphically innovative (maybe READY), and that I don't understand why it should be a criterion for rating. If someone made a game in the graphical style of The Last Express, that would be highly unique, very attractive, and I would want to give it 4 Blue Cups for the graphics, even though it's not technically "innovative". Maybe a word like "distinctive" would be better, although I still hesitate about the way it discriminates against games working in established styles.

The requirement for graphics originally made for the game should be loosened ("designed by the creators", maybe) to allow sequels to reuse sprites etc.

In general, I think there's an overemphasis on originality. This is a matter of opinion, I know, but personally I care much more about the result than about the process. If the music is perfect for the game, I don't think it really matters if it was originally made for it or for something else. It's not like we rate Tarantino movies down because the soundtrack features pop tunes rather than an originally composed score. The same with graphics: If the backgrounds look good, what matter that bits are copied from various Sierra games (like in KQ2VGA+)? It's only when it impacts quality that I care: bad paintovers of too-familiar sprites, or a game that takes place completely on backgrounds from other games, or cheesy MIDI tunes that are immediately recognizeable from some other source. If a game feels second-hand (and this goes for plot, too), that is a negative.

However, I know that others (especially the artists in the respective disciplines) feel differently. And it's difficult to find a solution that treats crappy Monkey Island sequels, more respectable games like the Maniac Mansion Mania series, and high-quality but not 100% original titles like KQ2VGA+ and 5DAS all fairly. Maybe a compromise would be to break out originality as its own bullet point, and evaluate it separately from the actual quality and effectiveness of the elements themselves.

Overall, a good start on the ratings system, and I think that with some discussion and common sense, we'll come to a system that highlights good games in a reasonable manner.

I am new, haven't created a single game and only played a few, but the proposed rating system looks overly complicated.

I would do it this way:
Every panel  member (good idea btw) assesses several aspects of a game (from 0 to 5)
- scripting
- graphics
- audio
- gameplay/fun
- whatever you think is important

Then you just average the panels marks for each category. After that you average the marks for the categories and have a single value for each game between 0 and 5. This would be fairly consistent. Bad audio or graphics alone would not ruin the evaluation of a game (0 or 1 cup) if the other aspects are good and vice versa.

Edit: If you have enough panel members you could also discard the best and worst marks for each category, like in figure skating.

I think this community is too insular. Personally, I would look at a game like, for arguments sake, Richard Longhurst and the Box That Ate Time, a game which this community seems to have a soft spot for or regard as some kind of existential comment on game design (SSH even made it Game of the Month way back when), and all I see is, well...a pile. It's a in-joke that nobody outside the old guard would get, or for that matter, care about. As far as I'm concerned, a game such as this is ripe for evaluation from the point of view of an "outsider", and as such I would rate it 1 out of 5, or 1 cup or whatever. The user percentage may be 80%+, but the "offical" rating would reflect the quality of the game from a general point of view.

As for "Joke" games, I say remove them for the general database and be done with it. A waste of time and resources, imho. I'm sorry if that sounds a little harah, but when good, original games are lost between piles of somebody's ego wankery, it's time for a change.

Rant over.

Snarky, the speed issue isn't primarily because it has to be done before a certain date, it's because the project is more likely to die out if it turns into a very lengthy process.

Also, this current interest is a reaction to my initiative. There have been plenty of discussions in the past, but they never really had any fruitful outcome. Sure, people have done a great job uploading and mirroring various catalogues of games (Scotch among others), but the official AGS database has only slowly deteriorated.
To make things happen you sometimes need more decisive actions and less talk.

I'm not against input (I'm very willing to give up my initial ideas about the rating - and have evidently done so in the designated forum to give room for better ideas) but if every single member of this community should be able to bring the progress to a halt just because they envision how their favourite game will be rated low, nothing will happen. After all, this is CJ's personal site, not the UN.

All that will happen is that the games database will have more working links, better classifications and a rating. People should be very happy that there exist a group who will deal with this.

PS:
What Limping said. There should be a more objective evaluation of the games, without looking at how many fans they gathered in the community at the time it was created.

Quote from: MillsJROSS on Sat 02/06/2007 15:35:07
It just feels like there's a lot of discussion behind closed doors when this is something for the community. It wasn't really brought forth to the community, but it's more like, this is what we're going to do, deal with it.

I appreciate this and understand where you're coming from. Basically, we've had several public discussions on how the games database should work, and we usually end up without a consensus and nothing happens. The reason this initiative was discussed behind closed doors was so that we could make a decision and get on with it.

I apologise if it feels like you guys have been left out of the discussion, but there's always room for changes, which is partly what this thread is about.

It wisnae me that made Richard game of the month! It was that oldest of old-guards, CJ...

Aren't the ratings being done with the player in mind, as opposed to developers? In that case, as mentioned before, I'm not sure how originality affects anything (I'd be more on the lookout for graphical consistency), and scripting... I suppose if it makes things easier for the player. I have no worries about favourite games, but I can think of quite a few which would be rated pretty low with the current classifications, which I'd think would be worthy of attention they wouldn't get: Pleurghburg, Apprentice 2, QfG4.5 (come on, it was fun :D), MMD, RON, Ben Jordan, META, Robotragedy 2, etc., etc. I'm pretty sure it'll not be just isolated cases which may suffer.

I dunno...I'm just giving my input for what it's worth. A rating is just a rating, and if something goes a bit too far off, there's always the user votes to balance things out.

Quote from: Andail on Sat 02/06/2007 19:34:26
Snarky, the speed issue isn't primarily because it has to be done before a certain date, it's because the project is more likely to die out if it turns into a very lengthy process.

Also, this current interest is a reaction to my initiative. There have been plenty of discussions in the past, but they never really had any fruitful outcome. Sure, people have done a great job uploading and mirroring various catalogues of games (Scotch among others), but the official AGS database has only slowly deteriorated.
To make things happen you sometimes need more decisive actions and less talk.

I'm not against input (I'm very willing to give up my initial ideas about the rating - and have evidently done so in the designated forum to give room for better ideas) but if every single member of this community should be able to bring the progress to a halt just because they envision how their favourite game will be rated low, nothing will happen. After all, this is CJ's personal site, not the UN.

All that will happen is that the games database will have more working links, better classifications and a rating. People should be very happy that there exist a group who will deal with this.

It might just be me, but this sound a tiny little bit like "Let's marry now because in a month we may not love each other anymore." ;)

If you want to rush the project, feel free to do so, but for a project of such proportion, I too think that it would be better to take the time to debate the thing openly, propose new ideas, reach a concensus or find a system that will please nearly everyone or at least a big majority and get the thing done correctly after the first try, rather than rush everything, end up with a system that only please an handful, get people involved and in a few years from now have a similar discussion like we're having right now to replace the rating system with another one, which would only piss off all the persons who invested energies in this proposed rating system.

It is also important that we reach a system that will convince as many persons as possible because it's these persons who'll propose their help and time to play and rate each games, and like some persons pointed earlier, if they don't like the rating system they won't be interested in joining the project.

So I say, let's not skip steps and let's openly propose 2 or 3 models and throw ideas to optimize those models. Then we could find a dozen of games of all kind and all backgrounds and review them with each of these models to see if they work well, see their strength, weakness and correct them if we need to, then we'll choose or vote for one model which will be the final one and get started with it. Who know, the original model could have been the best one all along, but if we don't compare it to the others, we'll never know for sure. It ain't the UN, but let's think of it more like presidential election. In the end one system will be chosen over the other by a majority and it's this system that will be used. If there must absolutly have one system, let's at least try them all, and give every systems their chance before picking one that we'll use for years.

However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

Quote from: BlueSkirt on Sat 02/06/2007 22:13:49
However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

Nobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

There's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.

I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

The formation of a rating, and cataloging, body should be arranged and instigated by one or more moderators, which I believe it has been, since we trust these people to maintain our forums, run the awards, etc.

I don't see why this should prove any different.

Well said that man.

QuoteAren't the ratings being done with the player in mind, as opposed to developers?

The short answer to this is no.  The long answer is sort-of.  The idea is to be as fair as possible to the developers while taking in mind the general popularity of the game and balancing that with the game's actual strengths/weaknesses.  It's impossible to be 100% objective but it's not impossible to strive for objectivity.  The system being established is intended to be as fair as possible to the author while remaining honest.

Also Mills, I wasn't implying that you're a scared child but I stand by the statement that you're preaching doom without seeing the system in action.  I really, really dislike it when people attack something without seeing a working example from which to form a complete opinion.  Right now you (and not you alone, really) are forming an opinion based on a single document that is a WIP of some basic criteria used for rating the games.  Meanwhile, if you have some suggestions to improve the document format then by all means share them.

QuoteWhat I'm missing is an explanation for what it takes for a game to get a certain rating. In this thread, people seem to assume that they need to fulfill all the criteria of that category.

Don't worry, Snarky.  This and some other things you mentioned are already being ironed out.


And yeah, the moderators do understand and care about the frustration and questions you guys have, I know I do.  Just try not to overreact before the picture is complete and there are some actual examples of the ratings in action.  Can we at least agree to that? ;)

Quote from: LimpingFish on Sun 03/06/2007 00:01:08

Quote from: BlueSkirt on Sat 02/06/2007 22:13:49
However, if people need to do something to stay interested, the database entries with download link that aren't working, and the joke games can be dealt with right now in the meantime that we all or that a majority of us agree on a system.

Nobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

There's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.

I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

Alright, alright.

*Blueskirt adds "concensus" to the list of word he'll never ever use again.

I did not say (or meant to say) it had to please everybody, I said it had to please a majority, 50%+1. Chances are we'll be able to see the rating system in action before it's on. If the rating system is really close to be finalized, then nevermind what I say, but if it were to take another bunch days and if in the meantime we can find another system that could be a good or better alternative, I fail to see how one additionnal day to rate the games with the second system, 2 additionnal days to do some corrections and up to 4 days to vote (depending of who between everyone who can apply to the panel, the panel or the moderators get the power to decide on which system to use), will considerably delay the process.

I too personally trust our moderators to decide what is the best for us and I don't have anything against the one proposed, even if I'd like to see it in action. I'd just rather avoid seeing this topic brought up again 7 years from now because we were too impatients to wait 7 more days when the rating system was created. Things are easier to correct right now than they will be when people will have invested years to rate all those games.

QuoteI appreciate this and understand where you're coming from. Basically, we've had several public discussions on how the games database should work, and we usually end up without a consensus and nothing happens. The reason this initiative was discussed behind closed doors was so that we could make a decision and get on with it.

I don't agree, because it seems one of the times we discussed things we actually got the system we have now. I mean we discussed among the forum, and things happened. Obviously, the system we have now wasn't as great as we expected, but we've learned from it, and it was a step up from our last system.

QuoteNobody said this was a democratic process. A system of rating that pleases everybody is more or less an impossibility. Either all 3000+ members of this community have a say in the process, or it goes to a group of regular, longtime members to sort out.

No one ever said this was a democracy, but we've mostly been very good about discussing things before a decision is made. Yes, at some point someone, CJ, has to give the greenlight to do something. However, as a longtime member of this forum, I very much would like to express my oppinions. So I am doing just that.

QuoteThere's no room for "Well every member doesn't need to have a say, but I think that...", which will only prove to slow the job at hand to a crawl and mire it in petty forum politics which may arise out of someboby's perceived dissatisfaction with the process.


I, for one, would like to see some action now, rather than sit through everybody's two cents on the subject.

Since when have we really had that much forum politics? I think what we need to do is discuss and then have action. Obtain as many oppinions as possible and move forward with that. I think there are people with good ideas, and I don't feel that we should not care about what people have to say.

QuoteThe formation of a rating, and cataloging, body should be arranged and instigated by one or more moderators, which I believe it has been, since we trust these people to maintain our forums, run the awards, etc.

I don't see why this should prove any different.

Well, for one, it's not up to moderators to decide what awards win or not. Most of the categories were also attained by open discussion. A moderator is a person of established trust, and they should be a part of this, but we should have a say as well.

Quote
Also Mills, I wasn't implying that you're a scared child but I stand by the statement that you're preaching doom without seeing the system in action.  I really, really dislike it when people attack something without seeing a working example from which to form a complete opinion.  Right now you (and not you alone, really) are forming an opinion based on a single document that is a WIP of some basic criteria used for rating the games.  Meanwhile, if you have some suggestions to improve the document format then by all means share them.

Of course I don't agree with this. All I'm getting now is promises that things are being done and snippets of information as to what is happening. I'm not in the know, so to speak. Perhaps, I'd be less prone to be against this material, if all the discussions were at least open to the public eye. I don't have to see something in action to form an oppinion about it. I can concede the fact that I may be wrong, I may be crazy, but it just may be a lunatic your looking for. If this system works, great! But I just in general, don't like feeling that my oppinions on the subject seem not as weighted as people in the current secret panel.

Ideas:
One, I don't like a forum age requirement for the panel. As mentioned, newer members aren't embedded into this community have a more subjective oppinion of games. As we're appealing to people who are essentially just browsing the game section, without being in a community, it seems newer members voting habits would be more in touch with said browsers.

Two, if a panelist is a hard working individual, than he/she should not have a predestined time to leave the panel. Don't get rid of people who are doing a good job. Just don't cap off the amount of people in the panel, either.

Three, I think a good system would almost be what rotten tomatoe has. In this fashion people can  either go by the critics or the masses. With the only criteria being, "I liked it," "I didn't like it."  And if people wanted to, right up comments/reviews so maybe the author could get some vital feedback, too all the merrier.

Four, the moderators should be reponsible for getting rid of dead links on the game database, whether that's by moving it to a deadlink section that can come back alive, if the link is reestablished, or just deleting them. They should also be responsible for genre selection. Moderators should, to me, have the power to move things accordingly, and to keep things running smoothly, and that's the end of the line.

And I'm done, for now.

-MillsJROSS

Okay, I'll discuss your points.  Note that I'm not the one who established the ruleset for selecting judges (Andail did) but I happen to agree with it.

The age requirement.  I think this is reasonable because while not every adult behaves responsibly, I would argue that most understand responsibility better than children/teens do and won't flake out as much.

Panel rotation.  I think some of the expectation here is to bring in new people on a regular basis to make the ratings representative of a community rather than 7 people.  I understand that if you're a hard working person you'd like to stay on the panel and this is perhaps something that could be flexible, though I think it really depends on how the community reacts to the initial ratings.  I definitely think a maximum panel limit is necessary to keep the group communicating effectively.

'I liked it/I didn't like it' doesn't really tell me anything, personally.  I'm more concerned about why something received the rating it did than what the reviewer rated it as.  There will be a comments section for judges to explain the rating if it's warranted and will provide a more detailed explanation behind the cup rating (were there serious flaws that prevented it from being completed, etc).

CJ's been working on the db system to allow judges to adjust genre, fix broken links and so on as they rate the games.  The author will be able to go back and alter the genre if they think the panel made a mistake.  As for giving moderators that same access I'm not sure it's something that will happen.  It's up to CJ, really.

If you want to form a negative opinion based on an incomplete model that's up to you, but I don't see it as very constructive.

Quote from: MillsJROSS on Sun 03/06/2007 07:20:10
No one ever said this was a democracy, but we've mostly been very good about discussing things before a decision is made. Yes, at some point someone, CJ, has to give the greenlight to do something. However, as a longtime member of this forum, I very much would like to express my oppinions. So I am doing just that.

As I say, I appreciate this and I concede that we could have done better with the way this was made public. I also appreciate that this smacks a bit of the nanny state saying "we know what's best for you", and that usually we do these things with a full public consultation here.

I think that one of the reasons we've gone ahead with it rather suddenly is that we got to a point where there was enthusiasm amongst the moderators to improve the database. If we had waited too long discussing it, it's only natural that people lose motivation and you end up with nobody willing to actually do the work when the time comes. So, it seemed a good idea to sieze on it and make something happen.

So in summary, I apologise if people feel left out. When we have the system working, there'll be plenty of time for us to review it and see if it works in practice; and if not, we can make changes.

QuoteI'd just rather avoid seeing this topic brought up again 7 years from now because we were too impatients to wait 7 more days when the rating system was created.

Actually, I think it's inevitable that it'll need to be changed in a few years time. For example, 4 years ago there was no need for a Review Panel because the number of games was small enough that it was practical for people to sift through themselves when looking for a game to play.

But now, times have changed and we need to devise a method to cope with that change. And I'm sure that in 5 years time, things will have changed again.

Quick question; Snarky and Mills, would you like to join the panel?

Quick answer: I'd like to help anyway I can. I graduated last month, started work, but I now have a lot of free time I never had while in school.

QuoteThe age requirement.  I think this is reasonable because while not every adult behaves responsibly, I would argue that most understand responsibility better than children/teens do and won't flake out as much.

I think I used the wrong word, here. I meant forum age. I do respect there be a minimum age requirement.

-MillsJROSS

Quote from: Andail on Sat 02/06/2007 19:34:26
Snarky, the speed issue isn't primarily because it has to be done before a certain date, it's because the project is more likely to die out if it turns into a very lengthy process.

Well, yes. I appreciate that. It does seem like it would be possible to start with things like cleaning up broken links, and maybe even playing some of the games to-be-rated while discussing the system, though. And if you're discussing it behind the scenes anyway, would more open dialogue really slow it down that much?

QuoteI'm not against input (I'm very willing to give up my initial ideas about the rating - and have evidently done so in the designated forum to give room for better ideas) but if every single member of this community should be able to bring the progress to a halt just because they envision how their favourite game will be rated low, nothing will happen. After all, this is CJ's personal site, not the UN.

I don't think anyone wants to bring the process to a halt. And I do think that to consider how the ratings of a few games generally considered good (e.g. AGS Award winners) would come out is a useful, quick-and-dirty way to get a sense for a ratings system (before actually applying it in practice).

You know, I'm not trying to complain, though. I was unhappy with a tone of hostility I sensed in the earlier posts, but I like the way things are seeming to progress. The panel system seems quite sane, I thought your draft document was a good start, and given that the people involved are reading this thread and hopefully considering the suggestions and criticisms, I am optimistic that the final system will reflect the preferences of the community. (Personally I like what I'm now hearing about augmenting the ratings with short explanatory comments, for example.)

Where I'm coming from is that philosophically, I am nearly always in favor of democracy, openness and consultation. Take how CJ runs the AGS engine. OK, it's his application, and he can choose to do whatever he likes. Generally, though, people suggest new features publicly, there may be a discussion, CJ engages with the suggestion, lets the suggester know whether or not the change or addition will be made, and explains why. If that kind of dialogue could be possible here (and I realize why it might be more difficult for this purpose), I think that would be great.

Quote from: ProgZmax on Sun 03/06/2007 03:37:00
Don't worry, Snarky.  This and some other things you mentioned are already being ironed out.

I'm glad to hear that. I offered the input not because I want to decide how the thing is run, but because I thought they were good and important ideas that could help improve the ratings process. The most urgent recommendation from me is to design the system so that it attempts to evaluate games on their own terms, valuing their individual strengths, rather than according to their weakest element among some set of pre-defined categories. I feel confident that the panel and organizers see the importance of this, especially since AGS games have a pretty extraordinary range, and many of the (in my opinion) "best" and most interesting titles are pretty unique in one way or another (whether it's META or The Shivah).

Quote from: Andail on Sun 03/06/2007 16:08:40
Quick question; Snarky and Mills, would you like to join the panel?

Thanks for the invite, Andail. I will have to decline, though. First of all, I don't qualify by reason of not having played anything close to most games in the database. Secondly, I simply cannot make the time commitment that is required. When you start rotating panel members I might be interested in joining for a while, if I'm out of a job at the time or something like that.

Quote from: MillsJROSS on Sun 03/06/2007 07:20:10
Well, for one, it's not up to moderators to decide what awards win or not. Most of the categories were also attained by open discussion. A moderator is a person of established trust, and they should be a part of this, but we should have a say as well.

I was simply comparing the process of setting up and maintaining those particular activities.
And having a say before something has even been put into effect is a little premature.

What's the alternative? We can discuss discussing a possible discussion about maybe thinking about the process of beginning to set up a discussion about setting up a panel? Are we that anally retentive about the symantics of the situation?

Or do we try to move foward with one of the possible systems of rating and cataloging and leave the subject open for input from the community, as a whole, with the possiblity of ongoing change and refinement?

Not to be crude, but we must shit or get off the pot.

In my opinion.

EDIT: If everybody with an interest applies to join the panel, thus declaring they have the time and interest to devote themselves to the process, instead of pointing out problems they have with it, then we'll all get a chance to participate and shape the outcome.

EDIT:

Quote from: SSH on Sat 02/06/2007 20:13:17
It wisnae me that made Richard game of the month! It was that oldest of old-guards, CJ...

My apologies, SSH :)

QuoteWhat's the alternative? We can discuss discussing a possible discussion about maybe thinking about the process of beginning to set up a discussion about setting up a panel? Are we that anally retentive about the symantics of the situation?

I think your perception of discussion is a bit over the top. We're discussing a plan of attack, openly, and then proceding with what we (CJ) thinks is best. However, as already mentioned, the way this thread was brought forth seemed a bit less open then how we usually run things.

It seems there is a sense of urgency to get things done immediately or something bad will happen. Yes, discussion might slow down action, but it might help refine our ideas before they are put to action. Regardless of what we do, we're going to end up fixing something down the line. 

QuoteNot to be crude, but we must shit or get off the pot.

I'm just trying to make sure we won't get anything on the seat.

-MillsJROSS

All those who are arguing about this, if they promise to give your games 5 cups, will you shut up?  ;) ::) :P

I think there is a point to what people are saying about giving some sort of over all rating of the experience of playing a game, rather that qualifying a set amount of credits for different categories. By setting a limit to how many cups you can get for different aspects of a game you are dictating what is to be considered a good game. In the long run this would lead to a cluster of similar games getting good ratings (the same, of course, goes for bad, mediocre, etc. games...). General properties of this "good game" would probably be a quite typical adventure game of old school type, with stunning graphics, home brewed music etc. and I think this can well have a serious impact on the exposure of games with a different take on the engine/genre/storytelling/graphics/*add interesting aspect of choice here*. An average over the panel's individual appreciation of a game would be more interesting, in my opinion, rather than a quantified set of points for predefined properties.

A quite interesting role of this community is the development of the adventure game genre as a whole. By restricting what is considered a good adventure game, by rewarding a certain type of games, the incentive to broaden and evolve this genre of games, that we all share a love for, is lowered.

However, this is all in good fun, now isn't it? So give it a bashing and see what comes out of it. I think it would be very interesting to see what games will acheive what rating. There are always other channels of promoting your game if you are unhappy with the exposure you get from this site.

Kudos to Andail who is putting an effort in to get something to happen!

Quote from: Andail on Wed 30/05/2007 09:50:02
This is preliminary outline to the rating document. (http://www.andail.com/agsclass.doc)

With respect to this, it would seem that any game using a Sierra or Lucas GUI does not have "a fully functional, originally designed interface", thus could get no more than one cup?

I could think of several popular games that this rating system would not do justice. For instance, Larry Vales has good plot and funny dialogue, but crummy graphics. Spooks is well written but appears to lack "advanced scripting". The King's Quest remakes are popular, but lack original plot since the plot was written twenty years ago by Roberta. And META simply doesn't fit in the box period.

It would seem that getting three or four bluecups is excessively rare, and that the categories for one and two would be haphazardly large, containing many games of varying levels of interestingness.

I believe the rating system is a good idea but could stand some refinement.


Oh yeah - would it be possible for a game author to improve his game to get better ratings (e.g. get a re-rate after bugfixing/typofixing/etc)?

Also, I'd like to volunteer for a future panel. I believe the current posts are filled, but at present I don't have time anyways, and in the fall I probably would.

To me, any game that runs smoothly and without any bugs features advanced scripting.

It looks like the rating system will have to go through various changes as time goes by just like AGS itself. Im sure that the people that rate the games will understand that changes will have to be made to the rules of rating due to things like the advancements made to ags. People should not worry about the ratings im sure that the people who are rating will do a good job and who knows if the ratings are rerevised then maybe the game will score higher who knows.
If you feel that your game should have a higher rating try looking thru your game again to see what you can improve.

Quote from: Vel on Mon 04/06/2007 13:43:46
To me, any game that runs smoothly and without any bugs features advanced scripting.

Then how would you define games that features special graphical effects, intuitive, uninterupted soundtrack that change as the events change (like the LEC's iMuse), arcade sequence, combat sequence, RPG elements, non-linearity, open ended gameplay and other similar features that aren't in most adventure games?

Quote from: Sinsin on Mon 04/06/2007 14:09:52
It looks like the rating system will have to go through various changes as time goes by just like AGS itself. Im sure that the people that rate the games will understand that changes will have to be made to the rules of rating due to things like the advancements made to ags. People should not worry about the ratings I'm sure that the people who are rating will do a good job and who knows if the ratings are rerevised then maybe the game will score higher who knows.
If you feel that your game should have a higher rating try looking thru your game again to see what you can improve.

I too begins to think the system will have to be upgraded every years as our standards will improve. There are plenty of things that can affect the rating but that aren't mainstream yet, like the use of speech pack or special features like I listed above. Maybe while games are rated, some games that present elements that can't be judged correctly yet, could be given a special tag, so the next time we upgrade the rating system to include the elements that became more mainstream, these games could be rated a second, third or fourth time to take into account the new standards. Games that fare too well in certain categories (like scoring 6 cups out of 5 if given the chance for example), could be easily enlisted for a second or third review and be rated again in case we decide to increase the total number of cups as the our standards increase.

Regarding unique GUI, I have no idea how rating the music will be done, but I suggest that it get rated the same way music will be rated. GUI should be rated for their precision, intuitivity, ease to understand, ease to use and good look, but originality should be considered as a bonus. A badly implemented unique GUI shouldn't pass merely because it is an unique GUI, and a well implemented GUI shouldn't sink because it's the same Sierra or LEC GUI we've seen so often.

2 other search categories or tags that could be mentionned next to the cup rating:
The "experimental" tag, so the games that don't fare very well in graphics, music, etc, but present something unique and new that wasn't seen before and that could please to a niche (META and Into The Light come to my mind here, BogEasy3D too even if it fared a bit better) could be labeled as such so their uniqueness would not be penalized by their other aspects.

Another one, similar to the one above, the "underdog" tag. For the games like Larry Vales, which doesn't fare very well graphically, musically, and use the default GUI, but that still deserve to be played.

QuoteThen how would you define games that features special graphical effects, intuitive, uninterupted soundtrack that change as the events change (like the LEC's iMuse), arcade sequence, combat sequence, RPG elements, non-linearity, open ended gameplay and other similar features that aren't in most adventure games?

Depends on how well these elements are implemented in the game, really. Either way there are at least a dozen of superb games on the database that feature none of those and yet are splendid titles. Come on, who cares how it is scripted as long as it works?

Quote from: Vel on Tue 05/06/2007 21:30:24
who cares how it is scripted as long as it works?

I do.

Okay, who, apart from 1337 programmers, cares how this game is scripted as long as it works?

Anyway, I'd like to propose a 'similar to' part in each game's page if it hasn't already been talked about. The advantages of such a part are obvious - if you like a certain game, you can easily play more like it. The underdogs does that pretty well imo.

Quote from: Vel on Fri 08/06/2007 20:16:08
Okay, who, apart from 1337 programmers, cares how this game is scripted as long as it works?

W00t, ph34r my t3h l33tn3ss!!

Seriously though, in my opinion advanced scripting can really enhance playability. For instance, in some AGS games, if you click some unexpected item on some hotspot, nothing happens at all. In somewhat better games, you at least get a textbox like "that didn't work". In yet better games, you get an answer like "I can't use the <item> on the <spot>" with the words filled in. It helps immersion. You'd be surprised at how an otherwise good game can be ruined by poor interface design, or other matters of smoothness/bumpiness of the coding.

I certainly care about scripting when I play games, and it's more than just 'did he do this all in rawdraw' type stuff.  Good scripting also involves avoiding game-stopping errors and using clever workarounds to AGS limitations.  I also like to see some inventiveness and yeah, I think people who show scripting prowess should be rewarded just as much as someone who can draw well or make great music, otherwise you're just saying that the only important aspects of a game are the ones you can see and hear.

Pretty much the only things that matter are those you see and hear! But then the code is driving all that, so it's fundementally important. Sure, you shouldn't be thinking "ooh, crappy looking 3d, this must have been hard to code" and giving it a higher score, but like progz and radiant said, coding is just as much about making things work slickly, making interfaces that don't get in your way, little cosmetic things, like characters turning to look at stuff they're describing, and so on. It has as much of an effect on the style of the game as good animation or writing does.

It's probably because AGS does a lot of the work for us, coding is quite underappreciated, but going beyond that standard can make things feel a lot more fresh.

Quote
Seriously though, in my opinion advanced scripting can really enhance playability. For instance, in some AGS games, if you click some unexpected item on some hotspot, nothing happens at all. In somewhat better games, you at least get a textbox like "that didn't work". In yet better games, you get an answer like "I can't use the <item> on the <spot>" with the words filled in. It helps immersion. You'd be surprised at how an otherwise good game can be ruined by poor interface design, or other matters of smoothness/bumpiness of the coding.

This is quite what I meant when I said that each game that didn't have interface problems nor bugs featured advanced scripting to me. Of course, games such as Linus Brockman(sp?) are astounding and even I, who don't usually care for those things, marvelled at the scripting genius. However, the puzzles ruined the impression. Anyway, while 1337 scripting skills should be appreciated, their lack should not deprive otherwise excellent games of a full blue cup score.

QuoteThis is quite what I meant when I said that each game that didn't have interface problems nor bugs featured advanced scripting to me. Of course, games such as Linus Brockman(sp?) are astounding and even I, who don't usually care for those things, marvelled at the scripting genius. However, the puzzles ruined the impression. Anyway, while 1337 scripting skills should be appreciated, their lack should not deprive otherwise excellent games of a full blue cup score.

Yes, but there is also more than just making things slick. It seems to me that we are in front of yet another aspect, with music originality and interface originality, where we'd like to have some elements of this aspect to count a bit more than others, so that slicks games don't fail even if they don't feature complicated arcade sequence, non-linearity or something like LEC's iMuse. I'm also wondering if it's a good idea. It seems to me that if we do this treatment for every games (music ain't original but it fits, interface ain't original but it's easy to use, game doesn't have incredibly complex scripting but it's slick), we'll simply reward mediocre games, or penalize awe-inspiring detailed games, or do both.

It's an interesting discussion. With every new post I seems to discover another little, subtle and insignificant details that can make a game better or worst.

BTW, how are going the discussions inside the panel? If it ain't too much asking and if progress were made, could we get a little update to see the improved outline to the rating document, or are there too many elements that are still being debated and aren't fixed in cement yet?

From a rating perspective, I don't think that scripting per-se should be considered. It doesn't matter to the player of a game how clever the scripting is, what matters is whether the game is fun.

Quotecoding is just as much about making things work slickly, making interfaces that don't get in your way, little cosmetic things, like characters turning to look at stuff they're describing, and so on. It has as much of an effect on the style of the game as good animation or writing does.

Indeed, scripting is one of those things where if you do it well, nobody will really notice; whereas if you do it badly, there'll be all sorts of complaints about it. So if we're talking about rating the game scripting based on things like being able to play through the game without encountering bugs and the gameplay being fluid, then I'd agree with it.
However, I'd think twice about awarding points to a game just because it has a cleverly scripted scene that's actually no fun to play.

You could always include a rating for scripting outside of the scope of the average rating. If the scripting is bad it will probably affect the rating anyway, and visa versa, as already mentioned. So perhaps having a stand alone score. Perhaps with this score, there can also be given comments as to how a game fell short of good scripting or succeeded in great scripting. At least, this way, it will help the game creator to get some grasp of what he/she may be doing wrong and needs to work on.

To me, it's apparent when there's good scripting and bad scripting, and it's not just lack of scripting. One little thing that usually get's me a little peeved is when people take the time to have their character walk to a hotspot, but the character blocks the hotspot, and so for every action you need to move the main character over. It is also about lack of scripting, or filling out the game for me. I want a message for every action I do, it doesn't have to be unique, that's expecting  a lot, but I want some form of communication that tells me things are happening. I consider this as leaving out essential, to me, and easy to achieve scripts.

-MillsJROSS

Quote from: Pumaman on Sat 09/06/2007 20:33:30
Indeed, scripting is one of those things where if you do it well, nobody will really notice; whereas if you do it badly, there'll be all sorts of complaints about it.

So scripting is basically like refereeing a football match.

Sorry for grave digging but I'm actually quite interested in the outcome of this new rating. Has the panel begun its work? How is it coming?

The panel has indeed begun its work. However, the results won't be made public until all the games have been rated, otherwise it could be confusing for new visitors to the site.

Oh speaking of ratings -- there was an old suggestion to also rate games by difficulty (easy/moderate/hard).

This just crossed my mind and I don't think it made it to this thread. I think it would be useful classification. Maybe the authors can do this themselves, like with long/moderate/short games.

Quotethere was an old suggestion to also rate games by difficulty (easy/moderate/hard).

I agree with you that it would be a very useful classification.

Quotethere was an old suggestion to also rate games by difficulty (easy/moderate/hard).

Too subjective, I would think. One man's moderate is another man's easy, and so forth.

Quote from: LimpingFish on Wed 04/07/2007 20:07:48
Too subjective, I would think. One man's moderate is another man's easy, and so forth.

That's not true at all. It is well known that Monkey Island 1 and Loom are pretty easy games, and that Spellbreaker is insanely difficult. And so forth. Besides, it'd be easy for the "rate this game" form to have a query on difficulty similar to the one on game length (i.e. "do you think this game is in the right category")

To prove Mr Fish's point, I wouldn't consider MI1 "easy". It's not mind-bendingly hard, but it has a ton of puzzles, most of which require you to actively engage your mind, and some that are downright sneaky. That immediately makes it far more difficult than many other adventures and AGS games. It's a typical example of a "moderate" game, I would say.

Still, I generally agree that difficulty ratings could be agreed upon, and even if they wouldn't be precise, they could still be useful.

A difficulty rating determined by player feedback would therefore be the answer. A single panel member's opinion of a game's difficulty level would be more or less pointless, as, like it or not, it would be a subjective opinion.

And the point of such a rating anyway? Wouldn't it discourage people from playing games rated as high difficulty, or those rated low, depending on what they were looking for? What if a game has mostly easy puzzles, but a handful of hard ones? Is this why we have hint threads and such?

If such a rating is to be implemented then, as I've said, leave it strictly to player feedback.

Quote from: LimpingFish on Thu 05/07/2007 18:45:59
A single panel member's opinion of a game's difficulty level would be more or less pointless

Yes, which is why I never suggested that in the first place. Like game length, the author could give an approximation.

Quote
And the point of such a rating anyway?

Obviously, to let people find the kind of game they like. Sometimes people want a quick 'n easy game. Sometimes people want a challenge. With the near-thousand games in the database, finding what you want is difficult enough already.

I still say one persons opinion (be it the author, a panel member, Jesus, Jesus' dog Andy) would be pointless. As I've said, such a feature would perform better as an ongoing user-based rating. Implemented as such, I don't have any problem with it.

Still, it strikes me as more pointless segregation of the game database. If a player doesn't want to devote a lot of time to a game, they should download a short game. Where's the fun in playing an easy game? Indeed, where's the fun for a developer making a game they know people will breeze through?

Maybe I'm just missing the point.

Quote from: Radiant on Thu 05/07/2007 19:18:46
Obviously, to let people find the kind of game they like. Sometimes people want a quick 'n easy game. Sometimes people want a challenge. With the near-thousand games in the database, finding what you want is difficult enough already.

From what I understand, the revamped games database will make finding the games you want to play a lot easier. We should wait and see how the work the panel is doing turns out, and decide if any new features should be added.

In my personal experience I've never had a game rated 'high difficulty' dissuade me from playing it.  I've never placed a huge amount of trust in the opinion/review of someone I don't know, though, so rather than let a rating or review convince me I usually look at a cross-section of reviews and player opinions and get a feeling for the average opinion of a game that I'm not terribly knowledgeable about.  If it's a game I've wanted to play, no rating is going to prevent me from at least trying it, however.  I think more people should form their own opinions of difficulty and such instead of this hive-mind reliance on other people to do things for them that's been creeping into society.  Hopefully the new rating system won't turn someone away from playing a game that sounds really interesting because it was rated poorly, but after they play it the rating might make some sense ;).  I think that difficulty (like play length) is something the author should decide but could be altered by player votes (much like game length).

I like this Rating Panel idea. It may not dissuade you, but it gives a professional view of what they think of the game. I prefer to go with what the panel has to say instead of the overall community. For instance in a abandonware site I saw the game Future Wars. It was rated 5 by the website and 3.2 by the community. I loved the game, and once I replayed the game, it was still something golden. The community most of the time doesn't understand why things are rated a certain way and sometimes rate it just for the fact its not there type of game, which is really unreliable. Newgrounds is a great example of this.

All in all, i'm all for the rating panel.

Maybe in two more years i'll be able to put my hand in the rating panel. Ive worked for a lot of abandonware websites and freeware sites rating videos games for about 6 years, so hopefully soon i'll be able to get back into it for the better good. :)

Geez long thread!

May I ask the admins to show us one "panel rating" (just out of curiosity).

EDIT: a game like 7 days a skeptic or some other well known ags product will do

EDIT2: maybe a "famous" game and a couple of short, slighty above average one, so that we can get a feel of this rating.
Also, where can I find the official rating guidelines (the one I downloaded seem only a preliminary draft).

Last, I do really fear most of the games will be doomed to the "1 cup swamp". And I think some "requirements" are quite dull (who cares about the originality of my GUI as long as it runs smoothly).

But as long the "cup-rating" will be showed along with the "people's rating" there will be no problem at all: I'll just stick to the last one (maybe a "filter by users' rating" option will be a good thing)

A little update:
The rating panel is now about halfway through the 750+ non-demo titles in the database.

Along with ratings, games have been further classified, and their links have been fixed.

So far, work has proceeded extremely smoothly.

Wow, I'd say that's good news. Halfway through, and it's not even two months later now. Good work, ratings panel! *distributes root beer floats*

I spotted something that may be of interest to the rating panel.
Several of the older games (such as this one (http://www.adventuregamestudio.co.uk/games.php?action=detail&id=233)) don't actually have a category (like short games, joke games, etc). You can spot them easily on this list (http://www.adventuregamestudio.co.uk/games.php?action=digest). It may be useful to add categories for these?

Also, I spotted four games with no listed author: James bond, Surreality (RON), QfG4.5 and this one (http://www.adventuregamestudio.co.uk/games.php?action=detail&id=84) which I suspect doesn't really belong.

Archive.org shows that James bond was made by foz (http://www.adventuregamestudio.co.uk/yabb/index.php?action=profile&u=1001). QfG4.5 appears to be made anonymously. Perhaps some oldbie remembers Surreality?

The rating panel will set a length if none is there or if it's wildly inaccurate as well as the genre since that's a new feature and it's not realistic to expect everyone to go back and edit their game.  As far as games without an author goes, that's usually the result of accounts being deleted/lost and there's not a whole lot that can be done about it unless CJ manually fixed each one.

Quote from: ProgZmax on Mon 30/07/2007 11:54:05
The rating panel will set a length if none is there or if it's wildly inaccurate as well as the genre since that's a new feature

Cool.

Quotethere's not a whole lot that can be done about it unless CJ manually fixed each one.

Er, there's, like, four of them. One of which I pointed out above, one is an advertisement, one is a game anonymously released. One left.

The author name is entered by people when they upload their game. If they choose not to fill it in and remain anonymous, who are we to argue?

Sorry for thread digging, but I'm curious about the outcome of this panel and figured it better to post here than start a new thread...
Is the panel still working their way though the games database? Any status update?

If this is inappropriate I guess a moderator can lock...

Cheers!

More than threequarters of the games in the database have been rated, so I'd assume they're getting close to putting this live :) I'm sure someone can give you a more detailed update than that, though!

Yes, the rating is still continuing. There have been a couple of people doing a really good job at playing and rating lots of games, but it's been hard to get the commitment from most people so the job is proceeding more slowly than we'd like.

We'll get there eventually!