There is something seriously ridiculous about AGS reviews

[lapsking]

 I made a game (The Will) that took 20 days and another game which took more than 2 years (The Order). They both got 3 cups from AGS panel which makes me laugh. I'm not comparing them to another developer's games which might make it subjective. But rating my first game the same as the second game just makes me wonder if they have any clue how they are reviewing the games. If they gave the first game 1 cup and the second game 3 cups it would make sense. But this is ridiculous. It doesn't make any difference to me cause  my both games are free, and I have no plans to become Ron Gilbert. When you take my joke as serious as my art, I realize you must be joking. Do you guys even know what's fart and what's art?

[cat]

Would you feel better if your other game was downgraded to 2 cups?
Obviously, the person(s) who rated your games generally liked them, but didn't think they deserve 4 cups (which only a small number of games get).

The rating isn't about how much effort went into a game - it's a recommendation for players to decide which game to play. For me personally, the panel comment is more relevant than the mere cup count.

[lapsking]

Would you feel better if your other game was downgraded to 2 cups?
Obviously, the person(s) who rated your games generally liked them, but didn't think they deserve 4 cups (which only a small number of games get).

The rating isn't about how much effort went into a game - it's a recommendation for players to decide which game to play. For me personally, the panel comment is more relevant than the mere cup count.

Yes, thank you. I would feel much better if you downgrade my first game to 1 cup. At least I feel like there is some sanity.

[fire7side]

Not knowing anything about your games, I would guess that the one that took 20 days or whatever was listed as a short game and the other listed as medium or long.  I look at those things and don't expect a short game to necessarily compete to a long game in terms of depth of story, etc.  I think the ratings are just does the game work.  Does it keep a person interested to the end.

[lapsking]

Not knowing anything about your games, I would guess that the one that took 20 days or whatever was listed as a short game and the other listed as medium or long.  I look at those things and don't expect a short game to necessarily compete to a long game in terms of depth of story, etc.  I think the ratings are just does the game work.  Does it keep a person interested to the end.

I appreciate your guess work, but you have to play both games to be able to judge characters (which is almost none existence in first game), puzzles, coding, writing, visual art and music (which is also none existence in first game). I understand people can even rate my first game more than second game, which might be 1 cup. But I believe if AGS officially wants to rate a game it must be much more professional. Or are we dealing with another ROTTEN tomato? Anyway, they made their rating based on the length of the game (really?) or whatever, I'm free to have to express my shock. Maybe I don't have a clue what I'm making, and the crap that I made to learn how AGS works is as valuable as something I put more than two years of my life in it.

[Creamy]

My 2 cents:

5 cups ratings don't allow too much leeway.
In the whole database, there are only 14 games with 5 cups - truly an exceptional rating.
There are no 0 cup games, probably out of respect for the benevolent creators who took the time to put their games on the database.
That leaves 3 possible ratings for most games.

About the Will and the Order: I've finished both games and I agree on the superiority of the Order.
Graphically and narratively.

The graphics in the Will were only serviceable and its story appeared a bit far-fetched, but it featured acceptable puzzles and I had an okay time with it. Plus it was your first game, so I can see the panel giving it the average to cheer you on.

The Order has a better atmosphere and oozes more personality. The puzzles are good but - as stated in my review - a couple of design choices feel odd.
Also, the conclusion could have been more impactful. The story is nothing to be ashamed of, but we don't get to know the  Nizari Order' leader or the Seljuk Prince. For someone unaware of Iran's History (like me), they are Tweedledee and Tweedledum and I didn't care helping one or the other in the end.
So I'd say The Order is a solid 3 cups.

[Snarky]

Just because two games got the same cup rating that does not necessarily mean they are equally good. Maybe one of them barely got three cups, while the other just barely missed getting four. (Creamy made the same point in more detail while I was writing.)

Anyway, complaining about your own ratings is always a bad look.

[lapsking]

Just because two games got the same cup rating that does not necessarily mean they are equally good. Maybe one of them barely got three cups, while the other just barely missed getting four. (Creamy made the same point in more detail while I was writing.)

Anyway, complaining about your own ratings is always a bad look.

As Creamy mentioned The Order gets solid 3 cups, so does it mean The Will got 3 and a half of cup?  It's just getting more ridiculous! Does it look like if I care that I look good or bad to you? Why do I even need to say much about Nizari leader?

[Snarky]

Let's imagine that the "real" panel rating of The Will is 2.55 cups, while the "real" rating of The Order is 3.46 cups. Because there are no half-cup ratings, both scores are rounded to 3 cups, but The Will is a "weak" 3 while The Order is a "solid" 3.

Does it look like if I look good or bad to you?

This discussion makes you look bad. When you release something, you have to accept that others will not agree with your own assessment of how good it is.

[lapsking]

Let's imagine that the "real" panel rating of The Will is 2.55 cups, while the "real" rating of The Order is 3.46 cups. Because there are no half-cup ratings, both scores are rounded to 3 cups, but The Will is a "weak" 3 while The Order is a "solid" 3.

Does it look like if I look good or bad to you?

This discussion makes you look bad. When you release something, you have to accept that others will not agree with your own assessment of how good it is.

Did I argue how good The Order is? Or was I telling you how bad The Will and hence your rating system is? Please read my comments again or the way you are defending your rating system might make you look even worse.

[eri0o]

Saying the word "your" says more about you than anything.

[lapsking]

Well, "he" was rating The Order 3.46 and The Will 2.55. I wasn't. That's why I said "your" "In case" what he said is true, they both still show up 3 😁 and liking "your" friend's comment doesn't change the fact that they both have 3 cups. 🙂

[lapsking]

Well, "he" was rating The Order 3.46 and The Will 2.55. I wasn't. That's why I said "your" "In case" what he said is true, they both still show up 3 😁 and liking or defending "your" friend's comment doesn't change the fact that they both have 3 cups. 🙂

[eri0o]

If you drive a car, and have been stuck in traffic you understand the concept that you ARE traffic.

The panel isn't "yours". You used the same wording in the first post too. This is a community, you are participating in the community, the panel is ours. It's a better conversation to discuss what do we want for our panel.

I understand there's a balance in motivating the game developers to continue creating games, and there's also a way for players to evaluate what to play. There may be more things that are subtly or perhaps unconsciously taken into consideration.

But I don't think approaching like you are some external entity is constructive.

[Postmodern Adventures]

Some people still think that my best game is Billy Master Was Right (a short, free game made during the lockdown). Much better than my other commercial games that took me about two years each. And I am happy with that. Maybe they saw something in it that wasn't there in Nightmare Frames or An English Haunting. In a way I'm proud that something so small and unpretentious struck a chord with some players.

[lapsking]

Some people still think that my best game is Billy Master Was Right (a short, free game made during the lockdown). Much better than my other commercial games that took me about two years each. And I am happy with that. Maybe they saw something in it that wasn't there in Nightmare Frames or An English Haunting. In a way I'm proud that something so small and unpretentious struck a chord with some players.

Good for you. I'm neither happy or sad, but rather amazed. Already busy writing another pretentious game. Anyone else is interested to convince me how great AGS panel ratings are? Don't hesitate before I get banned.

[Sinitrena]

This specific case aside (because I have played neither game), I do think there is a bit of a problem with the Panel ratings (that are showcased here fairly well):
They are untransparent.
First, while the player ratings are broken down in categories (Visual, Immersion, Puzzles & Pacing, Overall Enjoyment), the Panel ratings are not.
Aditionally - how does the panel even work? If I remember correctly from many, many years, there are some people on the panel (who? Is this noted anywhere?) and just one (Two?, I don't know) person from the panel needs to play a game to give the panel rating. So two games from the same author could be played by two different members of the panel, leading to (possibly) very different impressions and ratings.
Then there's the problem that a rating from 1-5 doesn't actually makes it possible to give a lot of variation in ratings, leading to vastly different games rated the same. As mentioned above, a game could be a very low 2 cups while another is a very high 2 cups - and they would obviously both receive 2 cups.
Then there's the problem that a short MAGS game is probably looked at differently than a full-length commercial game. The expactations for a full-length and/or commercial game are (understandably) much higher than for a short or MAGS or similar game, therefore it might (and maybe should be) more difficult for a full-length game to receive a higher rating.

In short (as someone who hasn't created a game in many years and therefore has no stake in the game), even though I generally like the panel ratings and often find them useful (in addition to the panel comment), I also do think they could and should be more transparent.

[fire7side]

Anyway, they made their rating based on the length of the game (really?) or whatever, I'm free to have to express my shock. Maybe I don't have a clue what I'm making, and the crap that I made to learn how AGS works is as valuable as something I put more than two years of my life in it.

It's just a different thing, like comparing a comic strip to a comic book.  You can have a great comic strip and a great comic book, but you can't really compare the rating and say the comic strip should get more stars because it took longer and has more depth.  You have to compare it to other games of about the same length.  And it's always going to be subjective.  It's just to help decide which game to download.  It's like critic ratings on a movie.  They are kind of all over but tend to be higher in a better game.  We have individual comments and the panel rating, plus we can look at the artwork on a few scenes.  Works for me.

[lapsking]

Anyway, they made their rating based on the length of the game (really?) or whatever, I'm free to have to express my shock. Maybe I don't have a clue what I'm making, and the crap that I made to learn how AGS works is as valuable as something I put more than two years of my life in it.

It's just a different thing, like comparing a comic strip to a comic book.  You can have a great comic strip and a great comic book, but you can't really compare the rating and say the comic strip should get more stars because it took longer and has more depth.  You have to compare it to other games of about the same length.  And it's always going to be subjective.  It's just to help decide which game to download.  It's like critic ratings on a movie.  They are kind of all over but tend to be higher in a better game.  We have individual comments and the panel rating, plus we can look at the artwork on a few scenes.  Works for me.

For me it's like rating a movie that is an hour and half or a movie that is 3 hours. Still they are both movies and I would rate it based on the writing, acting, music, etc. I didn't know it works like that. I learned something new. I'm convinced, I'm convinced that both should have solid 3 cups. How can I not be, after so many logical answers.

[LimpingFish]

But rating my first game the same as the second game just makes me wonder if they have any clue how they are reviewing the games. If they gave the first game 1 cup and the second game 3 cups it would make sense. But this is ridiculous.

Why? Do all artists inherently make better art as time progresses? And by "better", do we mean more polished? Is polished art better than unpolished art?

Maybe I don't have a clue what I'm making, and the crap that I made to learn how AGS works is as valuable as something I put more than two years of my life in it.

If your first game was crap, why did you release it? Did you know it was crap when you released it, or are you saying that you now consider it crap? Did you want people to play your crap, and call it crap, just so you could release a second game to earn kudos for how much better the second game is compared to the first? Will you retroactively announce this second game is crap as soon as you release a third one? Do I you see why some are saying this is a bad look for you, or do I have to elaborate on this dumb excuse for a gag any further?

I do think there is a bit of a problem with the Panel ratings

Fair enough, but some of your points raise their own problems. Such as, what is the acceptable granulation level for ratings? Half cups? Quarter cups? Ratings comprised of .1 increments? Just what would be the differences between a 3.4 and a 3.6 cup game? I'll go further, and ask what the difference between a 3 and 3.5 cup game is? To me, all that half tells me is that the game is good, but wasn't good enough to be a four? Maybe that's just me, though.

As to MAGS or commercial games, are we talking about handicaps or demerits? Should we be harsher on commercial games, or more forgiving of MAGS games?

Personally, if a game is worth playing, I don't feel the need to examine how it came into being. Factors such as development time, or the number of people involved shouldn't, to me anyway, bear any influence on the most important factor: enjoyment.

As to panel transparency, the panel was always intended to be anonymous, but I've long since waved my own anonymity. And while we do have guidelines, and we do occasionally discuss ratings as a whole, we also trust the abilities of each other to be fair and knowledgeable on the subject.

I think people should remember that 3 cups = Good. Numbers before 3 are less good. Numbers after 3 are...more good.

If your game gets 3 cups, you made a good game.