More Panel Ratings Discussion

[Igor Hardy]

It's just hard to remain optimistic in the face of blustering discordance. My time on the panel is limited, and I try to rate as wide a range of games as I can; just as every panel member does. If I was to dedicate the hours a relatively full-length game requires to appreciate it fully, only to have the reason that time was spent rendered void, I'd have a right to be a bit pissed. Fun can also be subjective, as I'm not going to enjoy every game I play. No reviewer enjoys everything they review. This aspect of their time spent reviewing can be thought of as "work". I wouldn't like to spend time working, only to have that work evaporate. If you haven't gleaned this fact from what I've being saying throughout this thread, then I can't really explain it further.

I can very well understand your disappointment. I'm sure there's a lot of work involved in preparing the panel reviews and I respect and appreciate what you do, as well as the serious approach to the reviews. But the database is primarily a place for authors to showcase their games. I think they have full moral right to decide they're better off promoting their works in a different way. It's not like they delete someone's full-length written review by removing a game.

[Shane 'ProgZmax' Stevens]

Now, due to the deletion, everybody knows who to contact. And I'm glad that there is a movement in that subject (the official information thread about reviewing the games).

This just isn't true, Harg.  If you'd have read the ongoing rating panel discussion you would have known that I was on the panel and willing to take PM's (as well as LimpingFish), something that was only restated here to remind some of you that we aren't a shadow operation .  Also, we'd been planning to release the guidelines for awhile now but holidays got in the way of a few panel members being able to offer their input on the subject.  I think you did manage to give the guidelines a kick to get posted in a timelier fashion but I wouldn't say your actions were constructive or beneficial, either to yourself or to the community.  I hope at some point in the future you will look back on this and see your response as a dramatic one and re-upload your games for other people to play and form their own opinions.

Again, if you'd like a more detailed workup of why your game received its rating just PM me your request and I'll ask the judge responsible to provide one for you.

[ThreeOhFour]

I'm not a panel member, but something that disappoints me is the fact that, while we do sometimes see people thanking the review panel, the only time that people remember them in any big way is when somebody complains (I'm guilty of this too). The rest of the time we seem to expect them to keep rating games without any recognition for their efforts, which is a pretty crappy attitude considering the friendly place this is.

If was was doing something and the only time anyone commented on what I was doing was to say "Hey! That's not how you do it! Don't do it like that anymore!" I'd say "Screw this".

And I'm not saying this so my games all get 27 cup ratings, it really is something to consider. We waited for ages to get this panel thing going on, it'd be a shame to lose it now that we've got it!

[Calin Leafshade]

^^^^ Agreed.

[Peder 🚀]

I demand one more cup on my game Where did Sam go? !!
"Joke" aside, I agree with Ben304 here..

The fact is, the panel cant just give any games good ratings to support the developers!
I mean, then the whole point of the system would be ruined.
Also, if we get the new website up now with the similar looking user rating I dont think 2 cups will look that bad if the user rating is 3 or more cups!

Also I can't really say if the panel was fair or not about the rating of SQIII as I haven't played it..

[Iliya]

The fact is, the panel cant just give any games good ratings to support the developers!

Peder, you are missing the point. No one wants better rating. I just don't want the games to be reviewed by a single person. If there is no way one game to be reviewed by many people, then only technical areas should be commented. Otherwise, the review will be very subjective.

[Andail]

Harg, he's missing your point because your point is very strange.
A review is always written by only one person. At least at all the major magazines or websites I ever visit. They may have a staff of reviewers, but ultimately, the published text will have been penned by only one person.
I hope you understand that a review or preview or commentary or what have you is always subjective. It's in the nature of reviews. However, by having a panel of very experienced and trusted reviewers, we at AGS have ensured that the reviews are as balanced and neutral and qualified as you'll ever get them.

It's fine that you don't want to expose your game to other opinions, but try to think through what it is you're asking here.

[Helme]

Thanks for the Rating Panel Information thread. It adds a lot of transperency to the work of the panel.

[SSH]

Here's an idea. Perhaps when a new game is added to the DB there can be two submit buttons: One for "I agree for my game to be reviewed by a single member of the review panel and their rating to be available for people to see and sort by and  I accept that I may not like their opinion because all opinions are subjective" and another with "I do not want my game to be reviewed by the review panel and accept that I will therefore not get a cup rating". Although people should presumably be allowed to change their minds later within reason.

[Peder 🚀]

Id say thats a good idea SSH,

[Igor Hardy]

Yeah, I agree. It would also save the Panel reviewers a lot of time and having to deal with bitter game authors.

[Ryan Timothy B]

If that were the case, when I go to sort games by cup rating and that one persons game isn't shown because it doesn't have a cup rating.  I guess it would be their own fault that I'll never play their game, due to not seeing it in the list.  Unless the game strikes my interest from the completed games thread.

There are two people I can see selecting the 'do not rate my game' option. 
One, someone who released a joke game, or deep down knows their game isn't worth anything at all (which I can say I'm ok with number one). 
Two, someone who just can't handle the truth and wants to hide the true rating their game deserves.  They also deep down know their game isn't worth a higher rating, but when they see the lower rating attached to their game, they can't handle it.

If I made a game and it received a rating I wasn't too pleased with, or even a review commenting on bad/broken areas which I had believed were perfectly done.  I wouldn't freak out.  Of course, I would probably drop my jaw in disbelief at first.  Then I'd play my game over again from beginning to end and see if there was any truth to the review, have I been blind to my own creation? 
If I ended up noticing the flawed areas of my game that the lesser review and rating was indicating, I'd probably engrave that information into my brain for my next game, so that I wouldn't do it again.
But, If I still couldn't see why it was rated so much lower than I anticipated, I probably would have pm'd progz because I've always known he's been on the panel.  And ask him to get the person who reviewed and rated it to explain it more in depth, or even possibly re review and rate the game.


Anyway, out of all honestly.  If we as a community of game players, have the option to vote yes or no to this idea of skipping out on the panel ratings and reviews... I say No.  I do not want some games to be able to avoid what every game must go through.  Otherwise everything the panel has done here in the past few years, would be useless.

Keep doing what you're doing guys (and it looks so good now with my new blue cups on the new template page.. mmm lol).

[Snarky]

There are two people I can see selecting the 'do not rate my game' option.  

• The third type would be people who, for whatever reason, do not have confidence in the panel/process, or who expect to receive an unfair rating.
• The fourth type would be people who are opposed to ratings in principle, and think that people should just try it for themselves without preconceptions.
• The fifth type would be people who think boiling an experience down to a single score is useless, and the only way to assess a game is through a detailed review.

I don't agree with any of these opinions, but I've heard all of them expressed by various people, and making the panel rating voluntary would be a simple way to address their concerns. I bet you'd hear a lot fewer complaints as well.

I wasn't sure about the idea at first (after all, the ratings are mainly for the benefit of prospective players, not for creators), but the more I think about it the more I think it makes sense. It probably wouldn't affect too many entries, and like you said, it'd be their loss. And after all, it's better to have the games in the database without a rating than to have people not post their game there in the first place, or delete it from the db in protest, right?

Instead of a rating, you could display a message stating "At the request of the creator, this game has not been rated".

Anyway, out of all honestly.  If we as a community of game players, have the option to vote yes or no to this idea of skipping out on the panel ratings and reviews... I say No.  I do not want some games to be able to avoid what every game must go through.  Otherwise everything the panel has done here in the past few years, would be useless.

Why would it be useless? You think that if we don't rate 100% of the games, 0% or 95% makes no difference?

[Helme]

After Snarkys post an option to choose wether a game will be rated makes more sense to me. But what about the user ratings? Should there be an option for no rating at all or no panel rating?

[Snarky]

Oh, and I can think of two more reasons I think are valid:

• The sixth type would be commercial game makers who agree that the ratings are fair, but know that they're graded on a tough curve, and worry that potential buyers won't realize that, say, a 3 cup-rating is quite good.
• The seventh type would be people who want a review, but not right now. For example, they might want to wait until after the AGS Awards, or if they're making an episodic game they might want to hold off until a couple of episodes are out. (Sure, they could probably just PM the panel contact and ask them to wait a while, but that might not occur to creators who aren't active on the forums.)

[Ryan Timothy B]

Because I believe it won't be a small percentage that choose the 'don't rate my game' option.  I can foresee it being 25% or more.

I've pressed that Lucky Dip feature in the new website layout more than 100 times now, and to tell you the truth, I haven't seen a game with a lesser player rating than blue cup rating.  Which is why I completely trust the panel, because the average gamer is just like "meh, 10 out of 10, I loved it" -- you see that comment all the time.  Every AGS game that I've played, I can agree more with the panel rating than I do with the player rating.  Which is why the panel was there in the first place.

I would almost rather the games not be in the database, then to see no ratings at all.  The keyword there being: almost.

[Snarky]

Well, you probably don't want to make it so that asking to be excluded from the panel rating is just as easy as being included, or you might get too many people who just don't feel like it for no particular reason. If it's just a little bit of effort, you'll filter out these low-commitment rejectionists while still giving people a real choice.

Maybe those asking not to be rated should have to fill in a text box explaining their decision (to "help the panel understand" why people choose not to participate). Or maybe there shouldn't be an option on the form, just a message saying that if they don't want a rating, email or PM this address.

[Andail]

This is a bit silly. If you choose not to display a rating, it's because you consider it a bad rating. I predict that 99% of those who opt out will first have accepted to receive a rating, then changed their mind when it turned out lower than expected. Fine way of wasting people's time.
Moreover, the database will have a flawed balance towards higher rated games, with a huge bunch of unrated games. I don't see how this helps anyone, even the developers.

You're suggesting that humans are more complicated than they are, Snarky. The world of a developer is very small - good ratings are good, bad ratings are bad. Whatever philosophical theories or sentiments you have about reviewing stand pale to this simple fact - we all want good feedback.

Also, some of your reasons are even better used as arguments for compulsary rating. So a person who wants to sell his game doesn't want bad publicity? Well, that's life. Are we going to support that developers who can't make games worth buying still manages to trick people into buying them, by erasing mediocre reviews?

[Igor Hardy]

Maybe those asking not to be rated should have to fill in a text box explaining their decision (to "help the panel understand" why people choose not to participate).

Maybe they should be presented with a form containg an emotional stability test. It wouldn't be widely known, but you would have to actually fail the test for your request to be listened to.

This is a bit silly. If you choose not to display a rating, it's because you consider it a bad rating. I predict that 99% of those who opt out will first have accepted to receive a rating, then changed their mind when it turned out lower than expected. Fine way of wasting people's time.

It shouldn't be possible to opt out once the rating is posted.

[Snarky]

Like Ascovel says, I was assuming that you couldn't ask to hide the rating after it had been assigned (though you should probably be able to request a rating even if you initially opted out), because otherwise the selection bias would obviously be enormous.

Andail, maybe you should have another look at what I wrote in my posts, since you seem to have missed a couple of details. (For example, I didn't talk about developers wanting to avoid bad ratings, but being afraid that the ratings would be misunderstood.)