Improving AGS savegames?

Started by Alan v.Drake, Thu 20/04/2017 17:58:32

Previous topic - Next topic

Alan v.Drake

Quote from: Atavismus on Thu 20/04/2017 17:36:49
And maybe an improved save system (to avoid corruption between version).

I don't agree with this.

Let's make one thing clear: AGS savegames are not savegames, they are *savestates*.
By design savestates can't help but corrupt when you change the game data. Making them pretend they work is bad, real bad (and also a waste of time).

Forcing a savestate not to fail will lead to all sort of mismatches which may lead to bugs and lots of headaches and despair.

If you need an upgradeable savegame system you'll have to code one yourself.
An upgradeable save system means you're storing the version of all the stuff you need to serialize along the data, so with each newer version you add the required transformations needed to upgrade the various subsets of the saved data.

That way older saves are able to execute all the required steps to correctly upgrade the data structure.

This cannot be done engine side, the engine has simply no clue what you're doing between versions, it cannot keep track of this stuff.


- Alan

Atavismus

Quote from: Alan v.Drake on Thu 20/04/2017 17:58:32
Quote from: Atavismus on Thu 20/04/2017 17:36:49
And maybe an improved save system (to avoid corruption between version).

I don't agree with this.

Let's make one thing clear: AGS savegames are not savegames, they are *savestates*.
By design savestates can't help but corrupt when you change the game data. Making them pretend they work is bad, real bad (and also a waste of time).

Forcing a savestate not to fail will lead to all sort of mismatches which may lead to bugs and lots of headaches and despair.

If you need an upgradeable savegame system you'll have to code one yourself.
An upgradeable save system means you're storing the version of all the stuff you need to serialize along the data, so with each newer version you add the required transformations needed to upgrade the various subsets of the saved data.

That way older saves are able to execute all the required steps to correctly upgrade the data structure.

This cannot be done engine side, the engine has simply no clue what you're doing between versions, it cannot keep track of this stuff.


- Alan
Good point Alan.
That's why I wrote "maybe" (and I hesitated to add something like "but maybe better code it by ourself").
Still, I'm not sure it's impossible (but as you said it could "lead to all sort of mismatches which may lead to bugs and lots of headaches and despair").

Snarky

Quote from: Alan v.Drake on Thu 20/04/2017 17:58:32
Quote from: Atavismus on Thu 20/04/2017 17:36:49
And maybe an improved save system (to avoid corruption between version).

I don't agree with this.

Let's make one thing clear: AGS savegames are not savegames, they are *savestates*.
By design savestates can't help but corrupt when you change the game data. Making them pretend they work is bad, real bad (and also a waste of time).

Forcing a savestate not to fail will lead to all sort of mismatches which may lead to bugs and lots of headaches and despair.

If you need an upgradeable savegame system you'll have to code one yourself.
An upgradeable save system means you're storing the version of all the stuff you need to serialize along the data, so with each newer version you add the required transformations needed to upgrade the various subsets of the saved data.

That way older saves are able to execute all the required steps to correctly upgrade the data structure.

This cannot be done engine side, the engine has simply no clue what you're doing between versions, it cannot keep track of this stuff.

What the engine can do is to give you hooks to make these transformations, and to deal with mismatches in the serialized data and the datastructures in your current version (e.g. what to do with variables that no longer exist, and how to initialize variables that you have no save state for). So not "forcing a savestate not to fail", but allowing you to put in fixes where it fails (or before it fails), instead of just crapping out directly.

That's still a hell of a lot better than requiring every game maker to write their own savegame system.

Radiant

Quote from: Alan v.Drake on Thu 20/04/2017 17:58:32
Let's make one thing clear: AGS savegames are not savegames, they are *savestates*.
Not necessarily.

For several years now I have managed to do updates to Heroine's Quest without breaking savegame compatibility. It's hideously complicated and you need to know exactly what you can and cannot change, but it can be done. For instance, adding an additional sound effect to your game will cause saved games to fail to load, and there's no technical necessity for that.  AGS can otherwise deal with missing (or surplus) sound effects just fine; there just happens to be a line like "if (game.sound_effect_count != savedgame.sound_effect_count) abort_game ();"

Anyway, that was a tangent. Looking over the above discussion, I entirely agree with the goal of making games playable on as many computers/platforms as possible; and I also agree that being fully backwards compatible is neither desirable, nor entirely possible in the first place. Again for example, using DX5 you can set up three layers in your room so that A appears above B, B appears above C, and C appears on top of A. This doesn't really make sense but it's possible; but in DX9 it is not. Nor should it be - breaking compatibility with uncommonly-used things shouldn't be a problem.

Snarky

Quote from: Radiant on Thu 20/04/2017 19:48:01using DX5 you can set up three layers in your room so that A appears above B, B appears above C, and C appears on top of A.

What happens if there's a part of the screen where all three layers are present? ??? 8-0

Alan v.Drake

Quote from: Snarky on Thu 20/04/2017 20:29:36
What happens if there's a part of the screen where all three layers are present? ??? 8-0

:=

- Alan

Atavismus

Sorry, I started something out of topic.
Maybe an admin could move these messages into another topic about how to deal with savegames corruption?

Crimson Wizard

#7
Quote from: Alan v.Drake on Thu 20/04/2017 17:58:32
Quote from: Atavismus on Thu 20/04/2017 17:36:49
And maybe an improved save system (to avoid corruption between version).

I don't agree with this.

Alan, what you say about savestates and need of game author to resolve errors is true, but we have talked about all this in the past already, and have a certain solution.

There is a working prototype for this system. It only needed certain improvements to be incorporated.

Here is the related thread:
http://www.adventuregamestudio.co.uk/forums/index.php?topic=53753.0

On engine side it checks for mismatches and then calls script callback to let game author resolve them. So, yes, that is a mixed solution, but that's best we could get to in AGS, and probably the only sane one in current situation.

Snarky

Quote from: Crimson Wizard on Thu 20/04/2017 22:54:15
There is a working prototype for this system. It only needed certain improvements to be incorporated.

I love this (prototype) feature, by the way. I'm not sure I would ever use it, because I can't see myself ever making a game big enough that it would be worth the work required to preserve savegame compatibility across versions, but it just warms my little coder heart that it should be possible.

I don't think I ever understood the issues you were talking about in your last posts. If you had a special "blank game state" save, couldn't that be generated essentially upon build, and just be one of the game's files? Then it wouldn't be weird that the game depended on it not being deleted, because of course you can't just go around deleting files from within the game distribution. But like I said, I don't really understand all the technical ins and outs involved.

Crimson Wizard

#9
Quote from: Snarky on Fri 21/04/2017 00:23:29
I don't think I ever understood the issues you were talking about in your last posts. If you had a special "blank game state" save, couldn't that be generated essentially upon build, and just be one of the game's files? Then it wouldn't be weird that the game depended on it not being deleted, because of course you can't just go around deleting files from within the game distribution.

To be honest, I do not know why but I never thought about generating it on game compilation... (I was considering to make it created at the first game launch).

In the end I was leaning towards integrating table of contents into the game file, because that would optimize loading times. Also, if you do not have ways to extract certain item from data, then you'd have to load the "default state" save every time before loading proper save, even if that is not necessary for the particular case.
You may argue that that is not very critical, considering required efforts to do otherwise, but... I just couldn't persuade myself into making something so clearly not optimal.

So, I thought about changing data file format, for both game data and savedgame, into freely ordered sequence of blocks, which store their types and, in case of blocks having array of game objects in them, table of offsets for each such object. That's not particulary complicated, just a routine task that need an accurate approach. In the end that would be same data, only wrapped into self-descriprive chunks.

Along the way (while I was discussing this with other developers), I was even suggested to switch to text representation instead (XML or JSON), because that would automatically grant TOC support. While I love that idea (for multiple reasons), I had to decline at that point, because -
1. That would require to completely change the way data is parsed, from purely binary form into mixed text + binary, and probably even zipped, form. This could mean new unknown issues, and I was not confident enough to put that in right away, considering the lack of human/time resource involved.
2. I was not sure if developers will like the idea of having their data exposed in such way.

Alan v.Drake

#10
CW, is this really a sane feature or is it just going to complicate things? I remember being very skeptical about it when you mentioned this stuff.
A savegame is worth nothing, it so far below below recognition, what's with all the hardon about upgradeable saves? This is madness.
Breaking compatibility is the expected behaviour. You know you're all going to regret this.

Why cant game developers just make hardcoded checkpoints for players so they can recover from there without fear of mismatches?

You want to handle old saves? Add some metadata to the save file instead, make it readable without actually loading the savegame and there you have it.

EDIT: Let me expand, with custom metadata handled by the game dev himself, there's no need to even load the savestate, he can check it's a older version and use the metadata to rebuild a consistent state (restart the chapter, whatever). No need to bother listing conflicts and mumbo jumbos. Also they could be useful for other reasons like showing how much gold the player had in that savegame, etc.

- Alan

Snarky

Quote from: Alan v.Drake on Fri 21/04/2017 06:29:58
CW, is this really a sane feature or is it just going to complicate things? I remember being very skeptical about it when you mentioned this stuff.
A savegame is worth nothing, it so far below below recognition, what's with all the hardon about upgradeable saves? This is madness.
Breaking compatibility is the expected behaviour. You know you're all going to regret this.

It's a big deal for commercial games (particularly if distributed on Steam, where updates are pushed automatically), because wiping out all savegames for all players is not really acceptable:

Quote from: Dave Gilbert on Sat 09/04/2016 19:58:30
This is one of our biggest issues with AGS and is the main reason why we don't update our games as often as we'd like. :(

And you can't really expect game makers to write their own savegame system for every game.

Crimson Wizard

#12
This was supposed to be an optional feature that helps you to upgrade your saves if you want to.

If you are concerned about wasted work, I would say that around 90% of code I wrote for this task became useful for other purposes, such as creating cleaner save format  and big refactoring of saving/loading process for both game data and savegames (some of it is already in the current version).


Quote from: Alan v.Drake on Fri 21/04/2017 06:29:58
Why cant game developers just make hardcoded checkpoints for players so they can recover from there without fear of mismatches?

You want to handle old saves? Add some metadata to the save file instead, make it readable without actually loading the savegame and there you have it.

This was also in plans - to support user data in saves.

Some will probably do this, for some games. But others won't. There are AGS users who do not have big experience with game making and working around saves.

Also, there are kinds of games and/or situations where user would hate if they will have to restore and find their character (and other objects) not in position they left them, simply because game developer fixed few bugs and released a patch.

The usual kind of problem people meet is not related to stuff like DLCs or big game updates. They make a complete game and release it, then someone finds a bug mid-game. To fix this bug they have to add couple of script variables. They add them - and all saves break. That makes zero sense, but this is because AGS works like that.

This is an often problem, and easy to get in. So I thought I'd add a ready system which lets you solve it with particular method.

It's either that, or you scrap everything and design completely new save system for AGS, with "metadata", as you say, and make it easy for everyone to use.
At that time I did not feel quite ready for doing that. Neither I am now.

I was dealing with existing system which already works in certain way, so I was looking for solution that could be used in this system.

Alan v.Drake

Quote from: Crimson Wizard on Fri 21/04/2017 09:21:28
If you are concerned about wasted work, I would say that around 90% of code I wrote for this task became useful for other purposes, such as creating cleaner save format  and big refactoring of saving/loading process for both game data and savegames (some of it is already in the current version).

If it make things cleaner, all the better.


Quote from: Crimson Wizard on Fri 21/04/2017 09:21:28
This was also in plans - to support user data in saves.

Some will probably do this, for some games. But others won't. There are AGS users who do not have big experience with game making and working around saves.

Also, there are kinds of games and/or situations where user would hate if they will have to restore and find their character (and other objects) not in position they left them, simply because game developer fixed few bugs and released a patch.

Those who want to update commercial games must learn to handle these things, no other engine holds your hand in this regard, AGS is easy because it does all the work for you, but this comes at a price. There is no direct control on how everything is being saved or loaded, even with a callback it won't be possible to guarantee that the savestate is free from undefined behaviours. Undefined behaviours are the source of all evils and should be reason enough to prevent the loading of broken savestates.

Adding custom user data in saves can enable those who need it to prepare a fallback by storing the essential information, recognize that the save is for an older version and perhaps restart the game from chapter X instead of restoring the state.


I don't want to be a dick and this is just my opinion, but making mismatching savestates able to be loaded means that *absolutely everyone* will exploit this feature, even if they don't understand how to use it or simply don't care. Then they will come here, complaining that something doesn't work right.
Never ever underestimate fools.


- Alan

Snarky

Quote from: Alan v.Drake on Fri 21/04/2017 13:53:33
Those who want to update commercial games must learn to handle these things

Or we can put in useful features that allow people to make adventure games without having to become experts in a pretty specialized programming niche. Isn't that what AGS is all about?

Quoteno other engine holds your hand in this regard

As we've been saying, AGS is actually really good in some ways.

Quoteeven with a callback it won't be possible to guarantee that the savestate is free from undefined behaviours.

99% of the time, changes will be pretty minor. As CW says, fixing bugs by adding a couple of variables, where it should be very doable to avoid undefined behaviors.

QuoteI don't want to be a dick and this is just my opinion, but making mismatching savestates able to be loaded means that *absolutely everyone* will exploit this feature, even if they don't understand how to use it or simply don't care. Then they will come here, complaining that something doesn't work right.

That sounds massively implausible to me, because:

1. Most AGS games aren't updated after release in the first place.
2. Since most game makers won't need this, and may not even encounter the problem, it's not something they'll be looking for.
3. Even if they do, why would they go to so much trouble to load old savegames unless their game is "professional-level"?

Mandle

If you are releasing a commercial game with the current AGS engine wouldn't it be best to add about 10 extra dummy sound files, 20 extra generic Global Variables, 30 extra dummy sprites, 10 extra dummy views, 10 extra dummy characters, and whatever else I'm forgetting, and then fill these in as needed for potential updates...

Just an idea, and if anyone has more experience with what exactly the right way to do this in is...please reply in this thread with the best method(s).

Or if I'm wrong, please explain why...

Crimson Wizard

Quote from: Mandle on Fri 21/04/2017 15:05:29...please reply in this thread with the best method(s).

Please don't... this thread is not about how to make it work with current restrictions, but how to develop savegame system further.

Alan v.Drake

Quote from: Snarky on Fri 21/04/2017 14:48:51
1. Most AGS games aren't updated after release in the first place.

You're forgetting game devs and their thousands tests before release!

Quote from: Snarky on Fri 21/04/2017 14:48:51
2. Since most game makers won't need this, and may not even encounter the problem, it's not something they'll be looking for.

They won't... until they hear about it on the forums. "heelo freinds, I made game and found fix for bad savegame load crash - cheers :·))" - first result from google (laugh)


Quote from: Snarky on Fri 21/04/2017 14:48:51
3. Even if they do, why would they go to so much trouble to load old savegames unless their game is "professional-level"?

Well that's exactly the point, I would hope they didn't bother, unless the game is "professional-level".
Other cases may be not-adventure games where the savestate could be useful but not mandatory, AGS is flexible so you can never tell.

- Alan

Dave Gilbert

"Most AGS games aren't updated after release in the first place."

I can say with certainty that the savegame incompatibility issue DISCOURAGES me from updating my games as often as I'd like to. I take all the usual precautions - lot of dummy global variables, characters, etc. There's always a small detail that I fail to foresee that breaks everyone's saves completely. And if I want to update to a new version of the AGS engine (as I did recently with Gemini Rue) it's impossible. Those savefiles are gone.

Crimson Wizard

#19
I can certainly tell that I am now persuaded to not do anything at all. All those discussions make my brain hurt, same does thinking about fixing AGS in any way.

I already said before that I lack the game developer's perspective. The only way to do something with that is to leave making purely engine, and develop games to see how engine can fit to it.


E: oh, scrap that. I don't care.

Dave Gilbert

#20
I can think of three instances in the past where we hired someone to fix this problem (or make it easier to deal with in some way) and each time we were told it was impossible or just too difficult/time consuming to be worth it. So we've learned to just accept it. I get the impression we'd need to design our own specific save system if we were going to do this. Which... is difficult, but not impossible.

Stupot

Patches and fixes should be a last resort, if they're going to break player saves. Preferably the game has been vigorously tested by people other than KS backers and friends.
I always feel sorry for people who buy a game on launch, because they are the true beta testers and they din't even know it.

SMF spam blocked by CleanTalk