Save system overhaul?

[Hobbes]

Thinking about Dualnames' post the other day in the Strangeland topic I was wondering if it were viable to overhaul the save game system for an upcoming release of AGS? The current way it works breaks saved games when games are updated, which I would love to see fixed. Now I understand people program loads of custom functions, plugins, etc into AGS, so I see why the saving-memory approach is used. But could we perhaps consider adding in a second save-game-system that people can choose in the editor? It could be started as a "basic one" that collects:

- Integers (Global & Room)
- Inventory Items
- Characters & locations
- Player current room, etc

I'm sure there's loads of other things that would need to be collected, but the more we can build a save system that creates version-independent saves, the nicer it'd be. My personal reasoning is that I'm toying with a modular approach (episodic structure) for a game and would be releasing multiple episodes within the same AGS game (similar to how Life is Strange 1 and 2 were released, for example, or the Walking Dead games). Currently, if I were to release Episode 1 and people would save the game and I'd then release an update a few months later with Episode 2, their saved games wouldn't work anymore.

Would there be any merit to such a design? The Editor could allow a "use memory-saving system (stable)" or "use modular saving system (experimental)" whilst we work on it, for example.

Looking forward to hearing some thoughts on this feature request/idea.

[Crimson Wizard]

We had numerous discussions before, here and elsewhere; you probably will be able to find some posts or even full threads on this. I do not want to go into full technical details right now, but I must leave this comment.

This article that summarize all knowledge regarding breaking saves and workarounds; this may be useful to assess current situation:
https://adventuregamestudio.github.io/ags-manual/GameSavesCompatibility.html



Before designing a non-breaking save system it is important to note what makes current save system break after a game update.

The answer to that is simple, but hides numerous technical issues underneath.

It is lack of reliable identification of game objects within the game data.

Putting this in a simplier words, there's currently situations when there's no reliable way to tell if the game data contains same object as in the save file, or a different one. Sometimes it's only possible to tell if their total number is different.
And that's why engine quits with error when it sees that object count is different; however, you may delete some object and create a new one, and engine will restore the save. Except, in the above case it will restore old data into the new object, because it is not taught to distinguish them; and in certain cases it's simply impossible to distinguish them.

If the game object has a reliable ID, then all you got to do, as a first step, is make sure that ID is both in the game data, and in the save. And when the save is loaded engine would find matches and restore the data into the correct object. Then there is the case when the object no longer exists in the game data, and the case when a new object exists in the game data but not in save. But these are a separate issue, and its resolution may depend on chosen design. I won't go there right now.

How came that objects in AGS are impossible to reliably identify? First of all, not all of them have mandatory script names.
Room objects, for example, have "Script name" field, but it is not required to be filled. Some of the areas don't have the name at all (walkable areas etc).
Views have the names, but these are not saved into the game, so you work with views using numbers.
But that's not too hard to add, and create a rule that a script name should exist always and be globally unique (provide some default generated name if the user won't give one).

Then comes the script. When script is compiled all of the variable names are discarded, they are not addressed as names in the engine, but by their memory offsets.
So for the variables to be identifiable:
* their names should be kept and saved in the compiled script data (e.g. as a table for matching offset:name);
* their names should also be saved when writing a save;

That might be not all though. Some variables are structs, or pointers to objects, and knowing their name is not enough, you also got to know their type, because if types don't match you may end up loading incompatible data there. Perhaps for simplicity sake this could be replaced with a simple size check, but that may lead to complicated results, as structs may also contain pointers, and if you load a wrong value into a pointer, that will crash the engine or cause bizzare effects on a game.
So for fully controlled result you need to have:
* names of variables;
* types of variables;
* at least some information of what the type (struct) itself contains inside (other variables and their types) -- this is so called RTTI - the runtime type information.



All the above is said about fully automatic system. Then, it might be possible to consider a one that is at least partially controlled by the game developer. That is - have a system where not the engine, but the gamedev identifies the objects and variables.

Such system, probably, would require to have functions for saving builtin game types, and variables of basic types; and of course of loading them. Developer would provide names, so engine would not have to care about that. It will be up to a human to tell exactly what to save or in which object to load the data under that name. Engine will have to know how to save that type of data: a Character, an integer, and so on.

[fernewelten]

As far as I know, Visionaire has robust save files (but AFAIK, they don't have user-defined public structs). How do they do it?

[Potajito]

A first step would be to have the editor tell you when are you doing something that breaks the save. On paper doesn't seem that hard, but I don't know how that works on the inside.

[Crimson Wizard]

I forgot, we have this article that summarize all knowledge regarding breaking saves and workarounds; this may be useful to assess current situation:
https://adventuregamestudio.github.io/ags-manual/GameSavesCompatibility.html

(also will add to my big post above)

A first step would be to have the editor tell you when are you doing something that breaks the save. On paper doesn't seem that hard, but I don't know how that works on the inside.

Almost everything will break the saves (see the article above), I have doubts if it's worth to put much effort into detecting and reporting these changes. Perhaps it's more productive to put this effort into bringing the engine closer to the better new system instead.

[fernewelten]

Let's suppose we had 10 characters, configured in the Editor. At the start of a game, all the characters are at factory default (as configured in the Editor).

As far as I understand the description in the manual page:

• When the game starts, all characters are at factory default (i.e., as configured in the Editor).
• When savefile is saved, then all 10 characters are saved, and it is saved that we have 10 characters.
• When savefile is loaded, then all its 10 characters are loaded, and it is loaded that we have 10 characters.

Now an 11th character is added.

• When an old savefile is loaded, then its 10 characters are loaded, and it is loaded that we have 10 characters in all, overwriting the information in the new code that there are 11.

Why? Let's do this differently. When a savefile is loaded, only change those entities that are in there (identified by their ID), but don't change the number of entities overall. All the missing entities would be simply left at factory default, i.e., as set up in the Editor. Throw a suitable event after the restore so that the programmer can decide what to do with those entities that are missing. It would be on the programmer to find out what would be sensible to do.

This could probably be done in a way that is robust against additions to characters, guis, inventory objects, and views, at the end of the respective array. Insofar as the restore operation itself is concerned.

Code that would, e.g., store the number of characters in a variable would still break when that variable is restored, but that would be on the programmer to resolve.

This would probably go a long way towards alleviating the pain that the game can't be modified once old savefiles exist.

[Crimson Wizard]

Why? Let's do this differently. When a savefile is loaded, only change those entities that are in there (identified by their ID), but don't change the number of entities overall. All the missing entities would be simply left at factory default, i.e., as set up in the Editor. Throw a suitable event after the restore so that the programmer can decide what to do with those entities that are missing. It would be on the programmer to find out what would be sensible to do.

I already had a code and experimental version that did that. But it was thought to be dangerous and confusing to users, so I cancelled it.

https://www.adventuregamestudio.co.uk/forums/index.php?topic=53753.0

It works exactly as you say: only restores as many items as possible, and leaves the added ones be. It called a script callback for each non-restored object.

The voiced concern was that if we add that users will have wrong expectation of how it works and have more trouble with it.
For example, it won't protect you if you rearrange objects by changing their IDs, or change the order of variables in script, or remove objects & variables etc.

EDIT: but yes, the code is still there in my personal repository...
https://github.com/ivan-mogilko/ags-refactoring/commits/feature--loadoldgamesave
and it does not make that many changes now (back then it also contained new savegame format, but that part has been applied).

I mean... some people were trying to persuade me to finish this, some were against. I guess it was the period of me being disheartened by the bad development progress, so I gave up on this.

[fernewelten]

The voiced concern was that if we add that users will have wrong expectation of how it works and have more trouble with it.

So we might perhaps add another flag to our "General" section. Call it, e.g., "Allow restoring outdated save files". This flag would be False by default and make the Engine throw up as soon as it needs to restore something that hasn't been saved (the current way it works).

However, if a game coder sets it to True and compiles a game with it, then by that they promise that they won't let themselves be confused and that they'll be happy and thankful for whatever we can offer them here. 

Crimson's prototype code even seems to have provided such a flag, too.



Reading to the old thread that Crimson Wizard linked to, I can only feel that his idea had much more acclaim than might be recalled now. Lots of voices saying "Let's do this!"

For me, the most elegant idea of that thread is Crimson's idea to save the game starting state in a savefile. (Use the same code to write it but call it differently from other savefiles so that users don't meddle with the file by accident.) Then, on restoring a game, use the starting state file as a "default" for all the entities that the game needs and the savefile doesn't supply.

I also like Dave Gilbert's idea of a "lock" that you can set in the Editor somewhere. If the lock is set, the Editor doesn't let the user do anything that might break saves.

[eri0o]

I've played with my own SQLite plugin and realized I could have saves with versions and handle migrations in DB like manner for savegames. It's a bit unusual but seems to work, the hard part in writing a custom savegame is it's very boring and slow to script, since you need to manually pickup all things field by field. (see my "module" here: https://www.adventuregamestudio.co.uk/forums/index.php?topic=55398.0)

[Hobbes]

Dear all,

Thanks for the engagement on this topic - the more I look at these ideas the more it seems that there's a way to get it done, perhaps?

Eri0o, in true "me fashion" not knowing anything about programming, is SQLite the "easiest" way forward or would something like CastleDB perhaps be a useful alternative? It seems to be integrated into a different system, but the GitHub version is remaining open source.

Dualnames mentioned that his work on the Strangeland save system is available on GitHub (I don't know where exactly) but perhaps there's useable code there to integrate moving forward? Together with Fernewelten's suggestions from the previous thread, it seems that there might be a good path to try:

1. The idea of having a "base save" that saves the game's starting stats.
2. Save games track changes from that base save onwards.
3. Save game format: SQLite, current format, CastleDB, etc?
4. Editor mode to "Lock" when you do save-breaking stuff
5. Find a method of adding recognisable identifiers to all things that "need" saving
6. Add a "use new save system (experimental)" to Editor once we're ready to beta-test

Would that encapsulate all the conversations had so far?

[heltenjon]

1. The idea of having a "base save" that saves the game's starting stats.

Isn't that what Gugames did with Kill Yourself?

I also seem to remember that Zniw had updates without breaking saves.

[eri0o]

There's no silverbullet, I should have been more specific, Database migrations are handwritten.

What I meant about using my already existing SQLite plugin is if you have experience writing SQL DB migrations, you can reuse this knowledge to write your own for your own game.

[Crimson Wizard]

Hmm, I do not think that format should be a part of this particular issue tbh (non-breaking saves), as assuming other problems solved, format could be anything in theory. The difference may be in perfomance and convenience.

5. Find a method of adding recognisable identifiers to all things that "need" saving

This should be point 1 imo, because without that anything else won't work for the wanted result.

4. Editor mode to "Lock" when you do save-breaking stuff

I fear this might be quite difficult and inconvenient to implement. Also why is this necessary if you goal is non-breaking saves?

[CrashPL]

I also seem to remember that Zniw had updates without breaking saves.

Yep, we had 6 major updates that introduced some additional goodies (like the OST as free DLC, the digital manual etc.), and several bugfixes/QoL improvements, without breaking the saves between versions (a 1.0.0 save will work just fine in the most recent 1.3.1). I managed to achieve it with a combination of the following

- following CWs advice, I've left several empty global variables, reserved for future use, as adding any new one at this point would render all older saves incompatible. That was a very wise decision, considering one bug would be notoriously difficult to fix without an additional flag. With the empty extra variables, it was a matter of a simple 5 minutes fix,
- similarly, I've also left several empty/unused sprite slots. They came in handy for things like additional backpack graphics (introduced in 1.0.4), or some extra assets for the Mac port,
- the game actually uses both standard AGS saves, as well as additional "save slots" files to store extra data. These allowed me to create the custom load routine, as well as allowed to implement the new game+/checking the encyclopedia/extras straight from the main menu. These can be freely modified, as they're not a part of the AGS core files.

So yeah, it is possible to future-proof your AGS game and update it without breaking older saves, but indeed, I'd welcome any change with open arms, especially the one that would actually let me add some new content in subsequent updates (maybe not as extreme as new rooms or characters, but at least some extra variables/scripts or sprites would be great)