AGS data format ID change

[Crimson Wizard]

I've come to the decision that we need to change the data version format in both source project and compiled game.
For legacy reasons they are currently defined as a simple number in a sequence, with higher number meaning later and more advanced data content.

However, this does not suit well the parallel development when older and newer versions of AGS are being worked on and updated simultaneously.
It is possible that we would like to change data format in update to current stable version. In such case the next number may already be taken by version in development.
Picking even higher number will technically work, but it creates a confusing situation when a format id in AGS release with lower version is higher than the one in the next WIP version.

Besides, the single sequence of numbers cannot explicitly describe the situation when two branches of development have unique features.
For example, AGS 3.3.4 may have data format-changing features A,B,C, and AGS 3.4.0.6 has features A,B,C and D,E,F. Then we release an update 3.3.5 with feature K. This feature is not present in any release of the 3.4.0 yet.

If we use single number sequence, as described above, we may have, e.g:

AGS Release	Data Version	Features
3.3.4	10	A,B,C
3.3.5	20	A,B,C,K
3.4.0.6	15	A,B,C,D,E,F

This may be confusing for developers, because they need to keep these differences in mind, and also there is no easy way to write a version check for Editor or Engine, to let it detect more advanced version, and therefore at least provide a sane error message when trying to open unsupported project.

Even less this would suit if someone would make alternative variants of AGS for their own use or amusement.



The most flexible implementation would be to store a list of feature string IDs inside game data, and have program compare this list to the ones it supports.
For example, game data may have this written in some data header:
- FEATURE__CustomResolutions
- FEATURE__NewPathfinder
- FEATURE__ScriptAPI_KeyPressEmulation,
and whatnot.

The question is, will or not this be too complex for AGS? Or will it be too complex for current moment?



Another alternative that I was thinking about is more simple, although less explicit: use long versions strings. For example, we may call data format 3.3.XXX, where XXX is an internal number of data format update. Or this may simply be equal to release number: 3.3.4, 3.3.5, 3.4.0.6, etc.
This version will be compared by comparing each part individually, therefore it will always be known whether data format comes from lower or higher version of AGS.

The downside of this implementation is that the program would still need to know which is a latest format of previous version it supports.
To clarify, in the situation I described above, AGS 3.4.0.6 will support formats 3.3.4 and 3.4.0.6, and when given a game with data format 3.3.5 it will know that although this data format belongs to previous major version 3.3, it is still not supported in 3.4.



I would like to hear other devs opinions on this problem, for the solution has to be aimed for long future use.

[Alberth]

With your strings listing features, do you list features used in the data file, or features supported by the AGS version used to create the data file?

[Crimson Wizard]

With your strings listing features, do you list features used in the data file, or features supported by the AGS version used to create the data file?

I thought of the latter first, because it is more simple idea. But I see what you mean. We may look if the first option may be useful too.
I do not think there will be any difference for the engine.

Speaking of the two alternatives I mentioned above, I came to think that essentially they are same thing, its just that the second one has limitations.
We could implement "component table" as a string/integer map, and to simplify things for beginning, define components for each active major version (3.3 and 3.4) with integer value meaning their update. This should be enough for starters.

Currently I am looking for any opinion on possible flaws in this method, because it happened few times in the past when I screwed up and had to fix core features.

[sonneveld]

I like the idea of a feature list.  We could define all current features as a classic profile.  That would allow you to start deprecating features. Possibly introduce profiles like how OpenGL has a core profile and a compatibility profile.  Newer engines for iOS and and Android will be able to explicitly state which features they will support.

Do we want to worry about people being able to degrade functionality but still allow the game to run (ie, to be able to run on an engine with reduced functionality because of memory?).  That would mean that the game would need to allow feature checks via scripting and only abort if the game explicitly tries to use a feature the game doesn't support. (Maybe have the option to abort up front or during runtime)

As for versioning features, I'm not sure that is a good idea. You'll just have the same versioning issues that you're trying to avoid if two people work on the same feature in two branches. It might be better to have the features fine grained enough that the two people working on the feature will just add a new feature string. (although APIs tend append version numbers to names, ie function_name and function_name_2, but you know if if you have function_name, it will always do what it says it did)

However, there could be feature limits.  Say a feature only allows X widgets on the screen at once, the game could check what limits are set, and set it's resource usage accordingly. That way the same game could work on both low and high end systems.

Would plugins also define features?

I'm not sure encoding any information in the version string is a good idea. Probably just best to stick to semantic versioning where minor version changes are guaranteed not to break things.

[Alberth]

Currently I am looking for any opinion on possible flaws in this method, because it happened few times in the past when I screwed up and had to fix core features.

The danger of the strings, especially if you take the 'used features' route that I seem to have proposed, is that the same string should always have the same meaning.

The questions that thus arise, what if you have to modify a old feature when it touches a new feature? Does the old feature get a new string? If so, can you still load the 'old string' feature? What if the game file doesn't use the new feature?
Tbh, I don't see an easy answer here, it may give rise to very subtle bugs.

As for your versioning number system, the 3.3.XXX number system basically makes versions for each major release independent (such as major release 3.3). You will need a XXX number for each previous major release that you support. While it is some work to do for a release, tables are not likely to become very big.
It all becomes a bit magic though, a user has no simple way to query the set of supported version (or you have to add that), so loading a data file into a newer version becomes a bit of a Russian roulette, although with a much less deadly end if it fails.

Using the complete version number as format eliminates the magic additional XXX, which is probably a Good Thing(tm), less numbers to get confused with, but the Russian roulette is not really solved, I think. You still need to know what versions of previous branches are understood by each version.

In all, I think it's quite complicated.


Did you consider dropping the change of data file version numbers within a released branch? For example, if 3.3.1 (First public release of the 3.3 branch, as I am not sure my number is actually correct.) has data file version 43, all 3.3.x versions (x >= 1) will have 43 as data file version.
Obviously, that will block certain changes, in particular, adding new features becomes impossible. It will lead to shorter life times for the various branches I think (At least, it will not make them longer.)

On the other hand, it greatly simplifies your data file version management problem, and reduces the porting effort. (New features are not portable to an existing version. If you want them, please upgrade.)

Development time is a scarce resource, reducing complexity of the builds and releases may help.

[Crimson Wizard]

@sonneveld
What you say (deprecating, plugin features, etc) is going far from my original purpose. I cannot say these are bad ideas, its just that it is something that I did not have in mind, nor thought through.

The simple case we are usually having is when we need to add one more option to the game (for example, to the General Settings). This means that
a) Editor should read this option as value from XML;
b) Engine should know about 1 extra integer to be read from binary stream, and how to interpret it.
Well, basic stuff.
All I need is a way to make both Editor and Engine to be able to determine (on project/game load), whether it is capable of doing that.


With this new versioning system, I would like to start small, but leave a possibility to expand later (to the possibilities you mentioned, or others we did not realize yet).
I am not sure yet whether to call these IDs "caps" (capabilities) or "components"; anyway, we could have an ID defined for every version of AGS where the game data was modified. This is not hard at all, for we already have a list of data versions, and for most we found which versions of AGS they belonged to (see GameDataVersion and GuiVersion enums).
I.e., something like this:
- GAME_DATA_250
- GAME_DATA_251
- GAME_DATA_272
- GAME_DATA_330
... and so forth.

If ever there would be need to have more precise list of components, newer engines may translate each of these IDs into list of sub-components;
for example if it reads "GAME_DATA_340" from the game data file, it will internally substitute it with "CHARACTER_DATA_340", "ANIMATION_DATA_340", and so forth.

For older games the engine would have to translate a data number to the string ID.

The questions that thus arise, what if you have to modify a old feature when it touches a new feature? Does the old feature get a new string? If so, can you still load the 'old string' feature? What if the game file doesn't use the new feature?
Tbh, I don't see an easy answer here, it may give rise to very subtle bugs.

But this is what we do in our engine today, in order to support backwards compatibility. Our AGS 3.3 engine can read game data as old as of AGS 2.5.
It remembers the game data number provided in the file, and then makes forks in execution for every feature that had differences, depending on game version.

Did you consider dropping the change of data file version numbers within a released branch? For example, if 3.3.1 (First public release of the 3.3 branch, as I am not sure my number is actually correct.) has data file version 43, all 3.3.x versions (x >= 1) will have 43 as data file version.
Obviously, that will block certain changes, in particular, adding new features becomes impossible. It will lead to shorter life times for the various branches I think (At least, it will not make them longer.)

Exactly the problem I am having is that I was considering to add a new option to next 3.3.* update, that would alter the data format a bit.
And it is true that this problem originates in longer development times. However, I would expect a situation when game developers would ask for some small additions to the old version they are using.
We could have features introduced in such an order, that includes breaking changes in the middle, making it difficult to upgrade. Or we screwed up and were not able to make a stable version of next release yet, so it would be easier to add the smaller feature on top of the previous stable version.

Also, it is not uncommon for AGS users to keep working with 1-2 year old (or even 3-4 year old) program. One of the purposes of AGS was always a hobbyist game making, and that could have irregular working schedule. A person can start a game in year 2015, then take a hiatus and continue in 2017; but he won't like to upgrade his AGS, because he already has a tonn of scripts, and we, for example, deprecated some of the script API in last versions, or introduced other breaking changes.

The situations we may get into are various. I would like to have a way to deal with them.

[Alberth]

I understand all trouble comes from keeping branches alive for a long time

Could you explain one bit though please? If you update 3.3, the users would have to install a new version too, right?
Why is upgrade 3.3.x.y -> 3.3.x.(y+n) feasible, and upgrade to 'latest stable' not feasible?

[Crimson Wizard]

Could you explain one bit though please? If you update 3.3, the users would have to install a new version too, right?
Why is upgrade 3.3.x.y -> 3.3.x.(y+n) feasible, and upgrade to 'latest stable' not feasible?

Um, "latest stable" IS 3.3.x.
We have a WIP version, which is 3.4.0, which was already released for public test.
If we take current situation, it is not a question of upgrading to "latest stable", but of upgrading to "latest unstable" (also having couple of breaking changes, which would require to modify scripts).

[Alberth]

Oh, sorry, I am clearly not up-to-date with AGS release numbers.

[Crimson Wizard]

Since that proved to be confusing in the past, I asked site admins to make me a new forum for release announcements:
http://www.adventuregamestudio.co.uk/forums/index.php?board=28.0

[Crimson Wizard]

To mark out implementation details,

1. When saving game, Editor write a header, containing set of string keys, "CAPS", that list the capabilities required to run this game properly. They include the game data versions, and extra features.
2. Engine defines a set of string keys, "CAPS", enumerating all supported data formats and features.
3. When loading the game, it reads the requires CAPS from game header, and compares with its own set. If any CAPS are not supported, engine quits, displaying an appropriate message.
4. When loading old game, which did not have CAPS definition yet, translate numeric data version into corresponding CAPS string to compare with engine CAPS set.

5. After compatibility check has passed, engine inits a "cache" of boolean/integer values based on game requirements; this struct is used further for faster checks, instead of checking capability strings every time.
This way we may keep all the existing version number checks in the engine, which switch to backwards compatible execution branches (at least for starters).

[Crimson Wizard]

Hmmm... a year later....
... although I never implemented this (I did some prototyping though), my thoughts keep returning to this thread. We have agreed on shorter development periods now, and while we may also take a recommendation to not change data formats for minor releases (or even try not to do parallel development of major and minor version), I would still like to have a backup solution if a need arises.

Maybe I will be looking into this again...