Splitting up Game.agf (and possibly CRMs and sprite file)

[mjomble]

I've found the current AGS file structure to be fairly inconvenient for concurrent editing (for example, sharing the project via SVN), mainly for the following reasons:

• Game.agf contains timestamps of audio files which seem to get updated a lot. Sharing this between different people leads to lots of needless editing conflicts.
  
  
  
  • This is amplified by the fact that the AGF file contains too many things. A tiny edit almost anywhere will modify the file and bring out the timestamp issue.
• It seems CRM files contain a compiled version of the ASC source. If one user modifies the background image and another user only modifies the script, both will get a modified CRM file that's impossible to merge because of its binary format. There's a workaround for this - if you only modify the script, you don't really need to check in the modified CRM - but it's still quite inconvenient.
• The sprite file can easily grow huge and it's unmergeable.

So I'd propose the following solutions:

• Break Game.agf down into smaller files. For example, one XML for AudioClips, one for Sprites, one for Views, etc (possibly into an XML subfolder). The timestamp issue would still be there, but it'd be limited to a much smaller scope.
  
  
  
  • I've also considered moving the timestamps into a separate "local settings" file that would be auto-generated if missing and never checked into source control.
• Break down CRM files, to separate the compiled code from the more static content. Possibly move all Room*.* files into a Rooms subfolder.
• Break the sprite file down, possibly something like one actual file per one "virtual" folder you see when viewing sprites in the editor.

I've started working on this for our project and currently I've got a proof of concept for the AGF split done.

But I realised there might be more interest in this in the community and we might eventually want something like this in the official AGS as well (provided it's backwards compatible). Is this the case?

[Sephiroth]

As I see it, the Head of the project would be the only one actually using the AGS project file, you would just have to send the resources to that person (audio clips, art, scripts).

Sometimes, not always, the programmer is the one who receives all the media content other members have produced, he's then importing and setting up everything in the game because this task requires access to the whole project file, he can then produce a compiled version for others to see the result.

But it would still be nice to have this kind of feature.

[Pumaman]

Game.agf contains timestamps of audio files which seem to get updated a lot. Sharing this between different people leads to lots of needless editing conflicts.

I think this timestamp thing is actually a bug in the editor, it seems to be doing something funny with the Summer Time +1 adjustment every time you load/save the game. It needs to be looked into.
If the actual audio files haven't changed, the timestamps shouldn't change either.

It seems CRM files contain a compiled version of the ASC source. If one user modifies the background image and another user only modifies the script, both will get a modified CRM file that's impossible to merge because of its binary format. There's a workaround for this - if you only modify the script, you don't really need to check in the modified CRM - but it's still quite inconvenient.

Yes, when you change the script, it is compiled and the bytecode is written to the CRM. The problem here is that the CRM functions as both a "source" file (in terms of room backgrounds, objects, etc) and a "compiled" file (in terms of the script). It would be nice to separate this out at some stage.

The sprite file can easily grow huge and it's unmergeable.

This can be an issue, yes. At the moment the editor and engine use the same format for the sprite file, so the benefit here is that if you have a lot of sprites, AGS doesn't have to do any work to build/run the game (since the engine can just access the sprite file that is already there).

[mjomble]

Phase 1 complete - Game.agf has been split.

You can find my changes here: http://dl.dropbox.com/u/3110371/ags/AGS_Game_agf_breakdown.zip - unzip these files over revision 48 from the SVN.

It's backwards compatible and creates new files in the XML subfolder, leaving Game.agf fairly small (in our project's case, under 6 KB).

If anyone with commit rights to the main SVN finds it useful, feel free to add this to the official codebase.

Could use a review by more experienced devs, though. I also left a TODO note in AGS.Editor.AGSEditor.LoadGameFile about a version check that should be added; it's up to you which exact values to use, based on the release plans.

Gonna look into splitting CRMs next, but my gut feeling tells me it'll have to wait until AGS.Native is open sourced.

[Pumaman]

As I said, I don't think this is addressing the root cause of the problem... what we should be doing is fixing the problem that causes the audio file timestamps to keep changing.

[mjomble]

The timestamp thing is definitely a problem that should be looked into, although I'm not currently volunteering to do that.

However, I consider it to be a separate issue that Game.agf contains data for almost every aspect of the game. It makes project sharing less convenient even without the timestamp problem that amplifies it.

So I'd just like to hear the devs' opinion of it - if it's not going in the official version, we'll start using a custom build for our project; otherwise, we'll stay on the standard version and wait for the update.

[mjomble]

Note: if anyone's interested in integrating this to the official version, contact me first.
I'd like to modify this so that the backwards compatibility logic would be separated from the main logic as much as possible. Otherwise it's a slippery slope towards an unmaintainable mess of intertwined legacy code.
But I'm not going to put much effort into refactoring this if I'm just talking to myself here

[mjomble]

As for the idea of splitting up some of the project files, I'm not sure that I see the point of it. It seems a very rare and odd occurrence that anything should actually go so wrong as to corrupt the AGF file. I'd reread your other thread, but I'm not going to be online too much longer. I'll take a look later on.

We haven't had any problems with file corruption, but the sheer size of the file (currently over 5MB for our project) leads to various issues.

For example, comparing two versions of the file gets pretty slow. I personally do this quite often. When other team members make generic changes in the editor and commit the results, I can check the diff of this file to see what exactly happened. Or before I check in my own changes, I want to see what actually changed in the files and spot any temporary hacks I made that weren't supposed to go in (like changing the starting room of the player character, etc).

On a slow connection, this can take a while, though, because even for a single line of change, I need to download two versions of the entire file (>10MB), containing things like indexes of all the sprites and views which I'm not interested in at the time.

Also, if for example, one person changes some sprites and audio files and another one changes character properties and some dialogue, it'd be nice to see from the file level that there are changes in Sprites.xml, Audio.xml, Characters.xml and Dialogue.xml rather than 4 edits to a massive, generic Game.agf where I'd have to review 4 large diffs to get that information. And there could be a good chance I'm not even interested in the sprite and audio changes and could just skip reviewing those files. Sure, commit comments can help, but you can't always rely on them.

There's probably not a lot of AGS projects that use version control and within those teams, many members may not bother to check these diffs. But for those that do, this would be an appreciated change. And making AGS more version control friendly could encourage larger teams to use it for larger projects.

Besides, you've already got the code that does this   ...well, except for these last changes I want to make. But if there's any chance this will go in the official version, I'll get right on them.