[Feature suggestion] less pre-compiled binary files, more direct assets

Started by Monsieur OUXX, Mon 23/12/2019 00:39:38

Previous topic - Next topic

Monsieur OUXX

Now that 3.5 is out (amazing work by the way) I'd like to push the idea of maybe considering making AGS more source-control friendly. At the moment it's always stressful to have several people use an AGS project at the same time. We haven't had a catastrophic failure yet, but we had a few conflicts or sprite IDs mixups, that sort of things. Leaving some files out of git is a hazard. Also not everyone knows how to use git, and less clever systems (like dropbox) are not too good at resolving conflicts.

Several things could be done :

  • getting closer to an "assets" system (as in Unity and Godot) where the files used are the actual files (script files, images) instead of a few annoying binary, pre-compiled files (.crm files, sprite cache, etc.). --> This would make tracking much easier (time stamps, etc.)
  • The main project file (game.ags) could be more concurrent-access-friendly
  • The IDs generation could be more concurrent-access-friendly
I'll explain below what I mean each time.

Of course, as always, the tricky part would be to still make it easy to move the project folder around without "losing" some files. For that, the so-called "assets" should be in a subfolder inside of the project folder. We would keep the import system and absolutely not replace it with a system of dynamic links.
Another tricky part would be to prevent the player from tempering manually with the assets.

But I think that very simple things could be done to kickstart this : For example, the sprite cache could be written in the Debug folder instead of the project's root folder, just like the .EXE file, since it's part of the output. The cache file would be replaced with Cache folder where all the files remain individual files. And to prevent the user from tempreing with them they could have cryptic names (such as "img0001.123" where 123 is the ID of the sprite?

Similarly, the project file (game.ags) could be split into several files (gui.xml, characters.xml...), all placed into a "Game.ags" folder in the project root.

Finally, I humbly suggest that the IDs should be recycled less . I think that when a sprite is deleted, its ID shouldn't be made immediately available again. Instead, the IDs should continuously increase, and only when they reach the maximum value should the unused IDs be recycled. This would be an imperfect-yet-cheap way of avoiding mixups with IDs. Because I use source control and shared folders (like Dropbox) it happened to me that some conflict let AGS believe that some views were still using some deleted sprites. It didn't corrupt the game (lucky me but it was showing views with bogus sprites that had been imported over the deleted sprites.
 

morganw

"less pre-compiled binary files, more direct assets"

^ I think this is the general direction that is being steered towards, since it solves issues for people using version control, removes some workflow issues in Editor, and also shifts the actual building of the game files as late as possible (which is better for building on platforms that aren't Windows, and for automated builds).

Currently there is no long-term design to work towards, and partly this is because the issue of how backwards compatible to be is not resolved. So at the moment I probably shouldn't comment too much, but just to say everything you raise is valid and already something I would like to change (in some cases I would like to go a little further too). There are also some issues with resource management with the engine, so potentially the asset management problem could be solved from both ends.

So to summarise, thanks for your suggestions and I hope that a link will be added here once the future development direction is decided...

Crimson Wizard

Quote from: morganw on Mon 23/12/2019 21:39:18
Currently there is no long-term design to work towards, and partly this is because the issue of how backwards compatible to be is not resolved. So at the moment I probably shouldn't comment too much, ...

So to summarise, thanks for your suggestions and I hope that a link will be added here once the future development direction is decided...

Backward compatibility should not restrict project goals.
I think we should just do it and define an "ideal" goal without fearing possible implementation problems, and then proceed from there. Maybe backward compatibility will turn out to be not that much of an obstacle, or it's possible to work around it. Maybe we will have to define temporary/intermediate steps. But if we have a proper goal we at least will see a big picture and know where we're going, instead of trying to find a step every time. This will also help in case of other contributors or users asking questions and giving suggestions (as Monsieur OUXX does now), as we will be able to compare clearly with what we want.

Note that, in terms of backward compatibility, Editor is probably easiest, because it's mostly a matter of converting one data into another. Which could be done even by an external tool (I think several people suggested that),
For engine you'd have to not only do that, but also provide old-style behavior, but that may be supported by a separate game library even.

Monsieur OUXX

Everything noted above is valid, but please do note that everything I suggested would require to change the Editor a little bit, but not the engine.
 

JanetC

I would love sprite files to be able to be stored separately. Unavowed has a 1.5gb sprite file and it needs to be uploaded in its entirety every time I change one sprite!

Crimson Wizard

Quote from: JanetC on Wed 30/12/2020 22:21:20
I would love sprite files to be able to be stored separately. Unavowed has a 1.5gb sprite file and it needs to be uploaded in its entirety every time I change one sprite!

Are you speaking of when working with editor, or final game?


If about editor, this is in TODO for the next major version of AGS, it's simply that someone has to work on this and change how editor works.
- make editor to cache files directly from the disk
- compilation process include generation of sprite pack which is then included into game.
Also, this probably will require a rule that all sprite assets are stored within the project files, and never outside.

AFAIK similar work is currently in progress to make rooms a set of separate assets stored on disk instead of a compiled blob.


If it's about the game, firstly there's already a workaround to have sprites as separate files in the game dir or subdirs, and then load them into dynamic sprites.

For a package solution, engine will have to
1) support to have more than 1 data package. This is practically supported in the wip branch, but not used for anything in particular yet.
2) support streaming sprites from more than one place. I guess that might be possible to do by adjusting how sprite caching system works.
3) also we need to be able to configure creation of these extra packs in the editor, or using a pipeline of standalone tools.


An ideal future goal would be to have game files packed in some easy packages, even like zip, which could be added by engine in any number simply by finding them in the game dir, and seen as one virtual filesystem, like many engines do.

SMF spam blocked by CleanTalk