Topic: Feature Request: Store sprites as separate files

[morganw]

But if there is no sprite manager, how do you tell it about these exceptions? I think I am missing something in this plan.

Well it would need something, but this something will not import, create or delete sprites. So if there is just a sprite viewer, it could just hand tasks off to a metadata manager, or you could be specifying any changes settings outside of the editor.

As a rough example:

• Save a new sprite into the sprites folder
• Editor see a new path and assigns it a number (possibly yielding to the filename or a metadata file)
• User right clicks a sprite and choose "top left pixel as the transparency" as they don't want the default settings
• User right clicks a sprite and manually edits the sprite number

So for these actions:

• If the asset pipeline is consistent (i.e. you are being given or making graphics in a consistent format) it is likely you would only need to set default import settings and not make any exceptions to this
• If you do need to change something for a particular image, this can just be stored as an exception to the default, either in the XML of the project file or in a metadata file next to the image on the disk (in the case of the extra file, you would check the image settings into version control independently of the project file)
• If you need to make changes for multiple images, this can also be stored as an exception to the default (as above, in project file or metadata file), but referenced against a sprite folder (which is now just a folder on the disk)
• In the case of using metadata files, you could potentially have specified the import modes and ID number reservations before the editor ever saw the images, so potentially it just needs to validate the data

Also, do you mean that file format itself stores frame count? If not, what did you mean by "read the frame count of the file"?

I meant that, if you had something like a GIF file which had multiple frames, you could read the frame count from the file and allocate it the corresponding number of sprite ID numbers (instead of having to extract each frame to disk to use it).

Could you write all this in a form of a task, with all points elaborated?

I will do this. I'm guessing you mean on GitHub?

[Crimson Wizard]

Well it would need something, but this something will not import, create or delete sprites. So if there is just a sprite viewer, it could just hand tasks off to a metadata manager, or you could be specifying any changes settings outside of the editor.

This is like a complete overhaul to how you work with sprites in AGS. Are you sure the people will accept this?
This may be a regular workflow for professional artists, but, for example, I got used to import sprites from clipboard, that's very fast and you don't have to manage files.
And the tiled import... I was just drawing all animation sequence in 1 document, and then tile-importing them, again not thinking about cutting into files.

Perhaps there may be a way to keep these import commands? They may be simply creating new files and metadata according to the settings, work like alternate entry point into this system (main is copying file into folder).

[fernewelten]

Efficieny might be an issue here:

If a sprites file can have gigabytes, it will probably be comprised of thousands of separate, tiny little files. Currently, we don't notice the sprite generation because it is done piecemeal, file by file as they are imported one by one. This will change, however, when the sprite file is generated in bulk at compilation time. Thousands of tiny files will have to be opened on each compile. The time needed might be significant.

So we probably need to cache the generated sprites file. But even that is not so easy as it looks:
If the user doesn't tell AGS that some file is „imported“, then at compile time, the AGS compiler will have to peruse thousands of tiny little files in order to find out whether new ones have shown up or old ones have changed their content. So that might mean, hash file contents and store that somewhere, e.g., in a database. This might start to become messy.

[Crimson Wizard]

If the user doesn't tell AGS that some file is „imported“, then at compile time, the AGS compiler will have to peruse thousands of tiny little files in order to find out whether new ones have shown up or old ones have changed their content. So that might mean, hash file contents and store that somewhere, e.g., in a database.

There may be slightly different approach: run a file change listener on a Sprites directory, and note files as they are added, removed or changed. Cache may be updated immediately, or at compile time using a prepared list of changes.

EDIT: oh right, the files could have been modified in between editor sessions too. So Editor would have to run full check when the project is opened.

[morganw]

This is like a complete overhaul to how you work with sprites in AGS. Are you sure the people will accept this?

I think so, and some of what I'm suggesting is optional, but the end result is you press save in the graphics program to use or update an image. Probably best survey some people though.

This may be a regular workflow for professional artists, but, for example, I got used to import sprites from clipboard, that's very fast and you don't have to manage files.

If you can press save instead of copy, it would be much the same process.

And the tiled import... I was just drawing all animation sequence in 1 document, and then tile-importing them, again not thinking about cutting into files.

The metadata could define how large a tile is, and that could be referenced per image or per folder.

Perhaps there may be a way to keep these import commands? They may be simply creating new files and metadata according to the settings, work like alternate entry point into this system (main is copying file into folder).

Maybe just have the option to convert an image. So for your tiled import, the image is already there and you just define the tile dimensions to reference it as multiple images. But the alternative I'm suggesting would let you specify the tile size on a sprite folder, and then all your sprite sheets which use that tile size would be automatically used as multiple sprites.

There may be slightly different approach: run a file change listener on a Sprites directory, and note files as they are added, removed or changed. Cache may be updated immediately, or at compile time using a prepared list of changes.

EDIT: oh right, the files could have been modified in between editor sessions too.

Presumably though, it is possible to know the position of a particular sprite within the file, without having to read the whole thing?

[Crimson Wizard]

I think I need to clear something up, because I forgot to explain it first.

@morganw, your idea sounds as a good "final" goal, perhaps for AGS 4(?), but I am concerned of the amount of work it will require, to develop, test and polish (and to teach users "new ways").

Frankly, my intention was to make an "intermediate" change, which would require relatively small effort, and get this into 3.4.2 update (or 3.5, whatever it be called), saving some trouble to people using source control with AGS games.

Taking into consideration what fernewelten said above, about sprite cache may take time to compile, I've changed my original suggestion a little. So this intermediate step could look like:
- The workflow stays generally same (import commands and menus);
- Game.agf stores references to all source files, if necessary - to which part was "cut out" to make a tile;
- if sprite was imported without source file (from a clipboard) then Editor creates it on its own, giving some dummy name (like spritexxxx.bmp).
- sprite cache is preserved and updated all along with import etc commands, BUT Editor supports full recreation of sprite cache from source files (assuming they are present). To keep it up-to-date with files modified on disk it will need a way to know when sprites were changed. For simplicity it could use same technique as with AudioCache: compare file's modification time. That's far from ideal. In the worst case sprite cache could be recreated with "Rebuild all files" command. (If that's too bad, then... hash?)
- question about deleting sprite from the editor... in which case the actual source file should be deleted? For example, game could have a note saying that particular sprite is autogenerated.



From this stage, it would be possible to move further into the system you suggested.

[eri0o]

I think the idea of building from separate sprites directly doesn't need to eliminate cache sprite file for doing things faster, it's just that you don't need to put the cache in source control, so that when loading a project without the cache file, it would be built the first time from sources. Imagine "reimport all sprites from source" if the cache.spr is not found.

[Crimson Wizard]

If we are going to aim for Morgan's suggestion, there are few side problems that has to be resolved along the way.
For instance, Editor is still using native drawing code for some components, like GUI and Views (maybe something else) and that code relies on runtime sprite cache. One of the preliminary steps could be reimplementing the way editor draws stuff, perhaps making it rely on its own sprite map.

(On the other hand, there was Gurok's idea to run the engine inside the editor and make it draw stuff...)

[Snarky]

I am all for storing sprites (and other resources) on the file system, but I just want to caution that there are definitely people who swear by tiled sprite sheets (I think ProgZmax is one of them, for example), so let's not do away with the whole import process before they've had a chance to weigh in. And no, we can't just assume that they will notice this thread.

Another thing to keep in mind when checking for changes to the sprite source files or generating the sprite cache is ensuring that sprite numbering isn't messed up.
And the system should be able to deal gracefully with changes to the resources/folder structure. So, for example, if I had a sprite folder called "Rogr walk" and I rename it to "Roger walk", it shouldn't be excessively tedious to ensure that the sprites are found and matched.

[morganw]

It wouldn't prevent you from using a spritesheet, you could either flag the folder for tiled import, or just flag a particular image for tiled import. Unless I'm missing something, keeping the spritesheet as a single image means you never have to re-import the whole thing if anything on the spritesheet is changed; perhaps people might be preferring spritesheets at the moment to get around workflow issues in the sprite manager. Presumably a tiled import will always give you sequential sprite IDs?

One of the preliminary steps could be reimplementing the way editor draws stuff, perhaps making it rely on its own sprite map.

I guess the question is, should the editor aim to show everything on screen exactly as it is in the game, or should it just be enough to arrange everything and then run it with the debugger attached. I think at the moment it is a bit of both.

[Snarky]

It wouldn't prevent you from using a spritesheet, you could either flag the folder for tiled import, or just flag a particular image for tiled import.

If you're going to have the capability to flag a sprite for tiled import (with x and y dimensions), and set the transparent color (because that's required functionality), and view the sprites, and set the sprite number (implies reordering the sprites?), and probably the option to then delete them from within the editor, because why not... isn't that more or less everything the sprite manager does?

Unless I'm missing something, keeping the spritesheet as a single image means you never have to re-import the whole thing if anything on the spritesheet is changed

Umm... isn't it the opposite? You do have to reimport the entire thing. I assume the "re-import from source" command works correctly with tiled images, so it might just be a two-click operation if you are well organized (don't move/rename the source file).

perhaps people might be preferring spritesheets at the moment to get around workflow issues in the sprite manager.

It seems pretty much six of one, half a dozen of the other to me. In my experience tiled import is about equally quick as importing multiple files in a folder as long as you already know the tile dimensions; otherwise it's much, much slower.

Presumably a tiled import will always give you sequential sprite IDs?

Should work the same as multi-import of files, I would assume? (Available sprite IDs from the lowest and up, so sequential unless you've deleted a lot of sprites recently.)

[morganw]

If you're going to have the capability to flag a sprite for tiled import (with x and y dimensions), and set the transparent color (because that's required functionality), and view the sprites, and set the sprite number (implies reordering the sprites?), and probably the option to then delete them from within the editor, because why not... isn't that more or less everything the sprite manager does?

Yes, but those settings are applied at time of import and then everything is manually specified. Just adding a mapping between files and IDs, with the ability to rescan and remap would give similar functionality as the import, but main difference would be if you can specify default settings and then just save images to start using them.

Umm... isn't it the opposite?

It doesn't look like the source file is tracked at all for a tiled import, so I can't see any benefit with regard to re-importing after an edit.

[Crimson Wizard]

It doesn't look like the source file is tracked at all for a tiled import.

But this is what I am planning to fix. I guess Snarky was assuming it will work eventually even if it does not now.

[morganw]

To track the source you will have to store the path though, as potentially the source image has changed the editor last saw it (so timestamp and hash may have changed). I think fundamentally we are both suggesting the same thing, and the mechanisms needed to reference the files would be similar. Given that an absolute path may not match on two different computers, ideally the source should be relative to the game folder, so really I think the the only difference is an implied import vs manual import, and then where the mapping is stored.

[Crimson Wizard]

To track the source you will have to store the path though, as potentially the source image has changed the editor last saw it (so timestamp and hash may have changed). I think fundamentally we are both suggesting the same thing, and the mechanisms needed to reference the files would be similar. Given that an absolute path may not match on two different computers, ideally the source should be relative to the game folder, so really I think the the only difference is an implied import vs manual import, and then where the mapping is stored.

Yes, I also do not see any difference in opinion about tracking source files. I've got an impression that the main argument here was about keeping certain import commands in the editor, but maybe Snarky meant something else.

Only to elaborate again, my intent for the moment is to implement a relatively smaller change, making sprite cache connected with source files consistently (in every possible import case). After this the editor will still rely primarily on the sprite cache when working with game, but it will be possible to fully reconstruct/update cache from the source files. If these source files are all located in the project's folder, user may put only these under source control, ignoring the cache. Probably this will require storing file hashes or timestamps in order to know when cache is out of sync.

From this point Editor may be evolved further into using files on disk directly.

[eri0o]

I like your idea CW

It's exactly what I always wanted : "no spritecache.spr found, rebuilding cache from sources in game.agf...Done!".

[Snarky]

I like your idea CW

+1

If there's a simple change that will make it possible to store an AGS project under source control without spritecache.spr, that's an enormous improvement in itself. Then we can work out exactly how a more radical/long-term solution should work (probably the experience in itself will also teach us a lot about the issues that crop up and the features that are needed).