AGS source control hosting

[Snarky]

I just got my monthly bill for hosting the AGS Awards project on Bitbucket: 31.25 USD. That's $375 per year that I'm paying to maintain the Awards. 

I would like to cut this cost. The reason it's so expensive is that the project requires several gigabytes of storage, because it includes large binary files, in particular acsprset.spr. Every time I add a sprite to the project, the whole file needs to be stored again (I've set up LFS, which I thought would help with this by doing binary diffing, but I guess it's not that smart).

Does anyone know a way around this problem? Or a cheaper host? (Note that the project needs to be private.)

[Kastchey]

Holy cow. Either we find an alternative or we'll need another bake sale.

What are other requirements, apart from private projects? Is Git support mandatory?

[Crimson Wizard]

The only workaround I know is to not adding compiled binary files to source repository at all: acsprset.spr, perhaps even not roomNNN.crm.
Full AGS project source in current state is not suitable for that. Back in a day when I was participating in a team project, I only pushed Game.agf, asset sources (separate sprite files in a subdirectory), and scripts. acsprset.spr and room.crms were kept in separate archives and tagged with wip release number, uploaded to some other hosting such as google drive. We only kept number of those corresponding to certain stages in development.

Latest version of AGS (3.5.0) in theory allows to fully restore sprites from sources if acsprset.spr is missing or does not have all the sprites mentioned in Game.agf. The first case is unstable (I don't remember if it works automatically), in the second case you will be warned of missing sprites. The missing sprites will be represented by sprite 0's graphic, and you will have to manually go around, doing "Replace sprite(s) from sources". If you do you also have to save repeatedly, or editor may run out of memory (the process is far from ideal).
Obviously this does not work if there are no sources and sprite was e.g. pasted from clipboard.

The only real solution to this is to completely revamp sprite and room storage in the project. Both is in plans for a while, but someone got to work on implementing this in the editor. I know that ChamberOfFear (aka persn on github) was working on rooms recently. I guess we just need to stop adding new stuff and focus on improving existing functionality (actually had to do this years ago).

[eri0o]

Snarky, have you tried Azure Repos (dev.azure.com)? The storage should be unlimited, the only limit there is I think access, but am not sure, if I am not mistaken it is free for one person to a team of 5 people - I don't know if they changed since all my games there are done by me alone. I don't even use git lfs, I just throw all files like normal and azure never complained.

[Snarky]

What are other requirements, apart from private projects? Is Git support mandatory?

Probably. I haven't looked into git alternatives, but I'm not keen on learning yet another source control system either.

Snarky, have you tried Azure Repos (dev.azure.com)? The storage should be unlimited, the only limit there is I think access, but am not sure, if I am not mistaken it is free for one person to a team of 5 people - I don't know if they changed since all my games there are done by me alone. I don't even use git lfs, I just throw all files like normal and azure never complained.

Thanks for the tip! I'll look into it.

The only workaround I know is to not adding compiled binary files to source repository at all: acsprset.spr, perhaps even not roomNNN.crm.
Full AGS project source in current state is not suitable for that. Back in a day when I was participating in a team project, I only pushed Game.agf, asset sources (separate sprite files in a subdirectory), and scripts. acsprset.spr and room.crms were kept in separate archives and tagged with wip release number, uploaded to some other hosting such as google drive. We only kept number of those corresponding to certain stages in development.

Hmmyeah. Of course, the project versions on git are pretty much useless without a matching acsprset.spr, or the ability to reconstruct it (particularly for something like the awards client, where a large proportion of the code is about managing these assets).

I suppose to an extent it depends on what the source control is used for. Backup, experimental branches, comparing with earlier versions, synchronizing between multiple developers, etc. That last case in particular is pretty tricky.

I did actually try a version of this solution at one point, keeping everything on Bitbucket but as two separate projects: "AGS Awards development" and "AGS Awards assets", where the assets project was only updated occasionally. It was fairly cumbersome and confusing.

Latest version of AGS (3.5.0) in theory allows to fully restore sprites from sources if acsprset.spr is missing or does not have all the sprites mentioned in Game.agf. The first case is unstable (I don't remember if it works automatically), in the second case you will be warned of missing sprites. The missing sprites will be represented by sprite 0's graphic, and you will have to manually go around, doing "Replace sprite(s) from sources". If you do you also have to save repeatedly, or editor may run out of memory (the process is far from ideal).
Obviously this does not work if there are no sources and sprite was e.g. pasted from clipboard.

This is definitely a step in the right direction, and I have started putting all my sprite assets in a subfolder of the AGS project folder (because otherwise it uses absolute paths, and breaks between computers, or if I move the project folder) for this reason, in anticipation of future usee.

The only real solution to this is to completely revamp sprite and room storage in the project. Both is in plans for a while, but someone got to work on implementing this in the editor. I know that ChamberOfFear (aka persn on github) was working on rooms recently. I guess we just need to stop adding new stuff and focus on improving existing functionality (actually had to do this years ago).

Yeah, this would be a great improvement for several projects I have worked on. If AGS was able to automatically regenerate all its needed binary files from the original assets (or eliminate them altogether), collaborative development would become a lot easier.

[eri0o]

CW, MorganW had this repo where he tested large import : https://github.com/morganwillcock/ags/tree/rebuild-spritecache

Not sure if you remember, the problem was AGS needs as much RAM as the acsprset.spr file is in size, because it doesn't save in the disk until the user has saved the project which can't happen in the middle of saving - but it should instead just write on the disk, or just make ags 64 bit and let it use more RAM.

Ah, Snarky, I think you can just let the repository be public and then no team restrictions apply, not sure - all my repos there are private.

[Snarky]

Eri0o, the project needs to be secret (edit: private, I mean) because it includes various (low-security) passwords as well as details about the winners of the juried awards. (Congratulations on your Best Innovation award, BTW!)

[AGA]

We already have a free (open source project) Bitbucket licence, which I set up a few years ago before it was decided to go with GitHub instead.  I'm busy this weekend, but I'll send the details to you ASAP.

If that doesn't work, I can probably set up a private Git repo on the AGS server for you to use.

[Crimson Wizard]

Even though the pricing may be critical issue for you, but the size of the repository is also an issue...

Anyway, there's another solution: do not rely on the Editor and have a tool that rebuilds acsprset.spr from the list of files. acsprset.spr's format is rather simple and well known (may be found in the engine sources). Reading Game.agf is parsing XML. The biggest problem is perhaps recreate sprite import algorithm, as it also has to read from all the supported image formats such as png and gifs. It's possible to make a quick version of such tool by reusing existing C# sprite import code from the editor. But it may be rewritten in C++ too if desired.

Having such tool is one of the upcoming tasks for AGS, as we are planning to detach certain tasks from the editor into stand-alone tools for easier automatic project building.

[ChamberOfFear]

The only real solution to this is to completely revamp sprite and room storage in the project. Both is in plans for a while, but someone got to work on implementing this in the editor. I know that ChamberOfFear (aka persn on github) was working on rooms recently. I guess we just need to stop adding new stuff and focus on improving existing functionality (actually had to do this years ago).

Just for the record.. I'm very close to finishing a first draft of the room stuff that can be sent in for a PR, what is missing is testing the file watcher and building .crm files from the open room format, the room editor is already working with the open format in my local branch and it is working well. I was planning on starting on the sprite stuff when I'm done with the rooms, but I had to take a break from AGS development because I got back issues so I can't sit too much in front of a computer until it's sorted out.

While we're on the subject, does the sprite storage revamp has a Github Issue that explains what needs to be done and what else might be relevant for the problem? I haven't seen one but maybe I'm bad at searching

[eri0o]

@Snarky whoa! Just discovered! XD I am trying to make a new and updated Web Port now to celebrate, it's almost working but just not yet, based on AGS 3.6.0, let me see if I manage to finish it today 

@ChamberOfFear not quite what you asked, but this issue has a lot of information on the matter: https://github.com/adventuregamestudio/ags/issues/702

[Crimson Wizard]

While we're on the subject, does the sprite storage revamp has a Github Issue that explains what needs to be done and what else might be relevant for the problem? I haven't seen one but maybe I'm bad at searching

I don't think there's a good ticket for this. One should cover at least following topics:
1. how sprites are stored in the project
2. how sprites are imported via the editor
3. how sprites are drawn in the editor
4. how sprites are compiled for the game

For example, p3 would definitely require abandoning use of AGS.Native for drawing anything. I get it that rooms are already done, but there are also GUI that is still drawn by native code.

Anyway, this is going offtopic. I might write one on the weekends, there may be much more to discuss.