Better Compression - Do We Need It?

[Scavenger]

I've been thinking about the kinds of compression AGS uses in it's sprites and backgrounds, and potentially the other data it creates to pack into it's EXE and it got me to thinking - is it really necessary to change the compression from the old RLE algorithm to something better?

Now, don't get me wrong, I love tiny filesizes, it's one of the greatest things to be able to be efficient in saving space. As time goes on, the more people are going to use high resolution, high bit depth graphics, and RLE just won't cut it for that kind of thing, resulting in games that are almost a gigabyte just through inefficient storage alone. I know people have posted on the forum saying "Is it bad to use the higher resolutions? Will my game be too big?".

Well, hard drives are huge now - and the place we need efficient compression the most is downloading the files. And we can just ZIP or RAR or 7Z or ARC or ARJ them up, and all that inefficiency will drop as it's being compressed into an archive, and people can then download the most efficient archive, saving on bandwidth and online filespace, but not their hard drive space, as it will uncompress into a huge behemoth again.

But still, something about it rubs me the wrong way. When AGS first implemented compression, I believe LZW was still patented, and PNG compression was new and relatively unheard of. Couple that with the low resolution, low colour nature of the graphics being produced for AGS games, and there really was no need for better compression. But now that we might well be targetting mobile devices, maybe better compression could help the Android/iPhone/Weird Gizmo market? They, unlike PCs, have limited storage space, orders of magnitude less. (Like maybe 2-32GB instead of terabytes.)

One thing that concerns me when thinking about compression is speed - if we, for example, implemented PNG (or any DEFLATE based compression algorithm) instead of RLE, how much slower would AGS load resources? Would we be able to compress the compiled script resources AGS uses so that it's script data doesn't take up massive amounts of space - without hindering the performance of the game engine as a whole? What could we compress without sacrificing speed, and how?

This, compared to more pressing issues, I felt quite low priority (bordering on pipe dream), but it's still interesting to discuss, looking forward.

How do you feel about the file sizes of AGS games?

[Gilbert]

I don't know the exact implementation, but I think CJ once mentioned that the backgrounds are already compressed with a scheme similar to ZIP (so I suspect it'd be something like Huffman) since long, so the gain in changing the compression algorithm to another could be minimal, unless of course, backgrounds can be optionally compressed by a lossy scheme, such as JPEG(preferably customisable roomwise), so that the games' producers can decide on their own whether it's worth trading off graphic quality for smaller file size in specific cases.

The RLE scheme used in sprites is mainly for speed concerns (this compression wasn't even available in the past). The problem is that the engine loads and unloads sprites "intelligently" on demand and depend on available memory, etc.(unlike in some engine that the game programmers have to manually tell the system to load assets when needed). If we use a tighter (and more computation demanding) scheme for sprites it can impact performance during gameplay (whereas for backgrounds the loadings are all done during room changes). Say for example, an aeroplane flies across the screen at a random time, then the engine has to load the sprites for this aeroplane at that instance (if it wasn't in memory yet), and the decompression may pause the game for a split second if the system is not fast enough. This is debatable though, as disk access requires time too. In some cases it may be faster to load something with a smaller file size from disk and then decompress it, then to load a large uncompressed file, but I think that requires some testings on different configurations to determine whether it's worth bringing in better compressions.

[Crimson Wizard]

I don't know the exact implementation, but I think CJ once mentioned that the backgrounds are already compressed with a scheme similar to ZIP (so I suspect it'd be something like Huffman)

Room backgrounds are compressed with LZW.

The problem is that the engine loads and unloads sprites "intelligently" on demand and depend on available memory, etc.(unlike in some engine that the game programmers have to manually tell the system to load assets when needed).

True, and, AFAIK, that already caused problems for some hires games. In some cases you need to alter the "Sprite cache size" parameter in config. "Anna's Quest" game forcedly preloads animation frames for the same reason (I suppose it makes simple operation over sprite to make engine load it in memory).
Calin suggested to add "Preload" script function for Views so that gamedev could tell engine to load sprites at certain points.
(Other side of this problem is that if sprite cache is not large enough engine may decide to unload some of the earlier cached sprites)