Compiled game size skyrocketing? (AGS 3.2.1)

[Scavenger]

I've been adding some code to my game recently, and after I completed it, I noticed that the game's filesize has jumped from 9mb to 40mb for no discernable reason. It compresses down to 3mb as normal, but the uncompressed filesize is amazingly huge. I looked at it in a hex editor, it seems to have freaked out after writing function names from a module that I had deleted (!!!) and wrote about 40 megabytes of 0s after it. After reimporting it, it managed to compile a working executable that was 1.70mb, and then was 40mb again after that. I'm not sure what I'm supposed to do, really, since I can't well restart my game project, and I don't see what I'm doing wrong.

Why is AGS writing a whole lot of pointless garbage in my EXE?

[Ryan Timothy B]

Have you tried  Build > Rebuild All Files? It might do the trick perhaps.

[Scavenger]

Did that, still having the problem, unfortunately.

Edit: More research, AGS appears to put a ridiculous about of blank space between module scripts in the EXE, in random locations. (In one compilation it was after PPCOLL.ASC, then INTERFACES.ASC. ) Turns out it was a similar, but distinct module that it was bloating after, instead of the one I deleted. I'm just not sure why it's doing this.

This is awful, as the game itself works but I can't have this huge bloated file for no reason. The game is only going to get bigger, and having 40 metabytes of 00s is not a good thing. I tried to see if it was having too many module scripts, but that wasn't it, as I made a test game with about 15 of the things.

Am I just coding my game wrong? Is there something terribly wrong with me? What could possibly cause the compiler to freak out like that?

[Crimson Wizard]

That's interesting, I may look into this if I have time;
but I must say that since many things in AGS are constant sized, like objects number, not only this prevents from making more objects, but this adds a lot of unused data. For example, there's data for all 50 hotspots for every room, even though only small number of these may be used. The amount of useless chunks grow with room count.

[abstauber]

Hmm.. this applies for my game as well and it only has three rooms.

2,7 MB compressed and around 40 MB uncompressed.
415 Sprites, 15 Sprite folders, 20 Views, 17 GUIs, 31 inventory items, 32 Characters and 0 Dialog topics

I hope this helps

[Scavenger]

That's interesting, I may look into this if I have time;
but I must say that since many things in AGS are constant sized, like objects number, not only this prevents from making more objects, but this adds a lot of unused data. For example, there's data for all 50 hotspots for every room, even though only small number of these may be used. The amount of useless chunks grow with room count.

The game is 4mb normally, and there's no extraneous stuff in my game - the compiled file literally explodes outward when I add in elements from Austauber's platform engine. I'm loathed to let the project slip out of my grasp, but if it helps, I can provide you with both the pre-TENG project files and the post-TENG project files.

[Crimson Wizard]

Well, I guess that three things that take most space are: room background, sprites and audio files (voices most of all).

I have an interesting idea. I could try and make a small utility that tells the offset and size of files in the compiled game.

[Crimson Wizard]

Answer found.

TENG script module has global data size = 35.627.832 bytes.

Look into the TENG.asc:

Code: ags

#define NUM_MAX_TILES	100000
#define NUM_MAX_SPRITES 128
<...>
struct ctile
{
  AudioClip *sound[LAYERS];
	short x[LAYERS];
  short y[LAYERS];
  bool mirrored_x[LAYERS];
  bool mirrored_y[LAYERS];
  
  bool is_solid[LAYERS];
  bool is_destructable[LAYERS];
  bool is_bonus[LAYERS];
  bool is_deadly[LAYERS];
  bool is_ramp[LAYERS];
  bool is_fg_ramp[LAYERS];
  bool is_ramp_support[LAYERS];
  bool to_bounce[LAYERS];
  bool is_platform[LAYERS];
  bool is_ladder[LAYERS];
  short bounce_counter[LAYERS];
  short emit_particle[LAYERS];
  short emit_particle_cnt[LAYERS];
  
  short loop[LAYERS];
  ViewFrame *vf[LAYERS];
  short frame_counter[LAYERS];  
  short tileno[LAYERS];
  DynamicSprite *spr[LAYERS];
  
  short frm_speed[MAX_FRAME_TILE];
  short frame[MAX_FRAME_TILE];
  short frames_in_use[LAYERS];
  
  short ani_delay[LAYERS];
  short ani_delay_counter[LAYERS];  
  bool ani_sync_enable[LAYERS];  
};
<...>
ctile tile[NUM_MAX_TILES];

"ctile" struct is lots of bytes.
and tile array is (lots of bytes) * 100.000;





E: The question remains, why does AGS writes global data to the game file... Never thought about this. Perhaps it cannot distinct compile-time initialized data (like when you write "int a = 10;") from non-initialized one.

EE: Confused two struct names, fixed the code above.

[Khris]

Funny; when I read this I wanted to post how TENG probably uses lots of data structures and stuff, but then I thought; shouldn't those only impact the size of save games (and the amount of RAM used by the game)?
Why do variables that are created at run-time increase the size of the game's exe?

[Crimson Wizard]

As I aded to my post above, I guess that's a historical issue: perhaps AGS cannot distinct initialized global variables from non-initialized.
Never paid attention to this.

Silly really.

Abstauber, Scavenger, I think the solution would be to use dynamic arrays instead (tiles = new ctile[] etc).

[abstauber]

Ahh, that's why. Thanks a lot!
2-3 years ago I've been told that dynamic arrays are super evil and I should use them, because they waste a lots of memory and resources. I can't currently find the post, but since then I rather use giant fixed arrays

[Scavenger]

As I aded to my post above, I guess that's a historical issue: perhaps AGS cannot distinct initialized global variables from non-initialized.
Never paid attention to this.

Silly really.

Abstauber, Scavenger, I think the solution would be to use dynamic arrays instead (tiles = new ctile[] etc).

That is very, very interesting. No wonder there were so many 00s in one block in the engine - it didn't occur to me that AGS would store that data. So, the fix is:

Code: AGS

//TENG.asc
// ctile tile[NUM_MAX_TILES];
ctile tile [];

function game_start () //The one in the module
{
tile = new ctile [NUM_MAX_TILES];
}

I didn't know structs could be dynamic arrays!

EDIT:
TENG.asc(6893): Error (line 6893): cannot create dynamic array of unmanaged struct
I don't think they can.

Is there any way to deal with all of this extraneous data, if it isn't possible to just... not have it (since it doesn't seem to serve any purpose other than to take up space, literally)? It's mostly just uninitialised data (so 0), so would it be possible to add RLE compression on it?

[Crimson Wizard]

2-3 years ago I've been told that dynamic arrays are super evil and I should use them, because they waste a lots of memory and resources.

lol what???

[abstauber]

2-3 years ago I've been told that dynamic arrays are super evil and I should use them, because they waste a lots of memory and resources.

lol what???

hehe, I know It was something like this:

Spoiler

A dynamic array by itself (before it is even assigned) takes up 4 KB of memory. So if you are defining seven dynamic arrays that's 28 KB of memory...
---
So I figured an assigned array would always save up memory, even if I don't have any clue how much memory those use.

[close]

[Crimson Wizard]

A dynamic array by itself (before it is even assigned) takes up 4 KB of memory.

What programming language it was about???
AGS allocates extra 8 bytes per dynamic array (to store length and some flags), C++ itself may allocate maybe 8 more for new/delete checks (don't remember now).
But 4 kb...

Strictly speaking working with dynamic arrays in AGS is generally slower, because it accesses it with managed "handler", not direct address. But how "slower" it is - it's very difficult to say without tests.

[abstauber]

Never too old to learn - dynamic arrays just got way more popular for me
But I hijacked this thread - I suppose the current problem is about not being able to create a dynamic array of a struct.

@Scavenger: If you know how many tiles your level uses, try to decrease MAX_NUMBER_TILES. This might save you some bytes.

[monkey0506]

I figured it would all be abstauber's fault.

As to using dynamic arrays, presently the only way of doing that in this case (the TENG module) would be to abandon the data structure entirely and use file-scope dynamic arrays instead. In lack of proper polymorphism, the only thing you'd lose is the logical grouping/scoping...oh, and the 2D array effect, but that's entirely simple to create a helper method for that. In fact you could create a file-scope static structure with helper methods for accessing the arrays while maintaining the logical grouping effect. If you split it into two modules you could even implement it using attributes, but that would place a ridiculous burden on the end-user for some developer syntactic sugar.

CW, I don't recall, but you said that dynamic array pointers are 8 bytes? I could have sworn that as with other AGS pointers that they were always 4 bytes. AFAIK isn't dynamic array access in AGS simply a dereferencing operation? Even C++ doesn't support run-time sized arrays without dereferencing.

[Crimson Wizard]

CW, I don't recall, but you said that dynamic array pointers are 8 bytes? I could have sworn that as with other AGS pointers that they were always 4 bytes. AFAIK isn't dynamic array access in AGS simply a dereferencing operation? Even C++ doesn't support run-time sized arrays without dereferencing.

Every reference in AGS is 4 byte id (aka handle), if you mean that.
However, the dynamic array object allocates extra 8 bytes at the head of buffer to store number of elements (4 bytes) and total bytes (another 4 bytes). It also uses one of those extra integers to store internal flags.

As for dereferencing, yes, it does that using handle as an index in pointer array. I seem to confused that with different thing: sometimes it does the inverted search (address to handle), but this only happens when a reference is being assigned new object, which probably never happens to array.
So, I should correct myself, that using dynamic array is almost as fast as a static one. I am risking to start nitpicking here though, because this difference is in calling two extra functions (to get inside managed pool), so it might make any significant impact only if there are many thousands of dereferences, or something like that.

[monkey0506]

I guess I never really thought about the dynamic array being responsible for tracking its own size, so 4 extra bytes would make sense. But why would it be storing "total bytes" in another, separate 4 bytes? Could this be related to the bug I discovered a while back where extremely large dynamic arrays just go insane? I don't recall the exact circumstances (I was just reading the post earlier, I edited it to update the code tags, but I'd have to look the link up again and I'm too lazy ) but there was something about dynamic array indices being multiplied by 4 on char arrays...or something like that.

And I used dynamic arrays in my EasyBake module (which is where I discovered said bug), with "many thousands of dereferences" and I never saw any huge performance hits. Of course I wasn't doing a lot else with it either. So, it may be biased.

[Crimson Wizard]

I guess I never really thought about the dynamic array being responsible for tracking its own size, so 4 extra bytes would make sense. But why would it be storing "total bytes" in another, separate 4 bytes?

It is not a template, that's why it needs to know the size of element. What may be questioned is why it stores elem_size * elem_count instead of just elem_size... Maybe to reduce computations, but that's a random guess. Also dynamic array never uses elem_size anyway, the element offsets are precalculated by compiler.

As for speed, I haven't said there will be perfomance drops if you change static array to dynamic, I said merely that there will be some difference... I suppose the perfomance will be affected by many other things, and dynamic array peculiarities won't be the decisive ones.