OGG audio makes the game freeze (Android)

[SinGala]

Hello everyone,

I am currently working on making an Android version of an AGS game, and I realized there was a freeze issue because of the ambiant sounds, I am not sure if it is normal or a bug, so here I am.

Here is a snippet of the current code to help explain the issue:

Code: ags

function room_repeatedly_execute()
{
if (variableLevel == true)
{
variableLevel = false;
player.Say("Test Line");
ambianceSound.Play(eAudioTypeAmbientSound, eRepeat);
player.Say("Second Test Line");
}
}

When the room is loaded, all the sprites/objects are visible in the scene. However, if we take the code above, the "Test Line" will appear, but then we will have around 10seconds of freeze where nothing will happen on screen before the "Second Test Line" text appears and the game continues.

After some debugging, it appears that the fact that the "ambianceSound" file is in .ogg is the reason behind the freeze issue, since deleting the line fixes the problem.
Changing the sound file for a .wav creates a seamless experience in-game, while a .mp3 will create a very short freeze (around 1 second max).

I however have quite a few ambiance files, and putting them all as .wav might be a little too heavy on the long run for this game. Furthermore, this game is also being released on Windows, and the .ogg files work perfectly there. The only issue arises when I try it on an Android build...

So my question is this one: is it a AGS issue, or is it normal behaviour? And if it is a normal behaviour, is changing my files to .wav the only real solution I have?


Thank you in advance for your help!

EDIT: I am using a Samsung Galaxy Android phone that can run Star Rail so I don't see how a AGS game could be a problem for it. I'm also using the 3.6.0 patch 6 version of AGS

[eri0o]

Is this the ACTUAL script code? Sound is played asynchronous in a different thread, and format should not matter.

Also important is which exactly version of AGS you are using? The latest 3.6.1 has the sound decoding library updated.

Probably relevant the exact specific Android device - maybe it's something so old you don't have RAM available - and also the exact specific sound that causes the issue. But technically it shouldn't be possible as it's always run in a separate thread and not synchronously.

[Crimson Wizard]

No, that's not an expected behavior at all.

But technically it shouldn't be possible as it's always run in a separate thread and not synchronously.

It is played on a separate thread, but there are synchronization points between game thread and audio, and if audio thread is hanging, the game may theoretically also freeze at these points. This immediate issue possibly could be resolved by introducing a wait timeout (but then some game logic may not follow the audio accurately, like reporting clip position, etc).

First of all we'd need to clarify if this is happening strictly on Android, and not other platforms like Windows.

The RAM question may be resolved by testing a dummy game which has nothing else but the simple scene where you play this sound.

@SinGala, could you upload this OGG file that is causing the issue and post here for us to try?

[eri0o]

I found an upstream-upstream issue that maybe causes this (https://github.com/nothings/stb/issues/1249), but of course having the actual file is the only way to verify if this is actually the issue.

But it should affect all things and not just Android.

[SinGala]

@Crimson Wizard @eri0o

Sorry I forgot to specify the Android/AGS infos, I edited the main post but here they are too:

I am using a Samsung Galaxy Android phone that can run Star Rail so I don't see how a AGS game could be a problem for it. I'm also using the 3.6.0 patch 6 version of AGS

First of all we'd need to clarify if this is happening strictly on Android, and not other platforms like Windows.

The issue seems to only be happening on Android, when I'm testing on Windows everything works perfectly fine and we don't have any freeze.

@SinGala, could you upload this OGG file that is causing the issue and post here for us to try?

Sound wise I am afraid I cannot share the exact file doing the issue, but if it can help, the issue is happening for every .ogg file set as an ambiant sound, so it is not related to just one.

[SinGala]

Also important is which exactly version of AGS you are using? The latest 3.6.1 has the sound decoding library updated.

I will try the 3.6.1 to see if it fixes the issue.

Sound wise I am afraid I cannot share the exact file doing the issue, but if it can help, the issue is happening for every .ogg file set as an ambiant sound, so it is not related to just one.

I'm joining a free music I found that does the same issue on my project here, in case it does help figuring it out:
https://drive.google.com/file/d/1N4_vEV3FRoxLrPoxZybS5vIQGHe8jLJT/view?usp=drive_link

[Crimson Wizard]

Sound wise I am afraid I cannot share the exact file doing the issue, but if it can help, the issue is happening for every .ogg file set as an ambiant sound, so it is not related to just one.

This may be a silly thing to ask, but what if you set this sound to another audio type (not ambient)?
Just to make certain that this depends only on a file, and not on some audio settings or logic in AGS.

[SinGala]

what if you set this sound to another audio type (not ambient)?

I actually didn't thought of trying this out, but playing Sounds or Musics does seem to work better.
We however still have a slightly weird behaviour, where the following lines of code (player talking with sound) are called but the animations/texts are arriving after a small freeze(around 2 seconds compared to the almost 10 of when the sound is in Ambient), at the same time where the sound finally starts (when it's called even before the player sound)

[Crimson Wizard]

what if you set this sound to another audio type (not ambient)?

I actually didn't thought of trying this out, but playing Sounds or Musics does seem to work better.

Hmm, could you tell, what is the "Bundling type" setting for all these audio types? It may be "in main data" or "in audio vox", and i wonder if that makes any difference.

Also, please clarify what is "player sound", are your speech lines played with voice, or do you mean something else?

[SinGala]

could you tell, what is the "Bundling type" setting for all these audio types? It may be "in main data" or "in audio vox", and i wonder if that makes any difference.

It is set as "in main data". I tried to set it as "in audio vox" too while debugging but it didn't change anything.

Also, please clarify what is "player sound", are your speech lines played with voice, or do you mean something else?

The player sound is an audio file set as "Sound", just think of it as an SFX to make it simpler. It doesn't use any of the Speech functions of AGS for the audio.

[eri0o]

Can you make a minimal AGS game that reproduces the said issue? I can't figure anything that could cause this so far.

[SinGala]

Can you make a minimal AGS game that reproduces the said issue? I can't figure anything that could cause this so far.

I'm linking a minimal project I tried creating. I don't seem to see a freeze, or at least not at all as strongly as in my main project. I do seem to see a small latency for the ambiance start when I'm using Android Studio's debug tool, but that's all I'm managing to get in a new project...

https://drive.google.com/file/d/1htqeSyGgfqAFTf1NunjbANSz80B-9mAF/view?usp=sharing

I'll keep trying tomorrow to figure out how to reproduce the issue in this project

I will try the 3.6.1 to see if it fixes the issue.

In other news, I seem to have trouble building for Android on this new version. I get the following error every time I try to launch the .apk on my phone :
"AGSNative
Loading game failed with error:
Main game file not found or could not be opened.
Filename: ac2game.dta..

The game files may be incomplete, corrupt or from unsupported version of AGS."

Until I manage to fix that, I won't be able to verify that the issue is fixed in the 3.6.1....

[Khris]

In case it helps: I can reproduce the issue the 3.6.1 version has.

The 3.6.0 version runs fine for me, no audio delay / stuttering whatsoever (Nokia 7.2 / Android 11).

[eri0o]

In other news, I seem to have trouble building for Android on this new version. I get the following error every time I try to launch the .apk on my phone :
"AGSNative
Loading game failed with error:
Main game file not found or could not be opened.
Filename: ac2game.dta..

The game files may be incomplete, corrupt or from unsupported version of AGS."

Until I manage to fix that, I won't be able to verify that the issue is fixed in the 3.6.1....

Hey, this was a mistake on the engine side, managed to reproduce and found a fix for 3.6.1. Should have a newly packaged Editor with the fix here soon, until the fix makes into a new release.

[SinGala]

Hey, this was a mistake on the engine side, managed to reproduce and found a fix for 3.6.1. Should have a newly packaged Editor with the fix here soon, until the fix makes into a new release.

Hey, alright that's good to know, I'll try it out today then with the fix to see if the issue persists there for the audio

EDIT: I just tried it, the freeze issue still persists...

I'll keep trying tomorrow to figure out how to reproduce the issue in this project

I did think of something about my issues with reproducing it. At some point during my debugging tests yesterday, I took away most of my scene elements (around 80%), and the freeze situation got less worse (around 5 seconds instead of 10). The issue still must come from the .ogg file, since taking away the line playing it fixes the freeze fully, but maybe the fact that my scenes have quite a good amount of objects/hotspots/etc. doesn't help the situation for reading the audio?

EDIT: If it can help figuring it all out, I tried launching the game through ScummVM on Android directly to test, and the freeze issue isn't there at all, the game plays smoothly. I'm also still trying to reproduce the issue on yesterday's project, I'll send it here as soon as I have it done


EDIT 2:
Alright, so, I think that I figured it out (kind of)!

After deleting every parts of the main project, I finally realized that all the Sounds were in the Main Datas, same for the Ambiance, and the Music was in Audio VOX. Switching all the Sounds to Audio VOX seems to almost fully fix the issue (around 1sec max of freezing against around 10 before)

The project does have a lot of files in the Sounds category (around 960), so maybe having them all in the same Main Datas as the Ambiance files was creating issues?

Not sure why the problem seems to be this much present only when using .ogg files though  And adding a whole ton of Sounds in the test project from a few messages ago doesn't seem to manage to recreate the problem either as of now


EDIT 3: Finally managed to reproduce the freeze on a new AGS project. It is less strong than the one I have on my main project, but it is finally visible. I ended up having to also put the Music files as Main Datas bundling type in order for the freeze to appear on this project (while it is in Audio VOX on my main project).

You can download the files here: https://drive.google.com/file/d/1k8ugJYCRfjQ-UKHhKHglt7bJdICzbH0d/view?usp=sharing

[Crimson Wizard]

If not the difference with other sound formats, I'd think this is purely an issue of slow resource search.
But that won't explain

Changing the sound file for a .wav creates a seamless experience in-game, while a .mp3 will create a very short freeze (around 1 second max).

[eri0o]

I'd think this is purely an issue of slow resource search.

I have a guess Android is compressing asset files and this is making jumping through offsets very expensive when accessing these files, I will try later to disable this in the Gradle scripts. I tried to profile a different game - I hadn't noticed the update in this thread - and noticed a lot of time was spent in Android's own stuff.

[Crimson Wizard]

I have a guess Android is compressing asset files and this is making jumping through offsets very expensive when accessing these files

But this means that it may be beneficial for Android ports to preload game resources once instead of re-reading or streaming them?

Overall it seems that the standard resource strategy of AGS is not fit for the mobile, and possibly for consoles too.

[eri0o]

I made a change that disables the additional compression Android does, only for .ags and .vox files. It should show up here eventually, please check it out: https://cirrus-ci.com/task/4912521337896960 (it may take a little while for the build to show up here)

But this means that it may be beneficial for Android ports to preload game resources once instead of re-reading or streaming them?

Overall it seems that the standard resource strategy of AGS is not fit for the mobile, and possibly for consoles too.

I don't know how things works in consoles, so can't say anything there. But the problem is if you are not re-reading/streaming you need to hold things in RAM. And if RAM is limited, how would one do that? My guess is the way would be to hold those compressed, and then you can decompress what gets used when needed.

Overall it would be better if there was some way per room where AGS could know everything that would be needed, then there could be some way to figure a strategy.

Here is the texture way of Android, https://developer.android.com/guide/playcore/asset-delivery/texture-compression, you can see it does lots of assumptions, like that the game is separated in levels and there are specific textures per level.

[Crimson Wizard]

Overall it would be better if there was some way per room where AGS could know everything that would be needed, then there could be some way to figure a strategy.

Yes, indeed, this would require to develop a different resource strategy for AGS games, where necessary resources are defined either by user script and settings or by automatic analysis of the scene. This is not going to be trivial, but may be necessary in the future, especially since people are trying making games in AGS which require heavy resources (hires gfx, etc).

[eri0o]

I don't know if the engine can load a "naked" game28.dta file (it refused to load when I tried), but I wonder if a fully unpacked (using agsunpack.exe) game would make sense to then pack in the Android asset, because this would lower the memory offset indirections - unfortunately this is not possible for the spritefile, so not sure it's worth it. This is more to then use the compression from Android (in case it's desired). I still think for Android it's best we handle the compression/decompression ourselves though.

[Crimson Wizard]

I don't know if the engine can load a "naked" game28.dta file (it refused to load when I tried)

I think it is supposed to, as there should not be any difference between loading this from disk or from package. Have you tried on Windows first?

[eri0o]

I haven't yet tried in Windows, that was Android.

@SinGala , if you have time, please try the AGS from here: https://cirrus-ci.com/task/6170912077119488

[SinGala]

I made a change that disables the additional compression Android does, only for .ags and .vox files. It should show up here eventually, please check it out: https://cirrus-ci.com/task/4912521337896960 (it may take a little while for the build to show up here)

Just made an .apk with this new build and it works perfectly, no freeze, no latency, nothing. Thank you!
It even fixed a lagging issue that happened while clicking on the buttons for the first time in my level scenes.

[eri0o]

Thanks for reporting, this should be in the next 3.6.1 release.

[eri0o]

@SinGala the new release of 3.6.1 should have the mentioned issues fixed!