Dynamically loaded sound clips, and issue with AudioClips array

[Crimson Wizard]

There's again a case when I must other people for any ideas, because I am in doubts or might not be seeing a good solution.

AGS does support creating and loading sprites dynamically.
In the old versions of script API it also supported playing an external sound file: PlayMP3File function. Since the new audio API was introduced in v3.2, all the imported clips are registered as AudioClip object, and they return AudioChannel when run. But there was never a way to register one dynamically at runtime, and PlayMP3File became inconsistent with the rest.

Now, in AGS 4 we cut out lots of deprecated API, and PlayMP3File among them, but there's no substitution for it yet. There was an open feature suggestion to support playing external sound files in AGS 4. The simplest solution is to have another function like that, except it would return AudioChannel. But this solution has a problem, as external clips will still be outside the system, and things normally applied to AudioClips would not be applied to them.

Thus I thought that a better solution would be to support dynamically created AudioClip objects instead. Similar to how sprites are created as DynamicSprite.CreateFromFile, there would be AudioClip.CreateFromFile returning AudioClip* pointer, which you would have to store in your own variable.

This solution seems good, but there's a problem: normal AudioClips are stored in Game.AudioClips[] array, where Game.AudioClipCount is there number. And they have a numeric ID, which tells their index in this array. And these IDs are used in some cases to attach a clip to something else, like ViewFrame.
You see where this is going: that's just another case of AGS relying on numeric IDs...

If AudioClip is dynamic, then just like a dynamic sprite it may be deleted in any time. If it's deleted, its ID becomes unused. What happens to array in such case? In case of sprites, they are all kept on their places. So internal list of sprites contains gaps (well, it contains gaps also because you can import sprites under arbitrary number). We could do same for AudioClips too, but then it's not obvious what to do with AudioClipCount: is it going to be a real number of available audioclips, or a number of elements in array, where some elements may be gaps?...

It looks like this problem exists in multiple instances, like Rooms, which I've been discussing only recently.


I'd wish there was a consistent solution to this. It could be that AGS requires overhaul of object storage and cross-reference, that would be essential if we want to support dynamic creation of everything (gui, characters, etc). But that's a lot of work, and may be a separate future task. For now I'd like to find something suitable at least in the short term.


Here are the possible approaches that I see, and all of them have their downsides.

1. Do not add dynamic AudioClips to the main list, have them only referenced by user's pointer variables. This means that their IDs will have to be unused (-1?). That's the simplest dumb solution, but it's bad, because it creates a discrepancy between objects of same type but different origin (imported at design time vs loaded at runtime). Dynamic AudioClips won't be found when iterating AudioClips[] array, and they cannot be attached where script or engine expects a ID - such as ViewFrames.

2. Have dynamic AudioClips in the list, and shift the list when a clip is deleted to avoid gaps. This helps AudioClipCount property stay meaningful. But this breaks clip IDs, although technically could be worked around, but makes their reference by ID prone to mistakes.

3. Have dynamic AudioClips in the list, and replace with null pointer when they are deleted. The next loaded clip will be assigned the first available ID from the gap (that's how sprites work).
This keeps IDs intact. But AudioClipCount's meaning becomes dubious. Also users will have to test the list's elements for null when iterating, but only if they use dynamic clips in their game.

4. The variant of 3, replace deleted clips with a dummy AudioClip object that contains no data, and serves only to avoid null pointer errors. This *seems* safer at first, but there is now a problem of not being to test an element for validity, unless we introduce some way to distinguish this "dummy clip". So maybe it's actually going to be more confusing than helpful.



Does anyone have opinion over those? Or any other ideas?

[eri0o]

I have some thoughts on this in general... No conclusion though.

• Dynamic Sprites are dynamic in two dimensions: they can be loaded dynamically, but also their content modified
• Sprites (static?) can be unlocked, this mean unloaded and reloaded, but their meta information in maintained
• There is an array of sprites for their width and height
• There is no "Dynamically loaded" sort of sprites, meaning ones that their content is not modified - so that they could be unloaded from memory but keep the PATH information, and reloaded on game load without storing their content on save-games
• Sprite ID is an unique integer but not an index
• Sprite ID is stored in objects (Button Image, object graphic, ...)

Now looking at the Dynamic loaded AudioClip idea

• They look like their content is not meant to be editable, so they could not save their contents on the save and just the path, and reload on game load
• The dummy audio clip idea could be static, so in case of a failure, one could still just compare the pointer to this dummy, and if it matches it doesn't need to check for validity
• A validity may be needed if these clips are not locked in cache - meaning they can be unloaded at any time
• The ID in audio clips is an unique integer
• The ID in audio clips is also an index
• The ID in audio clips is also stored as property of other things (ViewFrame, ...)

So the ID in audio clips being also an index is curious, and then there is the situation of AudioClipCount. I think AudioClipCount is meant to be something used to iterate through all AudioClips - perhaps you want to filter all of them, check what is of music type and throw them in an in-game music test menu or game radio.

Now the issue of iterating though things is AGS (Script?) doesn't have good ways of iterating through things, other than using a "for" - and even the "for" is recent, we used to only have while, until a few years ago.

So it seems some way of iterating through AudioClips that would set a starting condition and other to get the next (until you get a null) would be the way to go - and the initial condition could then even set this iterator to only iterate through some specific type, or at a specific order. I don't have a good idea for this API or its naming. But going this route I guess AudioClipCount would be removed.

Edit: or other iteration method. This is meant for option 3.

[Crimson Wizard]

Hmm, there are couple of comments I'd like to make:

1. I don't think there's much conceptual distinction between sprites loaded from "spritefile" and standalone sprite files. In the first case their "path" info is maintained by the engine based on spritefile's meta info, in another case it's maintained by the user scripts. I see this as 2 separate methods, but similar result. For example, engine may be modified to package sprites as files and reference them by filenames. OTOH, we could add a way to configure sprites' table of contents from script, by adding src file references, and let these act as "normal static sprites" (with the only difference that they may get unregistered, probably).

2. We may support editable or programmatically generated AudioClips as well, it's a matter of expanding the type. AudioClip is just a front (interface), what's behind may vary. In the end both sprites and clips could be just the same system: a resource items that either have an underlying asset, or only data in program memory.

What you say about iteration is true, if AGS had proper Container and Iterator types, then we could have Container reporting both real number of valid elements and number of slots, and a kind of iterator meant for going through only valid elements.



In regards to my own question, I am starting to lean towards a half-solution under option 1:
Support dynamically created AudioClips that get connected to a external file, but not placed into the Game.AudioClips[] array, and have ID -1. This has downside of not being able to learn its own index; in case users would like to have it identified with a number they would have to put it in a custom array, and maintain custom range of indexes.

The upside of this is that this gives us more time to figure out other issues without causing any new restrictions or compatibility issues. What I mean is that, as soon as we figure out what to do with the clips array, these dynamic AudioClips may also be added there without breaking anything.

BTW, in AGS 4 AudioClips can have custom properties, which in theory may be used to add custom identification too.
(speaking of which, they would have to be kept in mind when serializing dynamically created AudioClip struct)


On the other topic, previously I mentioned that ViewFrame is connecting sounds using their IDs. I got reminded that in the script API it actually lets assign sounds by AudioClip* pointer, and only has numeric id internally.
I suppose that this may be reworked into storing something else interally, something that would let attach these dynamic clips as well.

[Laura Hunt]

Are there any use cases for dynamic audioclips apart from loading external audio files? I'm asking mostly because I don't see why any dev would include separate mp3/ogg files with their games. Do you have any examples in mind?

[Crimson Wizard]

Are there any use cases for dynamic audioclips apart from loading external audio files? I'm asking mostly because I don't see why any dev would include separate mp3/ogg files with their games. Do you have any examples in mind?

Loading external files is for either modding support, or dynamically constructing the game without recompiling in the editor, or allowing to modify the game more rapidly during development, I suppose. I do not have an actual list of examples. @Baguettator was the user who reminded me of this, so perhaps he could explain his use case.

Besides extern files, dynamic audioclips could be used to store results of runtime modified sounds maybe, if we ever support that in AGS. Although filter chains applied to a playback instance would be better.

[Baguettator]

Yes I can explain my use case, it's pretty simple  : in my game, I allow players to use their own musics, organised internally in my game. So they put their MP3 musics in the game's folder, and in the game there is a Playlist management, the player can manages 3 different playlists. Each playlist is played in a precise context. In my case : playlist 1 for expeditions, playlist 2 for colony's stage, playlist 3 for menu stage.

Perhaps it's a bit "niche" point of view, but I like this setting

[Laura Hunt]

Oh, I see! Yeah, definitely a pretty niche case, haha. But interesting! Thanks for clarifying