How should a proper audio system look like?

[Snarky]

As it seems like AGS 4.0 is moving closer to release, I would like to revive this thread, as I feel an improved audio API should be part of that revision.

I think the basic idea of getting rid of AudioChannel in favor of something tied to the playback instance, and guaranteed to be non-null, is probably the right approach. The API might not need to change all that much, but there are a few different things to consider:

- AudioClip
- Voice clips
- Frame-linked audio (esp. on looping animations)
- AudioType
- Playback with repeat

For example, with a looping or repeating sound, is it always the same playback instance, or a new one each time? And how would you configure frame-linked audio (volume, etc)? If there was a persistent playback instance, that might offer a good API.

I think @Crimson Wizard has also sometimes mentioned more advanced effects, like changing playback speed or adding echo. I would suggest that this shouldn't be a priority to implement at this time, but it might be worth keeping in mind how it might be added in future.

[Crimson Wizard]

I think the basic idea of getting rid of AudioChannel in favor of something tied to the playback instance, and guaranteed to be non-null, is probably the right approach.

Not ready to give a full reply right now, just couple of quick notes.
I am not certain about getting rid of AudioChannel is a good thing or not. It may be a good thing to not have a channel as a return value from Play(), but at the same time I found audio channels to be an interesting logical concept in how multiple simultaneous playbacks are organized.

In regards to a "null pointer" problem, there's something that hit me recently, and I wonder why did not I think about this earlier. What if Play() returned a dummy audio playback? There are 2 reasons for Play to fail: failure to decode and failure to play, the former happens when the file cannot be opened at all, and the latter when the sound device is not working, or game set to use "no driver".
We are mostly interested to prevent script errors in the second case, because the first case will be noticed immediately during the game development. In audio software that works with filter chains there's a concept of "null" output, where the processed sound is simply discarded. So what we could do is to process the sound internally (which will even update its Position property), but then discard it through such "null" output. I did not check yet, but it's possible that our sound library supports that already.

[Snarky]

Not ready to give a full reply right now, just couple of quick notes.
I am not certain about getting rid of AudioChannel is a good thing or not. It may be a good thing to not have a channel as a return value from Play(), but at the same time I found audio channels to be an interesting logical concept in how multiple simultaneous playbacks are organized.

I think if there is something like an AudioPlayback instance for everything that's currently playing, that will basically take the place of AudioChannels. You might still want a way to iterate through all the playing sounds (for example if you need to adjust the overall volume), but I'm not sure a fixed list of AudioChannels (that exist regardless of whether anything is playing on the channel or not) is the right way to do that if the limit on simultaneous playback is removed.

AudioTypes seem more useful to me as a way to easily set up game logic for sound. MaxChannels would still help control whether a track replaces what is currently playing or not, for example. It's also important to consider how speech will work. It would be good if the system could support voiced background speech; some way to control the speech "channel(s)"/get speech playback instances would be needed.

In regards to a "null pointer" problem, there's something that hit me recently, and I wonder why did not I think about this earlier. What if Play() returned a dummy audio playback? There are 2 reasons for Play to fail: failure to decode and failure to play, the former happens when the file cannot be opened at all, and the latter when the sound device is not working, or game set to use "no driver".
We are mostly interested to prevent script errors in the second case, because the first case will be noticed immediately during the game development. In audio software that works with filter chains there's a concept of "null" output, where the processed sound is simply discarded. So what we could do is to process the sound internally (which will even update its Position property), but then discard it through such "null" output. I did not check yet, but it's possible that our sound library supports that already.

I think it's clear that if it must never return null but playback may fail for whatever reason, it must then return a "dummy." If I understand you correctly, though, what you're saying is that the dummy may still "pretend" to play, specifically in terms of reporting playback position. Yes, I think that could be useful because other game logic may depend on it (use it for timing, for example, like if a cut-scene has been designed to sync to a music track), but at the same time I think it's important that it should be possible to tell somehow that playback has failed.

I think there may be cases where the file fails to open/play but this isn't known at design time (e.g. if using AudioClip.GetByName() or Game.PlayVoiceClip(), or using an external AUDIO.VOX file; perhaps a way to load audio files from the file system or even streaming will also be reintroduced in the future), but there is already an AudioClip.IsAvailable property that can deal with that, if the AudioClip isn't simply null.

[Crimson Wizard]

I've been busy with other things, but now I will probably return to this issue. It's been a while since I've given any thought to the audio system, so will have to spend some time and think this through again.

I do believe that there has to be a way of limiting the number of simultaneous playbacks according to user-defined rule. The question is whether we support this as a native engine feature, or expose enough API to let users write their own "audio mixer", or both. Native feature may still be useful, because limiting e.g. music to 1 channel and having an automatic replacement is a very convenient functionality and used almost in every game.

There's one thing regarding the ideas mentioned above, which I do not like:

AudioTypes seem more useful to me as a way to easily set up game logic for sound. MaxChannels would still help control whether a track replaces what is currently playing or not, for example. It's also important to consider how speech will work. It would be good if the system could support voiced background speech; some way to control the speech "channel(s)"/get speech playback instances would be needed.

The reason why I do not like this approach is that if AudioTypes control number of simultaneous tracks, then that means there are still "audio channels" out there, except now they are "hidden" if AudioChannel is removed. We remove AudioChannels from script, and instead get secret "channels" in AudioTypes.

If we allow to access a list of AudioPlayback instances per AudioType, that makes AudioType to be a sort owning container of playback references (which seems like a strange organization to me). And if we would like to let users read the contents of these containers, then a) we have to expose AudioType as a struct in script, and b) we practically recreate audio channel in a less convenient way.

Why not keep AudioChannel, but make it
* not having a hard engine limit, but let users define as many as they like
* strip their API by moving playback control to AudioPlayback, and keep the channel object only as a "slot" which may contain a playback?

[eri0o]

One thing I would like is to be able to pass a volume right when using Play, and have that volume be a percentage that gets multiplied by whatever is the volume type percentage - so I could have a slider for the sounds per type in a menu.

[Crimson Wizard]

One thing I would like is to be able to pass a volume right when using Play, and have that volume be a percentage that gets multiplied by whatever is the volume type percentage - so I could have a slider for the sounds per type in a menu.

Having volume as a multiplier everywhere is a correct way IMO too. I've posted this somewhere before, the volume should be adjusted as a combination of multipliers:

System (master) volume -> Audio Type volume -> Clip volume

and optionally -> Emitter volume (e.g. character) -> Animation volume (for view frame linked sounds)

[eri0o]

Yeah, and also, the volume could be something optional right in the play call instead of requiring the need of channel?

I think from what I remember from previous discussions people forget to deal with the null audio channel. I think if there was some way to grasp for errors of audio - like if there is no audio device or the audio file doesn't exist - not sure how.

The other thing is some people like positional audio, but they understand as either it being relative to the player character or relative to the position on screen. There was also at some point a question about having different regions in a room with different music and some crossover transition. But these nuances could also be left for scripting.

The other thing I remember being asked is something like a filter for when the player is in a cave or under water. I don't think this is easy to do with mojoAl. I think this was done in strangeland through a plugin and maybe Laura Hunt asked about it too at some point.