3D/Positional Audio using OpenAL

Started by vga256, Thu 21/04/2022 23:47:17

Previous topic - Next topic

vga256

Context: I recently discovered that OpenAL (and via extensions?) supports Positional/3D audio. Although I spent some time browsing the ags github repo, I am not clear as to the progress made with the transition to SDL2 and OpenAL with 3.6.0. It *appears* from a glance that 3.6.0 is now using mojoAL for interfacing with OpenAL.

Question: Is it possible to write a plug-in for AGS that would enable some basic positional/3D audio functions that would allow run-time changes to audio during gameplay? I am working on a first-person adventure that would greatly benefit from this feature, and I'm willing to put in the wrench time to make it happen, if it is possible at all.

Crimson Wizard

#1
Quote from: vga256 on Thu 21/04/2022 23:47:17Although I spent some time browsing the ags github repo, I am not clear as to the progress made with the transition to SDL2 and OpenAL with 3.6.0. It *appears* from a glance that 3.6.0 is now using mojoAL for interfacing with OpenAL.

3.6.0 has fully transition to SDL2 and is using mojoAL for audio playback, which is an OpenAL interface and SDL2-based implementation.
The parts of Allegro library are embedded into the engine code now; these are parts related to bitmap manipulation (pixel drawing) and utf-8 handling.

Quote from: vga256 on Thu 21/04/2022 23:47:17
Question: Is it possible to write a plug-in for AGS that would enable some basic positional/3D audio functions that would allow run-time changes to audio during gameplay? I am working on a first-person adventure that would greatly benefit from this feature, and I'm willing to put in the wrench time to make it happen, if it is possible at all.

I don't think that's possible without changes in both engine and plugin API. Playback settings afaik are not exposed to plugin API in any way. Internally, in the engine playback does not have 3D settings. So even though the backend allows it, engine does not have this logic implemented anywhere.

AFAIK AGS supports sound stereo panning between left & right (see AudioChannel.Panning).

vga256

Quote from: Crimson Wizard on Fri 22/04/2022 00:15:12

3.6.0 has fully transition to SDL2 and is using mojoAL for audio playback, which is an OpenAL interface and SDL2-based implementation.
The parts of Allegro library are embedded into the engine code now; these are parts related to bitmap manipulation (pixel drawing) and utf-8 handling.

I don't think that's possible without changes in both engine and plugin API. Playback settings afaik are not exposed to plugin API in any way. Internally, in the engine playback does not have 3D settings. So even though the backend allows it, engine does not have this logic implemented anywhere.

AFAIK AGS supports sound stereo panning between left & right (see AudioChannel.Panning).

Gotcha! It sounds like this would be a very long term project if there was interest in it. In the interim I will do my best with the .Panning property - thanks.

eri0o

Can you elaborate what are you trying to achieve with 3D positional audio?

Like, as an example, Alan's Draconian Ags4 fork has sound from views attached to characters corresponding to either the character position on screen, or character position relative to player character. Is it this a similar case or more playing a sound at a specific position?

AGS has AudioChannel.SetRoomLocation, so if it's more on the single API implementation, perhaps a similar API could be made, which would be more flexible but more disconnected from internal AGS game logic.

vga256

Quote from: eri0o on Fri 22/04/2022 02:58:04
Can you elaborate what are you trying to achieve with 3D positional audio?

Like, as an example, Alan's Draconian Ags4 fork has sound from views attached to characters corresponding to either the character position on screen, or character position relative to player character. Is it this a similar case or more playing a sound at a specific position?

AGS has AudioChannel.SetRoomLocation, so if it's more on the single API implementation, perhaps a similar API could be made, which would be more flexible but more disconnected from internal AGS game logic.

So to clarify - the OpenAL 3D sound implementation is not simple Right-Left panning. It employs a complex set of algorithms to emulate positional sound in a 3D space using a Head-related Transfer Function. This is effectively "surround sound" produced algorithmically in stereo, instead of via many channels (5.1/7.1). You can watch/listen to an HRTF demo here. It is a much more powerful effect than a standard stereo pan.

After doing some reading, it is not clear to me if mojoAL supports HRTF, or if that functionality is enabled through an extension like ALC_SOFT_HRTF.

Crimson Wizard

#5
TBH I'm starting having doubts that using OpenAL interface in general is a good idea for AGS project. It came as a part of the SDL port done by Nick Sonneveld, and which we decided to use, and I trusted this choice; but the more i looked into it on my own, the more potential problems I saw. I think the biggest is that it's another layer of complexity which we cannot fully control. Good example is playback "Speed" property: it appeared OpenAL does not have a direct equivalent for it, so I had to implement it on top by resampling the sound data. The sound is "sped up" before passing into openal, but because we must buffer at least couple chunks of data there all time, there's a delay at which the "speed" parameter is applied to the output sound, and that delay depends on the size of this buffering. In Allegro4 it was applied for the very next played chunk perhaps at the very end of the data chain before it goes out to the sound driver, so almost immediately. I'm having concerns that situation may become worse if we decide to implement more sophisticated sound system in AGS script, with mixers, and so on.

Note that mojoAL uses SDL2 internally, which we also use directly in the engine, so the question to ponder is, what use is OpenAL for us, and how it will work in perspective.
If it's actually useful, perhaps there's a proper way to "extend" it ourselves and let us control the sound closer to the final output.

Personally, i used ffmpeg before and liked its "filter" design, where sound transformations may be applied freely at any stage of the data processing. Compared to that, OpenAL is a "black box" with everything which connects directly to the output.

eri0o

#6
MojoAL is just an OpenAL implementation. It uses Sdl2 to make everything work. Like SDL_Sound has received lots of updates recently, I believe MojoAl will receive too. The advantage of it is license (not every license is compatible with Apple devices) and easy of building it, and easy of porting if needed. Additionally, we can use a different OpenAl implementation if needed - the Emscripten port doesn't use MojoAl, and neither does the Android port, they use their own platforms OpenAL implementation and interfaces - they could use MojoAL too, but I thought since a native platform specific version was available it was alright to use that instead.

The sound decoding we use is separated in SDL_Sound and the sound sink (actual playback) is through SDL2 (except in Android and Emscripten). The OpenAL gives both control to the sound playback as to the final device that plays it. So if a different solution is used these are the points that need to be done in it.

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

vga256

Quote from: eri0o on Fri 22/04/2022 10:38:44

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

OK that makes a lot of sense, thanks! I'll save this wishlist item for Unity :)

eri0o

Quote from: vga256 on Fri 22/04/2022 14:31:01
Quote from: eri0o on Fri 22/04/2022 10:38:44

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

OK that makes a lot of sense, thanks! I'll save this wishlist item for Unity :)

I was answering CW, your approach of identifying HRTF in OpenAL spec is what I meant as the correct approach towards getting the feature in MojoAL.

Summary: there are no perfect libraries.

Crimson Wizard

#9
Quote from: eri0o on Fri 22/04/2022 15:21:55
I was answering CW, your approach of identifying HRTF in OpenAL spec is what I meant as the correct approach towards getting the feature in MojoAL.

Summary: there are no perfect libraries.

Sigh, I was in doubts to whom you were answering, so deleted my next reply.

The reply was this:

---

Quote from: eri0o on Fri 22/04/2022 10:38:44
About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

In my previous post I specifically mentioned "using OpenAL interface in general", not MojoAL. MojoAl's own problems is an issue on its own, but I was talking about using the OpenAL interface. I'm questioning the convenience of that interface in the context of AGS engine.

eri0o

I like node based audio libraries like AVAudioEngine for Apple devices and the Web Audio API in Web Browsers (Firefox, Chrome and Safari). But I don't know any library with similar functionality that is also cross platform, easy to port and has a license that is friendly to apple devices and consoles.

Crimson Wizard

#11
Quote from: eri0o on Fri 22/04/2022 15:57:49But I don't know any library with similar functionality that is also cross platform, easy to port and has a license that is friendly to apple devices and consoles.

This functionality may be written by the engine developers too, if it becomes necessary.

My point was not to say the mojoAL is bad overall, my intent was to voice my concerns about openal's interface may not be ideal for the purposes of this engine and its possible future features.
It's not like we're using alot of openal api at the moment. MojoAL is reusing SDL2 audio, as you mentioned yourself, so it may be seen as a wrapper over SDL2 audio component with buffering and certain sound effects algorithms. I believe that in theory it is feasible to recreate this functionality in a different form.

It also could be that there are ways to solve the mentioned issue within OpenAL as well. I believe that it's worth investigating this question.

Dualnames

Sorry to necropost this. I've been looking into the engine, and one thing strikes me as weird, the choice of MojoAl. Don't get me wrong it works for what it works for, but wouldn't switching to OpenAl - Soft give like the same features + all the cool effects (Reverb, Pass Filters, Echo, Delay, etc)?
Open Al has been rather 'abandoned' since 2012 or so.
Worked on Strangeland, Primordia, Hob's Barrow, The Cat Lady, Mage's Initiation, Until I Have You, Downfall, Hunie Pop, and every game in the Wadjet Eye Games catalogue (porting)

Crimson Wizard

#13
Quote from: Dualnames on Tue 23/08/2022 01:42:31
Sorry to necropost this. I've been looking into the engine, and one thing strikes me as weird, the choice of MojoAl. Don't get me wrong it works for what it works for, but wouldn't switching to OpenAl - Soft give like the same features + all the cool effects (Reverb, Pass Filters, Echo, Delay, etc)?
Open Al has been rather 'abandoned' since 2012 or so.

When we were moving to SDL2, we used much of the work prepared by another developer (Nick Sonneveld) to speed this process up. At that time I did not ask questions about this.
If I remember correctly, the choice of OpenAl's implementation was affected by some issue with the license and/or compilation on certain platforms (such as consoles, as Sonneveld was working on a Switch port of his own simultaneously).

If we keep using OpenAl interface in the future, then switching between mojoAL and OpenAl Soft is purely a matter of replacing one library on another. At this moment there is not much sense in that, as nobody would be able to use these additional effects without a proper scripting API. In regards to scripting, 1) it's too late for 3.6.0, and 2) tbh personally I'd rather revamp the audio api in whole before adding such things.

Dualnames

That makes sense. Imho, either moving to sdl sound mixer [which allows for callbacks (that still means manually writing the effects)] or imho OpenAl Soft [which has all these effects already implemented] would be a great step down the line.
Regardless, I do understand your point of view (just wanted to state my thoughts more explicitly), and I would love to help out if I can with such implementation.

I think (2) makes sense 100%.
Worked on Strangeland, Primordia, Hob's Barrow, The Cat Lady, Mage's Initiation, Until I Have You, Downfall, Hunie Pop, and every game in the Wadjet Eye Games catalogue (porting)

eri0o

#15
OpenAL Soft was very problematic on Windows last time I tried - it required a installation process.

But I don't understand the abandoned information, the main developer seen to commit everyday: https://github.com/kcat/openal-soft

SMF spam blocked by CleanTalk