Jibble

There is a new version of the forums ready for testing. Please post here if you're willing to help test! We could also use the help of coders!

Author Topic: 3D/Positional Audio using OpenAL  (Read 654 times)

3D/Positional Audio using OpenAL
« on: 21 Apr 2022, 23:47 »
Context: I recently discovered that OpenAL (and via extensions?) supports Positional/3D audio. Although I spent some time browsing the ags github repo, I am not clear as to the progress made with the transition to SDL2 and OpenAL with 3.6.0. It *appears* from a glance that 3.6.0 is now using mojoAL for interfacing with OpenAL.

Question: Is it possible to write a plug-in for AGS that would enable some basic positional/3D audio functions that would allow run-time changes to audio during gameplay? I am working on a first-person adventure that would greatly benefit from this feature, and I'm willing to put in the wrench time to make it happen, if it is possible at all.

Crimson Wizard

  • Local Moderator
    • Lifetime Achievement Award Winner
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: 3D/Positional Audio using OpenAL
« Reply #1 on: 22 Apr 2022, 00:15 »
Although I spent some time browsing the ags github repo, I am not clear as to the progress made with the transition to SDL2 and OpenAL with 3.6.0. It *appears* from a glance that 3.6.0 is now using mojoAL for interfacing with OpenAL.

3.6.0 has fully transition to SDL2 and is using mojoAL for audio playback, which is an OpenAL interface and SDL2-based implementation.
The parts of Allegro library are embedded into the engine code now; these are parts related to bitmap manipulation (pixel drawing) and utf-8 handling.


Question: Is it possible to write a plug-in for AGS that would enable some basic positional/3D audio functions that would allow run-time changes to audio during gameplay? I am working on a first-person adventure that would greatly benefit from this feature, and I'm willing to put in the wrench time to make it happen, if it is possible at all.

I don't think that's possible without changes in both engine and plugin API. Playback settings afaik are not exposed to plugin API in any way. Internally, in the engine playback does not have 3D settings. So even though the backend allows it, engine does not have this logic implemented anywhere.

AFAIK AGS supports sound stereo panning between left & right (see AudioChannel.Panning).
« Last Edit: 22 Apr 2022, 00:17 by Crimson Wizard »

Re: 3D/Positional Audio using OpenAL
« Reply #2 on: 22 Apr 2022, 00:42 »

3.6.0 has fully transition to SDL2 and is using mojoAL for audio playback, which is an OpenAL interface and SDL2-based implementation.
The parts of Allegro library are embedded into the engine code now; these are parts related to bitmap manipulation (pixel drawing) and utf-8 handling.

I don't think that's possible without changes in both engine and plugin API. Playback settings afaik are not exposed to plugin API in any way. Internally, in the engine playback does not have 3D settings. So even though the backend allows it, engine does not have this logic implemented anywhere.

AFAIK AGS supports sound stereo panning between left & right (see AudioChannel.Panning).

Gotcha! It sounds like this would be a very long term project if there was interest in it. In the interim I will do my best with the .Panning property - thanks.

eri0o

Re: 3D/Positional Audio using OpenAL
« Reply #3 on: 22 Apr 2022, 02:58 »
Can you elaborate what are you trying to achieve with 3D positional audio?

Like, as an example, Alan's Draconian Ags4 fork has sound from views attached to characters corresponding to either the character position on screen, or character position relative to player character. Is it this a similar case or more playing a sound at a specific position?

AGS has AudioChannel.SetRoomLocation, so if it's more on the single API implementation, perhaps a similar API could be made, which would be more flexible but more disconnected from internal AGS game logic.

Re: 3D/Positional Audio using OpenAL
« Reply #4 on: 22 Apr 2022, 04:05 »
Can you elaborate what are you trying to achieve with 3D positional audio?

Like, as an example, Alan's Draconian Ags4 fork has sound from views attached to characters corresponding to either the character position on screen, or character position relative to player character. Is it this a similar case or more playing a sound at a specific position?

AGS has AudioChannel.SetRoomLocation, so if it's more on the single API implementation, perhaps a similar API could be made, which would be more flexible but more disconnected from internal AGS game logic.

So to clarify - the OpenAL 3D sound implementation is not simple Right-Left panning. It employs a complex set of algorithms to emulate positional sound in a 3D space using a Head-related Transfer Function. This is effectively "surround sound" produced algorithmically in stereo, instead of via many channels (5.1/7.1). You can watch/listen to an HRTF demo here. It is a much more powerful effect than a standard stereo pan.

After doing some reading, it is not clear to me if mojoAL supports HRTF, or if that functionality is enabled through an extension like ALC_SOFT_HRTF.

Crimson Wizard

  • Local Moderator
    • Lifetime Achievement Award Winner
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: 3D/Positional Audio using OpenAL
« Reply #5 on: 22 Apr 2022, 07:59 »
TBH I'm starting having doubts that using OpenAL interface in general is a good idea for AGS project. It came as a part of the SDL port done by Nick Sonneveld, and which we decided to use, and I trusted this choice; but the more i looked into it on my own, the more potential problems I saw. I think the biggest is that it's another layer of complexity which we cannot fully control. Good example is playback "Speed" property: it appeared OpenAL does not have a direct equivalent for it, so I had to implement it on top by resampling the sound data. The sound is "sped up" before passing into openal, but because we must buffer at least couple chunks of data there all time, there's a delay at which the "speed" parameter is applied to the output sound, and that delay depends on the size of this buffering. In Allegro4 it was applied for the very next played chunk perhaps at the very end of the data chain before it goes out to the sound driver, so almost immediately. I'm having concerns that situation may become worse if we decide to implement more sophisticated sound system in AGS script, with mixers, and so on.

Note that mojoAL uses SDL2 internally, which we also use directly in the engine, so the question to ponder is, what use is OpenAL for us, and how it will work in perspective.
If it's actually useful, perhaps there's a proper way to "extend" it ourselves and let us control the sound closer to the final output.

Personally, i used ffmpeg before and liked its "filter" design, where sound transformations may be applied freely at any stage of the data processing. Compared to that, OpenAL is a "black box" with everything which connects directly to the output.
« Last Edit: 22 Apr 2022, 08:41 by Crimson Wizard »

eri0o

Re: 3D/Positional Audio using OpenAL
« Reply #6 on: 22 Apr 2022, 10:38 »
MojoAL is just an OpenAL implementation. It uses Sdl2 to make everything work. Like SDL_Sound has received lots of updates recently, I believe MojoAl will receive too. The advantage of it is license (not every license is compatible with Apple devices) and easy of building it, and easy of porting if needed. Additionally, we can use a different OpenAl implementation if needed - the Emscripten port doesn't use MojoAl, and neither does the Android port, they use their own platforms OpenAL implementation and interfaces - they could use MojoAL too, but I thought since a native platform specific version was available it was alright to use that instead.

The sound decoding we use is separated in SDL_Sound and the sound sink (actual playback) is through SDL2 (except in Android and Emscripten). The OpenAL gives both control to the sound playback as to the final device that plays it. So if a different solution is used these are the points that need to be done in it.

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.
« Last Edit: 22 Apr 2022, 10:40 by eri0o »

Re: 3D/Positional Audio using OpenAL
« Reply #7 on: 22 Apr 2022, 14:31 »

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

OK that makes a lot of sense, thanks! I'll save this wishlist item for Unity :)

eri0o

Re: 3D/Positional Audio using OpenAL
« Reply #8 on: 22 Apr 2022, 15:21 »

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

OK that makes a lot of sense, thanks! I'll save this wishlist item for Unity :)

I was answering CW, your approach of identifying HRTF in OpenAL spec is what I meant as the correct approach towards getting the feature in MojoAL.

Summary: there are no perfect libraries.

Crimson Wizard

  • Local Moderator
    • Lifetime Achievement Award Winner
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: 3D/Positional Audio using OpenAL
« Reply #9 on: 22 Apr 2022, 15:46 »
I was answering CW, your approach of identifying HRTF in OpenAL spec is what I meant as the correct approach towards getting the feature in MojoAL.

Summary: there are no perfect libraries.

Sigh, I was in doubts to whom you were answering, so deleted my next reply.

The reply was this:

---

About MojoAL it's better to not think of what it does and does not support, but if OpenAL supports it in some standard, and then notify the developer and figure how to support that if needed.

In my previous post I specifically mentioned "using OpenAL interface in general", not MojoAL. MojoAl's own problems is an issue on its own, but I was talking about using the OpenAL interface. I'm questioning the convenience of that interface in the context of AGS engine.
« Last Edit: 22 Apr 2022, 15:57 by Crimson Wizard »

eri0o

Re: 3D/Positional Audio using OpenAL
« Reply #10 on: 22 Apr 2022, 15:57 »
I like node based audio libraries like AVAudioEngine for Apple devices and the Web Audio API in Web Browsers (Firefox, Chrome and Safari). But I don't know any library with similar functionality that is also cross platform, easy to port and has a license that is friendly to apple devices and consoles.

Crimson Wizard

  • Local Moderator
    • Lifetime Achievement Award Winner
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: 3D/Positional Audio using OpenAL
« Reply #11 on: 22 Apr 2022, 15:59 »
But I don't know any library with similar functionality that is also cross platform, easy to port and has a license that is friendly to apple devices and consoles.

This functionality may be written by the engine developers too, if it becomes necessary.

My point was not to say the mojoAL is bad overall, my intent was to voice my concerns about openal's interface may not be ideal for the purposes of this engine and its possible future features.
It's not like we're using alot of openal api at the moment. MojoAL is reusing SDL2 audio, as you mentioned yourself, so it may be seen as a wrapper over SDL2 audio component with buffering and certain sound effects algorithms. I believe that in theory it is feasible to recreate this functionality in a different form.

It also could be that there are ways to solve the mentioned issue within OpenAL as well. I believe that it's worth investigating this question.
« Last Edit: 22 Apr 2022, 17:43 by Crimson Wizard »