[Feature Request] Videos non-blocking playback with GUIs

[eri0o]

Being able to playback videos in AGS while using GUIs at the same time could enable building FMV games in AGS. Bonus points for having the video just run in any Overlay (regular overlays or room overlays), which would allow something similar to using videos for things in the background or some specifics, like say, a communication device in a GUI that receives a message.

From what I remember, when a video is playing AGS enters in a special state, so the screen is exclusively for the video and it can't process other things beyond the key to skip the video.

This as other special states are distributed throughout the code (see: Refactor AGS for 1 game loop and game state managing). It looks like some states could be instead managed through a Finite State Machine, but others are things we kinda want to just happen at the same frame (so not exactly a different state). This refactoring is not necessary, but if there was some similar way to arrange this it could maybe work. Video is a bit tricky that the sound and image has to be synchronous, so not sure that could workout.

This thread is both to discuss use cases in game and to see what would be needed in ags details, and also figuring out things like the API for this.

• Thread about GUI elements on top of video

[Vincent]

I think that since this engine is mainly based on a certain type of video games (point and click) and since FMV games somehow fall into this category, the function of playing a video should work a bit more broadly, such as running a video without blocks or having more resources in manipulating a video. I think it would be nice if videos can have more resources of manipulation like the audio channels do. For example having something like this for videos too:

Video.Pause
Video.Resume
Video.Seek
Video.SeekMs
Video.Speed
Video.Stop
Video.ID
Video.IsPaused
Video.IsPlaying
Video.LengthMs
Video.Position
Video.PositionMs
Video.Volume

But of course also keep the things that it already does because they are useful as VideoSkipStyle and Flags.

[Crimson Wizard]

My opinion on this, shortly:

1. I believe there should be a sort of VideoPlayback type (name is an example), which is returned from PlayVideo, and lets you control the video, similar to how AudioChannel lets you control the audio playback.
On a side note, as we discussed this previously, we should consider introducing AudioPlayback type, representing a playing audio instance, which is returned from AudioClip.Play, instead of returning AudioChannel. Maybe AudioPlayback and VideoPlayback could have a parent type (MediaPlayback?) which shares common properties and functions too. This will make both video and audio handled similarly.

2. The video should be updated on the main game loop, regardless of whether it's blocking or non-blocking. The difference between these would be simply that some of the game parts are not run during blocking video, similarly to how some parts are paused during blocking Walk or Say command.
NOTE: the video decoding may be performed on a separate thread even. By "updated on the main loop" I mean that the playback state and displayed image should be updated when the rest of the game updates.

3. The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

4. The video's audio should be played through standard audio system, VideoPlayback should have a reference to AudioChannel it plays its audio on.
Maybe AGS will let configure which channel to play on, but that's a secondary question.
There are other details to decide here, like, should the video's audio have its own configurable "audio type", and so on.

[Vincent]

I think all CW opinions exposed are very pleasant and well organized which overall should work everything fine.
On a side note I was thinking of the easiest way for users to play a video as simply as playing an audio file in ags, in my mind I picture it something like this:

Maybe have a separate category for videos:


So you can simply add a video as you do with an audio file:


So every single video has his own properties (Regarding point 4 I think the video should have his own audio type too):


And inside the script we might eventually do something like this:

Code: ags

Video*video;
video = vIntro.Play(audioPriority, repeatStyle, blockingStyle, videoSkipStyle, flags);

     // vIntro.Play(eAudioPriorityNormal, eOnce, eNoBlock, eVideoSkipNotAllowed, 10);

[eri0o]

The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

[Crimson Wizard]

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

There is also a question of ownership. If user passes a dynamic sprite to the Play function, that means that the user also owns the sprite. This means that user may delete the sprite or edit it anytime by accessing its DrawingSurface. That may cause conflicts with what video is trying to do. For that reason I'd consider an alternative method of passing width and height into Play function, and let video object create its own sprite.

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

The decoding must happen on a separate thread to not unnecessarily load up or block the game's thread. Also, the video playback may work with its own fps setting, not necessarily matching the game's fps.

The solution I could come up with at this point is to implement a back-buffer mechanism for this sprite. That means that instead of having 1 bitmap (or texture), the sprite has 2 associated bitmaps (or textures). One of them is considered to be sync with the game, and another is the one that video decoder writes to. Upon entering a game frame update, these bitmaps switch: a previous back-buffer becomes an active sprite's surface, and the previous active surface becomes back-buffer, that a video decoder will write to when necessary.

[Crimson Wizard]

I was too lazy, but finally opened a ticket with this task:
https://github.com/adventuregamestudio/ags/issues/2155

[eri0o]

I have been just getting a general feel on what is the status of video codecs today - this is only tangentially related, but still.

So I found the following:

pl_mpeg: this is mpeg-1 codec, which is very outdated, but has expired patents,  and the code for it is also very small and builds to few kB.

dav1d: this is an AV1 codec written mostly in assembly with a bit of C, it's cross platform and requires nasm and meson to build. The built lib dav1d.dll is around 3MB and using strip gets it down to 2MB. License is not GPL, so we can use on iOS. Plus, AV1 has no patent license requirements or royalties!

libgav1: this is the AV1 library that is available in Android, but it can be built cross platform too as it's used in Google Chrome too. It's not as fast as dav1d - 1920x1080@60 is definitely heavy for it, even on my desktop. But it uses CMake and doesn't require nasm, can be built with any compiler with C++11 support! It's also smaller in size. And it has permissable license and as AV1, no patent license/royalties are necessary.

SDL3: it's not added yet, but there are talks of adding an API for SDL to access system libraries. Interesting to follow up.


All in all the general idea is to have some plugin interface for adding more codecs more easily, but I wanted to take a look on how is the current situation for video codecs regarding patents, licenses and royalties - h264 and similar are out of the picture as libraries because of those.

[Crimson Wizard]

I started writing a draft for this (based on AGS 4):
https://github.com/adventuregamestudio/ags/pull/2338

[Crimson Wizard]

[deleted as not working]

[eri0o]

Hi @Crimson Wizard ,

I tried copy and pasting mostly of what you made in this project here: Ags4VideoTest.zip

Unfortunately it gives me an error



Any ideas what is going on?

Log is below

Spoiler

Code: ags

[Main][Alert]: Adventure Game Studio v4.0 Interpreter
Copyright (c) 1999-2011 Chris Jones and 2011-2024 others
Engine version 4.0.0.3, 32-bit LE

[Main][Info]: Installing exception handler
[Main][Info]: Initializing backend libs
[Main][Info]: SDL Version: 2.30.0
[Main][Info]: Try connect to the external debugger
[Main][Info]: External debugger initialized
[Main][Info]: Initializing game data
[Main][Debug]: Looking for the game data.
 Cwd: C:/Users/user/Documents/AGSProjects/Ags4VideoTest
 Path arg: 
[Main][Debug]: Found game data embedded in executable
[Main][Info]: Located game data pak: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/_Debug/Ags4VideoTest.exe
[Main][Info]: Opened game data file: game28.dta
[Main][Info]: Game data version: 4000000
[Main][Info]: Compiled with: 4.00.00.03
[Main][Info]: Startup directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/_Debug/
[Main][Info]: Data directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/Compiled/Windows
[Main][Info]: Opt data directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest
[Main][Info]: Opt audio directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/AudioCache
[Main][Info]: Opt voice-over directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/Speech
[Main][Info]: Setting up game configuration
[Main][Info]: Logging to warnings.log
[Main][Info]: Was not able to init voice pack 'speech.vox': file not found or of unknown format.
[Main][Info]: Audio pack was not found, but explicit audio directory is defined.
[Main][Info]: Initializing TTF renderer
[Main][Info]: Initializing mouse: number of buttons reported is 3
[Main][Debug]: Initializing audio
[Main][Debug]: Requested audio driver: default
[Main][Info]: Audio driver: wasapi
[Main][Info]: AudioCore: opened device "Default OpenAL playback device"
[Main][Info]: Supported sound decoders:
[Main][Info]:  - MIDI decoder, using a subset of TiMidity : MIDI,MID,
[Main][Info]:  - Play modules through ModPlug : 669,AMF,AMS,DBM,DMF,DSM,FAR,GDM,IT,MDL,MED,MOD,MT2,MTM,OKT,PTM,PSM,S3M,STM,ULT,UMX,XM,
[Main][Info]:  - MPEG-1 Audio Layer I-III : MP3,MP2,MP1,
[Main][Info]:  - Microsoft WAVE audio format : WAV,
[Main][Info]:  - Audio Interchange File Format : AIFF,AIF,
[Main][Info]:  - Sun/NeXT audio file format : AU,
[Main][Info]:  - Ogg Vorbis audio : OGG,
[Main][Info]:  - Creative Labs Voice format : VOC,
[Main][Info]:  - Raw audio : RAW,
[Main][Info]:  - Shorten-compressed audio data : SHN,
[Main][Info]:  - Free Lossless Audio Codec : FLAC,FLA,
[Main][Debug]: Sound cache set: 32768 KB
[Main][Info]: Install exit handler
[Main][Info]: Initialize path finder library
[Main][Debug]: Load game data
[Main][Info]: Game title: 'Ags4VideoTest'
[Main][Info]: Game uid (old format): `2251531`
[Main][Info]: Game guid: '{bb2a0f89-ac0e-4141-8531-bc83ecc1dc02}'
[Main][Info]: Game GUI version: 119
[Main][Info]: Requested script API: 4.0.0 (4000003), compat level: 4.0.0 (4000003)
[Main][Info]: Game native resolution: 320 x 200 (32 bit)
[Main][Debug]: Mouse cursor graphic area: (0,0)-(319,199) (320x200)
[Main][Info]: Checking for disk space
[Main][Debug]: Device display resolution: 2560 x 1440
[Main][Info]: Graphic settings: driver: D3D9, windowed: yes, screen size: 0 x 0, game scale: round
[Main][Info]: Graphic settings: refresh rate (optional): 0, vsync: 0
[Main][Debug]: Try library path: d3d9.dll
[Main][Info]: Direct3D adapter info:
    Driver: nvldumd.dll, v31.0.15.3623
    Description: NVIDIA GeForce RTX 2080 SUPER
[Main][Debug]: Using graphics factory: D3D9
[Main][Debug]: Created graphics driver: Direct3D 9
[Main][Debug]: Supported gfx modes (32-bit): 
    640x480;640x480;640x480;720x480;720x480;720x576;800x600;800x600;
    1024x768;1024x768;1152x864;1176x664;1176x664;1176x664;1280x720;1280x720;
    1280x720;1280x768;1280x800;1280x960;1280x960;1280x1024;1280x1024;1360x768;
    1360x768;1366x768;1366x768;1440x1080;1440x1080;1440x1080;1440x1080;1440x1080;
    1600x900;1600x900;1600x1024;1600x1024;1600x1200;1680x1050;1680x1050;1920x1080;
    1920x1080;1920x1080;1920x1080;1920x1080;1920x1200;1920x1440;1920x1440;2560x1440;
    2560x1440;2048x1080;2048x1080;
[Main][Debug]: Attempting to find nearest supported resolution for screen size 1920 x 1200 (32-bit) windowed
[Main][Debug]: Maximal allowed window size: 2464 x 1393
[Main][Debug]: Attempt to switch gfx mode to 1920 x 1200 (32-bit) windowed
[Main][Info]: Graphics driver set: Direct3D 9
[Main][Info]: Graphics mode set: 1920 x 1200 (32-bit) windowed
[Main][Info]: Graphics mode set: refresh rate (optional): 59, vsync: 0
[Main][Debug]: Graphics driver texture memory (approx): 4185088 KB
[Main][Debug]: Render frame set, render dest (0, 0, 1919, 1199 : 1920 x 1200)
[Main][Debug]: Requested gfx filter: stdscale
[Main][Debug]: Graphics filter set: 'StdScale', filter dest (0, 0, 1919, 1199 : 1920 x 1200)
[Main][Debug]: Using gfx filter: StdScale
[Main][Debug]: Texture cache set: 131072 KB
[Main][Info]: Mouse speed control: enabled, unit: 1.000000, user value: 1.000000
[Main][Info]: Touch-to-mouse motion mode: absolute
[Main][Debug]: Mouse cursor graphic area: (0,0)-(1919,1199) (1920x1200)
[Main][Debug]: SetMultitasking: overridden by the external debugger: 0 -> 1
[Main][Info]: Multitasking mode set: 1
[Main][Info]: Setting up window
[Main][Debug]: SetMultitasking: overridden by the external debugger: 0 -> 1
[Main][Info]: Multitasking mode set: 1
[Main][Info]: Initialize sprites
[Main][Debug]: Sprite cache set: 131072 KB
[Main][Debug]: Initialize game settings
[Main][Debug]: Precache view 0 (loops 0-3) with 28 frames, total = 0 ms, average file->mem = 0 ms, bm->tx = 0 ms,
        loaded 0 sounds = 0 ms
[Main][Debug]:    Sprite cache: 13 -> 150 KB, texture cache: 0 -> 136 KB
[Main][Debug]: Prepare to start game
[Game][Debug]: (room:-10) Cursor mode set to 0
[Main][Info]: Engine initialization complete
[Main][Info]: Starting game
[Game][Debug]: (room:-10) Cursor mode set to 0
[Game][Debug]: (room:-10)[G 9] Game speed set to 40
[Game][Debug]: (room:-10) Transition-out in room -10
[Game][Debug]: (room:-10) Loading room 1
[Game][Debug]: (room:1) Mouse bounds constrained to (0,0)-(319,199)
[Game][Debug]: (room:1) Room camera released back to engine control
[Game][Debug]: (room:1) Now in room 1
[Game][Debug]: (room:1) Transition-in in room 1
[Main][Info]: VideoCore: init thread

[close]

Edit: Bizarrely it works when I try to debug using your source code in my C++ IDE!

Running the standalone exe the game just closes without message. I only get the "The game engine does not appear to have shut down properly" message when running the game project from the AGS Editor.

[Crimson Wizard]

[deleted as not working]


But this also points out that audio_core is not coded safely since 3.6.0 (since video_core is mostly written having audio core as an example).

[eri0o]

I closed things here. I then rebuilt it locally here using Visual Studio, and copied the acwin.exe on top of the old one in the downloaded editor directory. Then I ran the Editor and got the same error here when trying to run from inside the AGS Editor (clicking the play button for debug in the editor). I am not sure why. Edit: deleted the project _Debug and Compiled directories but it still gave me the error.

[Crimson Wizard]

Well, I dont have any crashes at the moment, whether running from the editor or from compiled folder.
I might test more later, and add safety checks where makes sense.

[eri0o]

Did you use the project I made linked in the previous message? Also to check it's not something specific from my video file.

I don't remember if ags4 still magically packages ogv files in the root dir of the project or if I should set an explicit DATA directory in general settings. Are the standard file tokens supported in the sketch video2 API?

[Crimson Wizard]

I've been testing the editor and engine that I build myself, and everything worked. The crash happened when the video was stopping.

Now I tested downloaded version, and it crashes right away when the video starts.

I don't know what's the difference, and don't have time to investigate right now, so I will do this later.
Will remove the download links for now.

[Crimson Wizard]

I found that it works with the old SDL2.dll (2.28.0) which I've been using for a while, but not with the SDL2.dll which comes from the CI (2.30.0).

EDIT:
The crash is happening when calling SDL_ConvertAudio with the new SDL2 2.30.

EDIT2:
The regular music plays, so this is either something specific to the sound format that the particular video uses, or any mistake in sound processing that did not cause trouble until now.

[eri0o]

Uhm, there was an issue with audio from float conversion reported for 2.30.0 but I didn't thought much at the time, I think it's best to rollback the 2.30.0 upgrade commit and try again in 2.30.1 later. The issue did got fixed but it was after 2.30.0 release, so it's waiting until a new release comes up.

https://github.com/libsdl-org/SDL/issues/9099

I will try to use git-blame to check around the audio code if some commit comes up.

[Crimson Wizard]

https://github.com/libsdl-org/SDL/issues/9099

This issue is about audio format AUDIO_S8, the APEG has audio in AUDIO_S16SYS.

[eri0o]

I think I figured it. Building the same tag 2.30.0 works when building from source, the issue seems to be in the specific DLLs that are shipped in the SDL release. This is the third time this happens, the SDL2.dll is not built in their CI and neither the VC package we use, for some reason they build in some of the devs machine and manually upload to GitHub and it's the binary release that has the problem (for some reason).