[Feature Request] Videos non-blocking playback with GUIs

[eri0o]

Being able to playback videos in AGS while using GUIs at the same time could enable building FMV games in AGS. Bonus points for having the video just run in any Overlay (regular overlays or room overlays), which would allow something similar to using videos for things in the background or some specifics, like say, a communication device in a GUI that receives a message.

From what I remember, when a video is playing AGS enters in a special state, so the screen is exclusively for the video and it can't process other things beyond the key to skip the video.

This as other special states are distributed throughout the code (see: Refactor AGS for 1 game loop and game state managing). It looks like some states could be instead managed through a Finite State Machine, but others are things we kinda want to just happen at the same frame (so not exactly a different state). This refactoring is not necessary, but if there was some similar way to arrange this it could maybe work. Video is a bit tricky that the sound and image has to be synchronous, so not sure that could workout.

This thread is both to discuss use cases in game and to see what would be needed in ags details, and also figuring out things like the API for this.

• Thread about GUI elements on top of video

[Vincent]

I think that since this engine is mainly based on a certain type of video games (point and click) and since FMV games somehow fall into this category, the function of playing a video should work a bit more broadly, such as running a video without blocks or having more resources in manipulating a video. I think it would be nice if videos can have more resources of manipulation like the audio channels do. For example having something like this for videos too:

Video.Pause
Video.Resume
Video.Seek
Video.SeekMs
Video.Speed
Video.Stop
Video.ID
Video.IsPaused
Video.IsPlaying
Video.LengthMs
Video.Position
Video.PositionMs
Video.Volume

But of course also keep the things that it already does because they are useful as VideoSkipStyle and Flags.

[Crimson Wizard]

My opinion on this, shortly:

1. I believe there should be a sort of VideoPlayback type (name is an example), which is returned from PlayVideo, and lets you control the video, similar to how AudioChannel lets you control the audio playback.
On a side note, as we discussed this previously, we should consider introducing AudioPlayback type, representing a playing audio instance, which is returned from AudioClip.Play, instead of returning AudioChannel. Maybe AudioPlayback and VideoPlayback could have a parent type (MediaPlayback?) which shares common properties and functions too. This will make both video and audio handled similarly.

2. The video should be updated on the main game loop, regardless of whether it's blocking or non-blocking. The difference between these would be simply that some of the game parts are not run during blocking video, similarly to how some parts are paused during blocking Walk or Say command.
NOTE: the video decoding may be performed on a separate thread even. By "updated on the main loop" I mean that the playback state and displayed image should be updated when the rest of the game updates.

3. The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

4. The video's audio should be played through standard audio system, VideoPlayback should have a reference to AudioChannel it plays its audio on.
Maybe AGS will let configure which channel to play on, but that's a secondary question.
There are other details to decide here, like, should the video's audio have its own configurable "audio type", and so on.

[Vincent]

I think all CW opinions exposed are very pleasant and well organized which overall should work everything fine.
On a side note I was thinking of the easiest way for users to play a video as simply as playing an audio file in ags, in my mind I picture it something like this:

Maybe have a separate category for videos:


So you can simply add a video as you do with an audio file:


So every single video has his own properties (Regarding point 4 I think the video should have his own audio type too):


And inside the script we might eventually do something like this:

Code: ags

Video*video;
video = vIntro.Play(audioPriority, repeatStyle, blockingStyle, videoSkipStyle, flags);

     // vIntro.Play(eAudioPriorityNormal, eOnce, eNoBlock, eVideoSkipNotAllowed, 10);

[eri0o]

The video should be decoded onto a dynamically created sprite, which may then be applied to any object. This will allow to play video on anything, and sort among other game elements on screen. VideoPlayback will have a Graphic property, or similar, to let reference this sprite. The sprite likely should be "owned" by the video until it is stopped, after which the sprite is disposed automatically.

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

[Crimson Wizard]

This sounds like a good idea, but just to try to figure out how is the "video sprite" size specified, does it uses the video resolution or is the dynamic sprite something we would create separately with whatever size and pass the dynamic sprite pointer to it? (VideoPlayback* vp = vcIntro.Play(dyn_spr);)

There is also a question of ownership. If user passes a dynamic sprite to the Play function, that means that the user also owns the sprite. This means that user may delete the sprite or edit it anytime by accessing its DrawingSurface. That may cause conflicts with what video is trying to do. For that reason I'd consider an alternative method of passing width and height into Play function, and let video object create its own sprite.

About the decoding, somehow the frame has to be set at the beginning or at the end of the game frame, so the contents of the dynamic sprite is predictable so people can assign it to things.

The decoding must happen on a separate thread to not unnecessarily load up or block the game's thread. Also, the video playback may work with its own fps setting, not necessarily matching the game's fps.

The solution I could come up with at this point is to implement a back-buffer mechanism for this sprite. That means that instead of having 1 bitmap (or texture), the sprite has 2 associated bitmaps (or textures). One of them is considered to be sync with the game, and another is the one that video decoder writes to. Upon entering a game frame update, these bitmaps switch: a previous back-buffer becomes an active sprite's surface, and the previous active surface becomes back-buffer, that a video decoder will write to when necessary.

[Crimson Wizard]

I was too lazy, but finally opened a ticket with this task:
https://github.com/adventuregamestudio/ags/issues/2155

[eri0o]

I have been just getting a general feel on what is the status of video codecs today - this is only tangentially related, but still.

So I found the following:

pl_mpeg: this is mpeg-1 codec, which is very outdated, but has expired patents,  and the code for it is also very small and builds to few kB.

dav1d: this is an AV1 codec written mostly in assembly with a bit of C, it's cross platform and requires nasm and meson to build. The built lib dav1d.dll is around 3MB and using strip gets it down to 2MB. License is not GPL, so we can use on iOS. Plus, AV1 has no patent license requirements or royalties!

libgav1: this is the AV1 library that is available in Android, but it can be built cross platform too as it's used in Google Chrome too. It's not as fast as dav1d - 1920x1080@60 is definitely heavy for it, even on my desktop. But it uses CMake and doesn't require nasm, can be built with any compiler with C++11 support! It's also smaller in size. And it has permissable license and as AV1, no patent license/royalties are necessary.

SDL3: it's not added yet, but there are talks of adding an API for SDL to access system libraries. Interesting to follow up.


All in all the general idea is to have some plugin interface for adding more codecs more easily, but I wanted to take a look on how is the current situation for video codecs regarding patents, licenses and royalties - h264 and similar are out of the picture as libraries because of those.

[Crimson Wizard]

I started writing a draft for this (based on AGS 4):
https://github.com/adventuregamestudio/ags/pull/2338

[Crimson Wizard]

[deleted as not working]

[eri0o]

Hi @Crimson Wizard ,

I tried copy and pasting mostly of what you made in this project here: Ags4VideoTest.zip

Unfortunately it gives me an error



Any ideas what is going on?

Log is below

Spoiler

Code: ags

[Main][Alert]: Adventure Game Studio v4.0 Interpreter
Copyright (c) 1999-2011 Chris Jones and 2011-2024 others
Engine version 4.0.0.3, 32-bit LE

[Main][Info]: Installing exception handler
[Main][Info]: Initializing backend libs
[Main][Info]: SDL Version: 2.30.0
[Main][Info]: Try connect to the external debugger
[Main][Info]: External debugger initialized
[Main][Info]: Initializing game data
[Main][Debug]: Looking for the game data.
 Cwd: C:/Users/user/Documents/AGSProjects/Ags4VideoTest
 Path arg: 
[Main][Debug]: Found game data embedded in executable
[Main][Info]: Located game data pak: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/_Debug/Ags4VideoTest.exe
[Main][Info]: Opened game data file: game28.dta
[Main][Info]: Game data version: 4000000
[Main][Info]: Compiled with: 4.00.00.03
[Main][Info]: Startup directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/_Debug/
[Main][Info]: Data directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/Compiled/Windows
[Main][Info]: Opt data directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest
[Main][Info]: Opt audio directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/AudioCache
[Main][Info]: Opt voice-over directory: c:/Users/user/Documents/AGSProjects/Ags4VideoTest/Speech
[Main][Info]: Setting up game configuration
[Main][Info]: Logging to warnings.log
[Main][Info]: Was not able to init voice pack 'speech.vox': file not found or of unknown format.
[Main][Info]: Audio pack was not found, but explicit audio directory is defined.
[Main][Info]: Initializing TTF renderer
[Main][Info]: Initializing mouse: number of buttons reported is 3
[Main][Debug]: Initializing audio
[Main][Debug]: Requested audio driver: default
[Main][Info]: Audio driver: wasapi
[Main][Info]: AudioCore: opened device "Default OpenAL playback device"
[Main][Info]: Supported sound decoders:
[Main][Info]:  - MIDI decoder, using a subset of TiMidity : MIDI,MID,
[Main][Info]:  - Play modules through ModPlug : 669,AMF,AMS,DBM,DMF,DSM,FAR,GDM,IT,MDL,MED,MOD,MT2,MTM,OKT,PTM,PSM,S3M,STM,ULT,UMX,XM,
[Main][Info]:  - MPEG-1 Audio Layer I-III : MP3,MP2,MP1,
[Main][Info]:  - Microsoft WAVE audio format : WAV,
[Main][Info]:  - Audio Interchange File Format : AIFF,AIF,
[Main][Info]:  - Sun/NeXT audio file format : AU,
[Main][Info]:  - Ogg Vorbis audio : OGG,
[Main][Info]:  - Creative Labs Voice format : VOC,
[Main][Info]:  - Raw audio : RAW,
[Main][Info]:  - Shorten-compressed audio data : SHN,
[Main][Info]:  - Free Lossless Audio Codec : FLAC,FLA,
[Main][Debug]: Sound cache set: 32768 KB
[Main][Info]: Install exit handler
[Main][Info]: Initialize path finder library
[Main][Debug]: Load game data
[Main][Info]: Game title: 'Ags4VideoTest'
[Main][Info]: Game uid (old format): `2251531`
[Main][Info]: Game guid: '{bb2a0f89-ac0e-4141-8531-bc83ecc1dc02}'
[Main][Info]: Game GUI version: 119
[Main][Info]: Requested script API: 4.0.0 (4000003), compat level: 4.0.0 (4000003)
[Main][Info]: Game native resolution: 320 x 200 (32 bit)
[Main][Debug]: Mouse cursor graphic area: (0,0)-(319,199) (320x200)
[Main][Info]: Checking for disk space
[Main][Debug]: Device display resolution: 2560 x 1440
[Main][Info]: Graphic settings: driver: D3D9, windowed: yes, screen size: 0 x 0, game scale: round
[Main][Info]: Graphic settings: refresh rate (optional): 0, vsync: 0
[Main][Debug]: Try library path: d3d9.dll
[Main][Info]: Direct3D adapter info:
    Driver: nvldumd.dll, v31.0.15.3623
    Description: NVIDIA GeForce RTX 2080 SUPER
[Main][Debug]: Using graphics factory: D3D9
[Main][Debug]: Created graphics driver: Direct3D 9
[Main][Debug]: Supported gfx modes (32-bit): 
    640x480;640x480;640x480;720x480;720x480;720x576;800x600;800x600;
    1024x768;1024x768;1152x864;1176x664;1176x664;1176x664;1280x720;1280x720;
    1280x720;1280x768;1280x800;1280x960;1280x960;1280x1024;1280x1024;1360x768;
    1360x768;1366x768;1366x768;1440x1080;1440x1080;1440x1080;1440x1080;1440x1080;
    1600x900;1600x900;1600x1024;1600x1024;1600x1200;1680x1050;1680x1050;1920x1080;
    1920x1080;1920x1080;1920x1080;1920x1080;1920x1200;1920x1440;1920x1440;2560x1440;
    2560x1440;2048x1080;2048x1080;
[Main][Debug]: Attempting to find nearest supported resolution for screen size 1920 x 1200 (32-bit) windowed
[Main][Debug]: Maximal allowed window size: 2464 x 1393
[Main][Debug]: Attempt to switch gfx mode to 1920 x 1200 (32-bit) windowed
[Main][Info]: Graphics driver set: Direct3D 9
[Main][Info]: Graphics mode set: 1920 x 1200 (32-bit) windowed
[Main][Info]: Graphics mode set: refresh rate (optional): 59, vsync: 0
[Main][Debug]: Graphics driver texture memory (approx): 4185088 KB
[Main][Debug]: Render frame set, render dest (0, 0, 1919, 1199 : 1920 x 1200)
[Main][Debug]: Requested gfx filter: stdscale
[Main][Debug]: Graphics filter set: 'StdScale', filter dest (0, 0, 1919, 1199 : 1920 x 1200)
[Main][Debug]: Using gfx filter: StdScale
[Main][Debug]: Texture cache set: 131072 KB
[Main][Info]: Mouse speed control: enabled, unit: 1.000000, user value: 1.000000
[Main][Info]: Touch-to-mouse motion mode: absolute
[Main][Debug]: Mouse cursor graphic area: (0,0)-(1919,1199) (1920x1200)
[Main][Debug]: SetMultitasking: overridden by the external debugger: 0 -> 1
[Main][Info]: Multitasking mode set: 1
[Main][Info]: Setting up window
[Main][Debug]: SetMultitasking: overridden by the external debugger: 0 -> 1
[Main][Info]: Multitasking mode set: 1
[Main][Info]: Initialize sprites
[Main][Debug]: Sprite cache set: 131072 KB
[Main][Debug]: Initialize game settings
[Main][Debug]: Precache view 0 (loops 0-3) with 28 frames, total = 0 ms, average file->mem = 0 ms, bm->tx = 0 ms,
        loaded 0 sounds = 0 ms
[Main][Debug]:    Sprite cache: 13 -> 150 KB, texture cache: 0 -> 136 KB
[Main][Debug]: Prepare to start game
[Game][Debug]: (room:-10) Cursor mode set to 0
[Main][Info]: Engine initialization complete
[Main][Info]: Starting game
[Game][Debug]: (room:-10) Cursor mode set to 0
[Game][Debug]: (room:-10)[G 9] Game speed set to 40
[Game][Debug]: (room:-10) Transition-out in room -10
[Game][Debug]: (room:-10) Loading room 1
[Game][Debug]: (room:1) Mouse bounds constrained to (0,0)-(319,199)
[Game][Debug]: (room:1) Room camera released back to engine control
[Game][Debug]: (room:1) Now in room 1
[Game][Debug]: (room:1) Transition-in in room 1
[Main][Info]: VideoCore: init thread

[close]

Edit: Bizarrely it works when I try to debug using your source code in my C++ IDE!

Running the standalone exe the game just closes without message. I only get the "The game engine does not appear to have shut down properly" message when running the game project from the AGS Editor.

[Crimson Wizard]

[deleted as not working]


But this also points out that audio_core is not coded safely since 3.6.0 (since video_core is mostly written having audio core as an example).

[eri0o]

I closed things here. I then rebuilt it locally here using Visual Studio, and copied the acwin.exe on top of the old one in the downloaded editor directory. Then I ran the Editor and got the same error here when trying to run from inside the AGS Editor (clicking the play button for debug in the editor). I am not sure why. Edit: deleted the project _Debug and Compiled directories but it still gave me the error.

[Crimson Wizard]

Well, I dont have any crashes at the moment, whether running from the editor or from compiled folder.
I might test more later, and add safety checks where makes sense.

[eri0o]

Did you use the project I made linked in the previous message? Also to check it's not something specific from my video file.

I don't remember if ags4 still magically packages ogv files in the root dir of the project or if I should set an explicit DATA directory in general settings. Are the standard file tokens supported in the sketch video2 API?

[Crimson Wizard]

I've been testing the editor and engine that I build myself, and everything worked. The crash happened when the video was stopping.

Now I tested downloaded version, and it crashes right away when the video starts.

I don't know what's the difference, and don't have time to investigate right now, so I will do this later.
Will remove the download links for now.

[Crimson Wizard]

I found that it works with the old SDL2.dll (2.28.0) which I've been using for a while, but not with the SDL2.dll which comes from the CI (2.30.0).

EDIT:
The crash is happening when calling SDL_ConvertAudio with the new SDL2 2.30.

EDIT2:
The regular music plays, so this is either something specific to the sound format that the particular video uses, or any mistake in sound processing that did not cause trouble until now.

[eri0o]

Uhm, there was an issue with audio from float conversion reported for 2.30.0 but I didn't thought much at the time, I think it's best to rollback the 2.30.0 upgrade commit and try again in 2.30.1 later. The issue did got fixed but it was after 2.30.0 release, so it's waiting until a new release comes up.

https://github.com/libsdl-org/SDL/issues/9099

I will try to use git-blame to check around the audio code if some commit comes up.

[Crimson Wizard]

https://github.com/libsdl-org/SDL/issues/9099

This issue is about audio format AUDIO_S8, the APEG has audio in AUDIO_S16SYS.

[eri0o]

I think I figured it. Building the same tag 2.30.0 works when building from source, the issue seems to be in the specific DLLs that are shipped in the SDL release. This is the third time this happens, the SDL2.dll is not built in their CI and neither the VC package we use, for some reason they build in some of the devs machine and manually upload to GitHub and it's the binary release that has the problem (for some reason).

[Crimson Wizard]

They have the actual commit that sais "Fixed memory corruption when resampling S16 t F32" (and this is the case with the video):
https://github.com/libsdl-org/SDL/commit/384fcea585573db4e5707c249746008f25a0fbac

I noticed that when I build SDL, it comes with SSE2 functions, such as SDL_Convert_S16_to_F32_SSE2.
The fix was related to SDL_Convert_S16_to_F32_Scalar. As a theoretical possibility, maybe this function will run if one builds SDL2 without SSE2 support. Or maybe some other reason why it would fallback to the Scalar functions.

I wish there were a way to tell which conversion func is getting called at runtime.

[Crimson Wizard]

Well, I opened a ticket for them: https://github.com/libsdl-org/SDL/issues/9127

No idea yet whether this is because of the error in Scalar conversion functions that they already fixed, or it's just a coincidence.

[eri0o]

Hey, I went I used an older SDL2.dll and used the Ags4 and made this simple test game: Ags4VideoTestBuilt.zip

It should play the entire video (I think), and it's now 60fps (the game, not the video). It's pretty cool that it works!

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

[Crimson Wizard]

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Video player must have a desync check itself in any case, comparing total length of audio and video output.

One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

Overlay should keep the size so long as that's the same sprite slot assigned.

[eri0o]

Ah, I did this minimal change and it worked when I ran in the web

Code: diff

diff --git a/Engine/ac/timer.cpp b/Engine/ac/timer.cpp
index df99bb140..263eed674 100644
--- a/Engine/ac/timer.cpp
+++ b/Engine/ac/timer.cpp
@@ -18,6 +18,7 @@
 #include "platform/base/agsplatformdriver.h"
 #if defined(AGS_DISABLE_THREADS)
 #include "media/audio/audio_core.h"
+#include "media/video/video_core.h"
 #endif
 #if AGS_PLATFORM_OS_EMSCRIPTEN
 #include "SDL.h"
@@ -71,6 +72,7 @@ void WaitForNextFrame()
     // Do the last polls on this frame, if necessary
 #if defined(AGS_DISABLE_THREADS)
     audio_core_entry_poll();
+    video_core_entry_poll();
 #endif

     const auto now = AGS_Clock::now();

I did a debug web build and pushed it here: ericoporto.github.io/public_html/video_test/

[Crimson Wizard]

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Alright, so this is happening simply because fps was stored as int, loosing precision (your video has a framerate of 29.97, which means it lost ~1 frame per second when converting). Pushed a fix for this, and now it plays fine to the end. Of course the sync mechanism should still be implemented later for safety.

[eri0o]

Played a bit more and it looks like the desync is indeed fixed!

Video Test Web with Distortion | ags4distfxvideo_project.zip

It appears I found an issue with how my distfx works horizontally that I need to fix...

This new video api is promising! I know it's a sketch but I mean playing videos assynchronously and assigning them to a sprite.

[eri0o]

Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

In any case, made a very rustic example of playing two videos at the same time, Ags4VideoTest2Videos.zip, it appears both stay in sync with audio, and the videos played fine in my computer.

Also made a test that runs a 1920x1080 video using this strategy on an Overlay to see if it works. It's here on google drive, if someone wants to run. In my computer it ran fine.

So far in terms of the feature it appears that it works. So it would be more about figuring the API.

[Crimson Wizard]

I did not have an internet connection for couple of days, but now I was able to push a proper script API. Here's how it looks like:

Code: ags

enum PlaybackState {
  ePlaybackOn = 2,
  ePlaybackPaused = 3,
  ePlaybackStopped = 4
};

builtin managed struct VideoPlayer {
  import static VideoPlayer* Open(const string filename, bool autoPlay=true, RepeatStyle=eOnce);
  /// Starts or resumes the playback.
  import void Play();
  /// Pauses the playback.
  import void Pause();
  /// Advances video by 1 frame, will pause video when called.
  import void NextFrame();
  /// Changes playback to continue from the specified frame.
  import void SeekFrame(int frame);
  /// Changes playback to continue from the specified position in milliseconds.
  import void SeekMs(int position);
  /// Stops the video completely.
  import void Stop();

  /// Gets current frame index.
  import readonly attribute int Frame;
  /// Gets total number of frames in this video.
  import readonly attribute int FrameCount;
  /// Gets this video's framerate (number of frames per second).
  import readonly attribute float FrameRate;
  /// Gets the number of sprite this video renders to.
  import readonly attribute int Graphic;
  /// The length of the currently playing video, in milliseconds.
  import readonly attribute int LengthMs;
  /// Gets/sets whether the video should loop.
  import attribute bool Looping;
  /// The current playback position, in milliseconds.
  import readonly attribute int PositionMs;
  /// The speed of playing (1.0 is default).
  import attribute float Speed;
  /// Gets the current playback state (playing, paused, etc).
  import readonly attribute PlaybackState State;
  /// The volume of this video's sound, from 0 to 100.
  import attribute int Volume;
};

Basic code example now is:

Code: ags

VideoPlayer *video = VideoPlayer.Open("video.ogv", true, eRepeat);
if (video)
{
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    over = null;
    video = null;
}

Most of the functionality works, except Seek, and few less important getters.
I found that APEG does not provide a Seek function, so either it has to be added there, or we should look for a different library for working with ogg theora decoder, or even write one ourselves. Maybe this should be a next task after this API is implemented.




On a side note, I have some problems with my PC, it looks like one of my hard-drives is dying, which causes all kinds of weird problems (i just did not figure out whether it's a system drive, or other one). So I may be disappearing sometimes.

Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

The basic idea is to count how much video and audio had played in time units, and when the difference reaches certain threshold, then either skip the data from one that is behind, or pause the one that got ahead.

[Crimson Wizard]

Ok, I reverted to SDL 2.28, and now the CI build seems to be working. If anyone is interested to try the new video commands out, here's a download link:

https://cirrus-ci.com/task/4992651611602944

[eri0o]

Created a new example using the new API here: VideoHdTest.zip

I simply get the Graphic of a looping video and throw it in a Room Overlay that has Z-Order that puts it in the background. Roger can walk on top of it.

Noticed a few things

• Frame Rate and Length are not implemented yet (they return 0 for me)
• It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

[Vincent]

Hello guys I was able to read the whole conversation just now!! I'm really happy that you guys managed to work out on this and from the tests I've tried from eri0o it seems like everything works great, really amazing work!!! As soon as I have some free time I'd like to try this version out and report something back, in the meantime still I want to thank you very much ❤️ both @Crimson Wizard and @eri0o

[eri0o]

Hey, CW did all the work, I am just playing around with it. 

[Crimson Wizard]

Frame Rate and Length are not implemented yet (they return 0 for me)

I've missed FrameRate property. But a situation around Length is not clear, as APEG implements it in a seemingly overcomplicated way, reading all video packets until the end. This is why it's disabled in settings. I must investigate if there are easier methods to get length from OGV theora at all. In the worst case we may say that not all formats return a valid value.

It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

So, about that. I implemented the FPS and sprite update similar to my original plan. This means that:
1. Video is played at its own FPS on a separate thread, changing frames whenever it's time.
2. There's a point of synchronization, once per game update (at Game Speed rate) where the game acquires whatever video frame is ready, and replaces the sprite's image, marking it as modified. If no new frame is ready, then it will keep the old image for a while more.
3. If the video's FPS is higher than the game's, then some frames may get replaced before game ever acquires them, causing visual skips.
4. If the video's FPS is lower than the game's, then same video frame may be displayed for more than 1 game update.

Higher video Speed requires it to load and prepare frames faster.
But also, if a low-FPS video begins to run with higher FPS, that means that the game receives a new frame not once in 2 or 3 updates, but once in 1 update. Naturally, that would increase rate at which a texture has to be updated.

For example, 24 FPS video and 60 FPS game means that game updates a sprite roughly once per 2.5 game updates.
If you increase playback speed to x2, making it run at 48 FPS, then game will have to update a sprite once per 1.25 game updates.

An additional note: if you don't need audio, then you may open video with "autoplay = false", and use NextFrame function to advance the frame at your wanted rate. NextFrame updates the sprite immediately, not waiting for sync during game update. This means you may also advance multiple frames in a loop. Something to keep in mind though, this is a synchronous function, and it will slow game down if you do multiple frames in one go.

PS. There still have to be ways to optimize the process, running full HD video seems bit slow atm.

[eri0o]

About optimization, I think we are not that far off of what is doable with Theora, VLC uses the same library as us from Xiph and it uses just slightly less CPU, but it's pretty similar if you throw an ogv video in it and in AGS, if you open the task manager and see how much CPU it's using.

Some newer codecs may perform better somehow if they do better use of cpu vectorization/intrinsincs and multithread in some way I guess.

I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

[Crimson Wizard]

I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

I think for AGS, quick possibilities are:
1. Create and buffer textures on the video thread, and return texture along with the bitmap when asked. This will put both sprite's bitmap and texture into the respective caches right away, and save time on converting sprite to texture during game update (and prebuffering will be saving more time overall).
2. Decode directly onto a texture. This will require to hack APEG though, because right now it allocates Allegro BITMAP itself. We'll have to provide either our own "pseudo" BITMAP which has its line pointers assigned to a locked texture buffer, or make APEG work with a raw buffer ptr instead. This will save on bitmap->texture conversion for Direct3D/OpenGL renderers, BUT, indeed, there has to be some mechanism for detecting that a sprite is backed up by a texture, so it should convert reverse if someone wants to raw draw.
There's an alternate approach, where VideoPlayer::Open may accept a flag telling to only create textures to speed things up.

EDIT:
3. Something that I missed, the video frame should be considered an opaque image, so ideally should use slightly faster BitmapToVideoMemOpaque conversion. Problem is that currently "opaque" is not a sprite's or texture's flag, but is applied on individual case basis when drawing stuff. So this may have to be redesigned first.

[Crimson Wizard]

Anyhow, the new Video API is now a part of AGS 4 Alpha:
https://www.adventuregamestudio.co.uk/forums/ags-engine-editor-releases/ags-4-0-early-alpha-for-public-test/msg636661708/#msg636661708