[Feature Request] Videos non-blocking playback with GUIs

[Crimson Wizard]

They have the actual commit that sais "Fixed memory corruption when resampling S16 t F32" (and this is the case with the video):
https://github.com/libsdl-org/SDL/commit/384fcea585573db4e5707c249746008f25a0fbac

I noticed that when I build SDL, it comes with SSE2 functions, such as SDL_Convert_S16_to_F32_SSE2.
The fix was related to SDL_Convert_S16_to_F32_Scalar. As a theoretical possibility, maybe this function will run if one builds SDL2 without SSE2 support. Or maybe some other reason why it would fallback to the Scalar functions.

I wish there were a way to tell which conversion func is getting called at runtime.

[Crimson Wizard]

Well, I opened a ticket for them: https://github.com/libsdl-org/SDL/issues/9127

No idea yet whether this is because of the error in Scalar conversion functions that they already fixed, or it's just a coincidence.

[eri0o]

Hey, I went I used an older SDL2.dll and used the Ags4 and made this simple test game: Ags4VideoTestBuilt.zip

It should play the entire video (I think), and it's now 60fps (the game, not the video). It's pretty cool that it works!

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

[Crimson Wizard]

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Video player must have a desync check itself in any case, comparing total length of audio and video output.

One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

Overlay should keep the size so long as that's the same sprite slot assigned.

[eri0o]

Ah, I did this minimal change and it worked when I ran in the web

Code: diff

diff --git a/Engine/ac/timer.cpp b/Engine/ac/timer.cpp
index df99bb140..263eed674 100644
--- a/Engine/ac/timer.cpp
+++ b/Engine/ac/timer.cpp
@@ -18,6 +18,7 @@
 #include "platform/base/agsplatformdriver.h"
 #if defined(AGS_DISABLE_THREADS)
 #include "media/audio/audio_core.h"
+#include "media/video/video_core.h"
 #endif
 #if AGS_PLATFORM_OS_EMSCRIPTEN
 #include "SDL.h"
@@ -71,6 +72,7 @@ void WaitForNextFrame()
     // Do the last polls on this frame, if necessary
 #if defined(AGS_DISABLE_THREADS)
     audio_core_entry_poll();
+    video_core_entry_poll();
 #endif

     const auto now = AGS_Clock::now();

I did a debug web build and pushed it here: ericoporto.github.io/public_html/video_test/

[Crimson Wizard]

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Alright, so this is happening simply because fps was stored as int, loosing precision (your video has a framerate of 29.97, which means it lost ~1 frame per second when converting). Pushed a fix for this, and now it plays fine to the end. Of course the sync mechanism should still be implemented later for safety.

[eri0o]

Played a bit more and it looks like the desync is indeed fixed!

Video Test Web with Distortion | ags4distfxvideo_project.zip

It appears I found an issue with how my distfx works horizontally that I need to fix...

This new video api is promising! I know it's a sketch but I mean playing videos assynchronously and assigning them to a sprite.

[eri0o]

Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

In any case, made a very rustic example of playing two videos at the same time, Ags4VideoTest2Videos.zip, it appears both stay in sync with audio, and the videos played fine in my computer.

Also made a test that runs a 1920x1080 video using this strategy on an Overlay to see if it works. It's here on google drive, if someone wants to run. In my computer it ran fine.

So far in terms of the feature it appears that it works. So it would be more about figuring the API.

[Crimson Wizard]

I did not have an internet connection for couple of days, but now I was able to push a proper script API. Here's how it looks like:

Code: ags

enum PlaybackState {
  ePlaybackOn = 2,
  ePlaybackPaused = 3,
  ePlaybackStopped = 4
};

builtin managed struct VideoPlayer {
  import static VideoPlayer* Open(const string filename, bool autoPlay=true, RepeatStyle=eOnce);
  /// Starts or resumes the playback.
  import void Play();
  /// Pauses the playback.
  import void Pause();
  /// Advances video by 1 frame, will pause video when called.
  import void NextFrame();
  /// Changes playback to continue from the specified frame.
  import void SeekFrame(int frame);
  /// Changes playback to continue from the specified position in milliseconds.
  import void SeekMs(int position);
  /// Stops the video completely.
  import void Stop();

  /// Gets current frame index.
  import readonly attribute int Frame;
  /// Gets total number of frames in this video.
  import readonly attribute int FrameCount;
  /// Gets this video's framerate (number of frames per second).
  import readonly attribute float FrameRate;
  /// Gets the number of sprite this video renders to.
  import readonly attribute int Graphic;
  /// The length of the currently playing video, in milliseconds.
  import readonly attribute int LengthMs;
  /// Gets/sets whether the video should loop.
  import attribute bool Looping;
  /// The current playback position, in milliseconds.
  import readonly attribute int PositionMs;
  /// The speed of playing (1.0 is default).
  import attribute float Speed;
  /// Gets the current playback state (playing, paused, etc).
  import readonly attribute PlaybackState State;
  /// The volume of this video's sound, from 0 to 100.
  import attribute int Volume;
};

Basic code example now is:

Code: ags

VideoPlayer *video = VideoPlayer.Open("video.ogv", true, eRepeat);
if (video)
{
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    over = null;
    video = null;
}

Most of the functionality works, except Seek, and few less important getters.
I found that APEG does not provide a Seek function, so either it has to be added there, or we should look for a different library for working with ogg theora decoder, or even write one ourselves. Maybe this should be a next task after this API is implemented.




On a side note, I have some problems with my PC, it looks like one of my hard-drives is dying, which causes all kinds of weird problems (i just did not figure out whether it's a system drive, or other one). So I may be disappearing sometimes.

Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

The basic idea is to count how much video and audio had played in time units, and when the difference reaches certain threshold, then either skip the data from one that is behind, or pause the one that got ahead.

[Crimson Wizard]

Ok, I reverted to SDL 2.28, and now the CI build seems to be working. If anyone is interested to try the new video commands out, here's a download link:

https://cirrus-ci.com/task/4992651611602944

[eri0o]

Created a new example using the new API here: VideoHdTest.zip

I simply get the Graphic of a looping video and throw it in a Room Overlay that has Z-Order that puts it in the background. Roger can walk on top of it.

Noticed a few things

• Frame Rate and Length are not implemented yet (they return 0 for me)
• It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

[Vincent]

Hello guys I was able to read the whole conversation just now!! I'm really happy that you guys managed to work out on this and from the tests I've tried from eri0o it seems like everything works great, really amazing work!!! As soon as I have some free time I'd like to try this version out and report something back, in the meantime still I want to thank you very much ❤️ both @Crimson Wizard and @eri0o

[eri0o]

Hey, CW did all the work, I am just playing around with it. 

[Crimson Wizard]

Frame Rate and Length are not implemented yet (they return 0 for me)

I've missed FrameRate property. But a situation around Length is not clear, as APEG implements it in a seemingly overcomplicated way, reading all video packets until the end. This is why it's disabled in settings. I must investigate if there are easier methods to get length from OGV theora at all. In the worst case we may say that not all formats return a valid value.

It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

So, about that. I implemented the FPS and sprite update similar to my original plan. This means that:
1. Video is played at its own FPS on a separate thread, changing frames whenever it's time.
2. There's a point of synchronization, once per game update (at Game Speed rate) where the game acquires whatever video frame is ready, and replaces the sprite's image, marking it as modified. If no new frame is ready, then it will keep the old image for a while more.
3. If the video's FPS is higher than the game's, then some frames may get replaced before game ever acquires them, causing visual skips.
4. If the video's FPS is lower than the game's, then same video frame may be displayed for more than 1 game update.

Higher video Speed requires it to load and prepare frames faster.
But also, if a low-FPS video begins to run with higher FPS, that means that the game receives a new frame not once in 2 or 3 updates, but once in 1 update. Naturally, that would increase rate at which a texture has to be updated.

For example, 24 FPS video and 60 FPS game means that game updates a sprite roughly once per 2.5 game updates.
If you increase playback speed to x2, making it run at 48 FPS, then game will have to update a sprite once per 1.25 game updates.

An additional note: if you don't need audio, then you may open video with "autoplay = false", and use NextFrame function to advance the frame at your wanted rate. NextFrame updates the sprite immediately, not waiting for sync during game update. This means you may also advance multiple frames in a loop. Something to keep in mind though, this is a synchronous function, and it will slow game down if you do multiple frames in one go.

PS. There still have to be ways to optimize the process, running full HD video seems bit slow atm.

[eri0o]

About optimization, I think we are not that far off of what is doable with Theora, VLC uses the same library as us from Xiph and it uses just slightly less CPU, but it's pretty similar if you throw an ogv video in it and in AGS, if you open the task manager and see how much CPU it's using.

Some newer codecs may perform better somehow if they do better use of cpu vectorization/intrinsincs and multithread in some way I guess.

I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

[Crimson Wizard]

I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

I think for AGS, quick possibilities are:
1. Create and buffer textures on the video thread, and return texture along with the bitmap when asked. This will put both sprite's bitmap and texture into the respective caches right away, and save time on converting sprite to texture during game update (and prebuffering will be saving more time overall).
2. Decode directly onto a texture. This will require to hack APEG though, because right now it allocates Allegro BITMAP itself. We'll have to provide either our own "pseudo" BITMAP which has its line pointers assigned to a locked texture buffer, or make APEG work with a raw buffer ptr instead. This will save on bitmap->texture conversion for Direct3D/OpenGL renderers, BUT, indeed, there has to be some mechanism for detecting that a sprite is backed up by a texture, so it should convert reverse if someone wants to raw draw.
There's an alternate approach, where VideoPlayer::Open may accept a flag telling to only create textures to speed things up.

EDIT:
3. Something that I missed, the video frame should be considered an opaque image, so ideally should use slightly faster BitmapToVideoMemOpaque conversion. Problem is that currently "opaque" is not a sprite's or texture's flag, but is applied on individual case basis when drawing stuff. So this may have to be redesigned first.

[Crimson Wizard]

Anyhow, the new Video API is now a part of AGS 4 Alpha:
https://www.adventuregamestudio.co.uk/forums/ags-engine-editor-releases/ags-4-0-early-alpha-for-public-test/msg636661708/#msg636661708