[SOLVED] Videos non-blocking playback with GUIs

Started by eri0o, Fri 03/02/2023 13:16:39

Previous topic - Next topic

Crimson Wizard

#20
They have the actual commit that sais "Fixed memory corruption when resampling S16 t F32" (and this is the case with the video):
https://github.com/libsdl-org/SDL/commit/384fcea585573db4e5707c249746008f25a0fbac

I noticed that when I build SDL, it comes with SSE2 functions, such as SDL_Convert_S16_to_F32_SSE2.
The fix was related to SDL_Convert_S16_to_F32_Scalar. As a theoretical possibility, maybe this function will run if one builds SDL2 without SSE2 support. Or maybe some other reason why it would fallback to the Scalar functions.

I wish there were a way to tell which conversion func is getting called at runtime.

Crimson Wizard

Well, I opened a ticket for them: https://github.com/libsdl-org/SDL/issues/9127

No idea yet whether this is because of the error in Scalar conversion functions that they already fixed, or it's just a coincidence.

eri0o

Hey, I went I used an older SDL2.dll and used the Ags4 and made this simple test game: Ags4VideoTestBuilt.zip

It should play the entire video (I think), and it's now 60fps (the game, not the video). It's pretty cool that it works!

I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

Crimson Wizard

#23
Quote from: eri0o on Sat 24/02/2024 00:56:02I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Video player must have a desync check itself in any case, comparing total length of audio and video output.

Quote from: eri0o on Sat 24/02/2024 00:56:02One curious thing though, I set the Overlay width and height once - the video is 640x360 but I resize down the overlay to 320x180. I thought I would had to keep resizing the overlays in rep_exec_always but it turns out it's keeping the size even though the graphic is updated. I am mentioning this because I think it was discussed at some point what happens when a new sprite slot is set to the overlay but I don't remember if this is intended or not when the sprite slot is kept but just it's contents are updated.

Overlay should keep the size so long as that's the same sprite slot assigned.

eri0o

Ah, I did this minimal change and it worked when I ran in the web

Code: diff
diff --git a/Engine/ac/timer.cpp b/Engine/ac/timer.cpp
index df99bb140..263eed674 100644
--- a/Engine/ac/timer.cpp
+++ b/Engine/ac/timer.cpp
@@ -18,6 +18,7 @@
 #include "platform/base/agsplatformdriver.h"
 #if defined(AGS_DISABLE_THREADS)
 #include "media/audio/audio_core.h"
+#include "media/video/video_core.h"
 #endif
 #if AGS_PLATFORM_OS_EMSCRIPTEN
 #include "SDL.h"
@@ -71,6 +72,7 @@ void WaitForNextFrame()
     // Do the last polls on this frame, if necessary
 #if defined(AGS_DISABLE_THREADS)
     audio_core_entry_poll();
+    video_core_entry_poll();
 #endif

     const auto now = AGS_Clock::now();

I did a debug web build and pushed it here: ericoporto.github.io/public_html/video_test/

Crimson Wizard

#25
Quote from: eri0o on Sat 24/02/2024 00:56:02I noticed that further in the video it becomes noticeable there is some audio and video desync. I don't know if this is an SDL problem though, maybe it's better to wait they solve your issue before looking into the desync...

Alright, so this is happening simply because fps was stored as int, loosing precision (your video has a framerate of 29.97, which means it lost ~1 frame per second when converting). Pushed a fix for this, and now it plays fine to the end. Of course the sync mechanism should still be implemented later for safety.

eri0o

Played a bit more and it looks like the desync is indeed fixed!

Video Test Web with Distortion | ags4distfxvideo_project.zip

It appears I found an issue with how my distfx works horizontally that I need to fix...

This new video api is promising! I know it's a sketch but I mean playing videos assynchronously and assigning them to a sprite.

eri0o

Quote from: Crimson Wizard on Sat 24/02/2024 05:24:21Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

In any case, made a very rustic example of playing two videos at the same time, Ags4VideoTest2Videos.zip, it appears both stay in sync with audio, and the videos played fine in my computer.

Also made a test that runs a 1920x1080 video using this strategy on an Overlay to see if it works. It's here on google drive, if someone wants to run. In my computer it ran fine.

So far in terms of the feature it appears that it works. So it would be more about figuring the API.

Crimson Wizard

I did not have an internet connection for couple of days, but now I was able to push a proper script API. Here's how it looks like:

Code: ags
enum PlaybackState {
  ePlaybackOn = 2,
  ePlaybackPaused = 3,
  ePlaybackStopped = 4
};

builtin managed struct VideoPlayer {
  import static VideoPlayer* Open(const string filename, bool autoPlay=true, RepeatStyle=eOnce);
  /// Starts or resumes the playback.
  import void Play();
  /// Pauses the playback.
  import void Pause();
  /// Advances video by 1 frame, will pause video when called.
  import void NextFrame();
  /// Changes playback to continue from the specified frame.
  import void SeekFrame(int frame);
  /// Changes playback to continue from the specified position in milliseconds.
  import void SeekMs(int position);
  /// Stops the video completely.
  import void Stop();

  /// Gets current frame index.
  import readonly attribute int Frame;
  /// Gets total number of frames in this video.
  import readonly attribute int FrameCount;
  /// Gets this video's framerate (number of frames per second).
  import readonly attribute float FrameRate;
  /// Gets the number of sprite this video renders to.
  import readonly attribute int Graphic;
  /// The length of the currently playing video, in milliseconds.
  import readonly attribute int LengthMs;
  /// Gets/sets whether the video should loop.
  import attribute bool Looping;
  /// The current playback position, in milliseconds.
  import readonly attribute int PositionMs;
  /// The speed of playing (1.0 is default).
  import attribute float Speed;
  /// Gets the current playback state (playing, paused, etc).
  import readonly attribute PlaybackState State;
  /// The volume of this video's sound, from 0 to 100.
  import attribute int Volume;
};

Basic code example now is:
Code: ags
VideoPlayer *video = VideoPlayer.Open("video.ogv", true, eRepeat);
if (video)
{
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    over = null;
    video = null;
}

Most of the functionality works, except Seek, and few less important getters.
I found that APEG does not provide a Seek function, so either it has to be added there, or we should look for a different library for working with ogg theora decoder, or even write one ourselves. Maybe this should be a next task after this API is implemented.




On a side note, I have some problems with my PC, it looks like one of my hard-drives is dying, which causes all kinds of weird problems (i just did not figure out whether it's a system drive, or other one). So I may be disappearing sometimes.




Quote from: eri0o on Mon 26/02/2024 20:23:09
Quote from: Crimson Wizard on Sat 24/02/2024 05:24:21Of course the sync mechanism should still be implemented later for safety.

I am curious what would this sync mechanism be, would this require one thread to talk with the other through some messaging?

The basic idea is to count how much video and audio had played in time units, and when the difference reaches certain threshold, then either skip the data from one that is behind, or pause the one that got ahead.

Crimson Wizard

Ok, I reverted to SDL 2.28, and now the CI build seems to be working. If anyone is interested to try the new video commands out, here's a download link:

https://cirrus-ci.com/task/4992651611602944

eri0o

Created a new example using the new API here: VideoHdTest.zip

I simply get the Graphic of a looping video and throw it in a Room Overlay that has Z-Order that puts it in the background. Roger can walk on top of it.

Noticed a few things

  • Frame Rate and Length are not implemented yet (they return 0 for me)
  • It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

Vincent

Hello guys I was able to read the whole conversation just now!! I'm really happy that you guys managed to work out on this and from the tests I've tried from eri0o it seems like everything works great, really amazing work!!! As soon as I have some free time I'd like to try this version out and report something back, in the meantime still I want to thank you very much ❤️ both @Crimson Wizard and @eri0o

eri0o

Hey, CW did all the work, I am just playing around with it.  :-D

Crimson Wizard

#33
Quote from: eri0o on Wed 28/02/2024 20:25:59Frame Rate and Length are not implemented yet (they return 0 for me)

I've missed FrameRate property. But a situation around Length is not clear, as APEG implements it in a seemingly overcomplicated way, reading all video packets until the end. This is why it's disabled in settings. I must investigate if there are easier methods to get length from OGV theora at all. In the worst case we may say that not all formats return a valid value.


Quote from: eri0o on Wed 28/02/2024 20:25:59It seems Speed affects the CPU usage (I haven't profiled, but I guess it's due to standard gfx BitmapToVideoMem functions)

I wonder, I have SetGameSpeed(60) at game_start, if a video updates a 24 FPS and I put it's graphic directly to the Overlay, will the Overlay update it's graphic (to texture) at 24 fps or 60 fps? (maybe things are more complicated than that too)

So, about that. I implemented the FPS and sprite update similar to my original plan. This means that:
1. Video is played at its own FPS on a separate thread, changing frames whenever it's time.
2. There's a point of synchronization, once per game update (at Game Speed rate) where the game acquires whatever video frame is ready, and replaces the sprite's image, marking it as modified. If no new frame is ready, then it will keep the old image for a while more.
3. If the video's FPS is higher than the game's, then some frames may get replaced before game ever acquires them, causing visual skips.
4. If the video's FPS is lower than the game's, then same video frame may be displayed for more than 1 game update.

Higher video Speed requires it to load and prepare frames faster.
But also, if a low-FPS video begins to run with higher FPS, that means that the game receives a new frame not once in 2 or 3 updates, but once in 1 update. Naturally, that would increase rate at which a texture has to be updated.

For example, 24 FPS video and 60 FPS game means that game updates a sprite roughly once per 2.5 game updates.
If you increase playback speed to x2, making it run at 48 FPS, then game will have to update a sprite once per 1.25 game updates.

An additional note: if you don't need audio, then you may open video with "autoplay = false", and use NextFrame function to advance the frame at your wanted rate. NextFrame updates the sprite immediately, not waiting for sync during game update. This means you may also advance multiple frames in a loop. Something to keep in mind though, this is a synchronous function, and it will slow game down if you do multiple frames in one go.

PS. There still have to be ways to optimize the process, running full HD video seems bit slow atm.

eri0o

About optimization, I think we are not that far off of what is doable with Theora, VLC uses the same library as us from Xiph and it uses just slightly less CPU, but it's pretty similar if you throw an ogv video in it and in AGS, if you open the task manager and see how much CPU it's using.

Some newer codecs may perform better somehow if they do better use of cpu vectorization/intrinsincs and multithread in some way I guess.

I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

Crimson Wizard

#35
Quote from: eri0o on Fri 01/03/2024 08:43:15I saw an example in SDL repository they called zero copy, if I understood it's when instead a hardware decoder is used in the GPU and the texture isn't back converted to a bitmap and pushed to a different texture. I guess in this case (hypothetical) the sprite slot would work as a reference to texture and fail dynamic sprites operations (which is fine) but still display in overlays? Just imagining if such thing existed.

I think for AGS, quick possibilities are:
1. Create and buffer textures on the video thread, and return texture along with the bitmap when asked. This will put both sprite's bitmap and texture into the respective caches right away, and save time on converting sprite to texture during game update (and prebuffering will be saving more time overall).
2. Decode directly onto a texture. This will require to hack APEG though, because right now it allocates Allegro BITMAP itself. We'll have to provide either our own "pseudo" BITMAP which has its line pointers assigned to a locked texture buffer, or make APEG work with a raw buffer ptr instead. This will save on bitmap->texture conversion for Direct3D/OpenGL renderers, BUT, indeed, there has to be some mechanism for detecting that a sprite is backed up by a texture, so it should convert reverse if someone wants to raw draw.
There's an alternate approach, where VideoPlayer::Open may accept a flag telling to only create textures to speed things up.

EDIT:
3. Something that I missed, the video frame should be considered an opaque image, so ideally should use slightly faster BitmapToVideoMemOpaque conversion. Problem is that currently "opaque" is not a sprite's or texture's flag, but is applied on individual case basis when drawing stuff. So this may have to be redesigned first.


Vincent

#37
Hello guys  @Crimson Wizard and @eri0o I got some free time to test this out and I would like to ask you some questions because I'm definitely missing something basic:

1) Althought I directly set the variable video.Looping = true; and opening the video with the repeat style set to eRepeat when the video ends it get black [EDIT] fixed by putting the code in rep_exe
Code: ags
  VideoPlayer *video = VideoPlayer.Open("VideoProva.ogv", true, eRepeat);
  video.Looping = true;
  if (video)
  {
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    //over = null;
    //video = null;
  }
ps: I see that if I remove the Wait(100) the video won't start which I think it make sense but how do I avoid having the wait so while the video is playing I can interact with some other things?

2) Is there a chance to play the video on a button graphic? Because now im playing the video on an overlay but the overlay is below the gui and I would like it to play the video in front of the gui, how can I do that?

Crimson Wizard

Hello

Quote from: Vincent on Fri 05/07/2024 09:40:16Althought I directly set the variable video.Looping = true; and opening the video with the repeat style set to eRepeat when the video ends it get black[/s]

There was a mistake in Video.Looping variable, it should be fixed in the latest ags 4 update:
https://www.adventuregamestudio.co.uk/forums/ags-engine-editor-releases/ags-4-0-early-alpha-for-public-test/msg636663912/#msg636663912

Quote from: Vincent on Fri 05/07/2024 09:40:16Is there a chance to play the video on a button graphic?

Video provides a Graphic property that you may assign to anything. In your code you are assigning it to overlay, but similarly you may assign it to the button:
Code: ags
MyButton.NormalGraphic = video.Graphic;

Quote from: Vincent on Fri 05/07/2024 09:40:16Because now im playing the video on an overlay but the overlay is below the gui and I would like it to play the video in front of the gui

This may also be done by changing Overlay.ZOrder to a higher value. Overlays are now sorted among GUIs on screen, so it's a matter of setting ZOrder higher than the topmost GUI.


Quote from: Vincent on Fri 05/07/2024 09:40:16
Code: ags
  VideoPlayer *video = VideoPlayer.Open("VideoProva.ogv", true, eRepeat);
  video.Looping = true;
  if (video)
  {
    Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
    Wait(100);
    //over = null;
    //video = null;
  }
ps: I see that if I remove the Wait(100) the video won't start which I think it make sense but how do I avoid having the wait so while the video is playing I can interact with some other things?

This is exactly same problem as with any other non-blocking actions. If you want to have a animation running while player interacts with other things, then you start it with eNoBlock and exit the function. Then check for animation completion in repeatedly_execute.

Same thing is true with the video. You declare VideoPlayer* variable outside of the function, assign new videoplayer to it in the function, and exit the function. Video will be playing while the rest of the game continues to run. Use this variable to check video's state in repeatedly_execute, and stop it when necessary.

Vincent

Thanks Crimson, as I said I was missing something really basic!!  :) It's been a while cause I was a bit off from Ags in the last period. I got fixed the thing to display the video on a button graphic, I was using the button.graphic instead of doing button.normalgraphic 🤦 as you mentioned, thanks a lot! Still, I am missing something basic to play the video in non block style, now im declaring the VideoPlayer* variable outside of the function but it still need the wait to be displayed, sorry in advance

Code: ags
VideoPlayer *video;
// called on every game cycle, except when the game is blocked
function repeatedly_execute()
{
    video = VideoPlayer.Open("VideoProva.ogv", true, eRepeat);
    if (video)
    {
      btnVideo.NormalGraphic = video.Graphic;
      //video.Play();
      //Overlay *over = Overlay.CreateGraphical(0, 0, video.Graphic);
      
      //Wait(100);
      //over = null;
      //video = null;
    }
}

SMF spam blocked by CleanTalk