Plugin: ags_theora Video Player

[scotch]

I made this for background animation in my own game, thought I'd package it up for release... hope it's useful for someone. Essentially it's a video playing plugin, with an internal ogg/theora decoder. Theora is a free video codec targetted to compete with well known MPEG4 codecs (xvid, divx etc). Quality isn't quite there yet but close enough, and the encoder is still being improved.

Generally be careful with it because as I said, it's for my own use and may not work in unusual situations. For a start it only supports videos encoded in YUV420 which happens to be most of them. Also seeking doesn't align to keyframes so there will be momentary tearing when seeking to a non keyframe, tell me if you need that fixed. Doesn't interpolate chroma though I suppose it should. If anyone actually uses it they can tell me what they want adjusted for their purposes.

plugin download / demonstration download (try it in d3d mode to see the rotating etc if you care)

Interface

Code: ags

/* values specifying the "layer" of the scene to draw the video over */
enum RenderStage {
    RsBackground,
    RsScene,
    RsUI,
    RsOverlay
};

enum RelativeTo {
    RtRoom,
    RtScreen,
    RtScreenPixels
}

/* Each Theora instance represents an open video file. */
struct Theora {
    // opening
    static Theora* open(string name);
    
    // positioning
    void NextFrame();       // move to next frame
    void Sync(float time);  // advance 0 or more frames to sync with time
    void Seek(float time);  // attempt to seek to a position
    
    // status
    readonly bool broken;   // the video is bad and can no longer render properly
    readonly bool ended;    // end of video was hit, no more new frames will appear unless looped
    readonly float time;    // current time of video (may be inaccurate)
    readonly float fps;     // fps of encoded video, you don't have to stick to it if you don't want to
    readonly int width;
    readonly int height;
    bool loop;              // enable/disable automatic looping

    // rendering
    void Draw(RenderStage stage, int left, int top, RelativeTo);
    void DrawEx(RenderStage stage, float cx, float cy, float xscale, float yscale, float rotation, RelativeTo); // scale/rotation can only be used if d3d renderer is being used!
}

Advantages of this over regular video playing are
. doesn't require user to have installed any codecs, while providing better compression than available by default
. completely free format for free and commercial use
. draw videos within your scene and onto sprites for special effects
. (hopefully) no black frames or other signs of switching between game and video

Disadvantages
. is a plugin, therefore only works on windows engine right now
. decoding is fairly slow, YUV->RGB conversion needs to be done in software which is an additional overhead, not sure what the minimum specs for a full screen 640x480 30fps video would be, but somewhat higher than the regular video player. However, I did have someone test the demo on an ASUS EEE PC with 900mhz processor and it seemed fine.
. does not do audio. AGS already does audio, if you want you can sync up a sound/music file with the video with script. The main purpose of the plugin is background animation which does not require audio.
. requires some scripting to get it going, videos don't play on their own (although I could make them if people really want)
. only supports 32 bit mode at present
. software mode doesn't scale videos, therefore playing your game at the wrong resolution (scale filters are fine) will look bad. I don't think AGS should have this option on the setup though, personally.
. one unpredictable lockup issue which occurs from time to time when exiting in d3d mode, which I will fix some time, but it is fairly rare for me.

Details

It's quite basic as it is, I only need it for background animations, but if anyone uses it and would like features go ahead and suggest them. Support for 16 bit mode and video file obfuscation might be worth doing.

Here's the room script from the demo game, as an example of how to play a video on the background (and how to scale and rotate one in D3D mode.

Code: ags

Theora* video;
float rot = 0.0;

function repeatedly_execute_always() {
    if(video) {
      video.DrawEx(DsBackground, 160.0, 120.0, 0.5, 0.5, 0.0, RtScreen);
      if(System.HardwareAcceleration) video.DrawEx(DsScene, 260.0, 120.0, 0.25, 0.25, rot, RtScreen);
      video.NextFrame();
      rot += 0.2;
    }
}

function room_Load() {
  video = Theora.Open("theora.ogg");
  video.loop = true;  
  SetGameSpeed(24); // you don't need to use the same fps as the video like this, but it's easiest this way
}

I should also say, if you have a video open and go to another room, that video is still open! When you no longer need a video set the pointer to null. You can do this on exit, and load it on enter if you want. Just in case someone loads a video background in every room and wonders why it's taking 100s of MB of RAM.

[Pumaman]

Looks great scotch, excellent work!!

In D3D mode I do get the mini video overlay, but it doesn't seem to rotate?

[scotch]

Oh yes, I appear to have broken that when I removed another bit from the room script before release, I'll update it. And updating it I see I broke another rotation related thing so I'll have to update it again later, anyway... rotation probably isn't likely to get used much, glad it works in general.

[LimpingFish]

Wow! Excellent work indeed, Scotch! Very smooth playback. Is there a minimum and maximum bitrate/fps setting for the video, or does the plugin only work with video encoded to a set bitrate/fps?

[scotch]

Any bitrate or fps is fine (if you replace theora.ogg in the example it will hopefully work), you don't need to lock your game speed to the video speed as I did there, that's just the most convienient way. If your videos are different speed you could do something like: "video.Sync(gameFramesPassed * (1.0 / IntToFloat(GetGameSpeed())));" instead of using NextFrame and it'd run at the right rate whatever your game speed is, just not updating every frame. Hopefully.

[GarageGothic]

Very cool. This looks like it will solve a lot of problems for people working with FMV or prerendered cutscenes. I've seen several people in the tech forums having problems with black flashes when starting a video file using the internal video player, but rendering straight to background should solve that. I'm also quite impressed with the amount of animation you can store at that quality in a 6MB video file.

The best thing would of course be full integration with AGS, so it wouldn't need Windows to work. But I wonder, would it without further integration be possible to store all videos in a container format like speech.vox, so that players couldn't open the files outside the game? (On the other hand, I always did feel like a 1ee7 hax0r when I found ways to open external cutscene files from commercial games using smacker or deluxe paint).

[scotch]

Well like I said, obfuscating the videos is an option if someone cares enough, it's slightly easier to work with plain files but I'm willing to put in the work if someone is making a good video heavy game. An ogg theora file renamed to .dat is fairly obscure as it is, but yeah it'll still play in VLC and I can understand wanting to hide them better.

The video there is actually lower quality than you might be able to squeeze out at the same bitrate because it's cropped then transcoded from another medium bitrate video.

I also think this kind of thing should end up in the core but for now a plugin isn't too bad, considering the state of the other ports, and how video doesn't work on them anyway.

[deadsuperhero]

Alright! Free video codecs supported by AGS! It'll make porting the games (someday) easier, and it won't be legally encumbered!

[deltamatrix]

Good job. But I still think AGS ought to bring native support for this codec. Native support for at least one open source codec is good for the Linux and Mac OS X users who can't play videos in AGS.

[scotch]

I agree, but I also think the state of the ports makes those kinds of arguments unconvincing for now, unfortunately. If theora support went into the engine how many years until a new Mac/Linux engine appears, if it ever does? Unless the engine is a collaborative project maintaining ports seems rather difficult. Mac/Linux users seem better off on emulators.

[Ali]

This looks very impressive (as does the first glimpse I've had of big buck bunny)!

Great work, this will really help broaden the kinds of games we can make.

[AJA]

Great job, scotch! This'll be really useful.

Found a bug, though: If there's a theora video playing in windows (I use ffdshow for decoding, if that matters) and video playback starts in the demo game, both the game and the video player crash.

[scotch]

That sounds quite odd, the plugin doesn't interact with any other codecs. I can't replicate it myself.

[AJA]

Okay, that's weird. Now it seems to be working just fine. Never mind.

[Dusk]

*** EDIT ***
I managed to find a workaround introducing another room, so feel free to ignore the following

***********

What can cause the video to be considered "broken"?

I've prepared a "video playing" room for some cutscenes... the idea is to set a String variable (in a struct with other things, but that's not important here) to the video to load, then move the player to this special room and let the movie play. When the movie ends, I want another changeRoom...

Of course my target are full screen video cutscenes... but I'm having problems.
If I start the game from the "movie playing room", the video playing works smoothly and the room change at the end works.

If I start the game from another room, and then perform a player.ChangeRoom to the video playing room, the game crashes. So I added a "broken" check, and the video is reported as broken.

Why?
What could be interfering?

Thanks... and BTW, great plugin here, I'm one of the "Yeah we want Theora support in AGS" guys 

Code: ags

// video playing room script file

Theora* video;

function room_Load() {
  videoCut.played = false;
  video = Theora.Open(videoCut.fileName); 
  video.loop = false; 
}

function repeatedly_execute_always() {
  if(video) {
     if (video.broken) AbortGame("VIDEO BROKEN!");
     if (!video.ended) {      
        video.DrawEx(DsOverlay, 160.0, 120.0, 0.5, 0.5, 0.0, RtScreen);
        video.NextFrame();
      } else {
        videoCut.played = true;        
      }
  }
}

function room_RepExec() {
  if (videoCut.played) player.ChangeRoom(203);
}

[scotch]

I don't know what's causing the problem with that setup but I will try it some time, glad you found a workaround for now.

[visionastral]

Hello Scotch, excuse my english please 

I found an issue with the theora plugin:
When you make a video background and run with Direct3D, Flipped sprites don't render!
Normal sprites are ok, it's just flipped sprites that disapear.
But it only happens when you run AGS in Direct3D.
It's weird, but it happens with everything you flip.
It's annoying because I'm coding the dialog system of my game and I was making big close-ups of the characters with huge sprites, fliping depending on the players position, but when you flip the sprite, it only disapears!

Another thing:

PLEASE PLEASE PLEASE allow video obfuscation!!!

I DO CARE A LOT!
Let me explain:
I'm making a game with plenty of full motion video, not only for intros and the like. For backgrounds, scene animations, quicktime events...
So, if I cannot obfuscate my video, every one could see the end of the game without playing it!
Of course, I know what some one could say: "you just have to change the extension bla bla bla"
No no no, if you just drop the file in windows media player, it will say "the file haven't the good extension" but it will play it!

Another thing wich would be realy great is sound support. Specialy the ability to mix the sound of the theora video with the background sound of AGS.
This way you don't loose continuity introducing video sequences because you don't break the background music.
However, this isn't so important for me than video obfuscation, because I'm already doing this by myself inside AGS along with a module wich load theora videos on demand and take care of playing them with their respective delays.

Another thing that makes me wonder:
How about a DESTROY method to explicitly clear memory from videos wich you won't run anymore??
I know about all the good things about "automatic garbage collector" but I'm from the old school, I would rather want to free memory by myself, just to be sure there isn't memory leaks  :

Of course, when the theora module will be finished, I'll upload it.
Right now it takes care of loading and managing up to 100 videos, allowing the user to load videos with only 1 function call and the module takes care of playing them at good delay.

However, I'm having troubles with the bottom of the video:
Imagine you have a background and don't wants to make a video of ALL the screen, but just the parts wich move. This will give you the ability to make multiple zones of animations at diferent playback rates, wich is much more interesting than having only one ciclic loop over the entire screen.
The module I'm making started with this goal.
However, the bottom of the videos don't render seamlessly, it makes me a green rectangle at the bottom of the video from left to right with variable height from video to video.
I suppose this is due to enconding chunks of the compression, that's why I made several trys making the video power of 2,4,8 and 16; but the green rectangle still appears at the bottom, in the best case, it becomes just a line, but still.

The main goal is to integrate little videos into the background seamlessly. It was a problem with the encoder changing gamma and contrast, but after spending hours trying, I finally found the correct setings to work around this, so the last issue is this green rectangle at the bottom of the videos.
After that, the module should be very useful for everyone.

Another thing that is a little problematic is the walk-behind zones being rendered over the videos.
I know it's because of AGS's design, but I wonder if something could be done to draw the videos directly to the buffer AGS take to draw the walk-behind zones of.


EDIT: I'm using the AGS beta release right now, perhaps something changed that made the flipped sprites to do not render over the videos, I have allready told Chris Jones about that, but, since it's a problem with your plugin, I thought I should told you about it.

[scotch]

Flipped sprites: I'll sort that out.
Obfuscation: Still not convinced it matters, but I might do it.
Sound: I don't have any intention of doing that because I have no idea how to do it reliably while AGS has the sound device.
Memory leaks: Set the pointer to null and it will be destroyed.
Walkbehinds: I suppose I could make it work... I can't see it getting used much but maybe I'll do it.
Green rectangle: I haven't seen this happen.

[naltimari]

PLEASE PLEASE PLEASE allow video obfuscation!!!

That's not just a matter of a video plugin, in my point of view. But in case you want to try it, I came across something that should allow you to do that.

It's a project called Pismo File Mount. I quote the site:

"Encrypted and/or compressed folders can be created that exist as simple files while locked, but become file system folders when unlocked. Contents can be viewed, edited, and executed, without copying to unprotected storage."

Among other things.

[visionastral]

Wow, that was brief but acurate  
Thank you very much Scotch.  

Naltimari, I'll check this out right now, thank you!

(how friendly the people in this forum is  )