PLUGIN: Snow/Rain 2.02

Started by Scorpiorus, Mon 03/02/2003 18:43:02

Previous topic - Next topic

Scorpiorus

The plugin basically starts from the engine API instance and browses through engine internals, like objects' vtables and code, to try and find all the required functions and data needed to perform drawing.
It can stand minor code changes or EXE-image re-basing (like ASLR - Address Space Layout Randomization) to a certain degree but of course will eventually fail as more and more code changes are made.
Again, its main purpose was to support previous set of AGS versions so that I wouldn't have to hardcode it for every single version of the engine that has been released.
My original idea was to only unlock plugin support for each new AGS version after I've thoroughly tested that plugin works with it correctly, and use standard AGS engine API Blit-family functions as a fallback (meaning nothing will actually be drawn until it's fixed).

So, while technically it is possible to keep the plugin working with even newer versions of AGS and still using such tricks, I'd better switch to something officially supported as soon as possible, and I'm really glad you are considering fixing Blit functions for 3.4.1 -- and to be honest that's what I was hopping for, since, yeah, changing graphic modes at run-time is a significant change and I'm sharing your concerns regarding the need to notify plugins somehow (as they may have their own resources allocated separately from the AGS graphics lists/caches, etc).


Oh, and I always wanted to say that before...

Crimson Wizard, I'd like to thank you for all the hard work, time and effort you dedicate to support AGS Community! And of course all other AGS developers contributing to it to keep AGS a wonderful game creation system! You've been doing a great work! Thanks!

Crimson Wizard

#121
Quote from: Scorpiorus on Sun 23/04/2017 19:59:28
The plugin basically starts from the engine API instance and browses through engine internals, like objects' vtables and code, to try and find all the required functions and data needed to perform drawing.

I found that when the plugin's onEvent callback is called for rendering stage, the first parameter is the event type, and the second parameter is a pointer to IDirect3DDevice9 interface.

Unfortunately, the callback argument is 32-bit integer, so that won't work if engine is compiled as 64-bit program.

EDIT: regarding the solution I am doing. I researched this a bit more, and plugin callbacks are already structured better than I initially thought.
When AGS creates a drawing list it pushes special items there, which indicate a plugin event. When it does all the rendering, and meets such item, it calls plugin callback instead.

Therefore, there seem to be no need to keep and manage bitmaps for every possible stage, but only one bitmap of a size of virtual screen. Interestingly, D3D and OpenGL renderers also have a dummy bitmap called "virtual screen", but I could not find yet what is it for (probably to make sure game don't crash).

Anyway, I am now trying to let plugins draw upon this "virtual screen" bitmap even if that's Direct3D running. Then, after callback, I am creating D3D texture from this bitmap and render it right away.
I've got results, but messy results, because apparently there is no direct way to know whether plugin has drawn anything on it. I guess we could try to set a flag whenever one of the drawing API functions is called (like GetVirtualScreen, BlitSpriteTranslucent, and so on).

As for the speed, it seems okay. I do not think this will be slower than software drawing anyway.

Scorpiorus

Quote from: Crimson Wizard on Tue 02/05/2017 15:44:38
I found that when the plugin's onEvent callback is called for rendering stage, the first parameter is the event type, and the second parameter is a pointer to IDirect3DDevice9 interface. Unfortunately, the callback argument is 32-bit integer, so that won't work if engine is compiled as 64-bit program.
And another practical issue with using the IDirect3DDevice9 interface directly by plugins is that plugins then may have to create their own texture resources. And since AGS introduces an abstract concept/layer of DriverDependantBitmap where with Direct3D9, for instance, it may actually be represented by several tiled textures of specific format (current internal implementation and hence is always a subject to change), plugins should carefully manage their own textures to be consistent with the engine's strategy and method of drawing. At the very least repeating engine's work on checking video hardware capabilities (like texture size, format, etc...) and, probably, even applying similar workarounds if necessary.
Otherwise, a game may fail to run on certain video hardware because of some Direct3D plugin, but would work perfectly in Direct3D mode without that plugin.
Also, things like AGS FlipScreen()-enabled rendering etc... may be a problem to support at such a low level.

QuoteI researched this a bit more, and plugin callbacks are already structured better than I initially thought.
When AGS creates a drawing list it pushes special items there, which indicate a plugin event. When it does all the rendering, and meets such item, it calls plugin callback instead.
Yeah, and I actually abused that NullSpriteCallback entry from within the plugin as a point from which to make a bunch of extra calls into private D3DGraphicsDriver::_renderSprite() to draw particles.

QuoteI've got results, but messy results, because apparently there is no direct way to know whether plugin has drawn anything on it. I guess we could try to set a flag whenever one of the drawing API functions is called (like GetVirtualScreen, BlitSpriteTranslucent, and so on).

As for the speed, it seems okay. I do not think this will be slower than software drawing anyway.
Would be really great if the speed is acceptable, and yeah some sort of a flag to optimize things seems like a good idea.

Crimson Wizard

#123
So, this is a build I was testing with your snowy/rainy room game: http://www.mediafire.com/file/hu8qntvzm0ofzod/acwin--d3dplugindrawing.zip
Branch: https://github.com/ivan-mogilko/ags-refactoring/tree/improv--d3dpluginblit

Traditionally, two news, good and bad one:
1) It works.
2) Software renderer draws everything to the same bitmap, composing a "flattened" image, therefore translucent sprites are blending with actual scene behind them.
But D3D has to pass separate bitmap to plugin. I could not figure what to do with that fact yet.
If I simply zero the bitmap, it will be opaque black with correct snow/rain on top, but room cannot be seen.
If I mark it as having alpha layer, room is seen, but rain will be completely invisible too, because plugin draws 16-bit sprites without alpha (I guess?).
So for now, to at least see the effect, I was filling bitmap with magic pink. This makes it transparent to see the room behind it, but caused those translucent sprites to blend with pink...

Any ideas?

Scorpiorus

Hmm... It seems like by painting sprites onto that transparent intermediate helper bitmap we would lose translucency levels of sprites (trans param) specified when calling several IAGSEngine::BlitSpriteTranslucent() for a single screen frame anyway.
And to save translucency info for each painted sprite (fur further blitting by GPU) we'd have to embed translucency (on a per-pixel-basis) into that intermediate bitmap data as alpha values?...
...because getting an alpha channel info from the original sprites won't help much, since it's a static translucency but IAGSEngine::BlitSpriteTranslucent() can be used to draw the same sprite multiple times with different trans values?

I'd like to look into it but I don't have the AGS Engine 3.4.1 build ready at the moment (simple build command doesn't work anymore since the external libs were separated from the AGS branch and I'd have to add them back to build the engine). I'll see if I can do it.

Crimson Wizard

#125
Quote from: Scorpiorus on Sat 06/05/2017 19:36:33
Hmm... It seems like by painting sprites onto that transparent intermediate helper bitmap we would lose translucency levels of sprites (trans param) specified when calling several IAGSEngine::BlitSpriteTranslucent() for a single screen frame anyway.
Not sure if I understand what you are saying correctly. To elaborate, multiple BlitSpriteTranslucent calls done during same event callback will draw upon same bitmap. But that bitmap will be cleared before each next callback.

I was thinking that maybe plugin could detect bitmap format, and draw alpha sprites if its 32-bit (or detect hardware-accelerated renderer, if there is such method). On engine side it will just mark this surface as having alpha layer, which would blend over rest of the room correctly.
Could not it be solved that way?

Quote from: Scorpiorus on Sat 06/05/2017 19:36:33
I'd like to look into it but I don't have the AGS Engine 3.4.1 build ready at the moment (simple build command doesn't work anymore since the external libs were separated from the AGS branch and I'd have to add them back to build the engine). I'll see if I can do it.
I have the collection of prebuilt libraries, if that may help:
https://www.dropbox.com/s/4p6nw6waqwat6co/ags-prebuilt-libs.zip?dl=0

Scorpiorus

Quote from: Crimson Wizard on Sat 06/05/2017 19:57:12Not sure if I understand what you are saying correctly. To elaborate, multiple BlitSpriteTranslucent calls done during same event callback will draw upon same bitmap. But that bitmap will be cleared before each next callback.
Yeah, what I mean is that since we don't have the access to the current "background" screen image at the moment of calling into BlitSpriteTranslucent() and hence can't blend sprites with the "background" immediately, we should instead prepare and pass translucency info for the GPU to blend the entire intermediate bitmap later (when the control returns to the engine from the plugin draw event).
And because it's a single flattened bitmap holding all the drawn sprites on it (with *different*, still virtual, trans levels), the only possible way(?) to mark translucency for each of the bitmap's area where a sprite is located seems to be adjusting the alpha channel data of the entire bitmap accordingly?

QuoteI was thinking that maybe plugin could detect bitmap format, and draw alpha sprites if its 32-bit (or detect hardware-accelerated renderer, if there is such method). On engine side it will just mark this surface as having alpha layer, which would blend over rest of the room correctly.
Yep, but then it would use a predefined static translucency from the sprite. Where BlitSpriteTranslucent() allows to specify a different trans factor for each of its call on the same sprite.

QuoteI have the collection of prebuilt libraries, if that may help:
https://www.dropbox.com/s/4p6nw6waqwat6co/ags-prebuilt-libs.zip?dl=0
Should save me quite some time! Thanks a bunch!

Scorpiorus

Ok, I did some research on this and it seems that preparing a 32-bit alpha-translucent overlay bitmap (_stageVirtualScreen) is not as simple as I thought at first.
The main issue is that, indeed, unlike with Allegro/DX5 blending, we don't have the background scene image and because of that cannot do blending in an incremental additive way (i.e. starting from the background).
And if multiple overlapping sprites are blended with the overlay bitmap (_stageVirtualScreen), it is not clear how (if at all practically possible) to do it correctly.
It is possible to accumulate their blending with each other, but I am not quite sure how difficult (including performance concerns) it would be to calculate _stageVirtualScreen's alpha values to represent the "final" alpha-translucency that would consider trans contribution of each overlapping sprite, because:

B - background scene image color, [0..255]
S1,S2,S3 - colors of 3 overlapping sprites/pixels, [0..255]
a1,a2,a3 - alpha values of 3 overlapping sprites/pixels, [0..255]
A1 is a shortcut for (255 - a1)
A2 is a shortcut for (255 - a2)
A3 is a shortcut for (255 - a3)
Res - result scene color, [0..255]
ALPHA - what we'd need to write into _stageVirtualScreen

in a classic way it would be:

Res = (((B*A1 + S1*a1)/255 * A2 + S2*a2)/255 * A3 + S3*a3)/255;

but since 'B' is unknown, we'd have transformed it into:

Res = B*(A1*A2*A3)/(255^3) + SomeKnownColor(S1,S2,S3,a1,a2,a3);

Next (A1*A2*A3)/(255^3) is (255-ALPHA)/255 (aka background scene opacity factor) that the GPU would calculate from ALPHA, to blend _stageVirtualScreen with the background scene;

So, ALPHA = 255 - (A1*A2*A3) / (255^2), is what should go into _stageVirtualScreen alpha component... it seems :p

And then in general:
ALPHA = 255 - (A1*A2*A3*...*An) / (255^[n-1]), where n - total number of overlapping sprites for the current pixel.

I'm not sure how to do it in a single pass... And if it's at all the correct way of doing this...


And here is a simple custom Allegro blend function that DOESN'T follow anything from above, so it won't work correctly with overlapping translucent sprites:
Code: CPP

unsigned long custom_blender_func_b32( unsigned long x, unsigned long y, unsigned long n )
{
    // From Allegro 4.4.2 manual:
    // x - blending modifier color;
    // y - base color to be modified;
    // n - (aka 'a') interpolation factor is in the range [0-255] and controls the solidity of the blending;

    // From Allegro 4.4.2 manual:
    // When a translucent drawing function is used (ex: draw_trans_sprite),
    // x is the color of the source,
    // y is the color of the bitmap being drawn onto,
    // n is the alpha level that was passed to the function that sets the blending mode
    // (the RGB triplet that was passed to that function is not taken into account).
    
    // TODO: revise implementation! Check Endianness!
    // using 'n' as a temporary to construct a new background pixel color
    n = n << 24;               // move Alpha to highest byte
    n = n | (x & 0x00FFFFFF);  // get sprite pixel color but don't use its Alpha component if any
    return n;                  // return new background pixel color
}

unsigned long custom_blender_func_STUB( unsigned long x, unsigned long y, unsigned long n )
{
    //return y;
    return 0xFFFFFFFF;
}


And adjusted IAGSEngine::BlitSpriteTranslucent():
Code: CPP

void IAGSEngine::BlitSpriteTranslucent(int32 x, int32 y, BITMAP *bmp, int32 trans)
{
    Bitmap *ds = gfxDriver->GetMemoryBackBuffer();
    
    // Hardware-accelerated graphics driver:
    if (HARDWARE_ACCELERATED_GRAPHICS_DRIVER && ALL_BITMAPS_ARE_32BIT && _stageVirtualScreen_and_its_HW_BITMAP_SUPPORT_ALPHA_CHANNEL)
    //if (bitmap_color_depth(ds->GetAllegroBitmap())==32)
    {
        set_blender_mode_ex(
            custom_blender_func_STUB,  // 15-bit (same color depth)
            custom_blender_func_STUB,  // 16-bit (same color depth)
            custom_blender_func_STUB,  // 24-bit (same color depth)
            custom_blender_func_b32,   // 32-bit (same color depth)
            custom_blender_func_STUB,  // 15-bit (different color depth)
            custom_blender_func_STUB,  // 16-bit (different color depth)
            custom_blender_func_STUB,  // 24-bit (different color depth)
            0, 0, 0, trans);
        
        draw_trans_sprite(ds->GetAllegroBitmap(), bmp, x, y); // TODO: use other blit func?
    }
    else // Allegro/DX5 graphics driver:
    {
        set_trans_blender(0, 0, 0, trans);
        // FIXME: call corresponding Graphics Blit
        draw_trans_sprite(ds->GetAllegroBitmap(), bmp, x, y);
    }
}


So, the above code assumes that VideoMemoryGraphicsDriver::DoNullSpriteCallback() calls CreateDDBFromBitmap() and UpdateDDBFromBitmap() with hasAlpha = true.

Unfortunately, even such simple blend implementation is much slower compared to the Allegro/DX5 AGS graphics driver.
Ah well... that's what we get for not following GPU's pipeline. But still, as a compatibility solution it's not that bad.

I'll see if I can come up with something else regarding the issue...

Crimson Wizard

#128
I am afraid I cannot follow everything you say right now. But I can add a small note regarding allegro alpha blending. That is true that Allegro functions discard source alpha, and that is why AGS uses its own alpha blending functions. CJ started implementing them, and I improved this to introduce true alpha blending for GUIs and DrawingSurface.

You may find this function in blender.h/cpp files, it is called "_argb2argb_blender".

Ofc. it IS slower than Allegro's blender, also because it does more job.

Alternative variant to what we are currently doing is, perhaps inserting a new drawing entry every time something like BlitSpriteTranslucent is called (instead of flattening it to one big bitmap) and let driver do cumulative blending as it see fit.

But if we are to go that way, I'd rather introduce explicit plugin API for actual hardware-accelerated render support. Because there are still API functions that are practically impossible to simulate, like Get/SetVirtualScreen.

Scorpiorus

To be honest, I wanted to suggest implementing plugin API blit functions compatibility mode to use driver's rendering functions instead. Because emulating flattened sprites with different translucencies by hacking them into _stageVirtualScreen's alpha channel seems too involved.

A specialized Engine API to manage drawing in a new way is a good idea too, since you removed the hard limit on the max number of drawing list entries in 3.4.1, but it would still be neat to support the original drawing functions such as DrawText(), DrawTextWrapped(), BlitBitmap(), BlitSpriteTranslucent() and BlitSpriteRotated() to work in hardware accelerated mode.
Maybe it's worth re-stating them to support only drawing to screen and if plugin tries to redirect drawing with SetVirtualScreen() just abort with error.

And even further, I don't know if there are any plugins working with allegro bitmaps in some specific way, but personally I always treated BITMAP* just like a pointer to pass around.
Doh! It even happened to me where I didn't notice to have it declared as Windows BITMAP* or something, and it worked and compiled since I didn't care of its internals but was passing pointer values :)

Crimson Wizard

#130
Quote from: Scorpiorus on Tue 09/05/2017 01:00:36
To be honest, I wanted to suggest implementing plugin API blit functions compatibility mode to use driver's rendering functions instead. Because emulating flattened sprites with different translucencies by hacking them into _stageVirtualScreen's alpha channel seems too involved.
Can you elaborate on how to do this?
OTOH you could even do pull request to my experiment branch, if you can figure how to make that work.

E: Or, do you mean same thing as I suggested as alternative?
Quote
Alternative variant to what we are currently doing is, perhaps inserting a new drawing entry every time something like BlitSpriteTranslucent is called (instead of flattening it to one big bitmap) and let driver do cumulative blending as it see fit.

Scorpiorus

Quote from: Crimson Wizard on Tue 09/05/2017 01:24:22E: Or, do you mean same thing as I suggested as alternative?
Ah, yeah. Probably adding/inserting new entries to draw may be too late since we are in the rendering phase already but calling _renderSprite() just-in-time, like I do in the snow/rain plugin should work, I think.

Crimson Wizard

#132
Quote from: Scorpiorus on Tue 09/05/2017 01:39:01
Quote from: Crimson Wizard on Tue 09/05/2017 01:24:22E: Or, do you mean same thing as I suggested as alternative?
Ah, yeah. Probably adding/inserting new entries to draw may be too late since we are in the rendering phase already but calling _renderSprite() just-in-time, like I do in the snow/rain plugin should work, I think.
I think either way might be possible with some work; currently I am substituting one bitmap right before _renderSprite get called, but I guess that would be possible to insert more, since there is a std::vector for draw entries now.

I guess the choice depends mostly on how that would be more convenient code wise. AFAIK _renderSprite is called after some variables are calculated, and these variables are local, thus calling it from external function may be problematic without larger code changes.

Scorpiorus

I'm not sure what is a common implementation of std::vector. It seems like it could be a plain array. Then insertion may take some time. And I'm going to try and draw 10k particles as a test for the plugin since it will be HW-accelerated. But, I don't think it will be too much of a performance hit with even std::vector anyway...

Quote from: Crimson Wizard on Tue 09/05/2017 01:50:35AFAIK _renderSprite is called after some variables are calculated, and these variables are local, thus calling it from external function may be problematic without larger code changes.
Well, I'm just calling it with NullSpriteEntry substituted with my bitmap. Seems to work. But then again maybe it only seems so...

Crimson Wizard

#134
Quote from: Scorpiorus on Tue 09/05/2017 02:12:45
Well, I'm just calling it with NullSpriteEntry substituted with my bitmap. Seems to work. But then again maybe it only seems so...

Ok, I checked more, there seem to be just some global flip settings. I do not remember how they work, but they could be made class members for this purpose.

Scorpiorus

Yeah, IIRC the flip params come from some global settings and I wanted to support them in the snow/rain plugin for AGS 3.4.0. But for now I just pass them as "no v/h flip".

Crimson Wizard

Something that I am still wondering about, if you call _renderSprite 10k times, how do you manage 10k driver-dependent bitmap creations? Or are you reusing only one? or do something else?

Scorpiorus

#137
Yep, like I mentioned before it is a very specific thing with particle systems, i.e. drawing tons of little images but in fact using just several (driver dependent) bitmaps as a source.
I don't allow drawing 10k instances of a particle in the plugin (only 2k max, at the moment), but I just thought that if it will be HW-accelerated I could probably increase the limit.

Using std::vector is perfectly fine if we are going to add elements to its end, but inserting them, especially one by one, would trigger copying the whole thing on the "right" each time we do it.
Another "bad" thing with std::vector is that it may well decide to reallocate its internal stuff to some other memory address (say, when we add new elements and it runs out of reserved space at its back), and then we can end up with dangling pointers if we had them pointing to vector's elements. I'm not sure maybe this is not a problem for us if we just pass element's addresses to _renderSprite() and don't resize the std::vector within that function.

EDIT:

So, it seems like flattening-bitmaps-to-scene-overlay approach has its own advantages here, where the engine doesn't have to care how to store/handle bitmaps passed via some plugin API Blit function.
The irony here, is that, practically, the bitmap passed to plugin Blit functions most likely was acquired with IAGSEngine::GetSpriteGraphic() and hence already loaded in sprite cache and may even have a device dependent representation ready to render. But since it's passed as an Allegro BITMAP all ties are lost and the engine has to deal with it as with some totally new entity (though, of course, this actually may very well be the case if it's some custom bitmap prepared by plugin).
And therefore there are, again, certain complexities of whether the engine should somehow try caching bitmaps passed via plugin blit functions, to use the same device dependent bitmaps and doesn't have to create new/destroy old DDB every single time...

Scorpiorus

Getting back to the flattened bitmaps idea... If it were possible to implement translucency more or less suitable, could we count on _stageVirtualScreen supporting 8-bit alpha channel (ARGB mode) with hardware accelerated drivers?

I noticed that the old snow/rain demo game (which is 16-bit color) runs in 32-bit color mode with the AGS Direct3D 9 driver.

Crimson Wizard

#139
I made it so it sets 32-bit display mode by default, and converts game to that mode, otherwise 16-bit bitmaps were loosing 1 bit of green color (there is no suitable texture format for R5G6B5).
OpenGL driver did that since its created.

Direct3D and OpenGL can now run even 8-bit games, converting them to 32-bit ARGB, except it does support palette cycling.

I left "downgrade to 16-bit" option in the engine, but IDK if that is ever comes useful. That could be removed too. Guess it was mainly for software renderer.

I was thinking that maybe there is already a suitable blender function in the engine, because I remember writing few, for blitting opaque sprite without alpha on surface with alpha and vice versa.

SMF spam blocked by CleanTalk