[EXPERIMENT] AGS 4: Custom Shaders support

[Crimson Wizard]

Overview

Following a recent request (see issue 2705),
I started working on an experimental feature which would allow to attach custom pixel shaders to game objects in AGS 4.0.

THIS POST WAS UPDATED TO COVER THE LATEST STATE

The PR with shaders support was merged into AGS 4 branch:
https://github.com/adventuregamestudio/ags/pull/2716
Latest experimental build download (of 15th May):
https://cirrus-ci.com/task/6225249240350720
Demo game project:
https://www.dropbox.com/scl/fi/q8lwoi6xwg4m7apezs6i8/ags4-shaders.zip?rlkey=12d4gxmfln1dh4h1zh0e2vpfs&st=9wdbt6e4&dl=0
Compiled game:
https://www.dropbox.com/scl/fi/yj98k9o43tiuufbo180cq/ags4-shaders-game.zip?rlkey=yspxgpfzduspun7o6a6s8kq4l&st=xuewfvb4&dl=0

Example preview:

Spoiler

[close]

Explanation and instructions

The idea overall is this:

1. User writes custom shader scripts

Shaders are written in either GLSL (OpenGL shader language) or HLSL (Microsoft's shader language for Direct3D). These scripts are distributed along with the game, either as external files, or packaged using "Package custom folders" option in General Settings.

OpenGL graphics driver uses GLSL scripts and compiles them into shaders at runtime.
Direct3D graphics driver uses HLSL scripts and compiles them into shaders at runtime.
Optionally Direct3D can also load up precompiled shader objects (FXO). These may be created using Microsoft's "fxc" utility ("Effect-Compiler") from the older DirectX SDK (deprecated and archived), or possibly a modern equivalent "dxc" ("directx compiler"). Frankly, I haven't tested the latter yet, so not fully sure if it supports old HSLS dialect for Direct3D 9, which is used by AGS.


2. In script there are two new structs declared: ShaderProgram and ShaderInstance.

ShaderProgram represents a compiled shader, while ShaderInstance represents a shader setup: which is a shader + a set of custom shader values.
For those who have an idea of how modern game engines work: you may think of ShaderInstance as a kind of a very limited "Material" type.
ShaderProgram is used to create ShaderInstances, and ShaderInstances are assigned to the game objects.

Following may have a shader assigned:
* Screen.Shader
* Viewport.Shader (e.g. Screen.Viewport.Shader)
* Camera.Shader (e.g. Game.Camera.Shader)
* Object.Shader, Character.Shader, GUI.Shader, GUIControl.Shader, Overlay.Shader
* Room.BackgroundShader
* Mouse.CursorShader


The new structs are declared like:

Code: ags

builtin managed struct ShaderProgram {
  /// Creates a new ShaderProgram by either loading a precompiled shader, or reading source code and compiling one.
  import static ShaderProgram* CreateFromFile(const string filename); // $AUTOCOMPLETESTATICONLY$
  /// Creates a new shader instance of this shader program.
  import ShaderInstance* CreateInstance();
  /// Gets the default shader instance of this shader program.
  import readonly attribute ShaderInstance* Default;
};

builtin managed struct ShaderInstance {
  /// Sets a shader's constant value as 1 float
  import void SetConstantF(const string name, float value);
  /// Sets a shader's constant value as 2 floats
  import void SetConstantF2(const string name, float x, float y);
  /// Sets a shader's constant value as 3 floats
  import void SetConstantF3(const string name, float x, float y, float z);
  /// Sets a shader's constant value as 4 floats
  import void SetConstantF4(const string name, float x, float y, float z, float w);
  /// Gets this instance's parent Shader
  import readonly attribute ShaderProgram* Shader;
};

You create a ShaderProgram using CreateFromFile static method. The filename you pass may contain a glsl, hlsl or fxo extension, but the engine will choose an actual file depending on the current graphics driver. For example, you call CreateFromFile("shaders/myshader.glsl"). If you run with OpenGL engine will look for "myshader.glsl". If you run with Direct3D engine will look for "myshader.fxo" (precompiled directx shader), and if it's not present, then for "myshader.hlsl" (directx shader source). This goes vice versa too. Such approach lets user pass the filename in any format, and not make extra if/else conditions in script checking current gfx driver (although you may if you need to).

ShaderProgram object is created always, even if shader compilation is failed for any reason. This is essential, because graphic driver may not support this shader, or not support shaders at all (such as Software graphics driver). This lets you write scripts not worrying about things failing if player switches to another driver. Of course the real visual effect will only appear if the actual shader was initialized successfully; otherwise this shader program will just do nothing.

After ShaderProgram is created, you have two options:
* use ShaderProgram.Default property which returns a always present default ShaderInstance. This instance is there to simplify things for you.
* create more ShaderInstances using ShaderProgram.CreateInstance() method.

Simple example would be like:

Code: ags

ShaderProgram* myshader;

function game_start()
{
    myshader = ShaderProgram.CreateFromFile("$DATA$/shaders/myshader.glsl");
    Screen.Shader = myshader.Default;
}

Why create more instances? Shaders may have "constants" in them, which may be thought as shader settings. These "constants" are not really constants in general sense, they are called "constants" because they don't change while shader is used in drawing. But you may change their values between the draws.
There are alot of purposes to use constants. Just to give couple of examples:
* a shader that tints a sprite by adding certain color - may have a constant "color", which you configure in script.
* a shader that changes the sprite look depending on time - then it will have a constant "current time", which engine will update each frame (find more info below).

So, suppose you have one shader, but want to configure this shader differently for different objects. That's where you need separate ShaderInstances.
You create 5 shader instances, and assign these to 5 objects, then set different constant values for these separate instances.

Here's an example:

Code: ags

ShaderProgram* myshader;
ShaderInstance* myshaderInsts[5];

function game_start()
{
    myshader = ShaderProgram.CreateFromFile("$DATA$/shaders/myshader.glsl");
    for (int i = 0; i < 5; i++)
    {
        myshaderInsts[i] = myshader.CreateInstance();
        object[i].Shader = myshaderInsts[i];
    }
    
    myshaderInsts[0].SetConstantF3("Color", 1.0, 0.0, 0.0); // red
    myshaderInsts[1].SetConstantF3("Color", 0.0, 1.0, 0.0); // green
    myshaderInsts[2].SetConstantF3("Color", 0.0, 0.0, 1.0); // blue
    // and so on
}

How to write shaders

In very primitive way "pixel shader" is an algorithm that is run over each pixel of a sprite, receives a real pixel color as an input, and returns same or another pixel color as an output. So what it does, essentially, is changing sprite pixel colors in some way. This change is not permanent (the sprite remains) but the result of a shader is used to draw this sprite on screen.

Unfortunately, I do not have enough expertise nor spare time to explain shader scripts from ground up. But AGS uses standard shader languages, and there must be thousands of tutorials online.

There are however few things that I must mention.

When writing shaders you may have your custom constants in it, but are also allowed to use a number of "standard" constants provided by the engine. The engine sets their values automatically for each draw.
In GLSL these must match the name and type, but their order is not important (they may be not present if not used too). That's because OpenGL finds these by name.
In HLSL these constants EITHER must match the type and the *register number*, OR you have to write an accompanying ini file called "<shadername>.d3ddef", which describes your shader. That's because Direct3D 9 needs a different utility library for finding constant automatically, but this library is outdated, so I decided to not use it to be safe. I will explain "d3ddef" file a little further.

Back to the constants, following is their list:
* float iTime - current time in seconds; note that it's not exact values that should matter, but the fact they it changes over time
* int iGameFrame - current game frame index (NOTE: must be float in HLSL, because apparently it D3D9-compatible mode does not support integer constants? at least that's what I read somewhere);
* sampler2D iTexture - sprite's texture reference;
* vec2 iTextureDim - texture's dimensions in pixels (type `float2` in HLSL);
* float iAlpha - sprite's general alpha (object's transparency);
* vec2 iOutputDim - final dimensions in pixels (type `float2` in HLSL); this constant is only set for the "whole screen shader" and tells the real resolution that the image will have when appear in window. If this shader will be applied to other game object, then this constant will have value of zero.

Other predefined input parameters:
* vec2 vTexCoord - is predefined for GLSL only, gets current texture coordinate for the pixel. HLSL should use TEXCOORD0 input parameter (see shader examples below).

Example of declaring parameters in GLSL:

Code: ags

uniform float iTime;
uniform int iGameFrame;
uniform sampler2D iTexture;
uniform vec2 iTextureDim;
uniform float iAlpha;

varying vec2 vTexCoord;

Example of declaring parameters in HLSL (notice the order of registers! - that matters if you precompile the shader):

Code: ags

// Pixel shader input structure
struct PS_INPUT
{
    float2 Texture    : TEXCOORD0;
};

// Pixel shader output structure
struct PS_OUTPUT
{
    float4 Color  : COLOR0;
};

sampler2D iTexture; // is in sampler register 0

const float  iTime:        register( c0 );
const float  iGameFrame:    register( c1 );
const float2 iTextureDim:  register( c2 );
const float  iAlpha:        register( c3 );

Example of my "Colorwave" shader in GLSL (for OpenGL):

Spoiler

Code: ags

uniform sampler2D iTexture;

uniform float iTime;
uniform int iGameFrame;
uniform vec2 iTextureDim;
uniform float iAlpha;

varying vec2 vTexCoord;

#define PI                 3.1415
#define PI2                6.283
#define FPS                120.0
#define WAVE_DIR          -1.0
#define TINT_STRENGTH      0.2
#define X_OFFSET_STRENGTH  0.00
#define Y_OFFSET_STRENGTH  0.02

void main()
{
    vec2 uv = vTexCoord;
    // convert from textcoord [-1;1] to [0;1] range
    vec2 uv_1 = uv * 0.5 + 0.5;
    // timer goes [0 -> 1) and resets, in FPS frequency
    float timer = mod(iGameFrame, FPS) / FPS;
    // wave cycles by timer + add starting phase depending on texture pixel position
    float wave_x = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.x));
    float wave_y = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.y));
    float wave_z = sin((WAVE_DIR * PI2 * timer) + (PI  * uv_1.x));
    
    vec3 tint = vec3(TINT_STRENGTH * wave_x, TINT_STRENGTH * wave_y, TINT_STRENGTH * wave_z);
    vec4 color = texture2D(iTexture, uv + vec2(wave_x * X_OFFSET_STRENGTH, wave_y * Y_OFFSET_STRENGTH));
    
    gl_FragColor = vec4(color.xyz + tint, color.w);
}

[close]

and same shader in HLSL (for Direct3D):

Spoiler

Code: ags

// Pixel shader input structure
struct PS_INPUT
{
    float2 Texture    : TEXCOORD0;
};

// Pixel shader output structure
struct PS_OUTPUT
{
    float4 Color  : COLOR0;
};

// Global variables
sampler2D iTexture;

const float  iTime:        register( c0 );
const float  iGameFrame:   register( c1 );
const float2 iTextureDim:  register( c2 );
const float  iAlpha:       register( c3 );

#define PI                 3.1415
#define PI2                6.283
#define FPS                120.0
#define WAVE_DIR          -1.0
#define TINT_STRENGTH      0.2
#define X_OFFSET_STRENGTH  0.00
#define Y_OFFSET_STRENGTH  0.02

PS_OUTPUT main( in PS_INPUT In )
{
    float2 uv = In.Texture;
    // convert from textcoord [-1;1] to [0;1] range
    float2 uv_1 = uv * 0.5 + 0.5;
    
    // timer goes [0 -> 1) and resets, in FPS frequency
    float timer = fmod(iGameFrame, FPS) / FPS;
    // wave cycles by timer + add starting phase depending on texture pixel position
    float wave_x = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.x));
    float wave_y = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.y));
    float wave_z = sin((WAVE_DIR * PI2 * timer) + (PI  * uv_1.x));
    
    float3 tint = float3(TINT_STRENGTH * wave_x, TINT_STRENGTH * wave_y, TINT_STRENGTH * wave_z);
    float4 color = tex2D(iTexture, uv + float2(wave_x * X_OFFSET_STRENGTH, wave_y * Y_OFFSET_STRENGTH));
    
    PS_OUTPUT Out;
    Out.Color = float4(color.xyz + tint, color.w);
    return Out;
}

[close]

Back to the "d3ddef" file required for the Direct3D's HLSL shaders. This file is obligatory if you have custom constants, as Direct3D cannot know about these. Another reason to write one if you like to specify compiler target (HLSL version).
"d3ddef" file is a simply "ini" file, which may contain few options:

Code: ags

[compiler]
target = compilation target ("ps_2_0", and so on)
entry = entry function name (e.g. "main")

[constants]
<constant_name> = register index (a number >= 0)

If "compiler" options are not present, Direct3D will use defaults.
If "constants" are not present, Direct3D will use default hardcoded register values. Note that if you write "[constants]" section in it, then you MUST mention ALL constants, including standard ones.

This may be an example of "colorwave.d3ddef" for my demo shader (I don't really use it in the demo game, but could have):

Code: ags

[compiler]
target = ps_2_b

[constants]
iGameFrame = 1
iTextureDim = 2
iAlpha = 3

[Crimson Wizard]

An update, I've revamped script API, for the purpose of supporting custom shader parameters, and now it looks like this:

Code: ags

builtin managed struct ShaderProgram {
  /// Creates a new ShaderProgram by either loading a precompiled shader, or reading source code and compiling one.
  import static ShaderProgram* CreateFromFile(const string filename); // $AUTOCOMPLETESTATICONLY$
  /// Creates a new shader instance of this shader program.
  import ShaderInstance* CreateInstance();
  /// Gets the default shader instance of this shader program.
  import readonly attribute ShaderInstance* Default;
};

builtin managed struct ShaderInstance {
  /// Sets a shader's constant value as 1 float
  import void SetConstantF(const string name, float value);
  /// Sets a shader's constant value as 2 floats
  import void SetConstantF2(const string name, float x, float y);
  /// Sets a shader's constant value as 3 floats
  import void SetConstantF3(const string name, float x, float y, float z);
  /// Sets a shader's constant value as 4 floats
  import void SetConstantF4(const string name, float x, float y, float z, float w);
};

Here ShaderProgram represents a compiled shader itself, and ShaderInstance is shader's setup with certain constant values. You may think of ShaderInstance as a kind of a limited "material" type.

Game objects (characters, etc) now assign ShaderInstance pointer to themselves, rather than a numeric "shader id".

Each ShaderProgram has a "default" instance which is always present, and may be used when either this shader does not have custom parameters, or you don't want to set them up. On another hand, if you want to use same shader on multiple objects but with separate sets of parameters, then you can create more "ShaderInstances".
ShaderInstance may be assigned to multiple objects, in which case they all will share same shader setup.

For example:

Code: ags

player.Shader      = myShaderProgram.Default;
cCharacter1.Shader = myShaderProgram.CreateInstance();
cCharacter2.Shader = myShaderProgram.CreateInstance();
cCharacter3.Shader = myShaderProgram.CreateInstance();

cCharacter1.Shader.SetConstantF("CustomConstant", 1.0);
cCharacter2.Shader.SetConstantF("CustomConstant", 2.0);
cCharacter3.Shader.SetConstantF("CustomConstant", 5.0);

More details are in the updated PR post:
https://github.com/adventuregamestudio/ags/pull/2716#issue-2987287896
Downloaded experimental build here:
https://cirrus-ci.com/task/6359903394070528

[Crimson Wizard]

Another update: the shaders feature is practically done, at least in the first iteration. Every drawn game object has a Shader property, also: Camera, Viewport, and Screen, so you may have shader effects on them all in any combinations. Savegames work too; furthermore, it's possible to save game with one graphic driver and load it with another, and have shader settings persist (shaders may work differently or not work at all, as with software renderer, but all the shader settings are remembered regardless).

There are couple of remaining issues, but they will have to be addressed separately.

PR with full explanation and usage instructions:
https://github.com/adventuregamestudio/ags/pull/2716
Download experimental build from CI:
https://cirrus-ci.com/task/5684067657580544

NOTE: the very first post in this forum thread contains outdated information. I will replace it tomorrow when will have more spare time.

[Crimson Wizard]

The feature has been merged into AGS 4 branch, and will be a part of the next AGS 4 Alpha Update.

I've rewritten the first post in this thread, please refer to it for the updated instructions, test build download, demo game, etc.