[EXPERIMENT] AGS 4: Custom Shaders support

[Crimson Wizard]

Overview

Following a recent request (see issue 2705),
I started working on an experimental feature which would allow to attach custom pixel shaders to game objects in AGS 4.0.

There is some progress already, and I have a working draft:
https://github.com/adventuregamestudio/ags/pull/2716
Experimental build download:
https://cirrus-ci.com/task/4700587138220032
Simple demo game:
https://www.dropbox.com/scl/fi/ifgmbhnmevzlv0jw0z1f6/ags4-shadertest.zip?rlkey=yxvrbna340ftyzzeuef7n787m&st=447hyl5j&dl=0
Compiled game:
https://www.dropbox.com/scl/fi/6norz18pkkqem7pc2hgeo/colorwave.zip?rlkey=8q04rjhkgi2vxoz97kbhhczqe&st=5ul9qe1s&dl=0

Preview:

Spoiler

[close]

Explanation

Although working, the present script API and implementation are not final, and may be subject to changes.

The idea overall is this:

1. User writes custom shader scripts in either GLSL (OpenGL shader language) or HLSL (Microsoft's shader language for Direct3D). These scripts are distributed along with the game, either as external files, or packaged using "Package custom folders" option in General Settings.

IMPORTANT: the current experimental build compiles opengl shaders from source script, but cannot do that for direct3d, so HLSL shaders must be compiled to a compiled shader object file (fxo) first. That may be done using Microsoft's "fxc" utility from DirectX SDK.
For that reason it may be easier to test this build using OpenGL graphics driver.

2. In script there's a new struct declared like:

Code: ags

builtin managed struct ShaderProgram {
  import static ShaderProgram* CreateFromFile(const string filename);
  import readonly attribute int ShaderID;
};

And every drawable object in game has this new property:

Code: ags

attribute int Shader;

3. In script you create a ShaderProgram object from a file, like this:

Code: ags

ShaderProgram *myshader;

myshader = ShaderProgram.CreateFromFile("$DATA$/shaders/myshader.glsl");

and then attach to the object or room viewport like this:

Code: ags

Screen.Viewport.Shader = myshader.ShaderID;
player.Shader = myshader.ShaderID;
gMainMenu.Shader = myshader.ShaderID;

Because not every graphics driver supports shaders, and for general convenience too, CreateFromFile always returns a valid ShaderProgram object, but if ShaderID is 0, then this shader is actually invalid. Assigning that will make renderer fallback to default shader.

Currently ShaderProgram.CreateFromFile accepts two file extensions: "glsl" for OpenGL shader sources, and "fxo" for D3D9 compiled shaders. To make things more convenient, user does not have to make a switch in the code: engine does it for them. If you pass "myshader.glsl", and engine is running a Direct3D driver, then it will look for "myshader.fxo" instead. This logic should be improved later.


How to write shaders

In very primitive way "pixel shader" is an algorithm that is run over each pixel of a sprite, receives a real pixel color as an input, and returns same or another pixel color as an output. So what it does, essentially, is changing sprite pixel colors in some way. This change is not permanent (the sprite remains) but the result of a shader is used to draw this sprite on screen.

Unfortunately, I do not have enough expertise nor spare time to explain shader scripts from ground up. But AGS uses standard shader languages, and there must be thousands of tutorials online.

There are however few things that I must mention.

When writing shaders you are allowed to use a number of constants (or uniforms in GLSL) provided by the engine.
In GLSL these must match the name and type, but their order is not important (they may be not present if not used too).
In HLSL, because engine currently only uses precompiled shaders, and is missing a utility library to get shader info, these constants must match the type and the *register* (but name may be any). This may change in future updates of this draft.

Here's a list of constants:
* float iTime - current time in seconds;
* int iGameFrame - current game frame index (NOTE: must be float in HLSL, because apparently it D3D9-compatible mode does not support integer constants? at least that's what I read somewhere);
* sampler2D iTexture - sprite's texture reference;
* vec2 iTextureDim - texture's dimensions in pixels (type `float2` in HLSL);
* float iAlpha - sprite's general alpha (object's transparency);

Other predefined input parameters:
* vec2 vTexCoord - is predefined for GLSL only, gets current texture coordinate for the pixel. HLSL should use TEXCOORD0 input parameter (see shader examples below).

Example of declaring parameters in GLSL:

Code: ags

uniform float iTime;
uniform int iGameFrame;
uniform sampler2D iTexture;
uniform vec2 iTextureDim;
uniform float iAlpha;

varying vec2 vTexCoord;

Example of declaring parameters in HLSL (names do not have to match if you precompile, but notice the order of registers!):

Code: ags

// Pixel shader input structure
struct PS_INPUT
{
    float2 Texture    : TEXCOORD0;
};

// Pixel shader output structure
struct PS_OUTPUT
{
    float4 Color  : COLOR0;
};

sampler2D iTexture; // is in sampler register 0

const float  iTime:        register( c0 );
const float  iGameFrame:    register( c1 );
const float2 iTextureDim:  register( c2 );
const float  iAlpha:        register( c3 );

Example of my "Colorwave" shader in GLSL (for OpenGL):

Spoiler

Code: ags

uniform float iTime;
uniform int iGameFrame;
uniform sampler2D iTexture;
uniform vec2 iTextureDim;
uniform float iAlpha;

varying vec2 vTexCoord;

#define PI                  3.1415
#define PI2                6.283
#define FPS                120.0
#define WAVE_DIR          -1.0
#define TINT_STRENGTH      0.2
#define X_OFFSET_STRENGTH  0.00
#define Y_OFFSET_STRENGTH  0.02

void main()
{
    vec2 uv = vTexCoord;
    // convert from textcoord [-1;1] to [0;1] range
    vec2 uv_1 = uv * 0.5 + 0.5;
    // timer goes [0 -> 1) and resets, in FPS frequency
    float timer = mod(iGameFrame, FPS) / FPS;
    // wave cycles by timer + add starting phase depending on texture pixel position
    float wave_x = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.x));
    float wave_y = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.y));
    float wave_z = sin((WAVE_DIR * PI2 * timer) + (PI  * uv_1.x));
    
    vec3 tint = vec3(TINT_STRENGTH * wave_x, TINT_STRENGTH * wave_y, TINT_STRENGTH * wave_z);
    vec4 color = texture2D(iTexture, uv + vec2(wave_x * X_OFFSET_STRENGTH, wave_y * Y_OFFSET_STRENGTH));
    
    gl_FragColor = vec4(color.xyz + tint, color.w);
}

[close]

and same shader in HLSL (for Direct3D):

Spoiler

Code: ags

// Pixel shader input structure
struct PS_INPUT
{
    float2 Texture    : TEXCOORD0;
};

// Pixel shader output structure
struct PS_OUTPUT
{
    float4 Color  : COLOR0;
};

// Global variables
sampler2D iTexture;

const float  iTime:        register( c0 );
const float  iGameFrame:    register( c1 );
const float2 iTextureDim:  register( c2 );
const float  iAlpha:        register( c3 );

#define PI                  3.1415
#define PI2                6.283
#define FPS                120.0
#define WAVE_DIR          -1.0
#define TINT_STRENGTH      0.2
#define X_OFFSET_STRENGTH  0.00
#define Y_OFFSET_STRENGTH  0.02

PS_OUTPUT main( in PS_INPUT In )
{
    float2 uv = In.Texture;
    // convert from textcoord [-1;1] to [0;1] range
    float2 uv_1 = uv * 0.5 + 0.5;
    
    // timer goes [0 -> 1) and resets, in FPS frequency
    float timer = fmod(iGameFrame, FPS) / FPS;
    // wave cycles by timer + add starting phase depending on texture pixel position
    float wave_x = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.x));
    float wave_y = sin((WAVE_DIR * PI2 * timer) + (PI2 * uv_1.y));
    float wave_z = sin((WAVE_DIR * PI2 * timer) + (PI  * uv_1.x));
    
    float3 tint = float3(TINT_STRENGTH * wave_x, TINT_STRENGTH * wave_y, TINT_STRENGTH * wave_z);
    float4 color = tex2D(iTexture, uv + float2(wave_x * X_OFFSET_STRENGTH, wave_y * Y_OFFSET_STRENGTH));
    
    PS_OUTPUT Out;
    Out.Color = float4(color.xyz + tint, color.w);
    return Out;
}

[close]

[Crimson Wizard]

An update, I've revamped script API, for the purpose of supporting custom shader parameters, and now it looks like this:

Code: ags

builtin managed struct ShaderProgram {
  /// Creates a new ShaderProgram by either loading a precompiled shader, or reading source code and compiling one.
  import static ShaderProgram* CreateFromFile(const string filename); // $AUTOCOMPLETESTATICONLY$
  /// Creates a new shader instance of this shader program.
  import ShaderInstance* CreateInstance();
  /// Gets the default shader instance of this shader program.
  import readonly attribute ShaderInstance* Default;
};

builtin managed struct ShaderInstance {
  /// Sets a shader's constant value as 1 float
  import void SetConstantF(const string name, float value);
  /// Sets a shader's constant value as 2 floats
  import void SetConstantF2(const string name, float x, float y);
  /// Sets a shader's constant value as 3 floats
  import void SetConstantF3(const string name, float x, float y, float z);
  /// Sets a shader's constant value as 4 floats
  import void SetConstantF4(const string name, float x, float y, float z, float w);
};

Here ShaderProgram represents a compiled shader itself, and ShaderInstance is shader's setup with certain constant values. You may think of ShaderInstance as a kind of a limited "material" type.

Game objects (characters, etc) now assign ShaderInstance pointer to themselves, rather than a numeric "shader id".

Each ShaderProgram has a "default" instance which is always present, and may be used when either this shader does not have custom parameters, or you don't want to set them up. On another hand, if you want to use same shader on multiple objects but with separate sets of parameters, then you can create more "ShaderInstances".
ShaderInstance may be assigned to multiple objects, in which case they all will share same shader setup.

For example:

Code: ags

player.Shader      = myShaderProgram.Default;
cCharacter1.Shader = myShaderProgram.CreateInstance();
cCharacter2.Shader = myShaderProgram.CreateInstance();
cCharacter3.Shader = myShaderProgram.CreateInstance();

cCharacter1.Shader.SetConstantF("CustomConstant", 1.0);
cCharacter2.Shader.SetConstantF("CustomConstant", 2.0);
cCharacter3.Shader.SetConstantF("CustomConstant", 5.0);

More details are in the updated PR post:
https://github.com/adventuregamestudio/ags/pull/2716#issue-2987287896
Downloaded experimental build here:
https://cirrus-ci.com/task/6359903394070528