Adventure Game Studio | Forums

AGS Development => Engine Development => Topic started by: monkey0506 on 26 Aug 2014, 15:44

Title: Built-in plugins/stubs?
Post by: monkey0506 on 26 Aug 2014, 15:44
It was recently suggested that I create a plugin stub for the AGSteam plugin, so games which use it can be run on non-Steam platforms. This is particularly relevant if trying to run a Steam game on a platform which doesn't support Steam.

My question is this -- There are already some plugins which are built into the engine source code, but I'm worried that we might bog things down if we start including and/or stubbing every "useful" plugin (AGSteam would have to be stubbed due to Valve's licensing). What is the general consensus on this? I already have a stub written, I just need to more thoroughly test that the full plugin can usefully tap into the stubbed features. Would it make sense for me to build it into the engine, or should I just provide it separately for those who need it?
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 26 Aug 2014, 16:04
My opinion on this is simple: I don't like to add stubs into the engine for every new plugin released. But it is not end of the world, and the number of plugins is small, so if there is no other way to make the game run, then so be it.
Besides, I cleaned up plugin stubs some time ago, and there's just a few "standard" stubs that are different only by default return value. They are used for all missing plugins.

The only problem I know about plugins is that engine currently is not capable to load libraries on iOS. I am generally uneducated in terms of how iOS works and if it is possible at all.
Title: Re: Built-in plugins/stubs?
Post by: onitake on 26 Aug 2014, 16:47
Quote from: Crimson Wizard on 26 Aug 2014, 16:04
The only problem I know about plugins is that engine currently is not capable to load libraries on iOS. I am generally uneducated in terms of how iOS works and if it is possible at all.
Apple doesn't allow iOS applications published on the App Store to load external code, be it some kind of bytecode or machine code.
So, at least on iOS, linking all executable code into a single binary is kind of a requirement - unless you just want to distribute your games to jailbreakers.
I'm not sure if Apple has recently softened their stance towards bytecode execution, but external machine code libraries are a strict no-go.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 26 Aug 2014, 16:52
Tobihan (BigMc) has proposed an interesting idea:
Quote
We could define a text file format containing only the function names
and return values that are appropriate for a stub. Could be provided by
the plugin author and the engine could automatically look for a file
with the ending .stub instead of .dll (e.g. AGSteam.stub). That way
users don't have to compile a stub library.

I think this can work, because since AGS 3.3.0 internally api function has a single abstract prototype; for the stub engine needs to know only name and which value to return. The latter should probably be restricted to numeric value and "null"/void.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 26 Aug 2014, 21:16
Presumably this would be one of the files that is built into the game data? So, for example, the plugin author would publish the DLL/SO file(s) alongside a STUB file which the developer would put in the editor directory. Then on building, the STUB would be built into the game file, and the DLL copied to the Compiled folder?

That seems like a good route to me (if I understand you correctly).
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 26 Aug 2014, 22:11
Hmm, I didn't think about putting that in a game package. Of course, that is doable too. But what if game dev simply forgets, or ditches this?
When AGS needs a game asset, it first looks into the game package, but then into the current directory (this refers to literally any data). This means that if there was no stub in the package, player may put one in the game directory himself.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 26 Aug 2014, 23:45
Well with your suggestion that the return types of the stubbed functions being restricted, it seems unlikely that a stub would do anything malicious, right? So the ability to have the stub built into the game package or run from the game directory is good. Isn't it? It seems like you might be trying to discourage this.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 27 Aug 2014, 00:40
Quote from: monkey_05_06 on 26 Aug 2014, 23:45
Well with your suggestion that the return types of the stubbed functions being restricted, it seems unlikely that a stub would do anything malicious, right? So the ability to have the stub built into the game package or run from the game directory is good. Isn't it? It seems like you might be trying to discourage this.
No, not discourage, just considering the possibilities.
The stub can't do anything malicious on its own, of course, it will just return a script value, either integer/float, or null pointer.
There's, however, a chance that someone simply edits the stub and makes it return different values. This would affect the results of game script, and only that. But this might be seen as a way to hack the game.
On other hand, similarily, someone can write an overriding plugin variant that does anything unexpected on function call. In the end, the stub feature, while being easier to hack, provides much less freedom of hacking.
Title: Re: Built-in plugins/stubs?
Post by: BigMc on 27 Aug 2014, 08:06
The variant with the separate file should definitely be used because that would probably make it possible to make some existing games work (more or less) on all platforms. Can plugin functions return other types than numbers and pointers? I'm wondering if all plugins could be replaced by stubs when this type restriction is in place (different behaviour of the game aside).
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 27 Aug 2014, 08:43
Quote from: BigMc on 27 Aug 2014, 08:06
Can plugin functions return other types than numbers and pointers? I'm wondering if all plugins could be replaced by stubs when this type restriction is in place (different behaviour of the game aside).
AGS Script does not have other return types than numbers and pointers. Well, void too.
The rule should be that if the stub defines a non-zero return value, it is treated as a numeric type. It should not let stubs return random memory address, that is.

Basically, the stub format may be:
Code: Text
  1. function_name number | nil
  2.  
where "nil" is for both pointer and void return types. In AGS interpreter it does not make a big difference.
Stubs do not need to define actual return type in case of pointer, nor function arguments, because these parameters are known at the script compilation.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 27 Aug 2014, 21:14
Quote from: BigMc on 27 Aug 2014, 08:06
The variant with the separate file should definitely be used because...

To be clear, the engine already has the ability to look for assets in the game directory if they are not found internally in the game's data file. The option to bundle it into the game file should be present, and the engine can fall back to the game directory if needed.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 28 Aug 2014, 02:10
For the sake of example, here's a patch that does stub parsing/registering:
http://www.mediafire.com/download/njfegtxays6sb51/0001-Engine-automatic-plugin-stub-support.patch
(applied on top of "develop-3.3.1")

It won't work to the end, though. First, script interpreter must be permitted to use this stub value instead of actually calling a function, which is easy to do.
More importantly, the script exports table keeps the original function name pointer instead of allocating its own string... :-\ Works well for string literals, but will cause memory access problems with dynamic strings. This needs a rewrite too.
Title: Re: Built-in plugins/stubs?
Post by: Radiant on 04 Sep 2014, 21:45
I think this is a great idea. There is currently a lot of interest in playing AGS games on different platforms, and to my knowledge there are only about three commonly used plugins that cause problems with this (i.e. Steam, Flashlight, and Rain/Snow).
Title: Re: Built-in plugins/stubs?
Post by: Wyz on 05 Sep 2014, 01:03
I've got an idea maybe.
Almost all plugins run on windows so they come in the form of a dll. Inside it is also the script header which defines all functions for AGS scripts. So stub generation could potentially be automatic: extract the script header from the dll and parse it for functions to create stubs for.
If that does not work cross platform it would be possible to create a tool (a 'fake' host to the dll) that implements the RegisterScriptHeader and RegisterScriptFunction functions and uses this to generate a stub file. This tool could run on windows (or perhaps wine) so the dll would load and the whole process could be automated.
What do you think?
Title: Re: Built-in plugins/stubs?
Post by: BigMc on 05 Sep 2014, 12:22
Radiant: there are reimplementations of Flashlight and RainSnow in the engine now so they're not the problem. But there are also games which use less common plugins.

Wyz: Normally the plugins all come with a readme file documenting the API that can be used to write stubs. Otherwise I guess there are already tools to extract the header from a dll. The rest could be done manually and the stubs could be collected somewhere.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 08 Sep 2014, 00:19
While we sort out what to do, I've created an open-source stub for the AGSteam plugin, which was the main offender in this case (as Flashlight and SnowRain have built-in workalikes).

https://github.com/monkey0506/agsteamstub
Title: Re: Built-in plugins/stubs?
Post by: Radiant on 27 Sep 2014, 13:28
Quote from: BigMc on 05 Sep 2014, 12:22
Radiant: there are reimplementations of Flashlight and RainSnow in the engine now so they're not the problem.
Yes, I've intentionally been avoiding plugins for several years now (and doing it in AGS code instead) because of Mac/Linux compatibility.

Quote from: monkey_05_06 on 08 Sep 2014, 00:19
While we sort out what to do, I've created an open-source stub for the AGSteam plugin, which was the main offender in this case (as Flashlight and SnowRain have built-in workalikes).
Thank you! But how does this work, am I supposed to compile this and ship it with a Linux build of a AGSteam game?

Come to think of it, it would be relatively easy to do a botted download of all games on the AGS database, and see which ones includes plugins.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 28 Sep 2014, 02:48
A "stub" is nothing more than a replacement for cases where the plugin can't, shouldn't, or won't run. So you would not push a Steam version of your game to Valve's servers with a stubbed AGSteam plugin. You could release the same game files as a non-Steam version by replacing the full plugin with a build of the stub. This would save you having to rebuild the game files or keep track of separate versions of the game files -- just make sure that you include the full plugin when publishing to Steam (including Steam for Linux) and the stub when publishing to non-Steam sources.

Hopefully that clarifies it.
Title: Re: Built-in plugins/stubs?
Post by: Radiant on 28 Sep 2014, 10:50
Wait, as far as I can tell the windows Steam plugin works fine when there is no Steam installed (i.e. it just returns false on all function calls). Is that not the case on other systems? Would I need a different plugin on Linux or Mac systems?
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 28 Sep 2014, 12:04
Quote from: Radiant on 28 Sep 2014, 10:50
Wait, as far as I can tell the windows Steam plugin works fine when there is no Steam installed (i.e. it just returns false on all function calls). Is that not the case on other systems? Would I need a different plugin on Linux or Mac systems?
The problem here, is that the "real" plugin that contacts the Steam servers cannot be open-sourced according to Steam license.
So, you may use the pre-built AGSSteam for linux, but it won't work on all Linux systems, because not all systems use same core C++ libraries. For those systems where it does not work, you'd have to provide an open-sourced version, i.e the stub.
Title: Re: Built-in plugins/stubs?
Post by: monkey0506 on 28 Sep 2014, 20:26
Right, sorry. The full plugin can still be run without Steam (although doing nothing) on compatible OSes (Windows and some versions of Linux), but if you're specifically publishing a non-Steam version it's still recommended that you use the stub instead of the full plugin. This allows the end-user to rebuild the stub for their system if the prebuilt stub (the one you would include in your distribution) does not run (e.g., versions of Linux using different core libraries, other unsupported OSes).
Title: Re: Built-in plugins/stubs?
Post by: Radiant on 28 Sep 2014, 21:39
Yes, but then you're assuming that players have sufficient programming skill to be able to recompile a stub from source code; I'm not sure if I'd be willing to make that assumption.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 28 Sep 2014, 22:04
Quote from: Radiant on 28 Sep 2014, 21:39
Yes, but then you're assuming that players have sufficient programming skill to be able to recompile a stub from source code; I'm not sure if I'd be willing to make that assumption.
If players don't have programming skill to build and install a program from source, they probably do not use non-standard Linux.

Besides, unlike Windows, on most Linux distributives you should only run 2-3 commands from console. I believe that an installation script and/or basic readme will be sufficient.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 19 Dec 2014, 12:18
I made it work:
https://github.com/ivan-mogilko/ags-refactoring/tree/improve-scapimap

Explanation:
Spoiler: ShowHide

Quote
The stub loading routine parses the symbol definition *.stub file and registers API stubs if no plugin library was found. This allows to substitute any missing plugin with dummy symbols.

Stub file format defines the text file consisting of any number of lines. Every line declares a stub function and its return value; the return value may be either numeric value or 'nil' that defines either null-pointer or 'void', depending on how the function is used by script interpreter.
If no value is provided, 'nil' is used by default.

    function_name [numeric_literal | nil]

Numeric value is first tested as integer, then as float, therefore to force floating-point type of value one should always include the decimal separator, e.g. write "1." or "1.0" instead of just "1".


Working example:
Spoiler: ShowHide

Code: Text
  1. JoystickName nil
  2. JoystickCount 0
  3. Joystick::Open^1 nil
  4. Joystick::Close^0 nil
  5. Joystick::Valid^0 0
  6. Joystick::IsButtonDown^1 0
  7. Joystick::Click^1 nil
  8.  


Can anyone suggest a game/plugin for a test?
Title: Re: Built-in plugins/stubs?
Post by: BigMc on 28 Jan 2015, 00:00
Cool! One of the games that one could test with is Barn Runner 5 (http://www.barnrunner.com/barn5.html), chapter 1. It uses Kweepa's AGS_OtherRoom.dll and Scorpiorus' ags_parallax.dll both of which are not ported as far as I know.
Title: Re: Built-in plugins/stubs?
Post by: Crimson Wizard on 17 Feb 2015, 15:24
Quote from: BigMc on 28 Jan 2015, 00:00
Cool! One of the games that one could test with is Barn Runner 5 (http://www.barnrunner.com/barn5.html), chapter 1. It uses Kweepa's AGS_OtherRoom.dll and Scorpiorus' ags_parallax.dll both of which are not ported as far as I know.
Sorry, I forgot to answer.

While it is technically possible, I do not think that it is a right thing to stub OtherRoom and parallax plugins, because this will disable their functionality. The game will load up, but gameplay may be completely broken.

The plugin stubs are meant for plugins that add optional effects, like rain/snow, Steam achievements, joystick controls, etc.