Author Topic: Built-in plugins/stubs?  (Read 4941 times)

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Built-in plugins/stubs?
« on: 26 Aug 2014, 15:44 »
It was recently suggested that I create a plugin stub for the AGSteam plugin, so games which use it can be run on non-Steam platforms. This is particularly relevant if trying to run a Steam game on a platform which doesn't support Steam.

My question is this -- There are already some plugins which are built into the engine source code, but I'm worried that we might bog things down if we start including and/or stubbing every "useful" plugin (AGSteam would have to be stubbed due to Valve's licensing). What is the general consensus on this? I already have a stub written, I just need to more thoroughly test that the full plugin can usefully tap into the stubbed features. Would it make sense for me to build it into the engine, or should I just provide it separately for those who need it?

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #1 on: 26 Aug 2014, 16:04 »
My opinion on this is simple: I don't like to add stubs into the engine for every new plugin released. But it is not end of the world, and the number of plugins is small, so if there is no other way to make the game run, then so be it.
Besides, I cleaned up plugin stubs some time ago, and there's just a few "standard" stubs that are different only by default return value. They are used for all missing plugins.

The only problem I know about plugins is that engine currently is not capable to load libraries on iOS. I am generally uneducated in terms of how iOS works and if it is possible at all.

Re: Built-in plugins/stubs?
« Reply #2 on: 26 Aug 2014, 16:47 »
The only problem I know about plugins is that engine currently is not capable to load libraries on iOS. I am generally uneducated in terms of how iOS works and if it is possible at all.
Apple doesn't allow iOS applications published on the App Store to load external code, be it some kind of bytecode or machine code.
So, at least on iOS, linking all executable code into a single binary is kind of a requirement - unless you just want to distribute your games to jailbreakers.
I'm not sure if Apple has recently softened their stance towards bytecode execution, but external machine code libraries are a strict no-go.

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #3 on: 26 Aug 2014, 16:52 »
Tobihan (BigMc) has proposed an interesting idea:
Quote
We could define a text file format containing only the function names
and return values that are appropriate for a stub. Could be provided by
the plugin author and the engine could automatically look for a file
with the ending .stub instead of .dll (e.g. AGSteam.stub). That way
users don't have to compile a stub library.

I think this can work, because since AGS 3.3.0 internally api function has a single abstract prototype; for the stub engine needs to know only name and which value to return. The latter should probably be restricted to numeric value and "null"/void.

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #4 on: 26 Aug 2014, 21:16 »
Presumably this would be one of the files that is built into the game data? So, for example, the plugin author would publish the DLL/SO file(s) alongside a STUB file which the developer would put in the editor directory. Then on building, the STUB would be built into the game file, and the DLL copied to the Compiled folder?

That seems like a good route to me (if I understand you correctly).

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #5 on: 26 Aug 2014, 22:11 »
Hmm, I didn't think about putting that in a game package. Of course, that is doable too. But what if game dev simply forgets, or ditches this?
When AGS needs a game asset, it first looks into the game package, but then into the current directory (this refers to literally any data). This means that if there was no stub in the package, player may put one in the game directory himself.

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #6 on: 26 Aug 2014, 23:45 »
Well with your suggestion that the return types of the stubbed functions being restricted, it seems unlikely that a stub would do anything malicious, right? So the ability to have the stub built into the game package or run from the game directory is good. Isn't it? It seems like you might be trying to discourage this.

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #7 on: 27 Aug 2014, 00:40 »
Well with your suggestion that the return types of the stubbed functions being restricted, it seems unlikely that a stub would do anything malicious, right? So the ability to have the stub built into the game package or run from the game directory is good. Isn't it? It seems like you might be trying to discourage this.
No, not discourage, just considering the possibilities.
The stub can't do anything malicious on its own, of course, it will just return a script value, either integer/float, or null pointer.
There's, however, a chance that someone simply edits the stub and makes it return different values. This would affect the results of game script, and only that. But this might be seen as a way to hack the game.
On other hand, similarily, someone can write an overriding plugin variant that does anything unexpected on function call. In the end, the stub feature, while being easier to hack, provides much less freedom of hacking.

Re: Built-in plugins/stubs?
« Reply #8 on: 27 Aug 2014, 08:06 »
The variant with the separate file should definitely be used because that would probably make it possible to make some existing games work (more or less) on all platforms. Can plugin functions return other types than numbers and pointers? I'm wondering if all plugins could be replaced by stubs when this type restriction is in place (different behaviour of the game aside).

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #9 on: 27 Aug 2014, 08:43 »
Can plugin functions return other types than numbers and pointers? I'm wondering if all plugins could be replaced by stubs when this type restriction is in place (different behaviour of the game aside).
AGS Script does not have other return types than numbers and pointers. Well, void too.
The rule should be that if the stub defines a non-zero return value, it is treated as a numeric type. It should not let stubs return random memory address, that is.

Basically, the stub format may be:
Code: Text
  1. function_name number | nil
  2.  
where "nil" is for both pointer and void return types. In AGS interpreter it does not make a big difference.
Stubs do not need to define actual return type in case of pointer, nor function arguments, because these parameters are known at the script compilation.
« Last Edit: 27 Aug 2014, 08:48 by Crimson Wizard »

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #10 on: 27 Aug 2014, 21:14 »
The variant with the separate file should definitely be used because...

To be clear, the engine already has the ability to look for assets in the game directory if they are not found internally in the game's data file. The option to bundle it into the game file should be present, and the engine can fall back to the game directory if needed.

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #11 on: 28 Aug 2014, 02:10 »
For the sake of example, here's a patch that does stub parsing/registering:
http://www.mediafire.com/download/njfegtxays6sb51/0001-Engine-automatic-plugin-stub-support.patch
(applied on top of "develop-3.3.1")

It won't work to the end, though. First, script interpreter must be permitted to use this stub value instead of actually calling a function, which is easy to do.
More importantly, the script exports table keeps the original function name pointer instead of allocating its own string... :-\ Works well for string literals, but will cause memory access problems with dynamic strings. This needs a rewrite too.

Radiant

  • Return once more to the Two Kingdoms!
    • I can help with publishing
    • I can help with story design
    • Radiant worked on one or more games that won an AGS Award!
    •  
    • Radiant worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #12 on: 04 Sep 2014, 21:45 »
I think this is a great idea. There is currently a lot of interest in playing AGS games on different platforms, and to my knowledge there are only about three commonly used plugins that cause problems with this (i.e. Steam, Flashlight, and Rain/Snow).

Wyz

  • anno 1986
    • I can help with making music
    • I can help with story design
    • I can help with translating
    • I can help with voice acting
    • I can help with web design
    • Wyz worked on one or more games that won an AGS Award!
    •  
    • Wyz worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #13 on: 05 Sep 2014, 01:03 »
I've got an idea maybe.
Almost all plugins run on windows so they come in the form of a dll. Inside it is also the script header which defines all functions for AGS scripts. So stub generation could potentially be automatic: extract the script header from the dll and parse it for functions to create stubs for.
If that does not work cross platform it would be possible to create a tool (a 'fake' host to the dll) that implements the RegisterScriptHeader and RegisterScriptFunction functions and uses this to generate a stub file. This tool could run on windows (or perhaps wine) so the dll would load and the whole process could be automated.
What do you think?
Life is like an adventure without the pixel hunts.

Re: Built-in plugins/stubs?
« Reply #14 on: 05 Sep 2014, 12:22 »
Radiant: there are reimplementations of Flashlight and RainSnow in the engine now so they're not the problem. But there are also games which use less common plugins.

Wyz: Normally the plugins all come with a readme file documenting the API that can be used to write stubs. Otherwise I guess there are already tools to extract the header from a dll. The rest could be done manually and the stubs could be collected somewhere.

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #15 on: 08 Sep 2014, 00:19 »
While we sort out what to do, I've created an open-source stub for the AGSteam plugin, which was the main offender in this case (as Flashlight and SnowRain have built-in workalikes).

https://github.com/monkey0506/agsteamstub

Radiant

  • Return once more to the Two Kingdoms!
    • I can help with publishing
    • I can help with story design
    • Radiant worked on one or more games that won an AGS Award!
    •  
    • Radiant worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #16 on: 27 Sep 2014, 13:28 »
Radiant: there are reimplementations of Flashlight and RainSnow in the engine now so they're not the problem.
Yes, I've intentionally been avoiding plugins for several years now (and doing it in AGS code instead) because of Mac/Linux compatibility.

While we sort out what to do, I've created an open-source stub for the AGSteam plugin, which was the main offender in this case (as Flashlight and SnowRain have built-in workalikes).
Thank you! But how does this work, am I supposed to compile this and ship it with a Linux build of a AGSteam game?

Come to think of it, it would be relatively easy to do a botted download of all games on the AGS database, and see which ones includes plugins.

monkey0506

  • SEND PIZZA.
    • Best Innovation Award Winner 2017, for his work to help AGS games reach the widest possible audience - through popular distribution platforms (Steam, Galaxy) as well as other operating systems (Android, Linux)
    • monkey0506 worked on one or more games that won an AGS Award!
    •  
    • monkey0506 worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #17 on: 28 Sep 2014, 02:48 »
A "stub" is nothing more than a replacement for cases where the plugin can't, shouldn't, or won't run. So you would not push a Steam version of your game to Valve's servers with a stubbed AGSteam plugin. You could release the same game files as a non-Steam version by replacing the full plugin with a build of the stub. This would save you having to rebuild the game files or keep track of separate versions of the game files -- just make sure that you include the full plugin when publishing to Steam (including Steam for Linux) and the stub when publishing to non-Steam sources.

Hopefully that clarifies it.

Radiant

  • Return once more to the Two Kingdoms!
    • I can help with publishing
    • I can help with story design
    • Radiant worked on one or more games that won an AGS Award!
    •  
    • Radiant worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #18 on: 28 Sep 2014, 10:50 »
Wait, as far as I can tell the windows Steam plugin works fine when there is no Steam installed (i.e. it just returns false on all function calls). Is that not the case on other systems? Would I need a different plugin on Linux or Mac systems?

Crimson Wizard

  • Local Moderator
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    • Lifetime Achievement Award Winner
    • Crimson Wizard worked on one or more games that won an AGS Award!
    •  
    • Crimson Wizard worked on one or more games that was nominated for an AGS Award!
Re: Built-in plugins/stubs?
« Reply #19 on: 28 Sep 2014, 12:04 »
Wait, as far as I can tell the windows Steam plugin works fine when there is no Steam installed (i.e. it just returns false on all function calls). Is that not the case on other systems? Would I need a different plugin on Linux or Mac systems?
The problem here, is that the "real" plugin that contacts the Steam servers cannot be open-sourced according to Steam license.
So, you may use the pre-built AGSSteam for linux, but it won't work on all Linux systems, because not all systems use same core C++ libraries. For those systems where it does not work, you'd have to provide an open-sourced version, i.e the stub.
« Last Edit: 28 Sep 2014, 12:11 by Crimson Wizard »