Integrating AGS with ScummVM?

Started by Crimson Wizard, Mon 25/12/2017 15:43:17

Previous topic - Next topic

Snarky

Quote from: Radiant on Wed 27/12/2017 13:27:43
By the way if there's a feature that AGS does not currently support and may or may not support at some unspecified point in the future, I don't consider such a hypothetical feature a reason against joining forces with ScummVM.

I agree up to a point. Part of the reason to switch to a new architecture, after all, is to enable things that aren't currently possible. If it's a feature we're pretty keen on, and that is realistic to do if we pick some alternative approach but not with ScummVM, I think it would offer an argument against that approach.

morganw

#21
Quote from: Crimson Wizard on Wed 27/12/2017 13:44:19
To be fair, that's the same issue as any engine may have when running on low-end systems. This is why having a way to turn off certain effects is important.
Yes, although in the case of SCUMMVM they do not seem to differentiate between platforms for game support (compatibility charts do not say where a game will not run on a particular platform). If the mindset is that all platforms are equal and then they are approached with a wishlist of features that would only work on a small subset of those, that could be fundamentally against what that project is trying to do. I may be entirely wrong though (and at some point old games will be games that use custom shaders).

Radiant

Quote from: Snarky on Wed 27/12/2017 14:06:50
Quote from: Radiant on Wed 27/12/2017 13:27:43
By the way if there's a feature that AGS does not currently support and may or may not support at some unspecified point in the future, I don't consider such a hypothetical feature a reason against joining forces with ScummVM.
I agree up to a point. Part of the reason to switch to a new architecture, after all, is to enable things that aren't currently possible. If it's a feature we're pretty keen on, and that is realistic to do if we pick some alternative approach but not with ScummVM, I think it would offer an argument against that approach.
What you describe wouild be a feature that we will support at some specified point in the future :cool:

morganw

Quote from: morganw on Wed 27/12/2017 14:43:16
Yes, although in the case of SCUMMVM they do not seem to differentiate between platforms for game support (compatibility charts do not say where a game will not run on a particular platform).
To argue against myself, there is a table which indicates some restrictions per platform:
http://wiki.scummvm.org/index.php/Platforms/Overview
...so hopefully if this goes in the direction of 'must have feature x', they might be willing to only support it on platforms which had access to recent compilers / frameworks. There are also built-in debug tools, which look useful, including one which can record and playback game actions to check for breakages.

The example of potentially needing custom shaders is quite interesting, as Allegro 5 looks to have custom shader support built-in, whereas SDL/SDL2 require that it is handled manually per renderer, or you use some middleware to do it for you.

snover

Hi all, ScummVM team developer here. I was notified about this thread and figured that I might offer some feedback and be available to answer any other questions you might have. There is a lot of great productive discussion occurring here already, so in this initial message I am going to just try to address some of the broader concerns I see raised, instead of trying to quote-reply to every thought. I will do my best to get more specific in response to any questions you target at me, since there are some good specific questions which have been raised but maybe not satisfactorily answered, so let me know which of those you would like me to address.

ScummVM's core system is designed to be pluggable with different graphics backends. Right now SDL and OpenGL are the two which are available, and I know a couple of people have been working on creating another one using Qt. I would not rate these APIs as being perfect or immutable. There is definitely room and reason to make improvements. I and a couple other active team members have been working on getting rid of some of the worst brokenness of the core graphics system, and having more interest and engagement from others would be a huge plus here. I would really like to make OpenGL the default renderer, which means doing some more work to bring it up to feature parity with SDL, and, I don't have the time or energy to do it myself.

One of my medium-term (3-6 month?) aspirational goals is to merge ResidualVM (the 3D game counterpart of ScummVM) into ScummVM proper. Assuming that this happens (informal feedback I've requested from team members on both sides has been all positive so far), there will be more feature availability for 3D games, and we do already have platform restrictions (as noted in one of the last posts) so there is no concern in my mind in that regard. It is obvious to me that the present and future state of games computing involves GPU programming, so ScummVM is going to need to be able to facilitate this sooner or later anyway or else never be able to support anything after the early 2000s. Having more developers on board with this vision and desire will make it easier to realise such changes.

ScummVM contains portions of code licensed under less restrictive licenses than GPL, so alternative licensing for some features is probably fine so long as the chosen license is GPL-compatible. I am not very familiar with Artistic License, but what I have read seems to indicate that version 2.0 is both GPL-compatible and also includes a relicensing clause which should allow for little trouble with any older code carried over (assuming AGS is using AL 2.0). I don't know what this would look like for an entire engine, I am at least not opposed to exploring reasonable solutions.

With regards to the non-library code and ports baggage, these are points of contention for me already and I plan on making a serious proposal to try to get some better goals in place which prioritise improving the end-user experience for the majority of our users, and improving the developer experience by offering more modern features to engage and retain developers, over choosing to have the absolute largest number of ports. When I look at our latest 2.0 release statistics and see we have 9787 Windows downloads against 30 Dreamcast downloads, it's pretty obvious to me that we are not serving anybody with the current approach. Having a new group of developers saying that they really want to help make ScummVM better, and are quite reluctant do so if they must be saddled with such legacy baggage, does help to strengthen the argument that the project needs to reorient.

I am personally also not opposed to the idea of using ScummVM as a base for some newer engine developments. I don't see this as being any different than development of brand new engines or enhancements to existing engines, as they all involve the same amount of risk in terms of adopting technical debt. So long as there is a reasonable process for change management, and the engine doesn't get into another state where old games become broken due to backwards-incompatible changes (= bad user experience), I'm willing to support such an initiative, especially if it means we'll get some more folks willing to put in the work to maintain the core system.

With all of these things, caveats apply: I am not the one in charge of ScummVM, so I cannot make any guarantees. I can only state my own desires and goals for the project direction, and express my openness in working with you to figure out if this idea makes sense, and if so, how to move forward.

Thanks for reading, and I look forward to your feedback.

morganw

To cross reference the threads, I am suggesting this as a potential way forwards. I wouldn't like to claim that I'm some sort of official spokesperson, but I am happy to suggest what I think is a workable solution (everyone should feel free to point out issues I am missing... :))

eri0o

Being a user of AGS, have to say that switching to ScummVM and having the portability it offers, along the possibilities of SDL (like joystick support or building for JS with LLVM Emscripten) is something that would be huge - note: I don't really know if any of this is true. I also really would like to be able to package the game for platforms so that the customer buying the game would never need to know any of these things.

Crimson Wizard

#27
Quote from: morganw on Wed 03/01/2018 23:44:22
To cross reference the threads, I am suggesting this as a potential way forwards. I wouldn't like to claim that I'm some sort of official spokesperson, but I am happy to suggest what I think is a workable solution (everyone should feel free to point out issues I am missing... :))

In another thread you mentioned asking fuzzie about the state of the port. The code in her repository did not change much since 2014 when I had an exchange of emails with her.
Now that I found them I see that it was not "almost complete" as I remembered for some reason.
This is what she said about it:

Quote
So, in the long term we'd definitely like to at least be a backwards
compatible interpreter. I was hoping that it would make it easier for you or
other upstream devs to do major refactoring of the code, eventually.

Valve don't provide any supported way to use SteamWorks with GPLed code
such as ScummVM. It isn't a requirement but it means no (easy) way to do
achievements on Steam. We have devs happily using ScummVM on iOS though,
for example. I carefully licensed all my own code under an AGS-compatible
license, in case any of it it was useful to you, but of course the rest of
ScummVM is GPL (and we can't change that).

<...>

So, the ScummVM 'AGS port' is really my attempt to rewrite AGS completely.
It dates from before your refactoring work, so it is (as you said) based on
JJS's code. But I decided that rewriting it would be a better idea than
refactoring it <...>

There is a short TODO list on our wiki at
http://wiki.scummvm.org/index.php/AGS but it's more complicated than that.
Especially things like the graphics code are still very incomplete.

The save/load functionality (which is complicated because I took a very
different direction for the scripting engine, as you know) is the most
important bit, because without it, it's difficult to debug very far into
games.

If you're interested in taking a look then I have a diff somewhere of my
local tree (which is needed to actually make most games work, mostly
stubbing some of the missing script functions). I'll try and send it to you.

Unfortunately she never sent anything to me, and our discussion ended like that (well, I got busy with other things anyway).

The last time I tried running ScummVM port I also found some mistakes in other parts of the code. I believe the first thing it needs is a checklist of functionality to test (maybe list of games).

morganw

Thanks for finding the messages, the to-do list page is still working so at least that gives a rough idea of what is missing. Unless anyone really thinks I should not, I'll try to contact her (starting with an apology for any previous hassle) to ask if she would be interested in looking at it again, or perhaps just providing some pointers if anything wasn't straight-forward. I'll also try building it at the weekend, to see what would be involved in testing it.

Radiant


Snarky

FYI, I already sent fuzzie an apology on behalf of the AGS community for any hostility she may have experienced last time around. I haven't heard back, and I'm not sure the email addresses I used are still active.

morganw

It looks like it was updated in 2014, but it still build okay following the ScummVM build instructions (although the backend it built with was SDL, not SDL2, but I think this is just because the default renderer was still SDL in 2014). Out of the three games I tried, one got stuck on the menu screen (I think due to an issue with GUI buttons), one crashed (I think because it called SetGameOption), and one very recent game was not detected as an AGS game. This might be because I am missing the part of the code which would add function stubs into older games, but I think finishing this port may still be promising option, purely because the code is commented all the way through and missing/rough functionality is labelled.

Is there any engine documentation anywhere which could be used to help with identifying missing features, issues with what is already done, or give details of the game data format? If I can manage to get my bearings with everything (both AGS and ScummVM) I would be interested in trying to work on this.

eri0o

morganw! If you can create a fork of this version of the engine, share on github and create a thread somewhere in the forums so people can keep an update on what you try that works and what does not, it would be cool!

Crimson Wizard

Quote from: eri0o on Mon 08/01/2018 00:46:06
morganw! If you can create a fork of this version of the engine, share on github and create a thread somewhere in the forums so people can keep an update on what you try that works and what does not, it would be cool!

We could even create a fork in "ags" community account. I think commit permissions are per-repository.

morganw

With the latest master merged against it it doesn't build, so I'll try to get working with the latest changes. Hopefully by doing that I'll pickup some ScummVM and AGS workings. I'm happy to shift everything into the AGS account if anyone else would also like to look at it.

morganw

It didn't need much doing to it to build, after merging with with their master branch. I have found it is very hit and miss and which games will open though, it couldn't find the game data in a 3.2.1 game, and exited trying to load a font for a 3.4.0 game. According to the checks that it makes, version 3.2.1 was the highest version it supported. I guess it is also misreading the version number as I'm not seeing the error message that should appear for anything newer than that.

In terms of older game data being patched, in this all done on load?

i.e. in functions like this one
Code: c++
MainGameFileError  ReadGameData(LoadedGameEntities &ents, Stream *in, GameDataVersion data_ver);
// Applies necessary updates, conversions and fixups to the loaded data


I guess that step one will be checking the existing data reader...

Crimson Wizard

#36
Could we possibly move the tech questions on code work to development forum? I feel it goes far outside of original topic.

ScummVM port most possibly does not support anything above 3.2.1. Since 3.3 and higher support would require adding not only game data, but also functional code (and our codebase may be very different), IMHO it may be better to aim 3.2.1 and lower versions support for starters.

I assume our "master" branch is used for reference, of course, because it has most percentage of code refactored.

Quote from: morganw on Thu 11/01/2018 23:06:01
In terms of older game data being patched, in this all done on load?

i.e. in functions like this one
Code: c++
MainGameFileError  ReadGameData(LoadedGameEntities &ents, Stream *in, GameDataVersion data_ver);
// Applies necessary updates, conversions and fixups to the loaded data


Yes, ReadGameData loads and UpdateGameData does some conversions. LoadedGameEntities is used to keep some temporary data in non-final state until all format checks are passed.

Another thing are numerous behavior hacks, which are performed realtime under conditions like -
Code: c++

if (loaded_game_file_version <= kGameVersion_262)
{
    // ....
}


morganw

Do you want everything shifted to the Github account, and then I'll open issues if I have any more questions?

Crimson Wizard

Quote from: morganw on Thu 11/01/2018 23:38:01
Do you want everything shifted to the Github account, and then I'll open issues if I have any more questions?
Sure, that's an option.
Do you mean your account, or "https://github.com/adventuregamestudio"?

morganw

If you can fork fuzzie's scummvm repo to the AGS account and give me write access, I'll just reapply the changes there. Also that gives some visibility over what I'm doing, by people who might notice what I've misunderstood.

SMF spam blocked by CleanTalk