Integrating AGS with ScummVM?

Started by Crimson Wizard, Mon 25/12/2017 15:43:17

Previous topic - Next topic

Crimson Wizard

Quote from: morganw on Mon 25/12/2017 15:39:36Personally I would like to think there was an opportunity to have a continually developed runtime using the SCUMMVM backend, that does not maintain backwards compatibility and can have a few more contemporary features.
This is one of the possibilities. I had big doubts in the past, particularily because they did not have hardware-accelerated gfx support, which I'd like to have in AGS. But now they seem to have it, or so I heard.

There may actually be two variants here:
- develop AGS runtime as a part of their framework;
- only take their backends system and develop runtime separately (if that's technically and legally possible).

But I guess this all is a separate topic. [Done! â€"Snarky]

Snarky

#1
I recently took a personality test that said I was extremely analytical, so let me take that approach.

I assume everyone is in favor of ScummVM adding support for older AGS games. But is joining AGS up with ScummVM, so that future development of the engine happens within the ScummVM project and on top of the ScummVM core (CW's first alternative), something we want?

The benefits as I see and understand them:


  • Much better platform support, including on mobile systems (though ScummVM 2.0 doesn't seem to have an iOS port at the moment). Experienced platform experts to help out with platform issues.
  • A well-organized architecture that can serve as a great basis to refactor and improve the AGS code.
  • We get some things "for free" in terms of e.g. graphics and audio, and ongoing support/development of the lower layers of the engine stack.
  • Decisively resolves or sidesteps some long-running open questions about the future direction for AGS (e.g. whether to switch to Allegro 5 or SDL).
  • Possibly higher profile/more attractive to skilled programmers, could attract more engine contributors.
  • Similarly, might be a boost to the community?
  • Edit: One more thing. It might be the most realistic way to get AGS support in ScummVM to happen at all.

And some objections/concerns that have been raised at various points in time:


  • CJ wanted to keep the development of AGS separate from ScummVM "for the time being" (this was when he was still thinking he would be actively involved in AGS development).
  • People have had technical concerns about whether the ScummVM core is well suited to supporting AGS (in particular hardware acceleration).
  • ScummVM is licensed under the GPL: its license restrictions means AGS would have to switch to GPL as well. CJ explicitly didn't want this. (However, the AGS-specific code could be dual-licensed, offered under both the GPL and the current Artistic License.)
  • If some fraction of AGS devs/users didn't accept the switch, it might end up splitting AGS as an engine and as a community.
  • It would be something of a one-way street: once the switch is made, it would be quite difficult to go back if we regret the choice for whatever reason.
  • Discussion about whether the GPL means that plugins also would need to be GPL-licensed. In particular, concern that this would not be possible for the AGSteam plugin, since the Steam API is proprietary.
  • Concern that having to clear changes to the core engine code with all the other ScummVM groups would limit the future development of the engine, or at least add overhead to the process. (If some change would be useful to game developers but would break ScummVM on the Dreamcast, is that something we want to have to give a damn about?)
  • I believe some people were somewhat wary of ScummVM "taking over" AGS, and decisions being made by people with no background in the AGS community. In general, concerns about culture clash; that different assumptions, priorities and concerns make the two communities somewhat incompatible.
  • Lots of unanswered questions about how building/running/distributing AGS games built on top of ScummVM would actually work.
  • And last but definitely not least: It would be a fairly big undertaking. Who's going to do it?

There was also a bunch of talk about whether the GPL would mean AGS games would also have to be open-sourced and GPL-licensed, but let's nip that in the bud: No, it wouldn't (given reasonable implementation choices in the engine). And things that were roadblocks in the past are no longer an issue, e.g. AGS is now on github, all the source code that is ever going to be released has been released, a lot of work has been done on refactoring, much of the porting work was already done, etc.

Snarky

#2
My 2¢:

I'm not a huge fan of the GPL, but if we dual-license the AGS code I don't think the license should stand in the way of potential major improvements to the engine. And I believe in respecting CJ's wishes, but only up to a point: it's been six-seven years since he was actively involved in the engine development or the community (to any great extent), and I think the opinions of the current contributors weigh more heavily. For all we know he might have changed his mind, anyway.

The Steam/GPL issue might be the most serious concern, but there ought to be some kind of solution to sidestep it, surely?

I don't think the ScummVM folks fully appreciate what integrating a "live" engine and game-making community entails, and I foresee issues. However, as Crimson Wizard points out (via his option 2), we don't actually have to give up full autonomy over AGS just because we switch to ScummVM. If it turns out we can't get along, or our needs for the engine core are too different, we could always fork it (though we'd stil be stuck with the GPL for the bits we take from ScummvM).

So I guess I'm cautiously positive to the idea, if there's general support for it and the engine developers believe it's technically promising and want to do it.

tzachs

When I started working on my MonoAGS rewrite, I considered basing it on ScummVM (among other options).
The biggest turn-off for me was that the goal of ScummVM is to support old games, so newer functionality is not getting developed, and the existing architecture might be limiting in terms of what you can do with it.
Is it possible to give the game developer the ability to write custom shaders, for example?

Radiant

I think for 99% of AGS games it would be absolutely terrific to have the multi-platform support that ScummVM offers.

Snarky

Quote from: tzachs on Tue 26/12/2017 03:46:35
The biggest turn-off for me was that the goal of ScummVM is to support old games, so newer functionality is not getting developed

Definitely a concern. I wonder to what extent that's just a function of the needs they've had so far, though. With AGS integrated, that would necessarily expand the goals of the project to include supporting our needs, right?

If they continue to add support for more and newer games (the interview mentions Blade Runner, and it wouldn't surprise me too much if something like the Runaway series or the later Broken Sword games were added at some point) they'll need some of that functionality anyway.

In any case, you're right that it's important to keep in mind that it's not just about supporting the things AGS can do today, but the things we hope it will be able to do in the future.

Quotethe existing architecture might be limiting in terms of what you can do with it.

Is it possible to give the game developer the ability to write custom shaders, for example?

Good question. Hope someone who knows ScummVM better can answer. (Do you mean a custom shader as e.g. for a scaling filter, as part of the engine, or for a particular game, to achieve some special effect?)

Danvzare

If there's one thing from this thread that I can see we all agree on (although please do correct me if I'm wrong), it's that we all think it would be awesome if ScummVM at least supported AGS 2.X games and prior.

There's a bunch of potential problems if they start supporting the newest games. But there doesn't seem to be much in the way of problems if the older games are supported. Also, it would better fit in with ScummVM since it is more or less there to help support older games so they are playable on older hardware. Something which is getting increasingly more difficult with the older AGS games.

If ScummVM at least supported the older AGS games, we could finally let go of trying to keep compatibility working with those older games.

But when it comes down to supporting the newer AGS games... I'll leave you lot to debate that. I'm woefully unqualified to come to a conclusion on that topic.

Crimson Wizard

#7
Assuming we are just toying with the idea for now...

Quote from: Snarky on Mon 25/12/2017 23:23:04
  • It would be something of a one-way street: once the switch is made, it would be quite difficult to go back if we regret the choice for whatever reason.

Well, it would be difficult to go back in any case, question is how difficult. ScummVM is a modular system, game engines are run as plugins, and communicate with the backend systems (graphics, etc) via some kind of interfaces. Which potentially means that an engine may be extracted, and new implementation of those systems written. Their collection of utility classes seem more or less independent from the rest of the code, and often emulates STL, which also potentially gives a possibility to replace use of e.g. ScummVM array classes with STL ones more or less easily.

License: fuzzie told me that she was writing her AGS port under double license: GPL/Artistic, - specifically in case we would like to use her code separately. We might need to check with the current head of the team to know if that's actually permitted.

C++ level: AFAIK ScummVM relies on an old language standard to support many systems. They also have practically rewritten parts of STL, if I remember correctly that was made because the binary produced with STL were too large for some very low-end platforms. Which might be... irritating (to say the least), if e.g. someone wants to use particular class or algorithm from STL which has not yet been implemented by ScummVM team.


There is number of things required by AGS runtime, assuming it is meant for future development. We would need to find out whether they are possible in ScummVM now, or could be made possible if not (and how - from technical and organizational side).

1. Setting up the game

1.1) Wrapping the game in such way that you will only run the game directly, and not ScummVM frontend (menu, etc).
1.2) Does it allow reading and writing own game's config?
1.3) Where and how does it write savegames? Also users have requested a way to setup custom location for storing saves (which I added to AGS).
1.4) Game plugins. Not critical, since many things could be integrated into engine, but a theoretical possibility would be nice. Also - Steam.
1.5) Integration with Steam and similar clients. This is not only about plugins themselves, but also legal issues.

2. Current runtime features

2.1) Changing display mode at runtime, by the game script command.
2.2) Make it work with Hardware-accelerated backend. Which are implemented by ScummVM btw, and how? Do they support accelerated rendering per-sprite, or just speed up final drawing/scaling on screen?
For instance, tinting and lighting are performed as custom shaders by Direct3D and OpenGL renderers. Can an engine use shaders in ScummVM?

3. Integration with IDE

3.1) Would that be possible to setup a communication between Editor and runtime? Currently it's done via named pipes, which is not portable, but could be changed e.g. to net sockets.

4. Possible further features

4.1) Hardware-acceleration. What if we decide to make AGS primarily hardware accelerated engine? How the backends with only software gfx are supposed to be dealt with in such case? Will engine be able to know the capabilities of backend, and make this information at least possible to query by the game script?
4.2) Various HA features, like shaders (also mentioned above), which may be provided by game (in data package or script).
4.3) Network support, for multiplayer games.


This is just what comes to mind first.

tzachs

Quote from: Snarky on Tue 26/12/2017 12:40:07
With AGS integrated, that would necessarily expand the goals of the project to include supporting our needs, right?
Maybe, though that would require a big commitment (and mind shift) from their side.

Quote
(Do you mean a custom shader as e.g. for a scaling filter, as part of the engine, or for a particular game, to achieve some special effect?)
For a particular game.

Cassiebsg

I won't dable too much into this discussion, since I have no skills to offer to AGS development. So I'll just add my two cents here for what is worth (not much, but what the hell. (roll) )

It would be nice to be able to have support for the older games in ScummVM at the very least and up to latest stable AGS version as the best option.
I would however hate it if I needed to have ScummVM installed to be able to play future AGS games... if that is what is meant with "integrating" the source code...
There are those who believe that life here began out there...

Snarky

Quote from: Cassiebsg on Tue 26/12/2017 21:04:19
I would however hate it if I needed to have ScummVM installed to be able to play future AGS games... if that is what is meant with "integrating" the source code...

What we mean by "integrating the projects/code" is to rewrite AGS so that it "talks ScummVM": it would use functions provided by ScummVM to do everything like play audio, show graphics, handle input, save and read files, etc., as well as do some of the basic logic of the engine (e.g. pathfinding), probably. In principle, that would mean that you could run an AGS game on anything that can run ScummVM (which includes a very wide range of platforms). From that point on, AGS would be a part of ScummVM, and we would work with the other ScummVM devs to decide on future developments.

So yes, new AGS games would need ScummVM to run, but we can pretty easily imagine that each AGS game would come with a cut-down version of ScummVM with only AGS support (just like every AGS game today comes with a copy of the AGS engine), so that you wouldn't need to first install ScummVM or anything. (The PC Gamer interview with the ScummVM project lead mentions that they've done something like this for some commercial games.)

blur

I like ScumVM's user interface. It lists all your games with saved game state, lets you alter preferences per game and globally. That is very nice.
Some days ago I came across an  apparent AGS 2.5 game (A Christmas Tale DEMO by Screen 7)¹, which could not be run with the current AGS engine (3.4.1.11). I had to resort to DOSBox to play it. I liked the game.

1) thanks to agsarchives.com!

Crimson Wizard

Quote from: blur on Wed 27/12/2017 01:04:02
Some days ago I came across an  apparent AGS 2.5 game (A Christmas Tale DEMO by Screen 7)¹, which could not be run with the current AGS engine (3.4.1.11). I had to resort to DOSBox to play it. I liked the game.

If you're implying that using ScummVM would help to run that game, then there's a misconception. ScummVM won't magically make old games run. One would still have to research the game's format and program its behavior in the new engine. This could be done in current AGS engine too.

cat

Quote from: Danvzare on Tue 26/12/2017 12:57:00
If there's one thing from this thread that I can see we all agree on (although please do correct me if I'm wrong), it's that we all think it would be awesome if ScummVM at least supported AGS 2.X games and prior.

There's a bunch of potential problems if they start supporting the newest games. But there doesn't seem to be much in the way of problems if the older games are supported. Also, it would better fit in with ScummVM since it is more or less there to help support older games so they are playable on older hardware. Something which is getting increasingly more difficult with the older AGS games.

If ScummVM at least supported the older AGS games, we could finally let go of trying to keep compatibility working with those older games.

I fully agree here. There is always discussion about how tedious compatibility for older games is in the engine. Wouldn't it be much better to declare old games as legacy and let ScummVM do what it does best - run old games? Then the AGS engine could focus on new development and features, getting rid of old version support for cleaner/easier code. Migrating old projects to the newest version could be done by external scripts or tools to keep the engine/editor clean of this stuff.

To me, this seems like a win-win situation.

The only downside I see is that we don't get the platform support of ScummVM for the newest AGS games.

Radiant

Well AGS is currently at 3.4 now. Instead of involving ScummVM to support up to 2.7, how about asking them to support up to 3.2? That's also stable legacy code, and it would make it easier to upgrade the ScummVM port to 3.3 or 3.4 later if we so choose.

Snarky

#15
Quote from: Radiant on Wed 27/12/2017 09:43:11
Well AGS is currently at 3.4 now. Instead of involving ScummVM to support up to 2.7, how about asking them to support up to 3.2? That's also stable legacy code, and it would make it easier to upgrade the ScummVM port to 3.3 or 3.4 later if we so choose.

I'm pretty sure the port/rewrite started by fuzzie was for AGS 3.2, actually.

I'd caution about this "ask them to do it" line of thinking. ScummVM doesn't currently have any devs who've expressed interest in taking on the project. If they saw some commitment from our side, that might change, so it's probably better to think of it as a collaboration to some degree, where AGS devs and ScummVM devs work together to port some version of the engine.

As for which version that should be...

Quote from: cat on Wed 27/12/2017 08:57:17
Quote from: Danvzare on Tue 26/12/2017 12:57:00
If there's one thing from this thread that I can see we all agree on (although please do correct me if I'm wrong), it's that we all think it would be awesome if ScummVM at least supported AGS 2.X games and prior.

There's a bunch of potential problems if they start supporting the newest games. But there doesn't seem to be much in the way of problems if the older games are supported. Also, it would better fit in with ScummVM since it is more or less there to help support older games so they are playable on older hardware. Something which is getting increasingly more difficult with the older AGS games.

If ScummVM at least supported the older AGS games, we could finally let go of trying to keep compatibility working with those older games.

I fully agree here. There is always discussion about how tedious compatibility for older games is in the engine. Wouldn't it be much better to declare old games as legacy and let ScummVM do what it does best - run old games? Then the AGS engine could focus on new development and features, getting rid of old version support for cleaner/easier code. Migrating old projects to the newest version could be done by external scripts or tools to keep the engine/editor clean of this stuff.

To me, this seems like a win-win situation.

The only downside I see is that we don't get the platform support of ScummVM for the newest AGS games.

Well, let's turn that around:

If we think it's a great idea for ScummVM to handle legacy AGS games and have them available on a lot of different platforms today, why wouldn't we look back five years from now and think the same thing about all the games that have been created in the mean time?

In other words, if we port a version of the AGS engine to ScummVM, wouldn't it make sense to continue building on ScummVM going forward? (We could still make it a separate engine branch in ScummVM to ditch all the legacy stuff.)

Yes, there are challenges and potential problems pushing AGS development forward within ScummVM. Are the problems created greater than the ones it solves? That doesn't seem obvious to me.

I guess it depends on what our plans are going forward. If I understand things correctly, Crimson Wizard, Gurok, Alan v. Drake and other AGS engine devs are leaning towards making 3.4.x the last "continuity" version of AGS, and doing a major cleanup/rewrite for AGS 4.0. If that involves adding a whole bunch of things that aren't supported and cannot easily be accommodated within ScummVM, whether that's hardware acceleration, shaders, or anything else, or if there's some other technology stack that looks more promising (such as the engine tzachs has in development), doing it without ScummVM is probably better. In that case, the ScummVM "legacy" support should target 3.4.x, right?

But if we don't have an obviously better alternative lined up, I'm not sure I see the upside in just sidelining all the work that will be done to bring AGS to the ScummVM platform.

Radiant

Quote from: Snarky on Wed 27/12/2017 10:32:54
I'd caution about this "ask them to do it" line of thinking. ScummVM doesn't currently have any devs who've expressed interest in taking on the project. If they saw some commitment from our side, that might change, so it's probably better to think of it as a collaboration to some degree, where AGS devs and ScummVM devs work together to port some version of the engine.
Good point.

QuoteIn other words, if we port a version of the AGS engine to ScummVM, wouldn't it make sense to continue building on ScummVM going forward?
Yes, it would.

This has the added advantage that if we add coding for e.g. custom shaders, they would then be available for all games in ScummVM. I'm sure some of the ScummVM devs and/or players would appreciate this, and it makes it more of a cooperation effort between the two engines.

morganw

It is worth bearing in mind that the vast platform support may limit adding features that wouldn't work on all platforms, unless such features are optional. Perhaps if they were treated as enhancements (i.e. a game is playable without custom shaders, but will look better with them) this may align more with SCUMMVM development, but I would imagine asking for new features for games that don't exist yet may cause objections.

Perhaps the best way to proceed would be to get a list of SCUMMVM funtionality, how that maps to what is needed for AGS 3.4, and what is missing from the more modern features people are asking for.

Radiant

By the way if there's a feature that AGS does not currently support and may or may not support at some unspecified point in the future, I don't consider such a hypothetical feature a reason against joining forces with ScummVM.

Crimson Wizard

#19
Quote from: morganw on Wed 27/12/2017 13:03:04
It is worth bearing in mind that the vast platform support may limit adding features that wouldn't work on all platforms, unless such features are optional. Perhaps if they were treated as enhancements (i.e. a game is playable without custom shaders, but will look better with them) this may align more with SCUMMVM development, but I would imagine asking for new features for games that don't exist yet may cause objections.

To be fair, that's the same issue as any engine may have when running on low-end systems. This is why having a way to turn off certain effects is important.
For instance, our OpenGL renderer has Tinting done through shaders, but these shaders require particular version of OpenGL to be installed on the system. This is why I programmed it so that if that version not present, the tinting functions simply do nothing (instead of crashing) if shaders could not be run.

In a more serious case it is usually up to game creator to switch certain things (like provide lo-res versions of gfx).

SMF spam blocked by CleanTalk