Porting engine to ScummVM

Started by dreammaster, Thu 19/11/2020 18:42:56

Previous topic - Next topic

dreammaster

Hey all. I'm a long-time ScummVM developer, and am currently (hopefully :) ) in the finishing stages of the current engine I'm working on. So I've been starting to cast around for a project to work on over the Christmas holidays, and thought it might be time to revisit the question of integrating AGS support into ScummVM. The previous threads in years past got somewhat contentious at times, but my understanding was that there was certainly overall support from the AGS community for older games to be supported at the very least. Though technically there are two options, with both the community codebase and fuzzie's earlier attempt, it seems like the community codebase is the way to go in that it covers a greater range of historical versions.

As such my question before I start such an effort, is whether there would be any issues with doing a conversion now. Particularly, I'm wondering what codebase point would be best to be used. I see that there's an ags3 branch, but it seems to be out of date compared to master, which has later 3.5 releases. Would the most 3.5 release tag be a good point to start for a ScummVM conversion? Though I'm also wondering if there's a near-term plan for a 4.0 release? Particularly if that ends up significantly changing the AGS format, would it be better to forget about it for now, and wait for the last 3.x release prior to working on a conversion? Something like that could end up being ideal, since there'd be less issues with a ScummVM implementation trying to play catch up with further commits and sub-releases on the main AGS repository.

Any guidance would be appreciated.

Crimson Wizard

#1
Hello, there's a close to complete (or so I believed) port to ScummVM here: https://github.com/adventuregamestudio/scummvm/tree/ags

It was started by Alyssa Milburn (aka fuzzie) as you know, then amended by several people in different time, including some of our community. We decided to bring a fork to our organization account hoping to finish it maybe and provide an alternative for running old games, but it was never done 100%.

I am unaware of it's exact status, IIRC the current goal was to cover AGS 3.2.1 API, but some things are still missing. I recall that the biggest missing feature was save/restore function.

Quote from: dreammaster on Thu 19/11/2020 18:42:56Though technically there are two options, with both the community codebase and fuzzie's earlier attempt, it seems like the community codebase is the way to go in that it covers a greater range of historical versions.

I believe it might be possible to copy necessary changes over to existing scummvm port, because every game logic in current ags code is under version check, so it's not hard to find parts related to this or that API. But a research may be required first to know what to do with some recent additions such as custom viewport/camera features and how to reimplement them in ScummVM (guess you would know better since you are a ScummVM developer).

Quote from: dreammaster on Thu 19/11/2020 18:42:56
As such my question before I start such an effort, is whether there would be any issues with doing a conversion now. Particularly, I'm wondering what codebase point would be best to be used. I see that there's an ags3 branch, but it seems to be out of date compared to master, which has later 3.5 releases. Would the most 3.5 release tag be a good point to start for a ScummVM conversion? Though I'm also wondering if there's a near-term plan for a 4.0 release? Particularly if that ends up significantly changing the AGS format, would it be better to forget about it for now, and wait for the last 3.x release prior to working on a conversion?

This is a very good question... Trying to explain in short:
- "ags3" is a forgotten branch, as in, I forgot to delete it apparently (we had number of hickups with planning). Conceptually, current master is ags3, as it continues full backward compatible support.
- There were plans to have small update in there, like 3.5.1, with certain API additions, but recently I leaned towards dropping that. The situation is explained here: https://github.com/adventuregamestudio/ags/issues/1121 in the ticket and comment(s).
- In shorter words, what we would like to do is this: 3.5.0 is the last "ags3" version (as in - version with full backward support). From now on this branch will only have fixes and perfomance updates, but no API, data format and game logic updates whatsoever.
- "ags4" becomes master, we make breaking changes there (cutting old stuff, redesigning some script commands etc) and this is where we do most work from now on. note "ags4" is just a meme at this point, as version number is currently not decided (imho differences are not enough to call it 4.0).
- we are currently in a big process of replacing backend from Allegro 4 to SDL2 in the master, if successful this will be AGS 3.6 with full backward support, but because this is only backend & library changes it's not needed for ScummVM port.
(We will also use SDL2 in the future "ags4" branch, ofcourse)

I was just planning to make a forum post about all above, which might still do in a few days.

dreammaster

Thanks for the informative reply. It sounds like it'll be more effective for me to work on the AGS3 "final version", after the switchover. Whenever that happens. I'm also familiar with SDL, and not at all with Allegro, so it would certainly make it much easier to make a port for ScummVM afterwards. Guess I'll hunt up something else to work on over Christmas.

Crimson Wizard

#3
SDL2 port did not change the game code alot (less than I even expected), as most system actions are kept behind interfaces, and we are still having parts of the stripped allegro 4 library embedded into the engine: primarily memory bitmaps and software drawing/blending functions, because SDL2 does not have a good alternative for these.
The biggest changes were to the window management and audio backend, latter was completely rewritten, but again the game logic code remained untouched.

Snarky

Quote from: dreammaster on Fri 20/11/2020 01:48:16
Thanks for the informative reply. It sounds like it'll be more effective for me to work on the AGS3 "final version", after the switchover. Whenever that happens.

Hmm, AGS 3.5.0 is already the current released version, and as Crimson Wizard said that is the "final" AGS3 version (according to plans), except for bugfixes and performance improvements and such.

Maybe you could save a little bit of time/effort by waiting for the SDL port, but given that there already is a mostly-complete ScummVM port by fuzzie, and that the bits that are missing are not (as I understand it) related to Allegro/SDL, I would encourage you to just go ahead and do a port of AGS 3.5.0 with that as a basis.

It's great to see some renewed interest in a ScummVM port, and it would be a shame to let the opportunity pass by.

dreammaster

I played around with the fork of fuzzie's work on adventuregamestudio/scummvm, and the results are just too poor for my liking. Of the games that had detection entries, one crashed trying to load data, one threw a wobbly about slider value, and the final one did manage to start showing the intro sequence but then crashed with "FollowCharacter unimplemented". It's just unfeasible for someone like me, with no prior experience with the AGS engine codebase, to try and debug and update a semi-restructured/different project. It'll result in endless frustrations for me. Far better to have an almost identical codebase based on the current community engine, where the backend Allegro or SDL calls have a ScummVM stub layer. That way if something goes wrong, I'll be able to compare execution between the two if something goes wrong in the ScummVM version. That kind of thing has been helpful in the past. For example, I once hacked DosBox to do output of port values written to the sound card in one of the Tsunami Media games, and doing likewise in my sound driver code in ScummVM, I was able to do a diff, and easily track down where discrepancies in my code started happening. But I digress. That there aren't any more major changes planned for 3.5.. it may indeed be the case that there's not much to be gained waiting for any SDL changes. I'll have a play around with the codebase, familiarize myself, and see how I go.

Crimson Wizard

The biggest problem of the current codebase in ags engine is that it's very dirty, and fuzzie did a big job refactoring it into cleaner code. I think I must mention this. Over time we were gradually replacing old code with new one, especially when adding more features, but there are still places that look just terrible.

dreammaster

Quote from: Crimson Wizard on Sat 21/11/2020 16:01:05
The biggest problem of the current codebase in ags engine is that it's very dirty, and fuzzie did a big job refactoring it into cleaner code. I think I must mention this. Over time we were gradually replacing old code with new one, especially when adding more features, but there are still places that look just terrible.

That makes sense. It's a trap I've fallen for myself in the past - back when I was converting a previous project to work on ScummVM I started making too many changes, then when the time came to debug it when I got it up and running, it became difficult to track down issues. I guess it was even harder if things were actively being changed in the source codebase. These days when I'm converting projects I initially tend to try and keep the original code as unchanged as possible, and simply create a thunk layer to emulate the original libraries. Then only worry about any cleanup once I've properly gotten everything to at least run. Over the last few days I've created a ScummVM skeleton engine, and seen how much code I code import and get to compile. It's been fun, although frustrating in that I still need to put it aside to finish my current project. But it seems like once that's done, I likely will experiment over the Christmas holidays to see if I can get it working for ScummVM.

Snarky

Hey, that's great to hear! Really hope it works out. Interesting to hear about your approach to converting projects; that makes a lot of sense to me.

I don't know whether it was intentional or a typo, but I also learned a new term.

dreammaster

Quote from: Snarky on Mon 23/11/2020 19:41:49
I don't know whether it was intentional or a typo, but I also learned a new term.

Yep, that was intentional. Call it a thunk, bodge, or shim layer. Heh, at times previously at work we called some of our nastier interface code the "Wibble" layer, from the Rowan Atikinson Black Adder series where he pretended to be insane by sticking pencils up his nose. :)

Joseph DiPerla

I just tried the latest Daily snapshot of SCUMMVM and it now supports AGS games. But I think it is limited to like up to 2.73 or something like that for now. https://www.scummvm.org/downloads/#daily
Joseph DiPerla--- http://www.adventurestockpile.com
Play my Star Wars MMORPG: http://sw-bfs.com
See my Fiverr page for translation and other services: https://www.fiverr.com/josephdiperla
Google Plus Adventure Community: https://plus.google.com/communities/116504865864458899575

eri0o

#11
This was the PR that introduced AGS in ScummVm codebase

https://github.com/scummvm/scummvm/pull/2738

And these are all the additional PRs there

I believe it should map any AGS 3 released game or at least one of the PRs specifically mentions Unavowed. I haven't actually looked into the code.

Edit:

I now looked a bit into the commits, there are some interesting things going on but I can't say I immediately grasped anything to be ported back, there are a lot of changes. I noticed a lot of troubles happened with 32 and 64 bits. I don't remember having trouble in the engine with 64 bit MacOS, but I haven't yet put my hands on the new M1 chips.

Danvzare

Quote from: Joseph DiPerla on Mon 08/02/2021 23:02:40
I just tried the latest Daily snapshot of SCUMMVM and it now supports AGS games. But I think it is limited to like up to 2.73 or something like that for now. https://www.scummvm.org/downloads/#daily
By that, do you mean that it doesn't play any game made before 2.73, or that it doesn't play any game made after 2.73?

Joseph DiPerla

Doesn't play games made after it.
Joseph DiPerla--- http://www.adventurestockpile.com
Play my Star Wars MMORPG: http://sw-bfs.com
See my Fiverr page for translation and other services: https://www.fiverr.com/josephdiperla
Google Plus Adventure Community: https://plus.google.com/communities/116504865864458899575

Crimson Wizard

Quote from: Joseph DiPerla on Tue 09/02/2021 20:20:59
Doesn't play games made after it.

This sounds strange, because afaik they took 3.5.0 code, and they recently added detection info for Unawoved. Maybe some features are not wired up yet? But I did not try that out myself.

Joseph DiPerla

#15
I downloaded two random games that had downloadable links.. Chick Chaser and Out of time. Chick Chaser seemed fine. I then loaded up Out of time and it couldn't detect the game. I think, as far as I could tell, they tell the SCUMMVM Launcher the titles of each game manually in their code.

EDIT:
https://github.com/scummvm/scummvm/blob/bdf9c0f3118cdc7cf43f7520f0d6675fb11b073e/engines/ags/detection_tables.h
Joseph DiPerla--- http://www.adventurestockpile.com
Play my Star Wars MMORPG: http://sw-bfs.com
See my Fiverr page for translation and other services: https://www.fiverr.com/josephdiperla
Google Plus Adventure Community: https://plus.google.com/communities/116504865864458899575

Crimson Wizard

Quote from: Joseph DiPerla on Tue 09/02/2021 20:51:00I then loaded up Out of time and it couldn't detect the game. I think, as far as I could tell, they tell the SCUMMVM Launcher the titles of each game manually in their code.

Yes, that's the real problem, unlike with ags engine they require a hardcoded detection info (exe filename and md5 summ). Guess this worked well with ancient engines where all the games were released 10+ years ago. But with AGS this will become a nuisance as more games are released in real time.
I explained how AGS engine detects games to the programmer who ported it, but idk if they are allowed to implement separate detection methods in scummvm. If they could, maybe they could've just parse the folder for ags data and read game titles from there.

Danvzare

Quote from: Crimson Wizard on Tue 09/02/2021 20:55:12
Quote from: Joseph DiPerla on Tue 09/02/2021 20:51:00I then loaded up Out of time and it couldn't detect the game. I think, as far as I could tell, they tell the SCUMMVM Launcher the titles of each game manually in their code.

Yes, that's the real problem, unlike with ags engine they require a hardcoded detection info (exe filename and md5 summ). Guess this worked well with ancient engines where all the games were released 10+ years ago. But with AGS this will become a nuisance as more games are released in real time.
I explained how AGS engine detects games to the programmer who ported it, but idk if they are allowed to implement separate detection methods in scummvm. If they could, maybe they could've just parse the folder for ags data and read game titles from there.
ScummVM doesn't seem to have any problem with detecting AGI fan games. So it's definitely possible for them to implement a more general detection system.

Crimson Wizard

#18
Quote from: Danvzare on Thu 11/02/2021 16:30:09
ScummVM doesn't seem to have any problem with detecting AGI fan games. So it's definitely possible for them to implement a more general detection system.

I dont know, from a quick look they have got fangames in the same kind of hardcoded table:
https://github.com/scummvm/scummvm/blob/master/engines/agi/detection_tables.h#L678
On the other hand there's some code that checks the presence of files, so maybe:
https://github.com/scummvm/scummvm/blob/master/engines/agi/detection.cpp#L158


SMF spam blocked by CleanTalk