The future of the AGS engine: STALEMATE!

[Calin Leafshade]

Yes, that was my thinking.

AGS 4 would be free to break compatibility but we *can't* abandon the 3.x branch and just go with AGS 4 because that would disenfranchise essentially everyone with an active project.

What we should do is work on cleaning up the existing code and port that. *Then* we can use that fully stable, ported, open library friendly version in AGS 4 in which we (carefully, remember modules and stuff) break backwards compatibility in order to improve things.

We shouldnt be too rash when it comes to breaking compatiblity. In fact I cant think of a single good reason to do so off the top of my head. What exactly is the problem with the current system that cant easily be converted?

[Radiant]

As there are 3 strong candidates,

I think a triumvirate would be a good idea in principle, but which three candidates are we talking about exactly?

Back to the engine discussion - what do folks feel would be more important from here on then,

In my opinion, portability (to Mac/Linux/Phones/whatever) should be a priority.


Just to throw a wild idea out there: it has been suggested in the past that AGS joins forces with SCUMMVM, which is a big open source engine which supports, among others, Sierra (AGI and SCI) and LucasArts games. They have a pretty big community with technical skills and support, and may be interested in helping us out. Would that be useful?

[tzachs]

It doesn't have to be either maintaing backwards compatibility completely or breaking it completely.
How it usually works, is when you advance to a new version you may break compatibility only with the oldest supported version of the bunch.
So for instance, with AGS 4.0, we may break compatibility with AGS 2.7, while still supporting AGS 3.0.
When we create AGS 5.0, we'll stop supporting AGS 3.0 while supporting 4.0, and so on.

It's like playing a game of Snake. When the head advances, so does the tail. If you decide to eat something and make you body bigger (i.e support more versions), the snake will get harder to control, and eventually it will crash.

[Wyz]

The Snake Tzach is referring to is Microsoft Windows, there is still 16bit compatibility stuff in there and that is really well let's say weird. The deal with breaking compatibility is that is does not nescesarry have to be permanent. One of two things might happen:

• The newer branch will get a 'compatibility mode' that emulates the old engine
  
• We stuff the old branch into scummVM

[monkey0506]

Right. I think there's a lot of misgivings about "breaking backwards compatibility". It's not like we'd be breaking things for the sake of breaking them. The point behind saying we're going to break it is so that we can refactor the code without having to worry about every single in and out and up and down of the way the current engine works having to be defined exactly the same. Largely the compiler should be able to remain backwards compatible with a new engine, just compiling the existing code and resources in a way that the new engine can understand.

Once we have a clean and consistent codebase, then certainly we could look at fixing what we broke, but as statically as AGS is coded, changing even a single line of the engine could break something, somewhere. That's why we say we're breaking backwards compatibility. Certainly we don't want to change or remove specific existing features of the program, but things have to change if we're going to make any headway. The source has been available far too long with far too little done about it.

One thing that may be likely to break is the ability to run older compiled games using the new engine. Or it may not, depending on if we can read the data that's appended to the older engine in a way that the new engine can make use of it. It may be something that we break now and fix later.

Keep in mind that even CJ broke backwards compatibility when it was appropriate to move the engine forward. And as widely used as it may still be, AGS 2.72 is not officially supported any more (CJ himself said so several times). He did put a lot of effort into making sure that he maintained it wherever possible, but there are times when it's easier to break it now, cutting the proverbial umbilical cord, and then fix it later. CJ understood that.

AGS 4 should not, IMO, guarantee backwards compatibility of the engine with any other AGS version, at least not in the initial releases. Again, this is something to consider. CJ had a well established method of implementing new features and fixes (including ones that broke things!) and then releasing minor builds for testing. The AGS 2.8 alphas were essential to the creation of what we have today as the AGS 3.x branch.

We should focus first and foremost on cleaning up the code. We need something clean and consistent, regardless if it breaks everything. We can work back from that point to fix what needs fixing, and through alpha and beta releases, get a concept of what it may be time to let go of.

[JJS]

The most important thing should be refactoring the current source.

- moving code out of headers (acroom.h!)
- splitting ac.cpp into several files
- renaming variables in a consistent manner
- possibly removing global variables
- enforcing one consisted coding and naming style
- identifying obvious "dirty hacks" and implementing them the right way

This is not so easy because the code is so intertwined and because of the generous use of global variables a lot of functions are not stateless.

edit: Well, basically what monkey_05_06 said.

[Snarky]

Code refactoring is something that should definitely be tackled going forward, but on the other I don't think it should stand in the way of short-term releases. I would propose a release plan as follows:

3.3: Q2 2012 - Very close to the current version, with high-priority bugfixes and features (mainly HD resolution support and some editor tweaks), with ports to Mac and Linux. Should generally be backwards compatible with 3.0, and to some extent with 2.7
3.4: Q4 2012? - Code refactored and libraries replaced as appropriate. Performance and API improvements (including Unicode support, please). Fully integrated ports to Mac, Linux, Android and iOS. Backwards compatibility with 3.0 maintained where possible
3.4.++: 2013 - Continued support of 3.x branch
4.0: 2013? - Futuristic, pie-in-the-sky version of AGS. May break compatibility with 3.x in major ways. Basically, anything that would require a complete rewrite, like merging with ScummVM, or rewriting the editor to be platform-independent

This doesn't mean that people couldn't start working on AGS 4.0 today (though it might be best to get the code refactoring done first), just that the goal for the near term, in my opinion, should be getting a version of AGS that fixes the most important bugs and missing features that people have wanted for years, but haven't been able to do anything about until now. Many of these have already been implemented in various unofficial/experimental branches, they just need to be merged together, tested, bugfixed (if necessary) and given an official release. I posted many of them somewhere else, but I'll repeat them here:

* Mac and Linux ports (and Android and iOS if possible, PSP if available)
* Fix DirectDraw/Direct3D issues so AGS runs reliably on Windows
* Support for widescreen and HD resolutions
* Fix music stuttering on room change bug
* Savegames breaking on basically any game update (if possible to fix without major rewrite)
* Other possible high-priority requirements for commercial games, based on input from Wadjet Eye (Dave/Janet), Himalaya (AGD2) and Skygoblin (theo)

Would the rest you agree with something like this?

[Wyz]

That is pretty much how I envisioned it yes.

[Shane 'ProgZmax' Stevens]

So for instance, with AGS 4.0, we may break compatibility with AGS 2.7, while still supporting AGS 3.0.
When we create AGS 5.0, we'll stop supporting AGS 3.0 while supporting 4.0, and so on.

Yes.  This leaves a clear way for users to 'upgrade' even if it requires having a few versions of ags to do so.

As far as issues to tackle, let's have a look at Snarky's list again:

* Mac and Linux ports (and Android and iOS if possible, PSP if available)
* Fix DirectDraw/Direct3D issues so AGS runs reliably on Windows
* Support for widescreen and HD resolutions
* Fix music stuttering on room change bug
* Savegames breaking on basically any game update (if possible to fix without major rewrite)
* Other possible high-priority requirements for commercial games, based on input from Wadjet Eye (Dave/Janet), Himalaya (AGD2) and Skygoblin (theo)


To that I would definitely add the ability to toggle between fullscreen and windowed mode on the fly and to be able to resize the window (stretch to fullscreen). 

(Note:  I'm not saying that the above is an agreed upon list, I'm just offering a what if scenario below)

As far as priority, I think it should be a priority to work out the disparities between ddraw and Direct3d first, then tackle usability improvements like savegames and windowed mode (as well as the music issue), perhaps even work in the widescreen and HD stuff BEFORE attacking the ports.  The reason being that the compilers will need modification each time we make an update, so stability and usability should get priority consideration.  Otherwise, consider the compiler as the initial focus; each time one of these fixes or features are worked in, compiler update.  It just makes more sense to me to get a significant amount of work accomplished so the compiler end doesn't need incremental modification.  A case can certainly be made for a separate group dedicated to producing a compiler for other platforms that is up to the level of the current official AGS release.


Since Dave has withdrawn his interest in management, I can't really think of anyone who I've worked with and could vouch for that hasn't already refused, so I'll just wait and see what happens.  Not saying my endorsement is a big deal, I just don't like to put my name behind someone I can't personally vouch for. 

[Snarky]

As someone already said, there are different degrees of compatibility:

4. The new engine will run games compiled in the old version
3. You can load old games into the editor and recompile them for the new engine without any edits
2. You can load old games into the editor, and it will automatically update them into the new format
1. You can load old games into the editor, but you might have to change some parts manually to make it compile
0. You can't load old games into the editor (incompatible)

And of course, in each of these cases the compatibility may be unintentionally buggy to a greater or lesser extent.

What level of backwards compatibility are we talking about here? Personally I would say that the next version (3.3 in my proposal) should aim for level-4 compatibility with 3.x and level-1 compatibility (which I think is what 3.2 has now?) with 2.7, while 3.4 should aim for level-2 compatibility with 3.x and level-1 compatibility with 2.7, and 4.0 should aim for level-1 compatibility with 3.x only.

[Stee]

We probably have a different perspective on things than most folks here.  We approach things from a customer perspective (i.e., making things better/more fun for the player) than a developer perspective (i.e., making the games easier to make).  While we definitely would like to improve the engine/editor, usability for the players would be our main priority. There are a number of common technical complaints we get almost every day from our customers and affiliates, and we can only shrug and say "Sorry, it's an AGS thing." 

-Dave

My vote if you want to call it that, goes to Dave. He's the only guy in here that's been suggested that I can actually see continued development on AGS happening under. There a lots of qualified people in here that I'm sure could do the job, but with no disrespect to any of you, I think you underestimate the task at hand, and I doubt many of you will be able to put the required time into the project.

Dave's job is essentially AGS. Bar one game in his collection, all his titles are made in AGS (I believe). He makes his living from it, and if theres anyone who is going to be more determined to get stuff done, its going to be someone who has that type of interest in it. Someone who requires it to put food on the table. Dave also has a lot of resources at hand and the backing of a lot of AGS vets.

The stuff Dave wants to work on is probably not the stuff we want to get done as "priority" (pretend those quotes are my fingers quoting), but I would rather see things get done than see the project rot into abandonment which is the likely alternative. The guy has a track record of getting stuff done, and that's the guy I want working on AGS more than anything.


Dave, I hope you reconsider.


PS: Can we get a seperate forum for all this AGS Engine Dev stuff?

[Dave Gilbert]

Hi everyone!  Since my last post on this thread, a number of you have emailed me or MSN'd me or PMed me asking me to reconsider.  So obviously you see something in me that I don't, or maybe I am just misunderstanding what the position entails.  While it's true that my livelihood is directly linked to AGS, I have zero technical knowledge about it at all.  I can program IN the engine, but I know absolutely zilch about what makes it tick.

That said, I do have lots of experience managing teams of people.  If I were to take a position like this, it would be more like an overseer than anything else.  Here's how I kinda see it working.

First off, there'd be a team of volunteers who work on the engine.  Maybe, say, ten people (to pick a random number).  Individually, they choose what bugs/features to work on and start working on them.  Obviously we'd need some kind of tracking system so we know who is working on what to prevent people from working on the same thing.

Once a volunteer finishes their job, the changes will be merged into a separate branch of the engine and tested by the team.  Once a number of changes have been made, we will release a beta build of that branch that the rest of the community can test.  Once it has been tested enough, we will make it the new "official" version of the engine.  This seems to be the way that CJ handled things (he'd add features to the engine until he felt it was significant, then release a beta candidate, which was tested ad nasuem until it was deemed release worthy), and it seems like the best way to move forward.

So my job, if anything, would be to keep an eye on the list of in-the-works changes/updates and make sure that they are being worked on.  If I have any kind of skill, it's staying on top of people until they finish something.   THAT I can definitely handle.  If that's the kind of person you need, and the kind of system you'd be happy with, than yes.  I'd be up for it. 

[Stee]

That was the kind of gist I was getting at Dave. I think that is ultimately the role of project manager. technical lead is the coding side of things.

[RickJ]

Dave, I think the only thing you may have left out is moderating discussions about future directions which I think is within the skill set you describe.    Also being a bit reluctant makes you highly qualified in my book.   

You have my vote     

[Snarky]

Hi everyone!  Since my last post on this thread, a number of you have emailed me or MSN'd me or PMed me asking me to reconsider.  So obviously you see something in me that I don't, or maybe I am just misunderstanding what the position entails.  While it's true that my livelihood is directly linked to AGS, I have zero technical knowledge about it at all.  I can program IN the engine, but I know absolutely zilch about what makes it tick.

That said, I do have lots of experience managing teams of people.  If I were to take a position like this, it would be more like an overseer than anything else.

...

So my job, if anything, would be to keep an eye on the list of in-the-works changes/updates and make sure that they are being worked on.  If I have any kind of skill, it's staying on top of people until they finish something.   THAT I can definitely handle.  If that's the kind of person you need, and the kind of system you'd be happy with, than yes.  I'd be up for it. 

As I said earlier...

People have argued that the project lead doesn't necessarily need to be the main developer or technical driver, so what does the role consist of?

My understanding is that the main responsibilities involve managing releases, facilitating communication and discussion while cutting through disagreements about the overall project plan, and generally wrangling contributors (including making sure that major roles are filled/tasks assigned, and picking up slack where necessary). In some situations the project lead may also act as spokesperson and publicist for AGS. I would think a large part of it is just staying on top of things, making sure progress is made and people are happy. It also depends on what the technical leads (of the engine, editor and ports) see as their role, e.g. decisions about new features or changes to existing ones.

No one seemed to really disagree with this (although Wyz defined the role a bit differently in the nomination thread), and it seems pretty much in line with what you outline.

Would you be able to take up the task straight away, and organize a new release by, say, summer? (Assuming enough technically competent people volunteered to contribute.) Before you said that you and Janet might not have time for the next few months. That's really my only concern; apart from that I think you are the safest bet, for some of the same reasons Stee mentions.

PS: Can we get a seperate forum for all this AGS Engine Dev stuff?

I think only CJ can create forums, and he hasn't been online for two months. 

[Intense Degree]

Disclaimer: My poor programming knowledge (compared to many here) and lack of activity in the community generally really count me out of having any voice in this debate. My "2 cents" below should therefore be treated accordingly.

I think the idea of an overseer or project lead who is not too involved with the actual programming is a good idea and would make for a healthy set up going forward.

Dave is a very credible lead for this project on every front, as a household name both here and in the world of adventure games generally, and with the experience of managing projects "in the real world". The tasks of programming etc. can then be split equally between others without any problems of someone trying to force it "their way" in terms of how the code works and the general structure.

As I say, I have no vote in this, but the model of someone overseeing and managing whilst others are given parts of the technical side to deal with seems like a good way forward to me.

[Dave Gilbert]

Snarky: Yep, I can pretty much start right away, if only to get the ball rolling.  Assuming everyone wants me, of course.   

We'll definitely need a subforum for engine stuff.  CJ is the only one with global mod status of this forum?  Does anyone know how to contact him?

[m0ds]

He said he would pop by on Monday, but hasn't yet...

[Shane 'ProgZmax' Stevens]

In the absence of direct assistance from CJ, one could just host the ags development forum elsewhere with a blog for updates and so on.  The tech forum here has never sufficiently handled the development process (in my view) though it has always been a good place for plugins and bug reports.  You could connect the forum to the sourceforge site for ags (or wherever) as well.


This is probably something Peder would be willing to do, though I'm not trying to speak for him.  This would at least move this beyond the discussion phase.

[Dave Gilbert]

So, it's been a week.  Should we just jump right in?  Since nobody here has the ability to create a subforum, I can set up a subforum on the WEG site for now.  We had some great momentum going and I would hate to lose it.

-Dave