The Engine/Editor Development Process

[mjomble]

Hey, I was hoping we could get some clarity on how the development of the official version of AGS is going now that it's been open sourced for a few months.

The big question: Who decides what goes in and what doesn't?

From the SVN history, I can see that apart from CJ, a few community members (e.g. tzachs, monkey_05_06) have commit access and are working on 3.2.2.

While the community's wishlist is endless, what interests me are cases where a member actually implements a feature they want and shares the source code. Obviously, you can't blindly accept every contribution without verifying its quality. And even a high quality contribution may require maintenance in the future if it's complex.

I'm assuming that CJ is too busy for this and the main reason for open sourcing the project was to delegate such tasks to someone else.

So if I want to submit a code change for potential inclusion in the official version, who should I contact?

So far I've tried this once, through a thread on the forum: http://www.adventuregamestudio.co.uk/yabb/index.php?topic=43416.0
But that didn't really go anywhere. Which is actually a bit of a good thing since I later decided I want to tweak it a bit; I'll add details to that thread.

But now I've got a second, smaller submission: when you add a new audio file to your project and the file is within your game directory, store only the relative path and not the full absolute path.
This has been a major pain for our project because we share the project files between different members of the team (and sometimes between multiple computers of one member) via SVN and the absolute paths are different on almost every machine.
There's a simple fix for this, but we'd need to use a custom-built AGS to benefit from it and I'd much rather prefer having this feature in an official build.

So do I send the code to someone, or is AGS heading into a future of dozens of obscure and incompatible forks?

[monkey0506]

Well I think the two biggest issues with anything like this are:

1) Community support, and
2) CJ's support

CJ is the only person who has the call on what gets checked into the official trunk. It is, after all, his program. However, he listens quite well when he has the time. The changes I've made are actually specifically tied to a project that is currently postponed in favor of more lucrative endeavours, but if you read the notes on what I've done then I imagine you could get some idea on what I'm working on. CJ has actually been a huge help to me in the changes I made.

As for the idea of splitting up some of the project files, I'm not sure that I see the point of it. It seems a very rare and odd occurrence that anything should actually go so wrong as to corrupt the AGF file. I'd reread your other thread, but I'm not going to be online too much longer. I'll take a look later on.

The secondary issue I guess would only be a problem if multiple people are trying to modify the sound files? Otherwise I'm not sure I understand the issue. The audio cache is automatically updated if the source file exists and is changed, so you're saying that the source file (within the project folder) may need to be modified by one of the members of your team, but if they don't have their copy of the project in the same path as the person who originally added the audio file then it's not automatically updated? This change sounds reasonable to me, though it wouldn't really be too hard to just delete the file and reimport it.

Anyway, I would want to see what other members think about these particular changes before I would want to check them in, so I'd say a bit of discussion is in order (and of course for the first issue you've already got that thread).

[mjomble]

(I'll open a separate thread about the music path issue - I wanted to keep the focus of this one on the overall process)

So the preferred method for proposing such changes is still to open a forum thread?

PS. Apart from the issues I've mentioned, there are more improvements coming up in the future if I find the time to implement them.

[mjomble]

So it seems at the moment, the process for submitting code to AGS is more or less:

1. Start a forum thread in the middle of various other technical questions and discussions.
2. Hope that CJ notices it and has time to get involved, discuss the feature, point out any required changes and all in all, decide what to do with it.

I see some room for improvement.

So here's a proposal:

1. An official group of active trusted community veterans is formed. Probably including the people with SVN commit permissions, etc.
2. This group will be tasked with actively monitoring the forum for code submissions. Could create a separate subforum for that to make it easier.
3. The members will carry out the initial evaluation of the submissions, filtering out pointless ones, recommending changes and otherwise ensuring proper quality.
4. The members notify CJ of the more viable submissions, requiring only a yes/no response (unless CJ wants to get more involved).

Or something similar. Whatever establishes more clearly who should be doing what in order to keep things moving. And mitigates the bottleneck of CJ's amount of free time to spend on AGS.

It seems to me like there's a lot of potential in the AGS community, but a lot of it is going to waste because of a lack of organization.

For the code changes I've proposed, I'd be happy to get any kind of a clear response, even if it's a "no". I'd just put our team on a custom build with features that only we need and stop bothering you all about it.
But right now it's in a vague "nobody really knows what's going on" area that's rather discouraging.

[Tartalo]

create a separate subforum

I agree. I come regularly here looking for something easy I could do, but have trouble even understanding what is going on.

[Alan v.Drake]

Yeah, we need a coordinator, a roadmap. Right now it's anarchy.


- Alan

[tzachs]

As I understand it, the current process is as follows:
1. Ask CJ for permission to commit code to SVN.
2. Open a thread to get feedback from the community (optional, but recommended)
3. When you feel satisfied enough with what you have done, commit the code to the current development branch.
4. When the time comes and CJ wants to release a new candidate version, he goes through all the changes made, decides what goes in and out, and fixes stuff if needed.

Yes, of course it can be improved, but currently that is how CJ wants it, and it worked thus far because there weren't so many changes made.

In the future, assuming there will be more and more changes, I guess that CJ will want to change stuff, but maybe not, it's really up to him.
I agree to the suggestion on a new subforum, that can definitely help.
As for how to ensure quality and the dedicated panel suggestion, the way I see it there are actually three different subjects here:
1. Is the feature really desired?
The whole community has a say here, where final say belongs to CJ unless he delegates this to some panel of responsible people.
2. Was the commited code up to the required standards?
Currently only CJ decides on that. A dedicated panel for this can be a good idea. That panel will have to decide on what are the required standards and how to enforce them.
3. Does the feature work as intended? Do all other features still work?
This is also something that the whole community has a say in, where there could also be another panel of volunteers with less programming knowledge but still want to help, to test the new features, and from time to time also test all existing features to verify they still work.

[Wyz]

I guess Tzach is right, and as long as this is how it works we could really use a sub forum. That way we could at least focus the interest of the (aspiring) developers to one place.

[Shane 'ProgZmax' Stevens]

I'm still pretty pro on the developer panel idea.  It's a good way to delegate responsibility of testing and accepting/rejecting features to a group of peers rather than one person who may or may not be available on a regular enough basis.  This is assuming CJ would still retain final 'approval' status.  One thing we don't want, though, is to mire down the development process in layers of red tape, so I would highly recommend the group is limited to 3-4 people who not only have development experience with ags (games and developing the software itself) but also are established members with something like 4 years invested in the forums.  The goal there is to have people you know aren't going to just disappear but also understand how to use AGS and have a knowledge of its inner workings.  

As long as those conditions are met, whoever organizes the approval process should be fairly dependable.


One other thing:  There should be a clear distinction of what the panel considers to be erroneous or undesirable as far as 'features' go (like implementing a full 3D makeover, for example) so people aren't encouraged to just waste their time.  Aside from that, most things that aren't redundant should be acceptable, especially fixes and performance improvements.  Perhaps there could be an initial idea submission process to save time and build interest in the idea.

[mjomble]

Allright, looks like we've got quite a lot of agreement on two things:

1. Creating a new subforum.
Is this something that only CJ can do?

2. Establishing some sort of a developer panel.
I guess to get started with this, we can either tell CJ about the idea and expect him to figure out who will be in it.
Or, to make things simpler, we can do some nominating and discussing to recommend a list of people.
I'm pretty much a noob in the AGS community and don't really know people, so it's up to you.
Reply to this thread and nominate yourself and/or other people who you think should belong in the panel.

[Calin Leafshade]

I suggest we just do it. In the last 6 months CJ has made about 8 posts on the forum, half of which were mittens related. The code is open, let's just do it. If he complains about our methodology later then we can correct it but at least we have progress.

Here's what i think needs to be done:

Set up a bug tracker
A wiki
Get the code documented.
Establish a roadmap for development.

However I think these things should be separate from adventuregamestudio.co.uk. CJ is a busy guy and should not be expected to handle trivial server issues when/if they arise but he doesn't want to delegate anything related to the server to other people, which is perfectly understandable. It is his server after all.

but these are the problems we need to address:

The code base is messy and there seems to be little impetus to fix it, never mind document it. Some people have been very enthusiastic about "the development process" but no one seems enthusiastic about fixing the code.

No one is seemingly interested in leading the project. We need a project lead who is a capable programmer and a decent leader. I know who I would choose for the job *cough*Wyz*cough* but that doesn't mean that they or indeed anyone else is willing/able to put in the work.

In sum, we have too many indians and not enough chiefs.

[mjomble]

In that case - anybody volunteering to be the chief?

As for setting up all those things, this page looks useful: Comparison of open source software hosting facilities

Out of those, I'm most familiar with Google Code, but only as a consumer and not as a project owner. If anyone has better suggestions or bad experiences with GC, let us know. Otherwise, I propose that the eventual chief starts a new project there.

As for the repository, it might be a good idea to clone the current one to the new environment where it would be linked to any additional tools. So the community would maintain their own repo, users, etc while merging in any changes that are made in the official repo.
And at certain points, the leader(s) could propose certain sets of changes to CJ and offer to merge them into the official repo. This way we wouldn't need to bother CJ with setting up SVN accounts for everyone, etc.
Or maybe I'm overthinking this.

[Calin Leafshade]

I agree that we should try to keep CJ away from petty administrative tasks and use his limited time more wisely.

Good luck finding your chief though. I remain sceptical.

[Alan v.Drake]

Make believe roadmap time:

(2012) AGS 3.2.x:

• Add last small functionalities
  
• Fix last bugs
  
  
  (2013) BREAK COMPATIBILITY !
  
• Clean up the code from the previous versions checks
  
• Hunt for stuff that had been impaired due to legacy support
  
• Decide the proper way such stuff should work
  
• Check for stuff that loses resolution (e.g. 100 instead of 255) and rethink it over
  
• 32bit AGS colors
  
• Add a flag to set normal/additive/subtractive opacity to guis,chars,etc
  
• Improve resolutions support and rendering routine
  
  
  (2017) AGS 4.0
  
• Rewrite from ground up with portable codebase, bells and whistles
  
  
  Discuss.
  
  
  - Alan

[Calin Leafshade]

No, the benefits you describe are far outweighed by their cost in man hours.

In order to work effectively as group on the 3.x branch the code will need to be documented which is no small task considering the only expert on the code is currently for all intents and purposes incommunicado.

also, the most demanded feature seems to be cross platform compatibility which your roadmap doesnt address until later.

So here are my thoughts.

I think we all agree that the editor and the compiler do their jobs perfectly well and dont need messing with. The editor is fairly new code and it's well structured and easy to modify in the future. So we can essentially ignore them.

So I think we should draw a line under 3.2, declare the code feature complete and provide LTS for it.

Then we should begin immediately on AGS 4. The goals of AGS 4 would be as follows:

1. provide a full backwards compatible runtime for AGS 2.72+ (the last widely used version) This is optional and perhaps could be taken up as a separate project by ScummVM if they were willing to do so.

2. Provide a portable and FOSS runtime for AGS 3.2+. AGS 4 will initially use an identical compiler, scripting language and game format to AGS 3.2 so AGS 4 just needs to interpret the current game format.

That seems, to me, to be the best way forward.

Of course its hypothetical without a team of programmers willing to work on it.

Although, it wouldn't be *that* much work, relatively speaking. CJ has done most of the hard work by providing a consistent system for us to work with. The script is already compiled to bytecode so all we need to do is rewrite the rendering engine and clean it up.

Thoughts?

[tzachs]

Well, before declaring CJ as incommunicado and going at it yourselves, have you tried PMing him?
Moving on without his blessing would be a bad move IMO.

[mjomble]

We'd definitely want his blessing, but it might be a good idea to first figure out what exactly we want to do and how we would do it.

[Snarky]

As far as I can tell, CJ doesn't check his PMs any more often than he visits the forums. If anyone has his email, maybe he can be reached that way?

As for the plans Alan and Calin propose, I agree with Calin that we don't want to delay a new engine for 6 years (wtf!). On the other hand, I don't think that means all efforts should go into a full top-to-bottom rewrite, and all incremental engine improvements or changes to the editor should be frozen.

The one thing I really think absolutely should be in the next major version is Unicode support. I imagine that will require changes to the compiler and possibly the editor as well as the engine. (And if we want to support it for bitmap fonts it would probably require extending the WFN format.)

[abstauber]

As far as I can tell, CJ doesn't check his PMs any more often than he visits the forums. If anyone has his email, maybe he can be reached that way?

Just text him and ask to read his PMs


Anyway I also think that it's a good idea to move to a public svn.

From a user's point of view google code is really a nice spot, but I hope that the development board can still be here in form of a sub-forum.

[Calin Leafshade]

Here's what CJ said on the subject in August (I'm sure he wont mind me posting the PM, its nothing salacious)

I think the main problem is a lack of project management. From speaking to the developers it seems that everyone is too afraid to step on AGS's metaphorical toes.

Thats why I suggested finalising the website first and cementing the related services like a bug/feature tracker and a more complete revision control system (it's not something i have much experience in but people seem to not want to touch an SVN repository with a barge pole). If those were updated it would work to quell the fears that AGS has become abandonware for all intents and purposes.

Then from there we could draw a line under the 3.x branch and just maintain it while starting a 4.x branch from scratch. Wyz has expressed an interest in taking a lead on that but because things are still somewhat in the air he and others are reluctant to commit.
if the 3.x branch were feature locked then the ScummVM team might be willing to take on the task of making an interpreter which would also push the project forward.

If I'm honest I think people involved in AGS on a personal level (as opposed to the recent influx of interlopers) are a little worried about taking any action out of respect for your baby.

Yeah I completely understand this -- the role I actually wanted to take on with the open-sourcing was that of project manager, but I just haven't had the time to even think about AGS lately. I'll have a think about it this week at Mittens and see if I can come up with a plan.

I had forgotten about this PM so yea, perhaps someone should ask CJ to make a definitive decision about all this so we can move on.

On my plan and Snarky's comments, I still think feature locking AGS would be wise until the engine were in a portable state.

Remember that its not a total rewrite. We dont need to mess with any of the low-level packing code, just the rendering and interface code. Essentially anything that is Allegro.

Of course we still have the problem of there being not enough people with a high enough skill level/enough time to take on the project.

[Lt. Smash]

IMO the most important improvement to AGS would be the extension of the current scripting language.
Please don't forget taking that into account... There are many things missing. Like for, switch, break, return, 2/3..D arrays, static extenders, array.length and many more.

[Wyz]

Well, I kind of agree with Alan v.Drake.
I think if we try to build on the existing code base it might even take longer because we first need to understand it very well, and there is a lot of code there. But also it would be silly to not have a bleeding edge while a bottom up project is in progress. Therefore I think we need to split development up into two (actually three, but I'll get to that) parts:

• Maintenance
  
• Future version
  
  Essentially the idea is that the two branches will meet somewhere in the middle. The future version will of course not reinvent everything, but it will make sure every part of the engine is treated, and keeps everyone motivated because it has a clear goal. Also because AGS is based on a lot of libraries that might not have stand the time as good as AGS, a bottom up project will tackle any issue with portability.
  Speaking of: I think portability is one of the most important things for future versions of AGS. The adventure genre has been revamped and this is caused mainly by the popularity of hand held devices. We must embrace this and try to get a future version working on all of them. The newest versions off Allegro support some of them, SDL support others, well it would be really nice if we wrap around both. Also it would run natively on any system of any importance that way which would be awesome.
  Also when adding things like unicode support, getting that into an existing code base is hell. When you start fresh it will be easy to keep it in mind.
  
  But as I said that is only one part of the process, there should always be a bleeding edge, keep on improving on the original code base. I've also been talking about the engine only, the editor is in fine shape. That does not have to be redefined in any way, but there should be developers keeping it in good shape. There are a few updates that can wait much longer, transparency should be fully featured and the blending modes as Calin made them into a plugin, the smooth rotation plugin etc. would do great in the engine natively. Well I don't feel it's up to me to decide stuff like that but I would like it. So that's the other part, and however it might feel ambiguous to work on two engines separately this is a quite common practise.
  
  I like to state a third branch: the 2.7 branch. People still use it and there have been hundreds of wonderful games made with it. It would be nice if there will be support for them for future platforms. I'm not going to say a lot about it since I've done before in other topics, but getting ScummVM to run it would be awesome.
  
  I know I am repeating myself with all this, but I just want it off my chest. Last things I want to say about it:
  Backwards compatibility is a bless and a curse at the same time. If you start fresh you should not worry about it initially because otherwise it will hold you back. You might change resolution and palette stuff because it will result in a more efficient program, but breaking compatibility. Well in that case a compatibility mode should come in handy. On the other hand: each game comes with an executable. The only reason this would be useful is to allow older games to be played with the newer engine for platform compatibility. This might be resolved by porting the 2.x and 3.x branches separately. Let's not worry about that for now!
  
  

  
  No one is seemingly interested in leading the project. We need a project lead who is a capable programmer and a decent leader. I know who I would choose for the job *cough*Wyz*cough* but that doesn't mean that they or indeed anyone else is willing/able to put in the work.
  

  
  Well I'm flattered, but unfortunately I feel like I don't have the time a project like this needs. But eventually if noone does come forward I will.
  
  

  
  Out of those, I'm most familiar with Google Code, but only as a consumer and not as a project owner. If anyone has better suggestions or bad experiences with GC, let us know. Otherwise, I propose that the eventual chief starts a new project there.
  

  
  Same for me, I haven't yet found any obvious flaws at Google Code. It has a wiki, bug tracker, code browser and a really really nice search feature for code. Also everyone with a Google account can use it immediately, well not that it matters. Most programmers already have a multitude of accounts for repo sites.
  Oh, also I really encourage the use of decentralised version control like git and mercurial. It means people can work separately without bother and decreases the amount of administration needed and that we can use. I prefer mercurial because it's lightweight and easy easier to learn.
  
  Loong loooooong post, sorry.
  Kudos if you've read it.

[mjomble]

Oh, also I really encourage the use of decentralised version control like git and mercurial. It means people can work separately without bother and decreases the amount of administration needed and that we can use. I prefer mercurial because it's lightweight and easy easier to learn.

In that case, we might want to try Atlassian instead.
It's also free for open souce projects.
And although my experience with both Git and Hg is rather limited, I also prefer Hg.

[Wyz]

Ah, I don't have experience with that site, but it looks nice.

[Monsieur OUXX]

Digging up (only one week old).

About the question of getting CJ's blessing : You might want his blessing for the development roadmap, but I think you don't need to wait to create such tools as a bug tracker, etc.

In other words: ask him about the goals, not necessarily about the underlying work organization/structure/toolset.

[dbuske]

When do the powers that be think a new version of AGS will be released?

[Calin Leafshade]

Since JJS and bero have a decent crossplatform build process and stuff i propose we adopt their branch as the official (or at least the actively developed) one.

[Sslaxx]

Since JJS and bero have a decent crossplatform build process and stuff i propose we adopt their branch as the official (or at least the actively developed) one.

I assume they're working together, then? Is there anywhere their stuff is available?

[Calin Leafshade]

Heres Beros version: http://gitorious.org/ags/ags

JJS's version is a fork of that repos.

Edmunditos mac version is also a fork

[timofonic]

Heres Beros version: http://gitorious.org/ags/ags

JJS's version is a fork of that repos.

Edmunditos mac version is also a fork

Are there plans to merge those forks eventually?

Gitorious is nice, but I think GitHub is cooler. It has many advantages like easy view of forks, pull requests, nice way to look commits and more!

You know, it's "Social Coding" and has a cool mascot called Octodex!