The Engine/Editor Development Process

[mjomble]

Hey, I was hoping we could get some clarity on how the development of the official version of AGS is going now that it's been open sourced for a few months.

The big question: Who decides what goes in and what doesn't?

From the SVN history, I can see that apart from CJ, a few community members (e.g. tzachs, monkey_05_06) have commit access and are working on 3.2.2.

While the community's wishlist is endless, what interests me are cases where a member actually implements a feature they want and shares the source code. Obviously, you can't blindly accept every contribution without verifying its quality. And even a high quality contribution may require maintenance in the future if it's complex.

I'm assuming that CJ is too busy for this and the main reason for open sourcing the project was to delegate such tasks to someone else.

So if I want to submit a code change for potential inclusion in the official version, who should I contact?

So far I've tried this once, through a thread on the forum: http://www.adventuregamestudio.co.uk/yabb/index.php?topic=43416.0
But that didn't really go anywhere. Which is actually a bit of a good thing since I later decided I want to tweak it a bit; I'll add details to that thread.

But now I've got a second, smaller submission: when you add a new audio file to your project and the file is within your game directory, store only the relative path and not the full absolute path.
This has been a major pain for our project because we share the project files between different members of the team (and sometimes between multiple computers of one member) via SVN and the absolute paths are different on almost every machine.
There's a simple fix for this, but we'd need to use a custom-built AGS to benefit from it and I'd much rather prefer having this feature in an official build.

So do I send the code to someone, or is AGS heading into a future of dozens of obscure and incompatible forks?

[monkey0506]

Well I think the two biggest issues with anything like this are:

1) Community support, and
2) CJ's support

CJ is the only person who has the call on what gets checked into the official trunk. It is, after all, his program. However, he listens quite well when he has the time. The changes I've made are actually specifically tied to a project that is currently postponed in favor of more lucrative endeavours, but if you read the notes on what I've done then I imagine you could get some idea on what I'm working on. CJ has actually been a huge help to me in the changes I made.

As for the idea of splitting up some of the project files, I'm not sure that I see the point of it. It seems a very rare and odd occurrence that anything should actually go so wrong as to corrupt the AGF file. I'd reread your other thread, but I'm not going to be online too much longer. I'll take a look later on.

The secondary issue I guess would only be a problem if multiple people are trying to modify the sound files? Otherwise I'm not sure I understand the issue. The audio cache is automatically updated if the source file exists and is changed, so you're saying that the source file (within the project folder) may need to be modified by one of the members of your team, but if they don't have their copy of the project in the same path as the person who originally added the audio file then it's not automatically updated? This change sounds reasonable to me, though it wouldn't really be too hard to just delete the file and reimport it.

Anyway, I would want to see what other members think about these particular changes before I would want to check them in, so I'd say a bit of discussion is in order (and of course for the first issue you've already got that thread).

[mjomble]

(I'll open a separate thread about the music path issue - I wanted to keep the focus of this one on the overall process)

So the preferred method for proposing such changes is still to open a forum thread?

PS. Apart from the issues I've mentioned, there are more improvements coming up in the future if I find the time to implement them.

[mjomble]

So it seems at the moment, the process for submitting code to AGS is more or less:

1. Start a forum thread in the middle of various other technical questions and discussions.
2. Hope that CJ notices it and has time to get involved, discuss the feature, point out any required changes and all in all, decide what to do with it.

I see some room for improvement.

So here's a proposal:

1. An official group of active trusted community veterans is formed. Probably including the people with SVN commit permissions, etc.
2. This group will be tasked with actively monitoring the forum for code submissions. Could create a separate subforum for that to make it easier.
3. The members will carry out the initial evaluation of the submissions, filtering out pointless ones, recommending changes and otherwise ensuring proper quality.
4. The members notify CJ of the more viable submissions, requiring only a yes/no response (unless CJ wants to get more involved).

Or something similar. Whatever establishes more clearly who should be doing what in order to keep things moving. And mitigates the bottleneck of CJ's amount of free time to spend on AGS.

It seems to me like there's a lot of potential in the AGS community, but a lot of it is going to waste because of a lack of organization.

For the code changes I've proposed, I'd be happy to get any kind of a clear response, even if it's a "no". I'd just put our team on a custom build with features that only we need and stop bothering you all about it.
But right now it's in a vague "nobody really knows what's going on" area that's rather discouraging.

[Tartalo]

create a separate subforum

I agree. I come regularly here looking for something easy I could do, but have trouble even understanding what is going on.

[Alan v.Drake]

Yeah, we need a coordinator, a roadmap. Right now it's anarchy.


- Alan

[tzachs]

As I understand it, the current process is as follows:
1. Ask CJ for permission to commit code to SVN.
2. Open a thread to get feedback from the community (optional, but recommended)
3. When you feel satisfied enough with what you have done, commit the code to the current development branch.
4. When the time comes and CJ wants to release a new candidate version, he goes through all the changes made, decides what goes in and out, and fixes stuff if needed.

Yes, of course it can be improved, but currently that is how CJ wants it, and it worked thus far because there weren't so many changes made.

In the future, assuming there will be more and more changes, I guess that CJ will want to change stuff, but maybe not, it's really up to him.
I agree to the suggestion on a new subforum, that can definitely help.
As for how to ensure quality and the dedicated panel suggestion, the way I see it there are actually three different subjects here:
1. Is the feature really desired?
The whole community has a say here, where final say belongs to CJ unless he delegates this to some panel of responsible people.
2. Was the commited code up to the required standards?
Currently only CJ decides on that. A dedicated panel for this can be a good idea. That panel will have to decide on what are the required standards and how to enforce them.
3. Does the feature work as intended? Do all other features still work?
This is also something that the whole community has a say in, where there could also be another panel of volunteers with less programming knowledge but still want to help, to test the new features, and from time to time also test all existing features to verify they still work.

[Wyz]

I guess Tzach is right, and as long as this is how it works we could really use a sub forum. That way we could at least focus the interest of the (aspiring) developers to one place.

[Shane 'ProgZmax' Stevens]

I'm still pretty pro on the developer panel idea.  It's a good way to delegate responsibility of testing and accepting/rejecting features to a group of peers rather than one person who may or may not be available on a regular enough basis.  This is assuming CJ would still retain final 'approval' status.  One thing we don't want, though, is to mire down the development process in layers of red tape, so I would highly recommend the group is limited to 3-4 people who not only have development experience with ags (games and developing the software itself) but also are established members with something like 4 years invested in the forums.  The goal there is to have people you know aren't going to just disappear but also understand how to use AGS and have a knowledge of its inner workings.  

As long as those conditions are met, whoever organizes the approval process should be fairly dependable.


One other thing:  There should be a clear distinction of what the panel considers to be erroneous or undesirable as far as 'features' go (like implementing a full 3D makeover, for example) so people aren't encouraged to just waste their time.  Aside from that, most things that aren't redundant should be acceptable, especially fixes and performance improvements.  Perhaps there could be an initial idea submission process to save time and build interest in the idea.

[mjomble]

Allright, looks like we've got quite a lot of agreement on two things:

1. Creating a new subforum.
Is this something that only CJ can do?

2. Establishing some sort of a developer panel.
I guess to get started with this, we can either tell CJ about the idea and expect him to figure out who will be in it.
Or, to make things simpler, we can do some nominating and discussing to recommend a list of people.
I'm pretty much a noob in the AGS community and don't really know people, so it's up to you.
Reply to this thread and nominate yourself and/or other people who you think should belong in the panel.

[Calin Leafshade]

I suggest we just do it. In the last 6 months CJ has made about 8 posts on the forum, half of which were mittens related. The code is open, let's just do it. If he complains about our methodology later then we can correct it but at least we have progress.

Here's what i think needs to be done:

Set up a bug tracker
A wiki
Get the code documented.
Establish a roadmap for development.

However I think these things should be separate from adventuregamestudio.co.uk. CJ is a busy guy and should not be expected to handle trivial server issues when/if they arise but he doesn't want to delegate anything related to the server to other people, which is perfectly understandable. It is his server after all.

but these are the problems we need to address:

The code base is messy and there seems to be little impetus to fix it, never mind document it. Some people have been very enthusiastic about "the development process" but no one seems enthusiastic about fixing the code.

No one is seemingly interested in leading the project. We need a project lead who is a capable programmer and a decent leader. I know who I would choose for the job *cough*Wyz*cough* but that doesn't mean that they or indeed anyone else is willing/able to put in the work.

In sum, we have too many indians and not enough chiefs.

[mjomble]

In that case - anybody volunteering to be the chief?

As for setting up all those things, this page looks useful: Comparison of open source software hosting facilities

Out of those, I'm most familiar with Google Code, but only as a consumer and not as a project owner. If anyone has better suggestions or bad experiences with GC, let us know. Otherwise, I propose that the eventual chief starts a new project there.

As for the repository, it might be a good idea to clone the current one to the new environment where it would be linked to any additional tools. So the community would maintain their own repo, users, etc while merging in any changes that are made in the official repo.
And at certain points, the leader(s) could propose certain sets of changes to CJ and offer to merge them into the official repo. This way we wouldn't need to bother CJ with setting up SVN accounts for everyone, etc.
Or maybe I'm overthinking this.

[Calin Leafshade]

I agree that we should try to keep CJ away from petty administrative tasks and use his limited time more wisely.

Good luck finding your chief though. I remain sceptical.

[Alan v.Drake]

Make believe roadmap time:

(2012) AGS 3.2.x:

• Add last small functionalities
  
• Fix last bugs
  
  
  (2013) BREAK COMPATIBILITY !
  
• Clean up the code from the previous versions checks
  
• Hunt for stuff that had been impaired due to legacy support
  
• Decide the proper way such stuff should work
  
• Check for stuff that loses resolution (e.g. 100 instead of 255) and rethink it over
  
• 32bit AGS colors
  
• Add a flag to set normal/additive/subtractive opacity to guis,chars,etc
  
• Improve resolutions support and rendering routine
  
  
  (2017) AGS 4.0
  
• Rewrite from ground up with portable codebase, bells and whistles
  
  
  Discuss.
  
  
  - Alan

[Calin Leafshade]

No, the benefits you describe are far outweighed by their cost in man hours.

In order to work effectively as group on the 3.x branch the code will need to be documented which is no small task considering the only expert on the code is currently for all intents and purposes incommunicado.

also, the most demanded feature seems to be cross platform compatibility which your roadmap doesnt address until later.

So here are my thoughts.

I think we all agree that the editor and the compiler do their jobs perfectly well and dont need messing with. The editor is fairly new code and it's well structured and easy to modify in the future. So we can essentially ignore them.

So I think we should draw a line under 3.2, declare the code feature complete and provide LTS for it.

Then we should begin immediately on AGS 4. The goals of AGS 4 would be as follows:

1. provide a full backwards compatible runtime for AGS 2.72+ (the last widely used version) This is optional and perhaps could be taken up as a separate project by ScummVM if they were willing to do so.

2. Provide a portable and FOSS runtime for AGS 3.2+. AGS 4 will initially use an identical compiler, scripting language and game format to AGS 3.2 so AGS 4 just needs to interpret the current game format.

That seems, to me, to be the best way forward.

Of course its hypothetical without a team of programmers willing to work on it.

Although, it wouldn't be *that* much work, relatively speaking. CJ has done most of the hard work by providing a consistent system for us to work with. The script is already compiled to bytecode so all we need to do is rewrite the rendering engine and clean it up.

Thoughts?

[tzachs]

Well, before declaring CJ as incommunicado and going at it yourselves, have you tried PMing him?
Moving on without his blessing would be a bad move IMO.

[mjomble]

We'd definitely want his blessing, but it might be a good idea to first figure out what exactly we want to do and how we would do it.

[Snarky]

As far as I can tell, CJ doesn't check his PMs any more often than he visits the forums. If anyone has his email, maybe he can be reached that way?

As for the plans Alan and Calin propose, I agree with Calin that we don't want to delay a new engine for 6 years (wtf!). On the other hand, I don't think that means all efforts should go into a full top-to-bottom rewrite, and all incremental engine improvements or changes to the editor should be frozen.

The one thing I really think absolutely should be in the next major version is Unicode support. I imagine that will require changes to the compiler and possibly the editor as well as the engine. (And if we want to support it for bitmap fonts it would probably require extending the WFN format.)

[abstauber]

As far as I can tell, CJ doesn't check his PMs any more often than he visits the forums. If anyone has his email, maybe he can be reached that way?

Just text him and ask to read his PMs


Anyway I also think that it's a good idea to move to a public svn.

From a user's point of view google code is really a nice spot, but I hope that the development board can still be here in form of a sub-forum.

[Calin Leafshade]

Here's what CJ said on the subject in August (I'm sure he wont mind me posting the PM, its nothing salacious)

I think the main problem is a lack of project management. From speaking to the developers it seems that everyone is too afraid to step on AGS's metaphorical toes.

Thats why I suggested finalising the website first and cementing the related services like a bug/feature tracker and a more complete revision control system (it's not something i have much experience in but people seem to not want to touch an SVN repository with a barge pole). If those were updated it would work to quell the fears that AGS has become abandonware for all intents and purposes.

Then from there we could draw a line under the 3.x branch and just maintain it while starting a 4.x branch from scratch. Wyz has expressed an interest in taking a lead on that but because things are still somewhat in the air he and others are reluctant to commit.
if the 3.x branch were feature locked then the ScummVM team might be willing to take on the task of making an interpreter which would also push the project forward.

If I'm honest I think people involved in AGS on a personal level (as opposed to the recent influx of interlopers) are a little worried about taking any action out of respect for your baby.

Yeah I completely understand this -- the role I actually wanted to take on with the open-sourcing was that of project manager, but I just haven't had the time to even think about AGS lately. I'll have a think about it this week at Mittens and see if I can come up with a plan.

I had forgotten about this PM so yea, perhaps someone should ask CJ to make a definitive decision about all this so we can move on.

On my plan and Snarky's comments, I still think feature locking AGS would be wise until the engine were in a portable state.

Remember that its not a total rewrite. We dont need to mess with any of the low-level packing code, just the rendering and interface code. Essentially anything that is Allegro.

Of course we still have the problem of there being not enough people with a high enough skill level/enough time to take on the project.