The reasons why this project is/was screwed up

Started by Crimson Wizard, Sun 31/12/2017 21:59:40

Previous topic - Next topic

Crimson Wizard

Following are the reasons why this project was screwed up from the very beginning. Just for the sake of history. I'd like to believe this may help someone to avoid same mistakes, but people seem to rarely learn on their own, so IDK.

Actually, I cannot think of anything that was made right here, from the first steps every mistake that could be made was made.


1. The problem with AGS was the reliance on old library that was already badly working on contemporary systems in 2012. Instead of immediately moving to better up-to-date library (SDL2 or Allegro5), or framework (like ScummVM) the old library was patched to somehow work on wanted platforms.
Usually the patching was performed by a single person. After such person took their leave, little information remained about their work, and whenever any kind of issue arised with the port, it took a lot of effort to figure out what to do.
Often the library was patched having only one particular port in mind. Sometimes a functionality was added right into this patch, different from the rest of the engine, causing inconsistences.

A lot of additional work related to device controls and graphic modes could be omitted, if we could rely on modern library. For example, I had to rewrite graphics renderers 5 or 6 times adding better display mode selection over those years.

The result: AGS is still based on same old library that work worse and worse on contemporary systems.


2. The problem of AGS was that Windows version was glued to particular old version of MSVS, which was even non-free. Instead of immediately looking for a way to remove that dependency, we kept it for several years.

Result: many potential contributors could not contribute because they could not even build it.


3. The problem with AGS was bad program code. Instead of immediately refactor it from top to bottom, this was only partially complete before we switched to compatibility fixes, and then - adding new functionality. Because of that a lot of time was wasted trying to hack new stuff into existing code. Supposedly, the bad code also scared away other potential contributors.

Additional moment: there was already a refactored code made by ScummVM developer, licensed under same license as AGS, which could be used, even if partially. But it was never used.

The result: AGS still has got a lot of non-refactored code which is barely known, and many features or fixes that were very wanted were never added because of that.


4. The problem with AGS was that the engine and editor contained a lot of legacy support that was practically useless for making game in current day, but cluttered up the code and made it difficult to develop it further.

Instead of splitting out legacy support into separate branch, we not only kept it in the engine, but increased its amount by adding support for even older games.
Because of this, adding any new feature required a lot of extra time, necessity to test whether ancient games still run with this engine after every update. This additionally mixed up with the non-refactored code producing more mess.

It may be hard to fully recognize consequences of this decision. A lot of system limits are there only because of legacy support and old plugin API support. There was a time when I spent over half of a year trying to remove these limits while keeping the legacy support, but in the end just threw the code out, so ugly it was.
Similarily, a lot of features could not be easily added simply because there was some fragile legacy stuff that could break.


Result: amount of time just wasted for nothing is impossible to calculate. Neither it is possible to know how much we could have done more if not for this.
Number of potential collaborators probably did not want to participate in project because of confusing code.


5. The problem with AGS was that with very little number of collaborators nevertheless we sometimes wasted time on something that could have been avoided. It was in our interests to take easier routes requiring less job where possible. Instead, the project started with one developer willing to write a class library specifically for AGS instead of using standard one. Another developer kept waiting several months for those classes, purposedly trying to avoid adding standard ones into engine. When it became clear that there will be none, instead of immediately switching to standard library, that second developer, for the reasons he cannot even explain today, decided to follow suggested idea and write these classes himself. In the end he did, kind of, but then realized that with them he will be drawned into maintaining utility classes instead of working on engine itself, so, again, just threw the code away. So, now we do use standard library, but nearly two years were lost.

This is only one such example. I could probably remember other not so useful stuff that I wasted time on, because someone said (in the end many things I did were not even added to the release).


6. The problem with AGS was the lack of automation. Everything had to be made manually. No automatic builds, let alone tests, nothing. If a person wanted to suggest features or changes, developers had to manually build and test their version to just make sure it compiles.
There was no way to know if all ports are still working or were broken by some of the recent additions except for building them all or waiting when someone tries to. And not everyone could build all the ports, naturally.
Making a full release package with all the data - demo game, templates, etc, - became such ordeal, that frankly, I tried to delay release every time until more changes accumulate, simply because I did not want to go through all this too often.


7. The problem with AGS was terrible organization. This was a project without actual leadership nor vision. There was no development plan, no documentation. Everyone just did some stuff as they see fit.
There was very little communication between game developers and AGS developers. With lots of ideas it was difficult to filter out things that were actually important to many people. A lot of critical nuances stayed unknown until occasionally mentioned years later.
A lot of tasks, even mentioned on forums, were never solved, even though several people could state their interest in that. Good example is this thread:
http://www.adventuregamestudio.co.uk/forums/index.php?topic=51340.0
Just look at this:
Spoiler

Quote from: Monsieur OUXX on Sat 29/11/2014 01:38:02
The help file is getting more and more out of sync with the actual Editor. <...>
The issue with the help file is that it's generated from some old LaTeX dialect, which in turn creates an old-school Windows help file (.chm files). <...>
We need to find a solution.
Quote from: sonneveld on Sun 03/07/2016 10:50:47
I can help.
Quote from: eri0o on Fri 21/07/2017 23:34:23
I used to use pandoc and sphinx, I was passing on the AGS topics and this is the first thing I thought "I know how to automate this!"
<...>
Can someone point me the source files?
[close]
From 2014 to 2018, and it did not move anywhere. Meanwhile, this is not even related to engine code or C++.

If organization were better, perhaps a number of other problems not directly related to coding engine could have been solved. (e.g. this: http://www.adventuregamestudio.co.uk/forums/index.php?topic=55646.0)
Simply because people could notice that such task exists.

Alan v.Drake

Yep, I can only agree with all the points.
We were just unlucky.
No one with extensive knowledge of game engines ever stepped in to steer us on the right path (suggestions of full rewrites do not count).
No one wanted to take leadership so everyone made his own attempts at trying to improve things, myself included.
We have a cleanup branch now, but I lack the eagerness I had 4 years ago.
Most templates are outdated and use old script api versions which confuse newcomers.

I'm also kinda rusty on actual AGS usage, I feel like I've spent more time wrestling with the source rather than actually making games with it... I kinda feel out of touch.
Thanks to my job I no longer find any joy in developing.


- Alan

eri0o

I am so sorry about the documentation thing,  could glue scripts, but they would probably run on Linux - low motivation for documentation. The terrible organization is really the biggest problem because it generates work that is wasted. Today I got an answer from Nick on Twitter saying he will DM me the stuff for building. I have to say I just wanted to use the tool instead of working on it.

AGA is right though that we could really use some tool to help with organization.

Snarky

Mistakes are the way we learn things.
These mistakes... I mean, they could be a lot worse. (I also think that in an exercise like this, you see all the problems with the path you ended up going, but you don't really see all the problems with the other paths you could have chosen.)

The main takeaway here seems to be that there's been a lack of high-level planning or vision that meant important decisions weren't made and things just sort of muddled along. So what can we do about that going forward?

One option is that you and the other engine contributors take some major decisions and make some longer-term plans, and then inform the community of what you've decided. Whether that's merging with ScummVM or switching to SDL or whatever else you feel would be beneficial.

Or if you feel you need someone to advise you, pick a person/some people you would trust, and ask if they could draw up a plan or pick some choices.

Or if you think it's someone the community needs to decide, create a poll and have us vote.

Quote from: eri0o on Mon 01/01/2018 02:41:14
I am so sorry about the documentation thing

Yeah, same. That's one part I thought I could help out with. I started learning about the different documentation formats and tools, and I was planning to devote some time to it once I'd finished my degree and found a job, but that ended up taking a year longer than I anticipated, and other stuff kept cropping up.

eri0o

#4
Question. Going from Allegro 4.4 to 5 involves any meddling in the Editor? The reason I ask is because building the Editor is a LOT harder (configuring MSVS) to me than the Engine (typing "make [enter]"). My stupid idea would be building against Allegro 5 and fix each error until it builds, which is something very stupid, but makes the task brainless - assuming the calls to Allegro are mostly wrapped in the code so there aren't that many, IDNK if this is true though...

Crimson Wizard

#5
Quote from: eri0o on Mon 01/01/2018 16:41:40
Question. Going from Allegro 4.4 to 5 involves any meddling in the Editor?
Yes and no. Editor still uses Native.dll library written in C++ which uses Allegro 4 to do some things (probably not much now, but still). On other hand, whether Editor depends on Allegro 4 does not make any obligations for the engine to use same.

Quote from: eri0o on Mon 01/01/2018 16:41:40
My stupid idea would be building against Allegro 5 and fix each error until it builds, which is something very stupid, but makes the task brainless - assuming the calls to Allegro are mostly wrapped in the code so there aren't that many, IDNK if this is true though...
It's hard to say if this is the optimal course. Their APIs are incompatible in other ways, not only like in function names. First of all, Allegro 5 is event-based while Allegro 4 is poll-based. You will have to at least write a new startup code and an event loop with Allegro 5, and change AGS game code to respond to events instead of polling device states in any arbitrary moment in code. So, in the end, you would have to restructure the code somewhat.
Other thing that comes to mind, IIRC Allegro 5 drawing algorithm is different, it assumes that you set target surface first and all the subsequential operations are made on that surface. In Allegro 4 you need to pass the destination bitmap for every operation. So the order of drawing operations has to be double checked to prevent mixing drawing to separate destinations.
Probably all the graphic renderers and software filters will have to be rewritten completely if not just removed and replaced by something different, because they are not needed with Allegro 5. Therefore all the graphic initialization routine should be different.


I've mentioned an alternative recently: https://github.com/adventuregamestudio/ags/issues/403#issuecomment-350581132

I actually began coding it for the sake of experiment, but paused when all this conversation about ScummVM started. Maybe I should've upload the code somewhere in case others want to use it/participate, but it is in such raw state now with literally nothing visual working except for a window created with some debug overlay, that I am not certain.

Joseph DiPerla

I am not really a C++ or C# developer. I am still learning how to print "Hello World" ;-) But would it at this point maybe be easier to consider keeping the editor as is, but rewrite the engine and Native.dll from scratch so that it is a fresh, new engine that still functions the same so all the familiarity of AGS is still there. Or would that be too difficult/defeat the purpose?
Joseph DiPerla--- http://www.adventurestockpile.com
Play my Star Wars MMORPG: http://sw-bfs.com
See my Fiverr page for translation and other services: https://www.fiverr.com/josephdiperla
Google Plus Adventure Community: https://plus.google.com/communities/116504865864458899575

Crimson Wizard

#7
Quote from: Joseph DiPerla on Tue 02/01/2018 01:01:30
But would it at this point maybe be easier to consider keeping the editor as is, but rewrite the engine and Native.dll from scratch
Native.dll is a part of editor, which should be eliminated and replaced by C# code.

Quote from: Joseph DiPerla on Tue 02/01/2018 01:01:30
so that it is a fresh, new engine that still functions the same so all the familiarity of AGS is still there. Or would that be too difficult/defeat the purpose?
Depends on what purpose that is.

I was advocating full rewrite many times before, but now I am not interested anymore, so it's up for other people.
Also, IDK if that even makes sense considering new engines are starting to get released, which already supercede AGS in several important points (portability, script language, possibly unicode support since that's C#).

Danvzare

While this is all 100% true, and really showcases just how messed up AGS is.
I can't help but feel a sense of pride with this community, who have not only gathered around this clearly flawed engine and editor, but continue to release many great games with it. Even many commercial ones!

Despite these many MANY shortcomings, we all continue to use AGS. And that's just amazing. :-D

eri0o

Two things: as to all open source projects, it seems we need a leader, benevolent dictator.

Second thing: for me AGS is the combination of adventure features with being good with pixels. If the being good with pixels is important for everyone, then it appears something like SDL makes more sense - no penalty from the lots of bitmaps. If the pixels are not that important, then Allegro 5 seems like the proper backend, but in this case, for me, there other engines that appear to work better.

The greatest strength for me in AGS is the editor that is very good and the scripting language that makes no distinction between things that happen synchronously or asynchronously, so the code looks like a sequence of thoughts.

[delete}

#10
Quote from: eri0owe need a leader, benevolent dictator

You are referring to no one else than... Gurok.


(his cat looks even cuter)

Crimson Wizard

Quote from: eri0o on Tue 02/01/2018 15:01:47If the being good with pixels is important for everyone, then it appears something like SDL makes more sense - no penalty from the lots of bitmaps. If the pixels are not that important, then Allegro 5 seems like the proper backend

What do you mean by "good with pixels"?

Danvzare

Quote from: Crimson Wizard on Wed 03/01/2018 14:36:54
What do you mean by "good with pixels"?
He means it's good for low-res games.

morganw

#13
Since the future direction is undecided and different people have different ideas about what would be best, perhaps in the short term it is best to concentrate on what exists now and how to preserve it. So I would suggest this 2 step plan:

- Get the existing ScummVM port working with current features
This gives a route to try and keep older games working, plus a replacement engine that could potentially work with the existing editor (potentially just the debug tools would need changing?). Someone has kindly done a lot of work on this already so this is perhaps the easiest way forward.

- Look at the situation again
If the ScummVM port can't accommodate the compatibility quirks for old games, at least there is now a working engine for games which are current which does not have the issues of Allegro 4. If it can have fixes for older games, that is a bonus (and if not anyone who has the source for their old game is free to upgrade it through the existing editor and get it working). You haven't asked ScummVM devs to add a continually developed engine that will get updated for new features, just one additional legacy engine (so very little rocking of the boat for everyone involved)

None of the above need impact Alan's AGS4 cleanup branch, someone suggesting their own engine as a replacement, or someone who wants to start from scratch. If the ScummVM backend looks good for continual development at this point, and there are no forthcoming alternatives which show advantages, there is now the option branch it and drop the compatibility quirks or speak to ScummVM devs about ongoing development and support. If anyone would like me to post/suggest this in another thread/forum, or try speaking to fuzzie about what was already done, please let me know and I will do so.

Crimson Wizard

Quote from: Danvzare on Wed 03/01/2018 16:24:05
Quote from: Crimson Wizard on Wed 03/01/2018 14:36:54
What do you mean by "good with pixels"?
He means it's good for low-res games.

Then, what is the difference between how Allegro5 works with low-res games and SDL2 works with them?

Monsieur OUXX

#15
I would like to say something important though. I really mean it:
What you did, CW, was not useless. It was not useless. You can be extremely proud of yourself.

You (and others) have made AGS live through Windows XP, Vista, 7, 10! What it means is that, during close to a decade, there was a tool that was doing (almost) perfectly what it was designed for. Thanks to you. God knows what have might happened to the community if the working branch has been put "on hold" for a full refactoring? It might have died. People might have lost interest. They might have stopped their game in the middle of development. I'm not only thinking about games such as Gemini Rue, but I'm even thinking about games like "Indiana Jones and the Fountain of Youth" which very recently switched to Unity. Without a continuously working AGS, these projects might have died.

I'm perfectly aware of how 20 years of spaghetti code can kill AGS on the long run. But I also know how a rewrite from scratch could have killed it instantly. We'll never know how things could have went. But we know how they did go: AGS kept working almost smoothly for 10 more fucking years. Amazing.

So, thank you. Thank you thank you thank you.
 

Crimson Wizard

#16
Quote from: Monsieur OUXX on Sat 06/01/2018 20:03:55
But we know how they did go: AGS kept working almost smoothly for 10 more fucking years.

It really wasn't. It's not working smoothly, and it has a lot of big problems that should have been addressed years ago, but never were, reasons including we were busy with things we should not be doing in the first place. Like ones I mentioned above.

Simple example: I am spending months hacking Direct3D and OpenGL renderers, and still there. If AGS was working on SDL2, all this time could have been saved for something more important. Maybe I would still have motivation to work.

Quote from: Monsieur OUXX on Sat 06/01/2018 20:03:55but I'm even thinking about games like "Indiana Jones and the Fountain of Youth" which very recently switched to Unity.

Well, is not that a true illustration for what AGS project is.

Monsieur OUXX

#17
Welp. If you really insist on not seeing your true merits, then I can't stop you ;-) but I think you're wrong to rain on your own parade.
 

Monsieur OUXX

I'd say the first strong leadership decision to take would be to state : "starting from this date, we stop upgrades to AGS 3.x". All of them. AGS 3 is dead. All in favor?
 

Crimson Wizard

#19
I think I know what troubles me most. In the past people said many times that AGS has its own niche, as an easy to use hobbyist tool for adventure games. Other tools, like Unity, require quite some programming skill to start with, or if you use advanced tools like AC on Unity, that costs money.
If there were a real alternative to AGS - that allowed to just jump in and make characters walk around rooms and talk to each other without much hassle - then I could just walk away in peace. But it really hurts me to think that we screwed this opportunity to keep such tool running. It makes me mad to think that I am also responsible for this. I know what you may say that I weren't, but since I participated, I had a chance, and did not use it properly.

E: For example, I heard Wintermute is still developed as an opensource project. But I never used it, so IDK if it works same as AGS.


Quote from: Monsieur OUXX on Sun 07/01/2018 14:27:19
I'd say the first strong leadership decision to take would be to state : "starting from this date, we stop upgrades to AGS 3.x". All of them. AGS 3 is dead. All in favor?
Okay, I wasn't really going to do any more upgrades anyway.

SMF spam blocked by CleanTalk