Thoughts on refactoring and futher improving the engine

[Crimson Wizard]

You guys have really made an atounding work on the source, I feel shamed I haven't finished porting my changes yet

Don't worry, refactored code is practically unused atm anyway, and it will take more time to test it before it may become "official".

[Crimson Wizard]

The console seem to be bugged. It works well with DirectX 5 renderer, but when I run the game with Direct3D 9 renderer and open the console, following eror occurs:

Error: Mismatched colour depths

The game I tested this has 16-bit color depth.

[Crimson Wizard]

I would consider separating as much as possible Allegro from the data structures. For example, all the GUIs are dependent on it because they don't only hold this info, but also draw it.

I just made the first (and big) step towards this by replacing use of raw allegro BITMAP by IBitmap interface throughout the code (class names may be changed later, that's not a big deal).
This also gave me better understanding of how much Allegro is nested in the engine (...quite).
There's yet much work to be done improving drawing code, it is sprawling over engine and also not very safe.

There's a big problem related to plugins. Plugin interface exposes BITMAP type and return allegro bitmap objects on demand. If there are plugins that use BITMAP directly (that is - by reading/writing returned BITMAP objects), changing plugin interface will break compatibility.
For now I left some workarounds there, but they are hacky and very much unsafe.

The code is in my personal repository, I am waiting for JJS to create new master branch based on refactored code before applying such a big change to the central repo.
https://github.com/ivan-mogilko/ags-refactoring/tree/refactory_bitmap

Meanwhile I was also working on replacing FILE ptrs with stream objects.
https://github.com/ivan-mogilko/ags-refactoring/commits/refactory_file

When all this is done I will (probably) turn towards script interpreter. Don't remember if I mentioned that, but it will be nearly impossible to work with game classes without changing the way script interpreter addresses their members.


PS. Oh, right, and still waiting for (dead)monkey's utility classes, *wink, wink*

[JJS]

I created the master branch based on the current refactory branch. But is also has automatically formatted code and I removed the print_welcome_text() copy protection.

Also in my personal branch here: https://github.com/jjsat/ags/tree/master I added classes for mutex and threads. They are implemented by having a central header that includes the platform specific implementations. Those implementations are descendants of a common base class and are made available to the engine with a typedef. I looked a bit into how to implement this in an efficient way and this seems to be a common approach. Comments are very welcome.

[Crimson Wizard]

I created the master branch based on the current refactory branch. But is also has automatically formatted code and I removed the print_welcome_text() copy protection.

Ouch, erm.... I thought code formatting will be applied a bit later.... I don't know how I will pull my branch there, it will fail to merge.

EDIT: hmm, technically there is a way, but it requires partially reverting commit (only common & engine folders).

[JJS]

Well, I have no problem with deleting the branch again. The formatting changes are automatic anyway and the other change I mentioned is easy to recreate.

E: Or would it merge more easily if you applied the formatting to your branch too?

Code: bash

#!/bin/bash

astyle \
--recursive \
--exclude=Windows/include --exclude=iOS --exclude=scintilla --exclude=Manual --exclude=nativelibs --exclude=buildlibs --exclude=libsrc --exclude=libinclude \
--lineend=linux \
--preserve-date --suffix=none \
--style=allman --add-brackets \
--indent=spaces=2 --convert-tabs --indent-col1-comments --indent-preprocessor \
--align-reference=name \
--pad-oper --pad-header --align-pointer=name --unpad-paren \
--formatted \
*.cpp *.h *.c *.java

[Crimson Wizard]

E: Or would it merge more easily if you applied the formatting to your branch too?

I have doubts on that.
The problem is not that ALL the code is auto-formatted, the problem is that this formatting could change same lines I did and those two variants will conflict with each other: there will be change that make old code formatted and a change that replaces old code with new one (formatted or not). Git won't know which changes to take and this has to be done by hand... all those hungreds of lines I've changed.... I am not going to do that for second time

[JJS]

I deleted the branch.

[Crimson Wizard]

Well, probably reverting one commit was sufficient, but thanks anyway.

I am done with bitmaps for now (that needs heavy testing, but at least I may see them drawn both in engine and editor). I need couple of days more to finish with file streams, then I'll make a branch pull and we may make the auto-formatting.

[BigMc]

Will you do the auto-formatting also for the Editor? In this case, tzachs changes should be merged in first.

[Crimson Wizard]

Will you do the auto-formatting also for the Editor? In this case, tzachs changes should be merged in first.

I strongly doubt that's needed, Editor has nice code already. I think only one C++ file of AGS.Native DLL may benefit from autoformatting, namely: agsnative.cpp.

[Crimson Wizard]

An update.

In my personal branch (for now) I've just finished modifying the script interpreter, letting better control over data it reads and writes. Skipping technical details, the most important consequence is that now it may remap the memory addresses on fly when script wants to read or write object's variables, or even replace direct memory access with calling getter/setter functions if needed (the least has not been implemented yet, because not needed on its own at this very moment).

Simply put: it is now possible to rewrite Character, etc classes and register them as script exports while maintaining full backwards compatibility.
For example, I made a test by adding 100 extra integers to the beginning of Character class and ran a game with calls to player and character[N] objects (both calling functions and accessing variables pre-OO style).

There's a minor concern about speed. I noticed that running game with excessive operations (in rep exec) now adds about 1-4% to CPU (I have 3.4 GHz PC). I have few ideas how to lower that down a bit, but hopefully it won't be a big problem anyway.


There's, however, bigger problem, something I keep forgetting over time. Plugin interfaces. Argh.
AGS exposes a number of structures (character, etc) letting plugins read&write their values directly. Good for speed, but terrrrible for future development. That means that we cannot just get rid of old structures.
For example, assuming that a new class for character is written, it still will have to have the old CharacterInfo structure inside itself, which address will be given to plugins on demand. Same thing goes for all other exposed objects (Room Object, ViewFrame, etc).
I really do not know if there might be a better solution here. Thinking of possible workaround I could only imagine an ugly thing: to give plugins a separate struct with a copy of values and then copy the values back to real class - before and after each event that has been hooked by plugin. But that's probably don't worth it.

[Calin Leafshade]

I think, when considering the impact on the plugin interface, we need to remember that there are very few plugins in active use.

The only one i can think of is the snow/rain plugin which will be arguably obsolete one the engine can blend alpha channels and doesn't access any game data at all.

The Lua plugin would possibly be an issue but that is currently in active development and so it can roll with any changes made to the plugin interface.

[Crimson Wizard]

I think, when considering the impact on the plugin interface, we need to remember that there are very few plugins in active use.

I was hoping that it would be possible to obtain/reimplement sources for all engine plugins that were used for AGS games. If that is achieved we could remake plugin interface in such a way that would fully hide data structures and rebuild all plugins correspondingly. I am not aware how much plugins were actually used. I can only tell - from viewing Modules&Plugins forum - that there weren't billions of them, but may be couple of dozens, which gives me slight hope that may be realistic scenario.

[Calin Leafshade]

The snow/rain plugin was recoded by someone (monkey i think) and released as open source. I think it's in the repos.

All my plugins that have been used (AGSBlend and AGSSpriteFont) were also open sourced and put in the ags repos.

I don't know of any other plugins being used in any ags games.

[BigMc]

Even if the old plugin interface has to stay for compatibility, it may be a good idea to deprecate it and (if a plugin interface is still needed) create a new one in addition.

EDIT: One could also remove the old plugin interface and rewrite the plugins that are needed once someone comes across a game using them.

[Crimson Wizard]

On supporting extended characters in strings (UTF, Unicode). I remember it was quickly discussed back in the summer 2012, and monkey_05_06 made an argument that multibyte strings (in utf-8 encoding) have an advantage over widechar strings in that they are more easily ported. Documentation on C/C++ widechar type tells that it has different size on various platforms: Windows defines wchar_t as 2 byte value, while Linux (and MacOS?) defines it as 4 byte value.

But I was thinking that it is possible to use wide strings internally in the engine, and read/write them as portable multibyte strings in game data / saved games.

Multibyte strings require less memory if only standard latin characters are being used, and probably about same amount of memory if other language/extended characters are used.
On other hand, mb strings may be slower to manipulate with, and require bit more complex algorithms, since their characters have variable widths.

The question is, what type of string is more beneficial in our case?

[Sslaxx]

Neither option strikes me as particularly great, honestly. But extended character support is essential. Which'd be best for cross-platform use (including PSP, iOS and Android) - the varying size of widechar, or handling multibyte characters?

[Snarky]

But I was thinking that it is possible to use wide strings internally in the engine, and read/write them as portable multibyte strings in game data / saved games.

Seems unnecessarily complicated. I don't really know enough to say which option is best, but just on gut feeling I would lean towards multibyte strings.

[BigMc]

UTF-8 is becoming the standard pretty much everywhere. Also it's not like you have to reinvent the code to handle this.