Choosing a way to go

[Crimson Wizard]

Alright, this is extremely stupid, but I found myself in a big confusion... again.

I needed some utility classes to continue work on AGS features (like custom resolutions and limits), - primarily hashmap, - so I wrote ones. But now, when I was about to push them, I started to wonder if I am doing a right thing rewriting what already exists. Does anyone remember why we decided not to use STL in AGS? Or did we? I think I've lost a track of reasoning there. When we started all this there was a talk about writing our own utility classes. Monkey wanted to do this, but after few months he told he can't. So I decided to do this instead, but wasn't that some kind of inertia, not rational thinking?

Indeed Hash Map is only included in C++11 library (as unordered_map), therefore we would need to move to C++11. This means we need to reorganize the way AGS is built. For instance, get rid of pre-built Allegro libraries for Windows and make everyone build them on their own from source code.
This will take some time... meaning other work will be delayed.

This all is extremely stupid, but I do not feel apt to decide such things. This all gets real results further and further, and I keep wondering if I'll ever finish what I started. I feel like I am rethinking same things over and over again, without being able to get to conclusion.
Meanwhile there was talk about writing a "new AGS" by other people. I don't know, have they started? Maybe what I do will become irrelevant after a short while?


It bugs me that I could do alot more if I weren't so unsure of what is right way to do.

[Alberth]

I needed some utility classes to continue work on AGS features (like custom resolutions and limits), - primarily hashmap, - so I wrote ones. But now, when I was about to push them, I started to wonder if I am doing a right thing rewriting what already exists. Does anyone remember why we decided not to use STL in AGS? Or did we? I think I've lost a track of reasoning there. When we started all this there was a talk about writing our own utility classes. Monkey wanted to do this, but after few months he told he can't. So I decided to do this instead, but wasn't that some kind of inertia, not rational thinking?

I wondered about that too, after finding two files implementing strings, but unilt until I actually understand the code, I assume it has a reason.

Indeed Hash Map is only included in C++11 library (as unordered_map), therefore we would need to move to C++11. This means we need to reorganize the way AGS is built. For instance, get rid of pre-built Allegro libraries for Windows and make everyone build them on their own from source code.
This will take some time... meaning other work will be delayed.

It seems to me that any switch is not going to happen at any short term, even if you can decide now you will make the switch. Given that you already have the classes, and the switch is thus not a pre-condition. I'd separate both issues, and implement what you want to make now without c++11, while discussing a switch like that. After the decision is "we will switch", I think you should start finding a path towards c++11. Your classes may get removed then again.

This all is extremely stupid, but I do not feel apt to decide such things. This all gets real results further and further, and I keep wondering if I'll ever finish what I started. I feel like I am rethinking same things over and over again, without being able to get to conclusion.

There are always bigger things in my experience. If you think far enough, you will get yourself in quick sand with every step that you try.
In my experience it helps if you focus on the most concrete next thing that needs to happen. Anything beyond that can wait until tomorrow.

Meanwhile there was talk about writing a "new AGS" by other people. I don't know, have they started? Maybe what I do will become irrelevant after a short while?

Rewrite from scratch is an often suggested idea, but the amount of work is staggering, and it remains to be seen whether it will give a better result. In addition, with a program being used by a community, you cannot just stop development and bugfixing the existing thing. If you do that for a few years, the community will die. People do not use a non-maintained program a few years with maintained free programs doing the same thing nearby (which I guess does exist in this case).

It bugs me that I could do alot more if I weren't so unsure of what is right way to do.

In my experience, nobody knows what the right way is. You can discuss it, and make nice plans, but only one minor change is needed to make all plans obsolete. In my projects I stopped planning ahead too far. I make changes only as much as needed to make the new thingie work for the one step I want to make 'today'. Tomorrow is a new day for deciding what to do next.
It never hurts to have a bit of a global high-level plan of course, but it should be high-level enough not to make abrupt changes.

[Crimson Wizard]

IIRC the reason I was somewhat reluctant to use STL was this note from ScummVM coding conventions:

ScummVM is written in a subset of C++. Due to limitations of the C++ run-times on various platforms, the following features cannot currently be used:

C++ exceptions (throw/catch): Not all C++ compilers support these correctly (esp. on embedded systems), and exception support incurs a noticeable overhead in binary size.
C++ RTTI (run-time type information, as in dynamic_cast<>): This incurs a (to us) severe overhead in binary size, since the static tables used for objects will get bigger.
Global C++ objects: Their constructors / destructors will not be called on certain targets, causing all kinds of bad problems and requiring ugly workarounds. (The GCC option -Wglobal-constructors helps finding code doing this.)
Furthermore, the standard C++ library is a big no-no. Besides usually heavily relying on the above mentioned features, it also sucks up rather more resources than we would like to, so we have our own replacements for various container classes etc.

But ScummVM is special in the way it is supposed to be run on all kinds of uncommon platforms.
Is this important for us? Our platforms so far are Win, Linux, Android, iOS, PSP and, potentially, Mac OSX.
I recall seeing complaints that programs with STL works too slow when ported on PSP, but it seem to be rather weak on its own.


E: Also here: http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html

* std STL allocators are painful to work with and lead to code bloat and sub-optimal performance.
* Existing std STL implementations use deep function calls. This results in low performance with compilers that are weak at inlining
* The STL puts an emphasis on correctness before practicality and performance

[Alberth]

I am aware of previous STL versions being bad, the first implementation I used in 1997 required inclusion of strings as first #include, or it wouldn't work :p
Also, linkers didn't understand they could throw away duplicate and unused functions. That somewhat holds today as well, it's very easy to get a 1MB executable if you compile with debugging information.

I just use the containers and strings though, I never saw much use in the algorithms.


A few months ago we have switched to C++11 in my own project based on a wish from my fellow project member. For me it's an experiment to see what C++11 has to offer. We mostly use the containers, strings, and unique and shared pointers, which I find very nice.
We aim at desktop system currently, where only linux is actually working, so hardware speed and memory use are not that relevant for us.


My first reactions on your quotes was "How old is that?", "What did they try?", and "How big were the differences?". STL offers more functionality than you generally need, look eg at the methods of any container template. Also, it's generic code rather than special purpose code (not optimized for a particular use case). Both make that STL templates are bigger, slower, or both. ScummVM doesn't mention a date, and the link seems to refer to something of 2007, which makes the C++ compilers they used at least 7 years old, which is very old (at least at Linux).

Some people are just so obsessed with performance, that they would write everything as custom solution rather than accept 0.1% performance loss. In my book that counts as premature optimization. You don't rewrite code that is not the bottleneck, except in extreme cases.


For AGS, I don't know the right answer, but I'll give it a shot. I only know about Linux and some of its considerations. Anything outside that area (such as Windows libraries, or PSP/iOS etc) is not taken into account.

Under the assumption it consists of an editor, a compiler, and a runtime system, I'd say there is no reason not to use STL for the editor and the compiler. You don't do development at a small slow system.
For the runtime, it may be different. However, depending on how smart the compiler is, it may also be less needed.

Given that the software is functioning, and already exists for many years, there is no immediate need to do any big rewrite or conversion, I think. There are replacements available, and they work. Rewriting implementations won't give any progress in features, and may introduce new bugs. If you want to make a step towards STL or C++11 (ie there are no blocking obstacles for all platforms), a more likely scenario would be that you only allow use of STL or C++11 (that is, remove the restriction of not using templates, for example), and let time decide.

[Crimson Wizard]

Ok, so, use STL unless something won't work well...

I feel inclined to use ref-counted strings though (I think they are more convenient to work with when used as return values; although I am aware they require careful treatment when shared between threads), and AFAIK STL strings are not reference-counted by standard (just some of the implementations have this).

[Alberth]

With C++11, you would do

Code: ags

std::shared_ptr<std::string>

for such strings
Threads is a good point though, I don't know what STL says about threads.

[Crimson Wizard]

With C++11, you would do

Code: ags

std::shared_ptr<std::string>

for such strings

AFAIK it has move constructors to evade copying when returning a value, if I uderstand the meaning of move constructor right.


On other topic, according to the readme supplied, the PSP port appears to be only "partially" working. Also, I do not recall people asking about PSP port much. The PSP port thread is not even made a Sticky (unlike Android and iOS). I wonder if we need to keep it.
Does anyone know how many people are using it and if there's an actual requirement for this port?

[Alberth]

Indeed, they added move thingies in C++11. Do do similar tricks with construction, where you can construct a new object directly in the container, ie

Code: cpp

std::vector<MyClass> v; v.emplace_back(constructor-args);

. Shared pointers do something similar with

Code: cpp

auto s = std::make_shared<std::string>("hi");

[monkey0506]

While I'm all in favor of starting to port the codebase to C++11 (it's been the official standard for three years now), I don't think that shared_ptr<string> really serves the same purpose as a reference-counted string. One of the benefits of a proper ref-counted string is that it also provides string pooling. For this we'd need a wrapper class to provide COW behavior if the shared_ptr is not unique. Also, while we're at it, we might consider using std::u32string instead. We could convert quite easily to/from UTF-8 as needed, but bear in mind that std::string doesn't guarantee its operations to operate on characters but just bytes (UTF-8 characters can consume up to 4 bytes), so UTF-8 strings could easily become corrupted. Using u32string would provide this guarantee for UTF-32 characters, and help prevent corruption.

[Crimson Wizard]

While this all is in consideration, the indisputable step is getting rid of precompiled 3rd party libs. (by "indisputable" I mean that this will be a nice thing to do anyway). Does someone want to experiment with that?

Speaking of C++11, it appears to me that there's no guarantee yet that all compilers we use for AGS support same portions of its standart. We would either need to restrict the usage to basic C++11 features, or attempt to use same compiler, if that's even possible, e.g. set up MinGW compiler on Windows to have relatively close results between Windows & Linux builds. Not sure if that is even a problem, so just speculating about possible hazards.

(Meanwhile I could just use either "std::map" or simple custom class temporarily, where I wanted to use hash map).

EDIT: I found out that MSVS 2008 already has a non-standart hash_map in stdext namespace. Latest GCC has unordered_map that compiles even for C++03 (I think, but I need to re-check).
I wonder if that would be possible to use typedef/macro trick to switch between them depending on compiler. But I'll have to test if all ports compile with that.

EDIT2: Duh! MSVS 2008 already has "unordered_map", just in std::tr1 namespace.

EDIT3: Ok, GCC on Linux has tr1::unordered_map too (which is natural, I guess, since TR1 was first variant of new standard), so I can use that until we switch completely to MSVC with C++11 support.
The onlt problem is that TR1 version has slightly different interface, so we'll have to fix that upon transition.
BTW, TR1 namespace has such things as shared_ptr too, which is very nice.