Topic: AGS High-Priority Improvement Needs

[Crimson Wizard]

Here's an experimental engine build based on AGS 3.6.0 to let anyone willing to test it:
https://www.dropbox.com/s/cnlkmjtz9ckcodv/acwin--unicodetra.zip?dl=0

(it got to come with SDL2.dll for now)


VERY IMPORTANT:
The build is in experimental state and does not cover all potential issues. Noteably your game itself (script, etc) should not use any extended characters (chars 127-255), or text will be broken. This is a temporary problem (or so I hope).
Unicode text works only in translations (not script, or property fields).

HOW TO USE:
Create your game's translation as usual, and save TRS as UTF-8 without BOM. Then you can type any unicode texts as a translation there.
Note that Editor does not care if it's UTF so long as there's no BOM (BOM is basically a small binary header in the file, and it breaks TRA compilation).
Remember to provide a proper Unicode font for this translation (not the hacky kind of font you usually use for AGS translations).

WHAT IS SUPPOSED TO WORK so far:
* Texts displayed with TTF fonts
* Both Left-to-right and Right-to-left text modes.
* Using most String functions over translated unicode strings (appending, formatting, comparing, etc).
* TextBox control should accept unicode characters and properly work with them (like deleting with Backspace).

TODO:
* Some String functions only accept or return byte-chars, which makes them incompatible with unicode. Such as: AppendChar and Chars[]. We'd need to either change these or add alternatives.
Also, not sure yet how to correctly support individual unicode characters in String.Format (unicode strings work with %s, just like normal strings).
* Text parser
* Saving and loading strings using File* (or maybe it already works, I have not tested it).

Example of a test project (AGS 3.5.1) with russian translation:
https://github.com/adventuregamestudio/ags/files/6643382/test--unicode.zip

Wait, do I understand correctly I'll be able to use traditional Chinese characters?

Hopefully yes. But really, you tell me if it works, because I don't know how to type these.

[Mehrdad]

Thanks a lot, CW for the fast feature.
Is it only works on the translation option?
I didn't use translation yet. I used 

Code: Adventure Game Studio

1. player.Say(" something ");

and doesn't work for the Persian language

Where is my wrong and what can I do?

[Crimson Wizard]

Thanks a lot, CW for the fast feature.
Is it only works on the translation option?

It's not a feature yet, it's only an experiment. Yes, for now it only works if you put language in translation (TRS), because that was easiest to do.

So, if you do not want to bother with real english language, I may only suggest to write simple placeholders in the script (like "Character tells his name") and then provide real text in proper language in translation.

For now this is made only for a test, and also maybe to allow translate existing games to new languages.


EDIT: You may download the ready project I posted above: https://github.com/adventuregamestudio/ags/files/6643382/test--unicode.zip
And create a more translations for it, or replace existing one.
To test string operations press Space key in the game.


UPDATE: Uploaded new version, now supports TextBox input so you can type unicode characters there.

!!! Just don't forget to assign a proper unicode font to GUI controls where you display these texts, otherwise you'll see garbage.

[Crimson Wizard]

I made a test with Persian characters, and found that Right-to-Left text direction is not working properly at the moment (it breaks some characters), so this got to be fixed.
It works left-to-right though, but obviously it means that text is inverted.
UPDATE: updated build: https://www.dropbox.com/s/cnlkmjtz9ckcodv/acwin--unicodetra.zip?dl=0

Spoiler: ShowHide

Idk if it's a correct text or not, I used Google translate, and maybe not the best font for this language:

I was not able to make Chinese work for some reason, could be I use wrong font or something else, I need to investigate this further.
UPDATE2:
Ok here you go Chinese:

Spoiler: ShowHide

I used this free GNU font: https://unifoundry.com/unifont/index.html

[Mehrdad]

Thanks a lot for try Persian font. It seems couldn't stick letters together. I typed improve your text in this picture

[Crimson Wizard]

Thanks a lot for try Persian font. It seems couldn't stick letters together. I typed improve your text in this picture

Sorry, to clarify, did it work for you in game, or you painted over the screenshot?

The letters not joining may be either a problem of a font, font's size, or font library that draws it.

I found some technical description of this problem here: https://stackoverflow.com/questions/49110006/arabic-joined-up-text-in-freetype
It sounds like this may be complicated with these kinds of languages.

We may also see if SDL2 library has better TTF support for this than our current library.
Some more technical info found: https://discourse.libsdl.org/t/sdl-for-farsi-or-persian-text/25292/4

[Mehrdad]

I painted on the picture. I want you see right typing. Your example shows wrong for Persian letters because letters don't stick together.
It doesn't relate to fonts

I really accept that Persian letters are complicated. Sometimes they are stick and sometimes not.

There is a great tools for type Persian in engines. Maybe help you
https://leomoon.com/downloads/desktop-apps/leomoon-parsinegar/

[Laura Hunt]

This is amazing, CW! The Chinese market is huge, and the idea that the "big languages" to go for are ES, IT, FR, DE, PT-BR and RU is quickly becoming obsolete, so being able to localize to Chinese without using weird hacks is going to be a massive game-changer.

[eri0o]

Edit: nevermind..._{cw, check the issue in GH, I think I got something}

[Manu]

Probably it's not a high priority, but talking about the translations, have you ever thought about a feature like "Move Default Language to Translation File"?
This action will move all the strings of the game from the code to a file such as English.trs, replacing the text with placeholders. Placeholders could use a few words of the text (just to make it more readable), followed by a random number to make them unique (something like "%HIMARYHOWAREYOU-157563%", or maybe something smarter I don't know).

In English.trs you will have 2 lines, the first one is the placeholder:

Code: Adventure Game Studio

1. %HIMARYHOWAREYOU-157563%
2. Hi Mary, how are you?

The other languages will be the same, but when you create a new translation, instead of 2 lines for each text, the system could generate 3:

Code: Adventure Game Studio

1. %HIMARYHOWAREYOU-157563%
2. // Hi Mary, how are you?
3.  

The second row is the full text, as a reference for the translator, but commented. I guess the compiler will remove first the comments, so it will end always with 2 lines.

This way, when you want to change the main language text, you just change the trs and you don't need to adjust the other languages, which is a pain now.

When you add a new text, eg. Say("Whatever"), you can just add it to the code, as always, this won't break the game. The engine will detect if it's a placeholder or not, and if it's not a placeholder it will show the hardcoded text. But when you do an "Update", the string will be moved to the translation file.


Maybe it's a crazy idea, but I thought it was worth suggesting it.

[Crimson Wizard]

@emabolo, similar thing was proposed before, and afaik this is how they often script translation systems in other engines - by using keywords and line identifiers in script instead of a literal text.

One of the alternatives is to have strings like "%ID% TEXT" where you can change TEXT part as much as you like, as only %ID% will be important for translations.

In theory you can already do that, except you have to do everything by hand and there will be "default language" selection in the engine and setup which should not be chosen by player.
Or script translation system that works similar to described alternative (I think some people already did this in their AGS games)

If implemented as a feature, transition to such system should be done with careful planning, and guess we may keep support of an old method of having texts in script in case someone wants to do it old way.

[Manu]

In theory you can already do that, except you have to do everything by hand and there will be "default language" selection in the engine and setup which should not be chosen by player.
Or script translation system that works similar to described alternative (I think some people already did this in their AGS games)

Right, in fact, more or less it's all already supported. It's more a utility to make the transition from hardcoded to external translation easier. And once you have migrated, AGS should just avoid running the game with placeholders instead of real text.

[eri0o]

@emabolo, Have you ever used Speech Center plugin?

I think switching to a token based dialog in the current editor will be confusing, in IDEs that have support for this you usually can edit the text of the token directly where it's used - it parses the AST and replaces the element directly in it, so you can read how things are, even with the token underneath. This sort of thing is not possible (or not possible easily) with Scintilla (specially our aging one), which is really guided towards text editors and not AST editors. So once you get a dialog with many sentences you will be alt+tabbing screens like no tomorrow. At minimum it would have to support a split view where you have the text and the text with the replaced tokens in other panel so you can see the context of sentences - faster than running the game.

I also get a feeling this auto token generation can reuse code from the speech numbering - and it's possibly not that hard to parse the sentence as is generated today and get just the speech line number and then join this with the character and handmade your own string token replacement - you now load this character+line_number from a dictionary and replace the string.

[Manu]

@emabolo, Have you ever used Speech Center plugin?

No, but if you think it's useful to manage translations I'll try it. Thanks

I think switching to a token based dialog in the current editor will be confusing, in IDEs that have support for this you usually can edit the text of the token directly where it's used

In my idea this should be an option, I wasn't thinking about switching to tokens by default. Also, I think this is an option that should be used when the game is complete or almost complete.

I can think of another advantage: imagine your game doesn't have translations, it's in English only, but you want to send all the text to a professional proofreader. In "hardcoded mode", when you receive the corrections, you will have to copy and paste everything, line by line, into the code. Argh. While if you activate the "tokens mode", it's just a matter of copying the file to the game folder. Much easier.

[Laura Hunt]

I can think of another advantage: imagine your game doesn't have translations, it's in English only, but you want to send all the text to a professional proofreader. In "hardcoded mode", when you receive the corrections, you will have to copy and paste everything, line by line, into the code. Argh. While if you activate the "tokens mode", it's just a matter of copying the file to the game folder. Much easier.

The "proper", or at least most convenient, way to go about this would be in fact to generate a .trs file, call it something like "EN_corrected", send it to the proofreader and tell them to add their corrections below each original line, as if they were translating the file. Then you just use that file as the game's default "translation": https://www.adventuregamestudio.co.uk/forums/index.php?topic=58812

[Crimson Wizard]

IMO identifying lines with tokens in the strings is more convenient and safe than relying on actual text, which may be changed anytime, which in current system will break any existing translations.
There are other reasons to not rely on the text itself: long known issue is that one line in one language could be translated with different lines in another language depending on the context. You won't have this problem with the unique tokens.

Naturally this token may be merged with the voice-over ID and then we have one token for everything (translation and voice-over).

Of course, the token does not have to replace the actual text, it may simply precede it as an optional method to identify the line. Then you also have the text which may be used either as a default language or just for the reference.

[Manu]

The "proper", or at least most convenient, way to go about this would be in fact to generate a .trs file, call it something like "EN_corrected", send it to the proofreader and tell them to add their corrections below each original line, as if they were translating the file. Then you just use that file as the game's default "translation": https://www.adventuregamestudio.co.uk/forums/index.php?topic=58812

Sure, but then you end up with a hardcoded English version (where the English text is used as tokens basically), and a "real" English version in the file. This is what I was going to do when I realized it was not a clean solution and a token version would have been nice

[Mehrdad]

@Crimson Wizard
Can you send me your Persian font ? I tested many fonts and AGS doesn't show all symbols on the Font page

[Crimson Wizard]

@Crimson Wizard
Can you send me your Persian font ? I tested many fonts and AGS doesn't show all symbols on the Font page

I linked it above: https://unifoundry.com/unifont/index.html
Here's direct download link: https://unifoundry.com/pub/unifont/unifont-13.0.06/font-builds/unifont-13.0.06.ttf

EDIT: I now remembered that for Persian test I simply used Arial.ttf from Windows fonts.

But AGS Editor cannot show all unicode symbols on the font preview, because it shows only first 256 indexes, while unicode font may contain thousands. I only changed the engine, not the editor so far.

[eri0o]

@Mehrdad, if you want to try, this is a build of the engine of ags3 with only support for translated Arabian languages: ags_full_persian.zip

It doesn't have bidirectional text support, so you need to set it in the Editor for the specific game. Below was using DejaVu Sans. It does breaks lots of things so may not be possible to roll it in ags3. It's using CW changes and a different font renderer for ttf fonts. It still has some bugs but it comes with text shaping. Try with GNU Unifont too, I couldn't tell if it was correctly shaped or not on non-anti aliased fonts.

Spoiler: ShowHide