Unicode Translations support test

Started by Crimson Wizard, Tue 22/06/2021 20:18:21

Previous topic - Next topic

Crimson Wizard

This was previously discussed in another thread, but I thought to make a dedicated post in hopes to draw more attention, because we really would benefit from more testers.

In short, I'm making an experiment to add partial Unicode support to AGS 3.6.0 now. Supporting unicode texts in all the game contents (game properties, script, etc) require extra work, so I decided to begin with translations, as TRS is a simple text file, which may be saved as UTF-8. Thus no big format changes are required.

There's a thorough technical explanation here: https://github.com/adventuregamestudio/ags/pull/1321

In a simplier words it works like this:
1. Engine can switch between ASCII/ANSI and UTF-8 text handling. ASCII/ANSI is a default mode for now.
2. Translations now have new "encoding" option that may tell engine what mode is expected.
3. When engine loads a new translation it checks the "encoding" option and switches to necessary mode.

This way both old ANSI and UTF-8 translations can work with the same engine, and in the same game.




DOWNLOAD:

UPDATE: This feature is now a part of 3.6.0 alpha, which may be downloaded here: https://www.adventuregamestudio.co.uk/forums/index.php?topic=58976.0
Because 3.6.0 is now based on SDL2 library, it temporarily requires SDL2.dll on Windows. You may get one here:
https://github.com/adventuregamestudio/ags/releases/download/v.3.6.0.0/SDL2-2.0.12-win32-x86.zip
Simply place SDL2.dll in the editor's program folder and it will be copied to the compiled game.

I strongly recommend to make a copy of your game project before opening in this version!
(in the worst case it will still be possible to rollback to 3.5.1 and 3.5.0, with certain manual hacks)


For example, a simple game which I used for a test:
https://github.com/adventuregamestudio/ags/files/6681700/test--unicode-source.zip
https://github.com/adventuregamestudio/ags/files/6681701/test--unicode-game.zip

Intercting with character or "orb" hotspot will display some translated lines.
Space key - starts String API test
T key - brings up GUI with text box, which fills a list by pressing Enter
D key - brings up GUI with list box, filled by game directory contents (this is to test dirs with unicode characters)



HOW TO USE UNICODE TRANSLATIONS:

1. Open an existing translation (TRS file) or create a new one.
2. On the top of the file there's a header with comments and options, it looks about like this:
Quote
// ** Translation settings are below
// ** Leave them as "DEFAULT" to use the game settings
// The normal font to use - DEFAULT or font number
//#NormalFont=DEFAULT
// The speech font to use - DEFAULT or font number
//#SpeechFont=DEFAULT
// Text direction - DEFAULT, LEFT or RIGHT
//#TextDirection=DEFAULT
If you created a new translation in this experimental editor, there will be another line "//#Encoding=", if not you may simply add it by hand:
Quote
// Text encoding hint
//#Encoding=UTF-8
// Source text encoding hint
//#GameEncoding=
3. Now convert and save the TRS text file as UTF-8. Note that it should be just UTF-8, not UTF-8 BOM (latter might cause corrupted texts in game).
4. Now add some translated lines in whatever "non-standard-latin" language you want: European, Cyrillic, Asian, etc, whatever you know. You may even combine multiple languages in one TRS.
5. Add a proper unicode TTF font to your game that supports the language(s) you used in TRS. Don't forget to actually assign it whenever you wish to see it (GUI, Game.NormalFont, and so on). You may also set #NormalFont and #SpeechFont in TRS file to this font's ID if you prefer.
6. Compile the new translation and a game.
7. Choose new translation in setup, and run the game.

SPECIAL NOTE on GameEncoding hint. This is required if your game texts contain extended ASCII characters (outside of standard latin alphabet). This option will help the engine correctly use these source game texts as translation keys.
Editor will fill it according to your current system locale, but if you must to fill it yourself here's instruction:
Open Game.agf in your game with a simple notepad-like text editor and look for the first line that will be something like: "<?xml version="1.0" encoding="windows-1252"?>". See this "windows-1252"? There may be a different number in your case. Remember this number, then go to the TRS and write it down like: "//#GameEncoding=.1252" (note the dot).
That should do the trick.

EXPECTED BEHAVIOR:

Editor:

1. Should update UTF-8 TRS file correctly with new texts, not damaging existing ones.

Engine:

Loading UTF-8 translation will make following unicode-aware:

1. Text display, at least with TTF fonts. I suspect WFN (bitmap fonts) will work too, if it contains glyphs in proper indices, it's just that I doubt if existing WFN editor allow to edit characters in slots >255.
Line splitting must work correctly, not corrupted in case there are several lines displayed, even if there's no spaces or linebreak characters added.

2. String API in scripts. Unicode texts got from translation may be combined, formatted, substrings extracted correctly, and so on. The remaining limitation is that there's no way to pass a unicode character as an argument to functions that expect 1 char, or receive them. For example, String.Chars[] array will likely work incorrectly at the moment.

3. Text input in the TextBox control. You may type unicode chars in there if you switch to non-English language in your system.

At the same time, any existing game or older translation file should still load and display as before (so long as they use their original fonts of course).

KNOWN ISSUES

1. There is a group of languages with special rules to gluing characters together, examples are Arabic and Persian, but maybe some more. This is currently not supported, their characters will be displayed but separated. This is something that has to be done as an additional step later.

2. If the game itself contains text with extended ASCII characters (127 -255) and you load UTF-8 translation, then the original text containing these extended ascii may cause problems. TBH I'm not yet sure what will happen in each particular situation, they may not display right, etc.
UPDATE: since the recent update, these lines will be correctly detected as "keys" for translation, so long as you provide correct "//#GameEncoding" option value (see instructions above).
But when there's no translation and original line is displayed - they will be displayed with extended characters replaced by "^".

Crimson Wizard

#1
UPDATE.

Made it work so that if the game texts are written with extended ASCII (like French, Spanish, German, Cyrillic etc characters) they are still correctly work as translation keys in UTF-8 TRS. But they *must* have translations, even if you simply copy the same line as translation. Original lines with extended ASCII will not display correctly while UTF-8 translation is enabled.

Or, in simplier words, you may have a game in e.g. French or Polish (ANSI), and provide UTF-8 Chinese translation, and that should work.

But to make this work you need to tell the engine what was the game's encoding. Currently this is done like this:
Open Game.agf in your game with a simple notepad-like text editor and look for the first line that will be something like: "<?xml version="1.0" encoding="windows-1252"?>". See this "windows-1252"? There may be a different number in your case. Remember this number, then go to the TRS and somewhere after "//#Encoding": write this option: "//#KeyEncoding=.1252" (note the dot), where 1252 is replaced with your number.
That should do the trick.
Probably I will make editor to add it automatically in the next update.

Crimson Wizard

#2
UPDATE.

Made Editor work correctly with UTF-8 TRS files when updating them with new lines. So hopefully now it won't be breaking your unicode text there.

(new download link in the first post)


EDIT: Fixed an error in the Editor regarding above; updated download links in the first post again.

Crimson Wizard

UPDATE: because the tests were pretty successful so far, this feature is now merged into 3.6.0 alpha, you can find its latest release here:
https://www.adventuregamestudio.co.uk/forums/index.php?topic=58976.0

Monsieur OUXX

#4
WOW. No one reacted to this but this is MAJOR news! It will unlock A LOT of potential for localisation.
 

Crimson Wizard

Quote from: Monsieur OUXX on Thu 02/09/2021 22:47:28
WOW. No one reacted to this but this is MAJOR news! It will unlock A LOT of potential for localisation.

Couple of people tested this, but yes, we still need more people to try this out and look for any problems.

Stranga

#6
Definitely testing this now! This is a big step, CW! Awesome stuff :)

Edit: So far so good! :D



Stranga

I think I've either found a bug or I'm missing some sort of special setting or something. When I export my translation files it seems to skip property strings (i.e item/hotspot/character ect. properties), is this a new thing? I can't exactly remember If it did before.

Crimson Wizard

#8
Quote from: Stranga on Tue 23/11/2021 22:18:56When I export my translation files it seems to skip property strings (i.e item/hotspot/character ect. properties), is this a new thing? I can't exactly remember If it did before.

Does this work in the previous versions (3.5.1 and earlier)?
In any case, I think they should, as someone may want to translate them using GetTranslation function.

EDIT: yes, this is very strange, but string custom properties were never added to translation. I checked ancient AGS 3.2.1 with the same result.
I believe this should be fixed. Temporarily the 2 workarounds I see is to either add these strings in script in a random place, or place them in TRS by hand.

Stranga

Thanks for the reply CW, and for addressing this via discord also.

Yes this did work in 3.5.1, I tested it just after the original post and the workaround worked perfectly by just manually adding the strings/lines to the bottom of the translation file.

Crimson Wizard

Quote from: Stranga on Wed 01/12/2021 10:36:26
Yes this did work in 3.5.1.

I have no idea how this could work in 3.5.1, because the editor code does not contain instructions for exporting custom properties, so I had to add them in 3.6.0.

To clarify, are you speaking of Custom Properties or of "Description"/"Name" property of objects?

Stranga

Apologies for the delay in responding. Yes, it was custom properties I set up for  "Description"/"Name" of Items to be specific, but I'm sure it obtain the same result for any custom properties.

Crimson Wizard

#12
Quote from: Stranga on Thu 06/01/2022 12:46:53
Yes, it was custom properties I set up for  "Description"/"Name" of Items to be specific, but I'm sure it obtain the same result for any custom properties.

I'm afraid there's a misunderstanding again. These are not "custom properties", they are regular properties. Custom properties are a completely different thing:
https://adventuregamestudio.github.io/ags-manual/CustomProperties.html

So i guess you were speaking about normal properties, that you see on a "properties" panel for the selected item?..

Could you try the latest 3.6.0 build and see if it still does not work?

SMF spam blocked by CleanTalk