Adventure Game Studio | Forums

AGS Support => Modules, Plugins & Tools => Topic started by: Snarky on Tue 18/04/2017 19:22:17

Title: MODULE: TotalLipSync v0.5
Post by: Snarky on Tue 18/04/2017 19:22:17
TotalLipSync

(http://i.imgur.com/LxqtIe0.gif)
(Character head/animation by Preston Blair. King Graham sprite ripped from King's Quest II VGA (http://www.agdinteractive.com/games/kq2/index.html) by AGDI. Background from AGS Awards Ceremony by Ali.)

Download TotalLipSync Module (v0.5) (https://github.com/messengerbag/TotalLipSync/releases/download/v0.5/TotalLipSync-0.5.scm)

TotalLipSync is a module for voice-based lip sync. It allows you to play back speech animations that have been synchronized with voice clips. TotalLipSync has all the same capabilities as the voice-based lip sync that is built into AGS, and offers the following additional advantages:

How to use
You are now ready to use the module. Add the code to initialize TotalLipSync on startup:

Code (ags) Select
function game_start()
{
  TotalLipSync.Init(eLipSyncRhubarb);    // Or whatever lip sync format you're using
  TotalLipSync.AutoMapPhonemes();
}

Or if you want a custom phonemes-to-frames mapping:

Code (ags) Select
function game_start()
{
  TotalLipSync.Init(eLipSyncPamelaIgnoreStress);

  TotalLipSync.AddPhonemeMappings("None",0);
  TotalLipSync.AddPhonemeMappings("B/M/P",1);
  TotalLipSync.AddPhonemeMappings("S/Z/IH/IY/SH/T/TH/D/DH/JH/N/NG/ZH",2);
  TotalLipSync.AddPhonemeMappings("EH/CH/ER/EY/G/K/R/Y/HH",3);
  TotalLipSync.AddPhonemeMappings("AY/AA/AH/AE",4);
  TotalLipSync.AddPhonemeMappings("AO/AW/UH",5);
  TotalLipSync.AddPhonemeMappings("W/OW/OY/UW",6);
  // Frame 7 unassigned to match default Moho mapping
  TotalLipSync.AddPhonemeMappings("F/V",8);
  TotalLipSync.AddPhonemeMappings("L",9);
}

To speak a line with lip syncing, you simply call the extender functions Character.SaySync() or Character.SayAtSync(), using a speech clip prefix:

Code (ags) Select
  cGraham.SaySync("&1 This line will be animated with lip sync");
  cGraham.SayAtSync(320, 100, 240, "&2 ... and so will this");    // x_left, y_top, width, message

And that's all there is to it! (If you don't use a speech clip prefix, or if there is no matching sync file, the speech animation won't play at all.)

Phoneme-to-frame mappings
The principle of lip syncing is that different sounds (phonemes) correspond to different mouth shapes. If we display an animation frame with the right mouth shape at the same time as that sound appears in the audio being played, the animation will seem to match the speech. The first step, then, is to identify the phonemes and timing of them in the speech (that's what the tools listed above are for), and the second step is to choose an appropriate animation frame for each phoneme. We usually don't use different animation frames for all the different phonemes, so we combine phonemes into groups that are all mapped to a single frame. The different tools have different sets of phonemes (or phoneme groups), so we have to define different mappings from phonemes to frames.

So here is the default mapping for each data format used by TotalLipSync. It has been set up for a speech animation with ten different frames, each representing a different mouth position. (This is a fairly standard setup.) If you stick to these frames and these mappings, you can use the same speech view no matter what lip sync tool or data format you use:

Spoiler
Frame
Description
Preston Blair (http://www.egusd.net/franklinhs/mediacom/Animation1/Animation1_Term1/Animating_Dialog.htm)
Rhubarb (https://github.com/DanielSWolf/rhubarb-lip-sync#mouth-shapes)
AGD2 (http://www.adventuregamestudio.co.uk/forums/index.php?topic=34516.msg451567#msg451567)
Rhubarb
phoneme ID
Moho
phoneme
Pamela
phonemes
0Mouth closed
(or slack)
[slack or same as 1](http://i.imgur.com/4N1FDRM.png)(http://i.imgur.com/mFj4T66.png)XrestNone
1M, B, P(http://i.imgur.com/pL3Xun9.png)(http://i.imgur.com/8JEJP1C.png)(http://i.imgur.com/O0Hc0h3.png)AMBPM/B/P
2Various consonants,
(Rhubarb: Ee-type
sounds)
(http://i.imgur.com/4oE8iRU.png)(http://i.imgur.com/kmwbqLh.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-B.png)(http://i.imgur.com/yJ09OGt.png)BetcK/S/T/D/G/DH/
TH/R/HH/CH/Y/N/
NG/SH/Z/ZH/JH
3Eh-type sounds,
(Non-Rhubarb:
Ee-type sounds)
(http://i.imgur.com/SgrvtEc.png)(http://i.imgur.com/lLqvM2F.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-C.png)(http://i.imgur.com/8iSeM1A.png)CEIH/IY/EH/AH/
EY/AW/ER
4Ah-type and
I-type sounds
(http://i.imgur.com/rI4vykO.png)(http://i.imgur.com/antVI3z.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-D.png)(http://i.imgur.com/I8HbWLe.png)DAIAA/AE/AY
5Aww-type sounds,
Ow-type sounds
(can also go in 6)
(http://i.imgur.com/EklC1df.png)(http://i.imgur.com/c7kxnRN.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-E.png)(http://i.imgur.com/61rjPIo.png)EOAO/OW
6U-type and
Oo-type sounds
(Non-Moho: W)
(http://i.imgur.com/g80Gpg6.png)(http://i.imgur.com/x7xVBHH.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-F.png)(http://i.imgur.com/Xm9dnSG.png)FUUW/OY/UH
7Moho: W(http://i.imgur.com/zHrqjrs.png)[same as 6][same as 6][same as 6]WQW
8F, V(http://i.imgur.com/GaAm0Bn.png)(http://i.imgur.com/xIHsWOg.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-G.png)(http://i.imgur.com/HZLTsls.png)GFVF/V
9L
(Th-type sounds
can also go here,
rather than in 2)
(http://i.imgur.com/ff6DT70.png)(http://i.imgur.com/d1LeVxT.png)(https://github.com/DanielSWolf/rhubarb-lip-sync/raw/master/img/ken-H.png)(http://i.imgur.com/SEE4vHV.png)HLL
[close]
Where to get it
TotalLipSync is hosted on Github (mainly just as a way for me to learn about how Github works):
https://github.com/messengerbag/TotalLipSync

You can download the current release from there:

Download TotalLipSync Module (v0.5) (https://github.com/messengerbag/TotalLipSync/releases/download/v0.5/TotalLipSync-0.5.scm)

Known bugs
None

Change log
0.5
-Added APIs to get the currently lip syncing character, the current phoneme and current frame.

0.4
-Fixed support for Sierra-style speech
-Minor bug fixes for edge-cases
-Documentation

0.2 (http://www.adventuregamestudio.co.uk/forums/index.php?topic=54666.msg636558761#msg636558761) (pre-release)
-Added support for Papagayo/Moho Switch (.dat), Annosoft SAPI 5.1 LipSync (.anno) and Rhubarb (.tsv)

0.1 (http://www.adventuregamestudio.co.uk/forums/index.php?topic=54666.msg636558645#msg636558645) (pre-release)
-Pamela support for LucasArts-style speech

Originally based on code by Calin Leafshade (http://www.adventuregamestudio.co.uk/forums/index.php?topic=36284.msg554642#msg554642) (though very little of it remains in the current version).
Thanks to Grundislav for providing a speech view used in development and testing of the module!
Title: Re: MODULE: TotalLipSync v0.4
Post by: Snarky on Tue 18/04/2017 19:23:02
A couple more things:

Pamela and Annosoft SAPI 5.1 LipSync use almost exactly the same phoneme sets, with only minor variation (TotalLipSync is not case sensitive). Pamela can tag vowels with three levels of stress (0-2), e.g. AY0, UW1. This is not particularly useful in AGS, and should be ignored (by setting TotalLipSync.Init(ePamelaIgnoreStress)) unless there's good reason not to. Anyway, here's the full list and what they represent (ones where Annosoft differs emphasized):

Spoiler
IPA
Example
PamelaAnno
[silence]-Nonex
ɑ:, ɒfather, box (AmEng)AAAA
æat, snackAEAE
ʌ, əhut, aloneAHAH
ɑ, ɔthaw, dogAOAO
cow, outAWAW
hide, guyAYAY
bbangBb
cheeseCHCH
ddamnDd
ðthese, batheDHDH
ɛ, ɛəbed, bearEHEH
ɜ:, ɚhurt, butterERER
ate, baitEYEY
ffineFf
ggoodGg
hhouseHHh
ɪ, ɪəit, fearIHIH
i:eat, freeIYIY
gee, jawJHj
kkey, crushKk
llipLl
mmonkeyMm
nnoNn
ŋping, pongNGNG
əʊoak, slowOWOW
ɔɪtoyOYOY
pputPp
rreadRr
ssapSs
ʃsharpSHSH
ttotalTt
θthinTHTH
ʊ, ʊəgood, poorUHUH
u:youUWUW
vvikingVv
wwe, questionWw
jyieldYy
zzooZz
ʒseizure, genreZHZH
(Based on the in-app list in PAMELA (https://users.monash.edu.au/~myless/catnap/pamela3/) and the Annosoft list here (http://www.annosoft.com/sapi_lipsync/docs/group__anno40.html).)
[close]

Also, if you use Rhubarb to do the lip syncing, this could be helpful:

Quote from: Snarky on Sat 15/04/2017 12:30:44You don't need to compile Rhubarb, just get the latest release for Windows or OSX: https://github.com/DanielSWolf/rhubarb-lip-sync/releases

The script doesn't call Rhubarb, you'll have to do all of that. Take all the speech clips, convert them to .wav if necessary, copy them into the Rhubarb directory, and for each one, call "rhubarb.exe myclip.wav > myclip.tsv" (where "myclip" is the name of the clip). You can also put the text corresponding to each clip in individual .txt files to assist with the speech recognition, and then you'd call "rhubarb.exe myclip.wav -d myclip.txt > myclip.tsv". Then once that's done, copy all the .tsv files over into the directory of your compiled AGS game, and the module will read them.

Obviously this process is tedious, and also Rhubarb takes quite a while to process each clip, so if you have more than a couple of dozen clips you'll definitely want to automate it (you could write a batch file to go through and process each .wav file in the directory)

In fact, I wrote a very simple version of such a batch file:

Code (bash) Select
for %%F in (clips/*.wav) do (
    rhubarb.exe clips/%%~nxF -d guide/%%~nF.txt > sync/%%~nF.tsv
)

This assumes that the voice clips are in a folder called "clips/" inside the Rhubarb directory, that the text files are in a folder called "guide/", and that there is a folder called "sync/" where the .tsv files will be written. It also requires a .txt file for each .wav file. So there are a lot of possible improvements. Save this in a text file in the Rhubarb directory and name it something like agsbatch.bat, and you can run it to process all the speech clips in one go (which might take a while!).
Title: Re: MODULE: TotalLipSync v0.4
Post by: Mehrdad on Wed 19/04/2017 15:15:42
Hey Snarky . It's great module . Nice Job!!
Title: Re: MODULE: TotalLipSync v0.4
Post by: Cassiebsg on Wed 19/04/2017 19:00:37
I may give it a test try at some point, if I can figure out how to properly animate mouths.  (laugh)
Looks like an awesome module though, thanks for the hard work.  (nod)
Title: Re: MODULE: TotalLipSync v0.4
Post by: Snarky on Wed 19/04/2017 19:18:06
Thanks! I hope it turns out to be useful to someone.

As for figuring out how to animate mouths, if you look in the second spoiler-hidden section of the post, there are three references which might help.
Title: Re: MODULE: TotalLipSync v0.4
Post by: Crimson Wizard on Wed 19/04/2017 19:20:39
The easiest way to test this, I think, is to draw said letters on sprites instead of mouth animation :).
Title: Re: MODULE: TotalLipSync v0.4
Post by: Cassiebsg on Wed 19/04/2017 20:07:10
Oh, nice! Just what I needed!  (nod) and the drawing one is just perfect reference for me to try and "copy" into the Blender models.  :-D
I've been doing 6 to 8 frames lipsync, so jumping to 10 doesn't feel that scary.  (laugh)
Title: Re: MODULE: TotalLipSync v0.4
Post by: Grundislav on Thu 20/04/2017 23:36:13
This is a great module, thanks so much for making it!

It's made me reeeeeeeally tempted to use it...
Title: Re: MODULE: TotalLipSync v0.4
Post by: bx83 on Sat 22/04/2017 04:26:26
Okay I've converted all files to TSV etc
One last question: where do I put the TSV files?
My game files are in K:\
The speech (as .OGG files) is in K:\gamename\Speech\
The WAV files for speech is in K:\gamename\Speech\WAV (I use .OGG files)

Do I put the TSV's in the Speech dir?...

I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?
Title: Re: MODULE: TotalLipSync v0.4
Post by: Snarky on Sat 22/04/2017 07:50:22
While you're working on the game, it's â€" in your case â€" "K:\gamename\Compiled\sync" or "K:\gamename\Compiled\Windows\sync" (I believe AGS will read both directories). "K:\gamename\Compiled\Windows" is probably the directory you will ultimately distribute once the game is finished (unless you're aiming for another platform), so that's where I would put it.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Sun 23/04/2017 00:36:08
Excellent, thankyou for all you've done :D
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Fri 19/05/2017 11:38:23
May I suggest changing
#define TLS_PHONEMES_LINE_MAX 50
To
#define TLS_PHONEMES_LINE_MAX 5000

This had me running out of space in a 13 second audio file.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Fri 19/05/2017 14:11:19
I seem to be late for this, but I'd just make a small note -

Quote from: bx83 on Sat 22/04/2017 04:26:26
I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?

Quote from: Snarky on Sat 22/04/2017 07:50:22
While you're working on the game, it's â€" in your case â€" "K:\gamename\Compiled\sync" or "K:\gamename\Compiled\Windows\sync" (I believe AGS will read both directories). "K:\gamename\Compiled\Windows" is probably the directory you will ultimately distribute once the game is finished (unless you're aiming for another platform), so that's where I would put it.

$INSTALLDIR$ is the directory where main game data file is located at (*.exe or else) the moment you run it.

AGS does not really check more than one directory normally; there are special rules when running from under the Editor only (debugger mode): in that case Editor passes couple of alternative paths to the engine (one includes AudioCache, for instance).
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Sun 04/06/2017 14:39:44
I have a strange problem.

I'm using a different movement view, and speech view, for my character. Before this, no bug; now, the compiler will randomly choose a point and then crash with a runtime error:
(http://www.redrom.ltd/img/screen1.png)


But then, if I toggle breakpoints in the file in one of the sections leading to the error (line 649), I can do the same thing, but no crash:


(http://www.redrom.ltd/img/screen2.png)

This would appear to be a timing issue, as I've checked everything else; all frames and movement and speech are the same as the original character, nothing is missing. No idea how the module works though; it appears to crashing on speechstyle=lucasarts...?

Ps. my game is lucasarts/rhubarb.

Please help.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Sun 04/06/2017 14:57:38
False alarm, it was to do with the number of frames in the speech view (I added 9, but whatever, it's 10 frames... but 9 frame descriptors XAB..GH. So who knows). Should have checked all, sorry :(
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 04/06/2017 15:07:20
Ah, good! As I was just about to write, there are two things to check first:

-What version of the module are you using?
-Does your speech view have enough frames (in each direction)? It should be 10 for the default mapping.

Rhubarb only has 9 different phonemes/mouth positions (it uses the same frame for W as for U/OO sounds), but the auto-mapping still assumes a 10-frame speech view (W, frame 7, will never be used, so you can leave it blank or make it the same as frame 6) for consistency with the other formats. Also note that the order of the frames is not as listed on the Rhubarb page, but as in the table (behind spoiler tags) in the first post in this thread.

However, I'll see about adding some checking so the module can give a more informative error message if this happens.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Tue 06/06/2017 11:50:56
I am curious - haven't yet tried a comipiled game...
If my sync/ folder is Compiled/; but my actual game is in Compiled/Windows/ - does this mean I include the sync directory in Windows/ too? Is the sync/ files just compiled into the game EXE?
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Tue 06/06/2017 12:04:41
The sync files are not compiled in (I don't think there's really any way to do that), so you need to include the sync directory when you distribute the game.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Wed 07/06/2017 03:12:31
But if the game searches for sync/ and not Windows/sync/ ?...
Oh well I'll change it.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Wed 07/06/2017 03:15:56
AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

But when you order "Build Exe" it builds it to Compiled/Windows. But it won't put extra files there automatically, so you would need to copy them over when preparing package.
(EDIT: actually just files from Compiled get copied into Compiled/Windows, but not subfolders.)
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Wed 07/06/2017 03:16:29
Exactly how does the EXE reference it's own install dir? Should I make it '/sync/' or 'C:/moci/Compile/Windows/sync' or...?
It seems the directory where the TSV files are stored is hard-coded in, so it will only work if the game is in '/programme files (x86)/moci/sync', and nothing else -- which makes it hard for testing.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Wed 07/06/2017 03:17:24
Ah, okay.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Wed 07/06/2017 03:21:12
Quote from: bx83 on Wed 07/06/2017 03:16:29
Exactly how does the EXE reference it's own install dir? Should I make it '/sync/' or 'C:/moci/Compile/Windows/sync' or...?

In script you reference install dir using $INSTALLDIR$ tag like
Code (ags) Select

File.Open("$INSTALLDIR$/sync/data.dat");


There is also "old style" without such tag:
Code (ags) Select

File.Open("sync/data.dat");

But that will work only to read files, not write.

Also, AGS does not allow to use absolute paths (like C:/...), it simply won't open anything even if such file exists. This is done for security reasons.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Wed 07/06/2017 07:43:44
Quote from: bx83 on Sat 22/04/2017 04:26:26
I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?
Quote from: bx83 on Wed 07/06/2017 03:16:29
Exactly how does the EXE reference it's own install dir?

:-\

Quote from: Crimson Wizard on Wed 07/06/2017 03:15:56
AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

Well, I just tested it, and if you have a "sync" folder inside "Compiled", it will be read when you run the game from the editor and reference files in "$INSTALLDIR$/sync". (But you are correct that the directory is not automatically copied into "Compiled/Windows/sync"; you have to do that yourself when you're ready to distribute the game.)
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Wed 07/06/2017 15:05:33
Quote from: Snarky on Wed 07/06/2017 07:43:44
Quote from: Crimson Wizard on Wed 07/06/2017 03:15:56
AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

Well, I just tested it, and if you have a "sync" folder inside "Compiled", it will be read when you run the game from the editor and reference files in "$INSTALLDIR$/sync". (But you are correct that the directory is not automatically copied into "Compiled/Windows/sync"; you have to do that yourself when you're ready to distribute the game.)

Oh right... probably I had a moment of amnesia; it was just couple of months ago what I was working around that, because in 3.4.1 Compiled folder is now Compiled/Data.

So, yes, when debugging from Editor the rules are a bit complicated, the "installdir" is actually "made" of three folders:
- working directory, which is project root folder: for example game takes font files from there
- Compiled folder: I think this was made to take speech.vox from there, and maybe something else, like translation files.
- AudioCache folder: it takes audio files from there.

Basically it looks into project root first, and if it did not find needed files, it also checks either Compiled or AudioCache, depending on what kind of material it is looking for.

Why it was made so: I think to increase compilation speeds when testing, because it does not have to gather/package all those files every time.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Wed 07/06/2017 15:22:00
For the record, I think this is a good thing, because it means you can keep all your "to be distributed" files in "Compiled/", and you don't need separate copies for each target platform (with the potential nightmare of keeping all the copies in sync).

(I might split this whole discussion off as a separate thread, since it doesn't really have much to do with this module specifically.)
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sat 13/01/2018 12:50:51
I wrote a little script to help people who use Rhubarb to do lip-syncing.

Rhubarb does the lip-sync tracking automatically by analyzing the audio, but you can also supply the actual text of the dialog to help guide it, improving the results. Using something like this batch file, you can then create the lip-sync tracking for the entire game automatically:

Code (Bash) Select
for %%F in (clips/*.wav) do (
    rhubarb.exe clips/%%~nxF -d guide/%%~nF.txt > sync/%%~nF.tsv
)


There hasn't been a convenient way to create these guide files from AGS, however. This script helps to automate the task.

It works with the voice acting HTML scripts generated by the Speech Center plugin (http://www.adventuregamestudio.co.uk/forums/index.php?topic=45622.0). Once you've created the voice acting scripts, place them in a subfolder (e.g. /VoiceScripts) inside your compiled game folder. Place a call to this function in your game, and run it to extract the guide files, like so:

Code (ags) Select
  ExtractVoiceScriptLines("$INSTALLDIR$/VoiceScripts", "$APPDATADIR$/guide");

(The output directory must be in $APPDATADIR$ or an existing subfolder of it. Where %APPDATADIR% is pointing can be set in winsetup.exe under Advanced.)

Here's the script (yeah, I know it's ugly, but it seems to work OK):
Code (ags) Select
#define SCRIPT_TOKEN_LINE_NUMBER "<span class=\"lineNumber\">"
#define SCRIPT_TOKEN_TEXT_START ":</b> "
#define SCRIPT_TOKEN_TEXT_END "</span>"

static String ExtractVoiceScriptLines(String sourceDirectory, String targetDirectory)
{
  int i=0;
  String tokenLineNumber = String.Format("%s", SCRIPT_TOKEN_LINE_NUMBER);
  String tokenTextStart = SCRIPT_TOKEN_TEXT_START;
  String tokenTextEnd = SCRIPT_TOKEN_TEXT_END;
  while(i < Game.CharacterCount)
  {
    String prefix = character[i].GetSpeechPrefix();
    String scriptPath = String.Format("%s/char%s.html", sourceDirectory, prefix);
    if(File.Exists(scriptPath))
    {
      File* scriptFile = File.Open(scriptPath, eFileRead);
      String lineNumberPrefix = prefix;
      while(!scriptFile.EOF)
      {
        String scriptLine = scriptFile.ReadRawLineBack();
        int startLineNumberTag = scriptLine.IndexOf(tokenLineNumber);
        if(startLineNumberTag != -1)
        {
          int startLineNumber = scriptLine.IndexOf(lineNumberPrefix);
          int endLineNumber = scriptLine.IndexOf("]");
          if(startLineNumber > 0 && endLineNumber > 0 && endLineNumber > startLineNumber + lineNumberPrefix.Length)
          {
            String strLineNumber = scriptLine.Substring(startLineNumber+lineNumberPrefix.Length, endLineNumber - startLineNumber - lineNumberPrefix.Length);
            int lineNumber = strLineNumber.AsInt;
            //Display("Writing %s%d", prefix, lineNumber);
            if(lineNumber>0)
            {
              String message = "";
              String dialogLine = scriptFile.ReadRawLineBack();
              int startTextTag = dialogLine.IndexOf(tokenTextStart);
              int endTextTag = dialogLine.IndexOf(tokenTextEnd);
              if(startTextTag > 0)
              {
                while(endTextTag == -1 && !scriptFile.EOF)
                {
                  dialogLine = dialogLine.AppendChar(' ');
                  dialogLine = dialogLine.Append(scriptFile.ReadRawLineBack());
                  endTextTag = dialogLine.IndexOf(tokenTextEnd);
                }
                int textLength = endTextTag - startTextTag - tokenTextStart.Length;
                if(startTextTag > 0 && endTextTag > 0 && textLength >= 0)
                  message = dialogLine.Substring(startTextTag+tokenTextStart.Length, textLength);
               
                String linePath = String.Format("%s/%s%d.txt", targetDirectory, prefix, lineNumber);
                //Display("Writing line \"%s\" to file %s", message, linePath);
                File* lineFile = File.Open(linePath, eFileWrite);
                if(lineFile != null)
                {
                  //Display("Writing line \"%s\" to file %s", message, linePath);
                  lineFile.WriteRawLine(message);
                  lineFile.Close();
                }
                else
                  Display("Couldn't write line \"%s\" to file %s", message, linePath);
              }
            }
          }
        }
      } // while(!scriptFile.EOF)
      scriptFile.Close();
    }
    i++;
  }
}
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Wed 20/06/2018 05:02:44
Hi Snarky,

The speech animation delay seems to be too slow for TotalLipSync - no matter what I set a characters to (1, 5, -25, 30), it doesn't make a difference.
What Speech Animation Delay does TLS use?
ATM I'm using 120fps ('GameSpeed'), so I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.


cJulius.ChangeView(1); //julius
cJulius.SpeechView=2; //julius speech
cJulius.AnimationSpeed=1; //normal animation speed
cJulius.SetWalkSpeed(7, 6); //normal walk speed


...though I still use some negatives - for an old man:

cJulius.AnimationSpeed=0; //old animation speed
cJulius.SetWalkSpeed(1, 1); //old walk speed


No negatives have caused problems, but still.
I have 30frames walking for all characters.

So aaaaaanyway, how does TLS do it's timing?
How does Rhubarb do it's timing,
considering it's file is in tenths of a second:


0.00 A
0.50 F
0.83 C
0.90 G
0.97 A
1.05 B
1.53 A
1.61 C
1.80 B
1.87 X
2.38 X
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Wed 20/06/2018 07:04:27
Quote from: bx83 on Wed 20/06/2018 05:02:44
Hi Snarky,

The speech animation delay seems to be too slow for TotalLipSync - no matter what I set a characrs to (1, 5, -25, 30), it doesn't make a difference.
What Speech Animation Delay does TLS use?

TotalLipSync doesn't use animation delay: it reads the timing data from the sync files and displays the corresponding frame.

Problems could potentially occur if the computer can't run the game at the game speed you've set. The animation might then fall behind the speech. So...

QuoteATM I'm using 120fps ('GameSpeed')

Don't.
Film runs at 24 fps. A lot of cheaper animation runs at 12 fps, literally one tenth of the framerate you have set: you're showing ten frames for each one of theirs! AGS by default runs at 40 fps, which is plenty for a point-and-click game, and twitch-based games like FPSs aim for 60 fps, which is also how fast most monitors refresh. Which is all to say that 120 fps is an insane framerate for an adventure game.

More significantly: with a high-res game like yours, many (most) computers won't be able to keep up, and this could affect lip sync (and other sync situations).

Quoteso I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.

No negatives have caused problems, but still.
I have 30frames walking for all characters.

Well, the "cure" is worse than the disease in this case. A better question might be, do you really need 30-frame walkcycles? If you cut it to 15 (every second frame) or even 10 (every third), do you lose noticeable quality?

QuoteHow does Rhubarb do it's timing,[/b] considering it's file is in tenths of a second:


0.00 A
0.50 F
0.83 C
0.90 G
0.97 A
1.05 B
1.53 A
1.61 C
1.80 B
1.87 X
2.38 X


Those are hundredths.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Wed 20/06/2018 08:45:46
Two more things...

Quote from: bx83 on Wed 20/06/2018 05:02:44
ATM I'm using 120fps ('GameSpeed'), so I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.


cJulius.ChangeView(1); //julius
cJulius.SpeechView=2; //julius speech
cJulius.AnimationSpeed=1; //normal animation speed
cJulius.SetWalkSpeed(7, 6); //normal walk speed


...though I still use some negatives - for an old man:

cJulius.AnimationSpeed=0; //old animation speed
cJulius.SetWalkSpeed(1, 1); //old walk speed

A. 0 is not a negative number.
B. At 120 fps, a 30-frame walkcycle will run 4 times a second. Since a cycle includes two footsteps, that's 8 footsteps/second. That seems faster than necessary. (Edit: By comparison, the world's fastest runners sprint at a max pace of about 4.4 â€" 4.8 steps/second (https://posemethod.com/usain-bolts-running-technique/).)
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Tue 31/07/2018 03:26:00
My Inventory window icons dissappear when the character is talking, even if it's just .Say
Is this an AGS thing or a TotalLipSync thing?
Title: Re: MODULE: TotalLipSync v0.5
Post by: abstauber on Tue 31/07/2018 07:51:22
Just a quick guess, but have a look at 'General Settings' in the AGS Editor. In the category 'Visual', check the setting "When player interface is disabled, GUIs should" and make sure it's set to 'Display normally'.
That could be at least one cause of the icons disappearing.
Title: Re: MODULE: TotalLipSync v0.5
Post by: bx83 on Tue 31/07/2018 07:56:30
That's the one, thankyou :D
Title: Re: MODULE: TotalLipSync v0.5
Post by: Olleh19 on Wed 14/10/2020 16:29:25
This is SO great, thank you very much Snarky!

I have a question tho. Some of the lipsync might not turn out "so great" (Swedish, probably the reason why..). Is there a possibiity to combine the automatic lipsync with manual adjustments with either Pamela or Papa?

I've tried manual with Papagaoya, and Pamela inside the "lip sync manager", before i've got the rhubarb running, and i think i prefered editing inside the lipsync manager with Pamela.

Is that possible?

I've read that some of the programs share the same phonomemes(probably spelled wrong), perhaps they could be used in conjunction for optimal result.

Funny fact. I get a grey tooth on my 16 Color "babe", i have no idea what is going on. It's not in the sprites, i've checked several times. Even went in and redrawn the colors and hit save, replaced the source. Deleted the sprites inserted them again in the project, etc.
Still on one sprite image i get a gray pixel. Any idea what could be the cause?  (laugh)

Thanks for a great module!

Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Wed 14/10/2020 17:29:10
Quote from: Olleh19 on Wed 14/10/2020 16:29:25
This is SO great, thank you very much Snarky!

Glad it's of use!

Quote from: Olleh19 on Wed 14/10/2020 16:29:25
I have a question tho. Some of the lipsync might not turn out "so great" (Swedish, probably the reason why..). Is there a possibiity to combine the automatic lipsync with manual adjustments with either Pamela or Papa?

I've tried manual with Papagaoya, and Pamela inside the "lip sync manager", before i've got the rhubarb running, and i think i prefered editing inside the lipsync manager with Pamela.

Is that possible?

I don't really have an overview of the lip sync editors that exist and what formats they can read. It wouldn't be terribly hard to write a tool to convert the files from one format to another so you could edit themâ€"and perhaps a tool like that already existâ€"but this module does not do it.

Quote from: Olleh19 on Wed 14/10/2020 16:29:25
Funny fact. I get a grey tooth on my 16 Color "babe", i have no idea what is going on. It's not in the sprites, i've checked several times. Even went in and redrawn the colors and hit save, replaced the source. Deleted the sprites inserted them again in the project, etc.
Still on one sprite image i get a gray pixel. Any idea what could be the cause?  (laugh)

Try to just display that sprite on an object, to see whether it's anything to do with the module or something else. Is the game actually 16-color?
Title: Re: MODULE: TotalLipSync v0.5
Post by: Olleh19 on Wed 14/10/2020 18:51:47
Quote from: Snarky on Wed 14/10/2020 17:29:10
Quote from: Olleh19 on Wed 14/10/2020 16:29:25
This is SO great, thank you very much Snarky!

Glad it's of use!

Quote from: Olleh19 on Wed 14/10/2020 16:29:25
I have a question tho. Some of the lipsync might not turn out "so great" (Swedish, probably the reason why..). Is there a possibiity to combine the automatic lipsync with manual adjustments with either Pamela or Papa?

I've tried manual with Papagaoya, and Pamela inside the "lip sync manager", before i've got the rhubarb running, and i think i prefered editing inside the lipsync manager with Pamela.

Is that possible?

I don't really have an overview of the lip sync editors that exist and what formats they can read. It wouldn't be terribly hard to write a tool to convert the files from one format to another so you could edit themâ€"and perhaps a tool like that already existâ€"but this module does not do it.

Quote from: Olleh19 on Wed 14/10/2020 16:29:25
Funny fact. I get a grey tooth on my 16 Color "babe", i have no idea what is going on. It's not in the sprites, i've checked several times. Even went in and redrawn the colors and hit save, replaced the source. Deleted the sprites inserted them again in the project, etc.
Still on one sprite image i get a gray pixel. Any idea what could be the cause?  (laugh)

Try to just display that sprite on an object, to see whether it's anything to do with the module or something else. Is the game actually 16-color?

Damn, maybe you are onto something. I don't think i've set it to 16colors. It's just the Ega Palette that's used for the art. I will try some experiments. It's no biggie anyway, game should be "re-done" in unlimited colours, but i thought it was a bit funny.
The reason i wrote ofc is i believe it might actually come from the module, but i have to experiment more. You gave some ideas to try, I should try take away licsync manager aswell, see if that does anything, could just as easily be that.



Edit: False alarm it was nothing. Let's laugh together. I'll show you guys how dumb i really am. >:( 8-0 :-[

Looks nice on the surface, right? WRONG


https://imgur.com/qCe9bbx (https://imgur.com/qCe9bbx)


Look again zoomed in:





https://imgur.com/F2ORCuy
(https://imgur.com/F2ORCuy)

Title: Re: MODULE: TotalLipSync v0.5
Post by: Olleh19 on Sun 01/11/2020 00:22:56
Hey Snarky!


First i want to share an important Accident/Lesson i've got today, and second a question. I think you should add this to the first post, btw (Unless i'm blind and don't see it, already being there). I didn't realise the importance of wave length.
Today i've made a little birthday cutscene "game", with Silly Voices, so ofc i used your Total lipsync. Everything went fine until i ran into a wall completely. I Used Display "test",
and thought "Yeah, the code is running, but why is the text stuck on the screen forever after the speech is done?!" (at least that's how i feelt it). I went out for a bluecup of tea,

and came back to my computer and to my surprise the code had actually taken up on where i wanted it to go instantly. So......That got me thinking. Maybe something is off with the sync files? So i edited them. I saw in the end there were huge gaps. Between B and X, timing wise. Changed that to more "normal values", didn't solve it. It turns out that all voiceclips needs to be cut VERY precise in the endings or else, no matter what your syncfile info says. The biggest impact seems to be the sound source's ending length. So even if you have a little bit of silence in the end. (In my case wasn't that little, 5seconds).  (laugh)

The Silence get's picked up as "sound", when it ain't supposed to be. So make sure to cut your wave/ogg files, so you don't end up with text on the screen for 5seconds longer then expected!
Always cut your wave/ogg files at the endings,. I use Audacity for that, great freeware. Lesson: Don't leave space left in the sound source, unless you intentionally want the text to stay longer on screen ofc.



Question: I have played around with the lipsync and the results are pretty ok! But i'm still thinking i want to see if it's possible to do it differently i am doing pixelart only. I don't need a "specific mouth shape for S or T or A or B". All i want is movement, and stops. I want to be able to run the actual animation all the way thru that is frame 2 to 8 on all single letters, apart from one letter (that is never used in the game text ofc) That should represent the closed mouth shape. So forexample.  letter A is typed = Edit: Images 2,3,4,5,6,7,8 get's played in a row. If letter is B same thing.
I'm using Rhubarb and the standard settings for it. You might wonder why i want to do this? Well to avoid the stops between words ofc. The silences are almost always spot on. It's the rest that ain't. Rarely at least!

I am guessing i should be able to modify it to be able to not have the animation "stop" at certain points by playing all frames on every letter, and just keep going and then "reset" itself once a new letter is executed (If possible). Or it all becomes a giant glitch fest, you would know! (laugh)

the letter ü or öäå perhaps would be fiting for the silent mouth shape. Or else it would be a disaster, ofc!

Is it do able? Please let me know!





Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 01/11/2020 07:52:37
Quote from: Olleh19 on Sun 01/11/2020 00:22:56
First i want to share an important Accident/Lesson i've got today, and second a question. I think you should add this to the first post, btw (Unless i'm blind and don't see it, already being there). I didn't realise the importance of wave length.

It turns out that all voiceclips needs to be cut VERY precise in the endings or else, no matter what your syncfile info says.

This doesn't really have anything to do with TotalLipSync. If you use voice speech in AGS, the character is considered to be speaking for as long as the audio clip lasts.

Quote from: Olleh19 on Sun 01/11/2020 00:22:56
Question: I have played around with the lipsync and the results are pretty ok! But i'm still thinking i want to see if it's possible to do it differently i am doing pixelart only. I don't need a "specific mouth shape for S or T or A or B". All i want is movement, and stops. I want to be able to run the actual animation all the way thru that is frame 2 to 8 on all single letters, apart from one letter (that is never used in the game text ofc) That should represent the closed mouth shape. So forexample.  letter A is typed = Edit: Images 2,3,4,5,6,7,8 get's played in a row. If letter is B same thing.

I don't really understand what you're asking for (you say that Rhubarb gets the silences right, so why do you need a hack for the closed mouth shape?), but in any case it doesn't sound like what this module is for.

The module merely plays animations defined in external sync-files. The result will depend on the quality of those sync files; it is not something the module can control, and it has nothing to do with the text you put in the dialog. If you type one line in the game text, use a different voice clip, and a sync file for another different clip, it will of course play the animation from the sync file (though it will be cut off when the voice clip ends), and it won't match either what is written or what is said: it's only by keeping all three consistent that things stay in sync, but the module has no way to know that.

If you want the animation to be based on the written dialog text, you can use AGS's built-in text-based lipsync.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Olleh19 on Sun 01/11/2020 12:47:11
Quote
This doesn't really have anything to do with TotalLipSync. If you use voice speech in AGS, the character is considered to be speaking for as long as the audio clip lasts.

Ahh, ok. My bad, thanks for informing! It's still useful to know for those who try to sync up and get's confused why it might stay on screen for longer then expected. (nod)


QuoteI don't really understand what you're asking for (you say that Rhubarb gets the silences right, so why do you need a hack for the closed mouth shape?), but in any case it doesn't sound like what this module is for.

The module merely plays animations defined in external sync-files. The result will depend on the quality of those sync files; it is not something the module can control, and it has nothing to do with the text you put in the dialog. If you type one line in the game text, use a different voice clip, and a sync file for another different clip, it will of course play the animation from the sync file (though it will be cut off when the voice clip ends), and it won't match either what is written or what is said: it's only by keeping all three consistent that things stay in sync, but the module has no way to know that.

If you want the animation to be based on the written dialog text, you can use AGS's built-in text-based lipsync.


I've tried that alternative before and yes the animation plays correct, but the timings for when the speech is in a short breath or pauses for a millisecond, are not present, and the animation just keep on playing no matter if there is sound or not in the voiceclip. Unless i failed to see how to change that from happening. But again if that logic is "true" then how come it wasn't with the Lipsync? When you clearly said the Lipsync does not have anything to do with how long the text stays on screen. So no matter if there is silence in the audio file or not, the animation will just keep playing. If my basic logic isn't completely faulthy.  (laugh) Which is not the intended behaviour for what i'm trying to achieve. I'm trying to have just the timing for the X's  Intact (closed mouth shape). I will try a new example. Thank you very much for your patience so far, Snarky!

Edit: Here's an example when i did this "theory" in Sony Vegas. I exported 10 frame talk (or however many frames it was i dont remember). IN a row, and went in manually and just found the X's (if talking lipsync, but i didn't use lipsync, but you get the idea, hopefully). and at places where there should be closed mouth shape, i inserted that manually, but the lipsync could do that for me! The rest is just constant animation in a row flow with abrupt mouthclosings here and there.

https://www.instagram.com/p/CFcCD8fpnYi/

Example. OK here is a example of what i mean editing a Syncfile manually. I guess the textfiles would only need to be the right amount of letters, and not "the actual letters". Because then this clearly would not work.

Code (ags) Select

0.00 X
0.67 B            //<--you replace these letters so they all come in the correct "animation row" as exported from Photoshop (Until the X shows up, and not have them "spaced out" acording to their timing numbers (as it is here, 00.67 - 00.93 a "gap").
0.93 A           //Perhaps i could edit these manually, and save a template where all letters come in a row (if it's not possible any other way). So that would mean some work, i wouldn't mind. I'm very concerned about it looking "better". If possible.   
                      //these come in a row like in the actual Photoshop talk animation. That's why the X or "the closed mouthshapes" timing of the synctool would be very benificial.
                     // to create the illusion of sync, even tho like you as in your confused state of mind have noticed. There barely is [u]no syncing[/u], apart from the X's. Are you following my idea now? If not here comes the "fixed" version (in theory that is).

                     
                      //Replacement idea of previous example (I have no idea, if this would work in theory, please let me now). Forexample say i wanted a 5 second Delay Speech animation)
                     //Going in manually and editing in notepad the syncfile TSV this way. Now i want the photoshop animation to come in a ROW.

0.00 X
0.05 A - Frame 1   //Photoshop frame 1!
0.10 B - Frame 2  //Photoshop frame 2,
0.15 C - Frame 3  //Photoshop frame 3, etc etc.
0.20 D - Frame 4    //so now let's pretend a silence comes here acording to the sync tool.
0.25 X -                  // Closed Mouth. and Restart the photoshop pattern..Or perhaps pickup where the last frame ended? Wouldn't matter, aslong as there is mouth movement (in my mind!).
0.30 - Frame 5
0.35 - Frame 6
0.40 - Frame 7
0.45 - Frame 8
0.50 - Frame 9
0.55 - Frame 10
                           //And this pattern keeps on going until the next X Shows up with it's correct timing (as you can see all other timings are replaced.

                          //Is it possible to do theese kind of edits manually in notepad and just replace the syncfiles??




This (if working) Would look 10times better for what i'm doing at least. But this is also a question of taste.
Would it look realistic? Probably not. Is pixelart realistic 3d faces that needs perfect sync? No. But it still would be nice to have a illusion of sync (a little sync with the X's, and no sync with the rest and just a constant flow of animation).


Edit 3: Clearly it works to edit it in Notepad which is cool. However i do get an script error when i try to add more rows to accomplish the task of having movement all the time until the next X comes.




Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 01/11/2020 18:11:48
Instagram links are (usually) only viewable by people with Instagram accounts. Please use a public image host, like Imgur, ImgBB, PostImage, or FreeImage.

Quote from: Olleh19 on Sun 01/11/2020 12:47:11
But again if that logic is "true" then how come it wasn't with the Lipsync? When you clearly said the Lipsync does not have anything to do with how long the text stays on screen.

I said the TotalLipSync module bases its animation on the sync files, not the dialog text. But what we're talking about here is the built-in AGS lipsync support. AGS has two different kinds of lipsync:

1. Text-based lipsync is based on the dialog text and ignores the voice audio (you don't even need to have any voice clips).
2. Voice-based lipsync is based on the voice audio (actually, the sync-files created for the voice audio clips) and ignores the dialog text.

Currently, TotalLipSync offers an improved version of voice-based lipsync only (though I have an in-development version that also supports text-based sync). But the main point is that you have to choose: there's no feasible way to combine both methods.

Now, if I understand your issue correctly, it's that you don't actually want the animation to lip-sync when speaking: you just want it to play a generic "talking" animation loop. The only thing you want the "lip sync" functionality for is to determine when the characters are actually speaking, and when they are silent.

I have two suggestions:

1. Don't have significant silences in your speech files. Divide the lines whenever there is a long pause, and trim the audio files to contain near-constant dialog. Forget about lipsync and just use regular AGS talk animations.
2. As you have tried to do, edit the sync-files to only distinguish silent periods (frame X in Rhubarb terms) from speaking periods, and then fill the speaking periods with a standard sequence of frames. Since this is tedious work, it would be best to automate it unless the number of lines to sync is very small.

If you are getting errors with the edited sync files, you have made some error in the editing. Without seeing the files or the error message there is no way to tell what that might be, but one possibility is that your text-editing app uses the wrong format for the line endings (one way to test this would be to take a sync file that is confirmed to work, and delete and reinsert a line ending and see if it breaks).
Title: Re: MODULE: TotalLipSync v0.5
Post by: Olleh19 on Sun 01/11/2020 22:36:30
Quote

I have two suggestions:

1. Don't have significant silences in your speech files. Divide the lines whenever there is a long pause, and trim the audio files to contain near-constant dialog. Forget about lipsync and just use regular AGS talk animations.
2. As you have tried to do, edit the sync-files to only distinguish silent periods (frame X in Rhubarb terms) from speaking periods, and then fill the speaking periods with a standard sequence of frames. Since this is tedious work, it would be best to automate it unless the number of lines to sync is very small.

If you are getting errors with the edited sync files, you have made some error in the editing. Without seeing the files or the error message there is no way to tell what that might be, but one possibility is that your text-editing app uses the wrong format for the line endings (one way to test this would be to take a sync file that is confirmed to work, and delete and reinsert a line ending and see if it breaks).

Notepad ++ is the editor of choice, btw. Sounds like some great advice there! Thanks as always, i will try some more!.
The instagram link works on my Laptop (Secondary Computer) that is NOT logged into Facebook or Instagram. However if going thru https://www.instagram.com/andreasblack1/ you can't hit the play button it seems and need to login. I don't have the mp4 left on my system since it crashed some month ago, and i'm back with a new HDD. But anyway, it doesn't matter.

It's funny you'd say that tho. Cause that's exactly what i've tried earlier today. But you get this feeling that you still want to manually be able to control the overall speech animation delay sometimes in sections, i guess in a way it could still be done, just like you say with AGS "own" lipsync support. If i type "SpeechAnimationDelay" out just before "a fast" line" or a word is said, in code. However that means i could only have "that word" in that line, or else all the others words get affected which may not be fast in nature, which could in theory i believe be better done with the Total Lipsync in the actual syncfile manually (nod). I should take a look at your chart and line them up in the syncfile and experiment more, and try add rows of 01.10, 01.20, 01.30 etc etc.

I'm after the ability to change speed of the animation on different words (so in a way i guess that could be called "Animation syncing", but different then lipsyncs original feature, *frame syncinc* (?)). Cause at the moment it get's stuck a lot of times on "one frame", which looks aweful. I hate to swear in Church, but Thimbleweed Park alltho a great game, had the same fate. Sometimes the mouth shapes are stuck for too long, and it looks down right aweful.  Just because this worked in Sony Vegas tho, doesn't mean it would in an AGS editor. (laugh). But what i've just explained was exactly what i did in Sony Vegas. The ability to sometimes speed up the 10 frame sequence mp4, sometimes slow it down, and then *insert closed mouth* at silent places automatically with Total Lipsync (i had to do that manually in Vegas ofc). Would be a totally killer feature! (nod)

Maybe the easiest way to describe it should have been to say this: Bypass one frame function for words, and just have an overall speed animationdelay setting for chosen sections of the text (if text + audiosync will be available). Forexample you could type something like, "break" in the syncfile and it would "recognise" that OK that was that line/or word, and then the next line or word that comes get's a new potential animationspeed, or keeps the same unless the user choose to change it. To the users liking, and when there is no sound under -40-50db *insert noisegate* = closed mouth frame or a animation for closing the mouth, how cool would that be!. I know i'm talking out of my ass here :). I'm no coder! I just enjoy AGS so much. Had a lot of fun lately, and even getting the lips to move "somewhat" is an epic win it itself Snarky! Thanks to your hardwork that was possible! But then you start dreaming "wouldn't it be possible to do this", and so on, so forth. What i've done so far with the Total Lipsync is probably good enough for the majority of people. It's just that my perfectionist spirit see's other possibilities, and now i've gotten them out of my system. Feels better now. Thank you! :)


Title: Re: MODULE: TotalLipSync v0.5
Post by: Dave Gilbert on Thu 28/04/2022 00:45:16
Hello all! I've been using this for awhile in conjunction with Rhubarb, and it seems to be working *perfectly* aside from one big issue.

If I reload an earlier saved game, or press F9, it somehow loses the lip sync data. The voiceover plays but characters' mouths stop moving entirely. Any idea why this would happen?

Any light-shedding appreciated!

edit: To clarify, if I start a fresh game the lip sync plays perfectly with mouth movement. But if I reload a save, or press F9 to restart, then the mouths stop moving (although the VO plays).
Title: Re: MODULE: TotalLipSync v0.5
Post by: Dave Gilbert on Thu 28/04/2022 18:59:38
Sorry to double post, but this seems to be caused by a bug in the engine rather than the module. The work around was to add these lines of code to an eEventRestoreGame in addition to game_start.

  TotalLipSync.Init(eLipSyncRhubarb);   
  TotalLipSync.AutoMapPhonemes();

The engine bug is being fixed so this info may be redundant soon!
Title: Re: MODULE: TotalLipSync v0.5
Post by: eri0o on Fri 29/04/2022 01:15:18
Hey, I know no one asked, but just thought about mentioning. Since this module can leverage files in the game installation directory, if someone doesn't want to do that, and instead want to have the files packaged with the game, since AGS 3.6.0 it's possible. Simply, create a directory at the root of your project, name it say "lipsync". Then note in the property below in General Settings. In the game code, when asked for the path for the files, use "$DATA$/lipsync", to find the files inside the AGS Game package.

Spoiler
(https://i.imgur.com/boc8Wsr.png)
[close]
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Fri 29/04/2022 05:35:38
That's really great to avoid a ****ton of tiny files in your distribution. It's also great because it means I don't have to code the pack/unpack feature I've had on my TODO list for years.
Title: Re: MODULE: TotalLipSync v0.5
Post by: newwaveburritos on Sat 03/06/2023 03:39:02
I had a bit of trouble getting this to work in AGS 3.6.0.35 and I can't really explain it so I thought I would mention it.  I think the problem was basically that it was not loading the files correctly.  Eventually, I discovered that it was writing in the warnings.log what was going on:

Game : (room:2)[R 369] FileOpen: FAILED: c:/Brick Sundown 3.6/Brick Sundown/Compiled/Windows/sync/Bric1.dat

When I had the files name 1.DAT and 1.WAV it wasn't working but when I renamed them BRIC1.DAT and BRIC1.WAV it started working.  The name of the character is "Brick" so it sort of makes sense and seems to work fine now once I determined that maybe I should rename the file to the one that it's looking for.

I'm using Papagayo for the lip-syncing file.  The .DAT file is in Compiled>Windows>sync and the .WAV is in Compiled>Windows>Data

Anyway, it's working great now and I really like the module so thanks for making it!
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sat 03/06/2023 06:54:23
Yes, AGS requires voice clip file names to be the first four letters of the character script name (https://adventuregamestudio.github.io/ags-manual/VoiceSpeech.html) (excluding the first letter if it starts with "c"), followed by a number. Since it isn't something specific to this module, but a standard part of AGS, I didn't explain it, but perhaps I should mention it.

There is also a complication that on some platforms (but not Windows) the file names are case-sensitive. I should add some better way to deal with this.
Title: Re: MODULE: TotalLipSync v0.5
Post by: newwaveburritos on Sat 03/06/2023 20:07:44
Newwaveburritos: Very mysterious behavior.
Snarky: Pretty standard stuff.  Did you read the documentation?
Newwaveburritos: Very mysterious indeed...
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 04/06/2023 09:24:26
Well, it was a failure of imagination on my part not to consider that users might be unfamiliar with how voice speech works in AGS. It should be mentioned/referenced in the documentation. Thanks for pointing it out!
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Sun 04/06/2023 09:44:43
Quote from: Snarky on Sat 03/06/2023 06:54:23There is also a complication that on some platforms (but not Windows) the file names are case-sensitive. I should add some better way to deal with this.

IIRC for every asset name engine should try both case-sensitive and case-insensitive search. At least that worked for the files stored on disk.
But it may be skipped for the files stored inside the package, which frankly may be considered inconsistent, and something to be fixed.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 04/06/2023 09:59:53
This (https://www.adventuregamestudio.co.uk/forums/engine-development/file-access-on-linux/) is the problem I was referring to. Later versions of the engine may have been changed to be case-insensitive, but it should still be handled by the module in case it's used on older AGS versions. (And yes, if it doesn't find files case-insensitively in the packaged assets, that's probably going to be an issue since sync files are ideally suited for that kind of packaging.)
Title: Re: MODULE: TotalLipSync v0.5
Post by: Crimson Wizard on Sun 04/06/2023 10:06:14
Weird, I thought that problem was already addressed. I must check what the current situation is.

EDIT: I double checked, and the engine itself looks up for the *assets* (anything inside the package) using case-insensitive comparisons, always. This was true at least since AGS 3.3.0, and possibly even earlier in the initial Linux ports.

I suppose the problem may exist in File.Open and similar *script commands* when they look in the dir.

I don't know why this was not addressed earlier, when the above thread was created...
Title: Re: MODULE: TotalLipSync v0.5
Post by: newwaveburritos on Wed 12/07/2023 17:44:59
Now that I've learned how it works I think it's a fantastic module and really you get a lot of character out of some relatively easy steps.  I do have a question, though.  Is it possible to have two characters talking over top of each other at once?  I was thinking you could make the speech line a single WAV file but the characters seem to block one another when called in tandem.  It's not a big deal but I was trying to script an argument lol.  Thanks for the module all the same!
Title: Re: MODULE: TotalLipSync v0.5
Post by: AndreasBlack on Tue 26/09/2023 12:32:55
I'm not sure what's going on in my syncing, help! I'm exporting wrong perhaps? Since AGS wanted a dat file as i understood the instructions. I push the export Anime Studio button in Papagayo. Problem is the sync doesn't look nothing like in Pagagayo once it's in AGS. It looks more like 10fps everything is slowed down. Both AGS and Papagayo is set at 60fps. What's wrong? 9 Frames in the sprite view, perhaps i should have showed those in the video..Edit: Hmm, maybe there's too many closed mouths in there, nope doesn't seem to change anything. Is it the Thumbleweed Template perhaps that's causing something?

If i take away the sync file the character doesn't move his mouth at all, so obviously it has some form of function, but not the correct one.

TotalLipSync.Init(eLipSyncMoho); //Papagayo setting, right? i have 9 mouth movements in the speech view..
TotalLipSync.AutoMapPhonemes();


Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Tue 26/09/2023 15:03:51
Check out https://github.com/messengerbag/TotalLipSync#void-totallipsyncsetdatafileframerateint-framerate
Title: Re: MODULE: TotalLipSync v0.5
Post by: AndreasBlack on Sat 30/09/2023 22:52:55
Trying to SaySync the Standard Replies like "I can't use that" inside the Thumbleweed TemplateSettings.asc that works for multiply characters, and not just Character.SaySync. However i can't seem to find an option for it. I'm guessing it's doable, it's just my none coder brain is clueless
Title: Re: MODULE: TotalLipSync v0.5
Post by: Dave Gilbert on Sat 30/09/2023 23:36:53
Hi! I had a question about the way Rhubarb is set up in this module:

The rhubarb mapping is written like this:

Code (AGS) Select
void _autoMapPhonemesRhubarb()
{
 TotalLipSync.AddPhonemeMapping("X",0);
  TotalLipSync.AddPhonemeMapping("A",1);  // mbp
  TotalLipSync.AddPhonemeMapping("B",2);  // other consonants
  TotalLipSync.AddPhonemeMapping("C",3);  // EH/AH/EY etc. (bed, hut, bait)
  TotalLipSync.AddPhonemeMapping("D",4);  // AA/AE/AY (father, bat, like)
  TotalLipSync.AddPhonemeMapping("E",5);  // AO/OW (thaw, slow)
  TotalLipSync.AddPhonemeMapping("F",6);  // UW/OY/UH/OW (you, toy, poor)
  // Frame 7 unassigned to match Moho mapping
  TotalLipSync.AddPhonemeMapping("G",8);  // F/V (fine, very)
  TotalLipSync.AddPhonemeMapping("H",9);  // L (letter)
}

When my talking sprites are drawn and saved to individual files, they are labelled with the appropriate letter name. So they are called A.png, B.png, C.png, etc, all the way to X.png.

Thus, when they are imported, they are imported in alphabetical order. A first, and X last. When I assign the sprites to the loops, I just assign them in order. Like so:

(http://www.wadjeteyegames.com/temp/talk-frame-order.png)

Since the sprite order is different than what the code above expects, I changed the code to this:

Code (AGS) Select
void _autoMapPhonemesRhubarb()
{
  TotalLipSync.AddPhonemeMapping("X",8);
  TotalLipSync.AddPhonemeMapping("A",0);  // mbp
  TotalLipSync.AddPhonemeMapping("B",1);  // other consonants
  TotalLipSync.AddPhonemeMapping("C",2);  // EH/AH/EY etc. (bed, hut, bait)
  TotalLipSync.AddPhonemeMapping("D",3);  // AA/AE/AY (father, bat, like)
  TotalLipSync.AddPhonemeMapping("E",4);  // AO/OW (thaw, slow)
  TotalLipSync.AddPhonemeMapping("F",5);  // UW/OY/UH/OW (you, toy, poor)
  TotalLipSync.AddPhonemeMapping("G",6);  // F/V (fine, very)
  TotalLipSync.AddPhonemeMapping("H",7);  // L (letter)
}

This SEEMS like it should work, but I am noticing that the final result for lipsyncing isn't as good as it could be. So I'm wondering if the problem lies with the rhubarb end, or if me changing the code like this messed things up. For example, what was the reason that frame 7 was originally unassigned? Was it essential to keep that?

Anyway, light shedding appreciated!

-Dave


Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 01/10/2023 07:04:49
Quote from: Dave Gilbert on Sat 30/09/2023 23:36:53So I'm wondering if the problem lies with the rhubarb end, or if me changing the code like this messed things up. For example, what was the reason that frame 7 was originally unassigned? Was it essential to keep that?

Hmm, changing the mapping should be fine (but see below), though I would strongly recommend that instead of changing the AutoMapPhonemes implementation in the module code, you set up a manual mapping outside of the module when you initialize it—so, instead of calling AutoMapPhonemes, you use those exact AddPhonemeMapping calls yourself to set it up the way you want it.

The reason why the auto-setup is arranged the way it is (skipping frame 7) is because the other lipsync data formats, Moho and Pamela/Annosoft, distinguish between "ooh" sounds and "w" sounds, but Rhubarb uses the same phoneme/mouth shape for both. The auto-setup maps the frames the same way for all the formats, allowing you to set up the animation view once, and then use whichever lipsync data format you like. You can even switch formats along the way, if for example you want to hand-sync some scenes in Pamela format and do others automatically in Rhubarb (though you'd need to reset the module with the new format and mappings whenever you switch). I don't remember precisely why it's frame 7 that is the "optional" one rather than the last frame, but I would guess I based the order on some standard or convention for lipsync setups.

Anyway, to get to the point of your question: looking over the code, it does assume that Frame 0 is the non-speaking frame, as in the AGS convention for NormalView and SpeechView; this frame is set when the animation file does not specify a frame. (This is necessary because some of the data formats can have gaps, but I don't remember if this applies to Rhubarb.) You could try to change it here:

Code (ags) Select
void _playFrame(int frame)
{
  // Look up the frame based on phoneme, unless argument is -1, in which case use frame 0
  if(frame == -1)
  {
    _currentPhoneme = "";
    _currentFrame = 0; // <-- Change this to 8
  }
  // ...

If that doesn't help, I would be inclined to believe the problem is with Rhubarb, but another potential problem could be that the animation and the audio drift out of sync because AGS isn't running at its set game speed (which is pretty likely with a high-res game like Old Skies, if that's where you're using it). If so, that would probably be fixable fairly easily by basing the calculation of the current time code on the system clock rather than on counting game frames.
Title: Re: MODULE: TotalLipSync v0.5
Post by: eri0o on Sun 01/10/2023 08:30:48
I don't think you can get system clock from the API today, but like, does getting the elapsed time in milliseconds from when the game started running something that would help there?
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 01/10/2023 09:24:53
Quote from: eri0o on Sun 01/10/2023 08:30:48I don't think you can get system clock from the API today

You can: DateTime.Now (https://adventuregamestudio.github.io/ags-manual/DateTime.html)

Incidentally, this is one of those cases that demonstrates how much harder things get when you go from a basic module to a more general one: If using the system clock fixes any desync problems related to game slowdown, and is simple to do (it is), doing so should be fine, right?

Well, not quite. Because what if the game, including audio, is paused in the middle of the animation? (For example by tabbing away from the window, if not set to run in the background?) Then the system clock progress will not match the audio progress. And if the module is extended to support syncing during background speech, we get another complication, because then the game can be saved and restored in the middle of a sync animation.

Probably the most robust solution would be to use the system clock unless there is a major discrepancy with the last frame (say, more than a second), and in that case use the game speed as a fallback.
Title: Re: MODULE: TotalLipSync v0.5
Post by: eri0o on Sun 01/10/2023 10:31:35
@Snarky uhm, does getting the time of the voice clip in milliseconds would be better? Because I think you can get it from it's audio channel.

I saw this in the manual (https://adventuregamestudio.github.io/ags-manual/MusicAndSound.html?highlight=audiochannel&case_sensitive=0),

QuoteOne channel is always reserved for the speech voice-over and you cannot change that, so you have 15 at your disposal.

I wonder if this means the voiceclip AudioChannel is fixed. The other approach is to ask the player to pass this audio channel and let them figure it out.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Snarky on Sun 01/10/2023 11:10:26
Quote from: eri0o on Sun 01/10/2023 10:31:35Because I think you can get it from it's audio channel.

Yes, that could work (voice clips always play on channel 0), though I've never quite trusted AudioChannel.PositionMs (a long time ago I had some trouble when it didn't seem to work properly).

I do also want to support syncing without an audio file, for more general-purpose synced/scripted animations, but the frame counter should be fine for that purpose.
Title: Re: MODULE: TotalLipSync v0.5
Post by: Dave Gilbert on Sun 01/10/2023 15:36:14
Quote from: Snarky on Sun 01/10/2023 07:04:49
Quote from: Dave Gilbert on Sat 30/09/2023 23:36:53So I'm wondering if the problem lies with the rhubarb end, or if me changing the code like this messed things up. For example, what was the reason that frame 7 was originally unassigned? Was it essential to keep that?

Hmm, changing the mapping should be fine (but see below), though I would strongly recommend that instead of changing the AutoMapPhonemes implementation in the module code, you set up a manual mapping outside of the module when you initialize it—so, instead of calling AutoMapPhonemes, you use those exact AddPhonemeMapping calls yourself to set it up the way you want it.

Ok, that I can do. Thanks! Just to be sure, I did this on game_start():

Code (ags) Select
  TotalLipSync.Init(eLipSyncRhubarb);    // Or whatever lip sync format you're using
  //TotalLipSync.AutoMapPhonemes(); <--- commented out!
  TotalLipSync.AddPhonemeMapping("X",8);
  TotalLipSync.AddPhonemeMapping("A",0);  // mbp
  TotalLipSync.AddPhonemeMapping("B",1);  // other consonants
  TotalLipSync.AddPhonemeMapping("C",2);  // EH/AH/EY etc. (bed, hut, bait)
  TotalLipSync.AddPhonemeMapping("D",3);  // AA/AE/AY (father, bat, like)
  TotalLipSync.AddPhonemeMapping("E",4);  // AO/OW (thaw, slow)
  TotalLipSync.AddPhonemeMapping("F",5);  // UW/OY/UH/OW (you, toy, poor)
  TotalLipSync.AddPhonemeMapping("G",6);  // F/V (fine, very)
  TotalLipSync.AddPhonemeMapping("H",7);  // L (letter)

QuoteThe reason why the auto-setup is arranged the way it is (skipping frame 7) is because the other lipsync data formats, Moho and Pamela/Annosoft, distinguish between "ooh" sounds and "w" sounds, but Rhubarb uses the same phoneme/mouth shape for both. The auto-setup maps the frames the same way for all the formats, allowing you to set up the animation view once, and then use whichever lipsync data format you like. You can even switch formats along the way, if for example you want to hand-sync some scenes in Pamela format and do others automatically in Rhubarb (though you'd need to reset the module with the new format and mappings whenever you switch). I don't remember precisely why it's frame 7 that is the "optional" one rather than the last frame, but I would guess I based the order on some standard or convention for lipsync setups.

Okay. So since I never use anything aside from Rhubarb, I don't need to worry about that? I can use my version of the code?

QuoteAnyway, to get to the point of your question: looking over the code, it does assume that Frame 0 is the non-speaking frame, as in the AGS convention for NormalView and SpeechView; this frame is set when the animation file does not specify a frame. (This is necessary because some of the data formats can have gaps, but I don't remember if this applies to Rhubarb.) You could try to change it here:

Aha! Yes I have noticed that when the characters stop speaking they always end up on the wrong mouth shape. I added some code to manually change the frame when they stop speaking, but your solution is better! Thanks!

edit: Hmm. I notice that the mouth shape still finishes on the wrong frame. That's on me (because the way the mouth shape frames are imported) so if there's no way to fix that, I'll just continue using the workaround code I created.