Author Topic: MODULE: TotalLipSync v0.5  (Read 4351 times)

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
MODULE: TotalLipSync v0.5
« on: 18 Apr 2017, 19:22 »
TotalLipSync


(Character head/animation by Preston Blair. King Graham sprite ripped from King's Quest II VGA by AGDI. Background from AGS Awards Ceremony by Ali.)


TotalLipSync is a module for voice-based lip sync. It allows you to play back speech animations that have been synchronized with voice clips. TotalLipSync has all the same capabilities as the voice-based lip sync that is built into AGS, and offers the following additional advantages:
  • It works with LucasArts-style speech (as well as with Sierra-style and full-screen speech modes).
  • In addition to the Pamela (.pam) and Papagayo/Moho Switch (.dat) formats supported by AGS, it also reads Annosoft SAPI 5.1 (.anno) and Rhubarb (.tsv) lip sync files.
  • In particular, Rhubarb support means that lip syncing can be 100% automated (with decent results): no manual tracking of the speech clips is required.
  • It is more flexible: You can switch speech-styles mid-game, change the phoneme mapping, use files with different data formats, etc.
  • You don't have to do the phonemes-to-frames mapping manually: The module comes with a default auto-mapping.

How to use
  • Create the lip sync data files for the speech clips. You can use one of these tools (personally I would recommend Papagayo for manual tracking, and Rhubarb for automatic lip syncing, but the Lip Sync Manager plugin is good too):
    The filename of each sync file should be the same as the speech clip except for the extension, and you need to place them in your compiled game folder (by default, in a folder names "sync/" inside the game folder).
  • Create the speech animation for your character(s), with different animation frames for the different phonemes (see below), and set it as their speech view.
  • Download and import the TotalLipSync module into your AGS project.
  • Make sure your game settings are correct: the AGS built-in lip sync (in the project tree under "Lip sync") should be set to "disabled".
  • If you are going to use Sierra-style (or full-screen) speech for your lip sync animations, you must create a dummy view. Make sure to give it exactly one loop and one frame. If you name the view TLS_DUMMY it will automatically be used by the module. Otherwise you can set the view to use with TotalLipSync.SetSierraDummyView().
You are now ready to use the module. Add the code to initialize TotalLipSync on startup:

Code: Adventure Game Studio
  1. function game_start()
  2. {
  3.   TotalLipSync.Init(eLipSyncRhubarb);    // Or whatever lip sync format you're using
  4.   TotalLipSync.AutoMapPhonemes();
  5. }

Or if you want a custom phonemes-to-frames mapping:

Add spoiler tag for Example code:
Code: Adventure Game Studio
  1. function game_start()
  2. {
  3.   TotalLipSync.Init(eLipSyncPamelaIgnoreStress);
  4.  
  5.   TotalLipSync.AddPhonemeMappings("None",0);
  6.   TotalLipSync.AddPhonemeMappings("B/M/P",1);
  7.   TotalLipSync.AddPhonemeMappings("S/Z/IH/IY/SH/T/TH/D/DH/JH/N/NG/ZH",2);
  8.   TotalLipSync.AddPhonemeMappings("EH/CH/ER/EY/G/K/R/Y/HH",3);
  9.   TotalLipSync.AddPhonemeMappings("AY/AA/AH/AE",4);
  10.   TotalLipSync.AddPhonemeMappings("AO/AW/UH",5);
  11.   TotalLipSync.AddPhonemeMappings("W/OW/OY/UW",6);
  12.   // Frame 7 unassigned to match default Moho mapping
  13.   TotalLipSync.AddPhonemeMappings("F/V",8);
  14.   TotalLipSync.AddPhonemeMappings("L",9);
  15. }
To speak a line with lip syncing, you simply call the extender functions Character.SaySync() or Character.SayAtSync(), using a speech clip prefix:

Code: Adventure Game Studio
  1.   cGraham.SaySync("&1 This line will be animated with lip sync");
  2.   cGraham.SayAtSync(320, 100, 240, "&2 ... and so will this");    // x_left, y_top, width, message

And that's all there is to it! (If you don't use a speech clip prefix, or if there is no matching sync file, the speech animation won't play at all.)

Phoneme-to-frame mappings
The principle of lip syncing is that different sounds (phonemes) correspond to different mouth shapes. If we display an animation frame with the right mouth shape at the same time as that sound appears in the audio being played, the animation will seem to match the speech. The first step, then, is to identify the phonemes and timing of them in the speech (that's what the tools listed above are for), and the second step is to choose an appropriate animation frame for each phoneme. We usually don't use different animation frames for all the different phonemes, so we combine phonemes into groups that are all mapped to a single frame. The different tools have different sets of phonemes (or phoneme groups), so we have to define different mappings from phonemes to frames.

So here is the default mapping for each data format used by TotalLipSync. It has been set up for a speech animation with ten different frames, each representing a different mouth position. (This is a fairly standard setup.) If you stick to these frames and these mappings, you can use the same speech view no matter what lip sync tool or data format you use:

Add spoiler tag for Animation frames:
Frame
Description
Rhubarb
phoneme ID
Moho
phoneme
Pamela
phonemes
0Mouth closed
(or slack)
[slack or same as 1]XrestNone
1M, B, PAMBPM/B/P
2Various consonants,
(Rhubarb: Ee-type
sounds)
BetcK/S/T/D/G/DH/
TH/R/HH/CH/Y/N/
NG/SH/Z/ZH/JH
3Eh-type sounds,
(Non-Rhubarb:
Ee-type sounds)
CEIH/IY/EH/AH/
EY/AW/ER
4Ah-type and
I-type sounds
DAIAA/AE/AY
5Aww-type sounds,
Ow-type sounds
(can also go in 6)
EOAO/OW
6U-type and
Oo-type sounds
(Non-Moho: W)
FUUW/OY/UH
7Moho: W[same as 6][same as 6][same as 6]WQW
8F, VGFVF/V
9L
(Th-type sounds
can also go here,
rather than in 2)
HLL
Where to get it
TotalLipSync is hosted on Github (mainly just as a way for me to learn about how Github works):
https://github.com/messengerbag/TotalLipSync

You can download the current release from there:


Known bugs
None

Change log
0.5
-Added APIs to get the currently lip syncing character, the current phoneme and current frame.

0.4
-Fixed support for Sierra-style speech
-Minor bug fixes for edge-cases
-Documentation

0.2 (pre-release)
-Added support for Papagayo/Moho Switch (.dat), Annosoft SAPI 5.1 LipSync (.anno) and Rhubarb (.tsv)

0.1 (pre-release)
-Pamela support for LucasArts-style speech

Originally based on code by Calin Leafshade (though very little of it remains in the current version).
Thanks to Grundislav for providing a speech view used in development and testing of the module!
« Last Edit: 19 May 2017, 12:14 by Snarky »

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.4
« Reply #1 on: 18 Apr 2017, 19:23 »
A couple more things:

Pamela and Annosoft SAPI 5.1 LipSync use almost exactly the same phoneme sets, with only minor variation (TotalLipSync is not case sensitive). Pamela can tag vowels with three levels of stress (0-2), e.g. AY0, UW1. This is not particularly useful in AGS, and should be ignored (by setting TotalLipSync.Init(ePamelaIgnoreStress)) unless there's good reason not to. Anyway, here's the full list and what they represent (ones where Annosoft differs emphasized):

Add spoiler tag for Hidden:
IPA
Example
Pamela
Anno
[silence]-Nonex
ɑfather / box (AmEng)AAAA
æat, snackAEAE
ʌ / əhut / aloneAHAH
ɔ / ɑthaw / dogAOAO
cow, outAWAW
hide, guyAYAY
bbangBb
cheeseCHCH
ddamnDd
ðthese, batheDHDH
ɛ / ɛəbed / bearEHEH
ɜ: / ɚhurt / butterERER
ate, baitEYEY
ffineFf
ggoodGg
hhouseHHh
ɪ / ɪəit / fearIHIH
i:eat, freeIYIY
gee, jawJHj
kkey, crushKk
llipLl
mmonkeyMm
nnoNn
ŋping, pongNGNG
əʊoak, slowOWOW
ɔɪtoyOYOY
pputPp
rreadRr
ssapSs
ʃsharpSHSH
ttotalTt
θthinTHTH
ʊ / ʊəgood / poorUHUH
u:youUWUW
vvikingVv
wwe, questionWw
jyieldYy
zzooZz
ʒseizure, genreZHZH
(Based on the in-app list in PAMELA and the Annosoft list here.)

Also, if you use Rhubarb to do the lip syncing, this could be helpful:

You don't need to compile Rhubarb, just get the latest release for Windows or OSX: https://github.com/DanielSWolf/rhubarb-lip-sync/releases

The script doesn't call Rhubarb, you'll have to do all of that. Take all the speech clips, convert them to .wav if necessary, copy them into the Rhubarb directory, and for each one, call "rhubarb.exe myclip.wav > myclip.tsv" (where "myclip" is the name of the clip). You can also put the text corresponding to each clip in individual .txt files to assist with the speech recognition, and then you'd call "rhubarb.exe myclip.wav -d myclip.txt > myclip.tsv". Then once that's done, copy all the .tsv files over into the directory of your compiled AGS game, and the module will read them.

Obviously this process is tedious, and also Rhubarb takes quite a while to process each clip, so if you have more than a couple of dozen clips you'll definitely want to automate it (you could write a batch file to go through and process each .wav file in the directory)

In fact, I wrote a very simple version of such a batch file:

Code: Bash
  1. for %%F in (clips/*.wav) do (
  2.     rhubarb.exe clips/%%~nxF -d guide/%%~nF.txt > sync/%%~nF.tsv
  3. )

This assumes that the voice clips are in a folder called "clips/" inside the Rhubarb directory, that the text files are in a folder called "guide/", and that there is a folder called "sync/" where the .tsv files will be written. It also requires a .txt file for each .wav file. So there are a lot of possible improvements. Save this in a text file in the Rhubarb directory and name it something like agsbatch.bat, and you can run it to process all the speech clips in one go (which might take a while!).
« Last Edit: 18 Apr 2017, 19:37 by Snarky »

Mehrdad

    • I can help with animation
    •  
    • I can help with characters
    •  
    • I can help with story design
    •  
    • I can help with web design
    •  
Re: MODULE: TotalLipSync v0.4
« Reply #2 on: 19 Apr 2017, 15:15 »
Hey Snarky . It's great module . Nice Job!!

Cassiebsg

  • Cavefish
  • Fleeing the Cylon tyrrany...
    • Cassiebsg worked on a game that was nominated for an AGS Award!
Re: MODULE: TotalLipSync v0.4
« Reply #3 on: 19 Apr 2017, 19:00 »
I may give it a test try at some point, if I can figure out how to properly animate mouths.  (laugh)
Looks like an awesome module though, thanks for the hard work.  (nod)
There are those who believe that life here began out there...

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.4
« Reply #4 on: 19 Apr 2017, 19:18 »
Thanks! I hope it turns out to be useful to someone.

As for figuring out how to animate mouths, if you look in the second spoiler-hidden section of the post, there are three references which might help.

Crimson Wizard

  • AGS Project Tracker Admins
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    •  
    • Lifetime Achievement Award Winner
    •  
    • Crimson Wizard worked on a game that was nominated for an AGS Award!
      Crimson Wizard worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.4
« Reply #5 on: 19 Apr 2017, 19:20 »
The easiest way to test this, I think, is to draw said letters on sprites instead of mouth animation :).

Cassiebsg

  • Cavefish
  • Fleeing the Cylon tyrrany...
    • Cassiebsg worked on a game that was nominated for an AGS Award!
Re: MODULE: TotalLipSync v0.4
« Reply #6 on: 19 Apr 2017, 20:07 »
Oh, nice! Just what I needed!  (nod) and the drawing one is just perfect reference for me to try and "copy" into the Blender models.  :-D
I've been doing 6 to 8 frames lipsync, so jumping to 10 doesn't feel that scary.  (laugh)
There are those who believe that life here began out there...

Grundislav

  • AGS Baker
  • Mittens Lord
  • Brogan
    • Lifetime Achievement Award Winner
    •  
    • Grundislav worked on a game that was nominated for an AGS Award!
      Grundislav worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.4
« Reply #7 on: 20 Apr 2017, 23:36 »
This is a great module, thanks so much for making it!

It's made me reeeeeeeally tempted to use it...
A Golden Wake
Visit my Games Page           Read my Development Blog

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.4
« Reply #8 on: 22 Apr 2017, 04:26 »
Okay I've converted all files to TSV etc
One last question: where do I put the TSV files?
My game files are in K:\
The speech (as .OGG files) is in K:\gamename\Speech\
The WAV files for speech is in K:\gamename\Speech\WAV (I use .OGG files)

Do I put the TSV's in the Speech dir?...

I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.4
« Reply #9 on: 22 Apr 2017, 07:50 »
While you're working on the game, it's – in your case – "K:\gamename\Compiled\sync" or "K:\gamename\Compiled\Windows\sync" (I believe AGS will read both directories). "K:\gamename\Compiled\Windows" is probably the directory you will ultimately distribute once the game is finished (unless you're aiming for another platform), so that's where I would put it.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #10 on: 23 Apr 2017, 00:36 »
Excellent, thankyou for all you've done :D

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #11 on: 19 May 2017, 11:38 »
May I suggest changing
#define TLS_PHONEMES_LINE_MAX 50
To
#define TLS_PHONEMES_LINE_MAX 5000

This had me running out of space in a 13 second audio file.

Crimson Wizard

  • AGS Project Tracker Admins
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    •  
    • Lifetime Achievement Award Winner
    •  
    • Crimson Wizard worked on a game that was nominated for an AGS Award!
      Crimson Wizard worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.5
« Reply #12 on: 19 May 2017, 14:11 »
I seem to be late for this, but I'd just make a small note -

I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?

While you're working on the game, it's – in your case – "K:\gamename\Compiled\sync" or "K:\gamename\Compiled\Windows\sync" (I believe AGS will read both directories). "K:\gamename\Compiled\Windows" is probably the directory you will ultimately distribute once the game is finished (unless you're aiming for another platform), so that's where I would put it.

$INSTALLDIR$ is the directory where main game data file is located at (*.exe or else) the moment you run it.

AGS does not really check more than one directory normally; there are special rules when running from under the Editor only (debugger mode): in that case Editor passes couple of alternative paths to the engine (one includes AudioCache, for instance).
« Last Edit: 19 May 2017, 14:15 by Crimson Wizard »

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #13 on: 04 Jun 2017, 14:39 »
I have a strange problem.

I'm using a different movement view, and speech view, for my character. Before this, no bug; now, the compiler will randomly choose a point and then crash with a runtime error:



But then, if I toggle breakpoints in the file in one of the sections leading to the error (line 649), I can do the same thing, but no crash:




This would appear to be a timing issue, as I've checked everything else; all frames and movement and speech are the same as the original character, nothing is missing. No idea how the module works though; it appears to crashing on speechstyle=lucasarts...?

Ps. my game is lucasarts/rhubarb.

Please help.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #14 on: 04 Jun 2017, 14:57 »
False alarm, it was to do with the number of frames in the speech view (I added 9, but whatever, it's 10 frames... but 9 frame descriptors XAB..GH. So who knows). Should have checked all, sorry :(

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #15 on: 04 Jun 2017, 15:07 »
Ah, good! As I was just about to write, there are two things to check first:

-What version of the module are you using?
-Does your speech view have enough frames (in each direction)? It should be 10 for the default mapping.

Rhubarb only has 9 different phonemes/mouth positions (it uses the same frame for W as for U/OO sounds), but the auto-mapping still assumes a 10-frame speech view (W, frame 7, will never be used, so you can leave it blank or make it the same as frame 6) for consistency with the other formats. Also note that the order of the frames is not as listed on the Rhubarb page, but as in the table (behind spoiler tags) in the first post in this thread.

However, I'll see about adding some checking so the module can give a more informative error message if this happens.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #16 on: 06 Jun 2017, 11:50 »
I am curious - haven't yet tried a comipiled game...
If my sync/ folder is Compiled/; but my actual game is in Compiled/Windows/ - does this mean I include the sync directory in Windows/ too? Is the sync/ files just compiled into the game EXE?

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #17 on: 06 Jun 2017, 12:04 »
The sync files are not compiled in (I don't think there's really any way to do that), so you need to include the sync directory when you distribute the game.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #18 on: 07 Jun 2017, 03:12 »
But if the game searches for sync/ and not Windows/sync/ ?...
Oh well I'll change it.

Crimson Wizard

  • AGS Project Tracker Admins
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    •  
    • Lifetime Achievement Award Winner
    •  
    • Crimson Wizard worked on a game that was nominated for an AGS Award!
      Crimson Wizard worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.5
« Reply #19 on: 07 Jun 2017, 03:15 »
AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

But when you order "Build Exe" it builds it to Compiled/Windows. But it won't put extra files there automatically, so you would need to copy them over when preparing package.
(EDIT: actually just files from Compiled get copied into Compiled/Windows, but not subfolders.)
« Last Edit: 07 Jun 2017, 03:19 by Crimson Wizard »

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #20 on: 07 Jun 2017, 03:16 »
Exactly how does the EXE reference it's own install dir? Should I make it '/sync/' or 'C:/moci/Compile/Windows/sync' or...?
It seems the directory where the TSV files are stored is hard-coded in, so it will only work if the game is in '/programme files (x86)/moci/sync', and nothing else -- which makes it hard for testing.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #21 on: 07 Jun 2017, 03:17 »
Ah, okay.

Crimson Wizard

  • AGS Project Tracker Admins
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    •  
    • Lifetime Achievement Award Winner
    •  
    • Crimson Wizard worked on a game that was nominated for an AGS Award!
      Crimson Wizard worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.5
« Reply #22 on: 07 Jun 2017, 03:21 »
Exactly how does the EXE reference it's own install dir? Should I make it '/sync/' or 'C:/moci/Compile/Windows/sync' or...?

In script you reference install dir using $INSTALLDIR$ tag like
Code: Adventure Game Studio
  1. File.Open("$INSTALLDIR$/sync/data.dat");
  2.  

There is also "old style" without such tag:
Code: Adventure Game Studio
  1. File.Open("sync/data.dat");
  2.  
But that will work only to read files, not write.

Also, AGS does not allow to use absolute paths (like C:/...), it simply won't open anything even if such file exists. This is done for security reasons.

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #23 on: 07 Jun 2017, 07:43 »
I notice '$INSTALLDIR$/sync' - is this "C:\Program Files (x86)\Adventure Game Studio 3.4.0\sync" or "my installed game\sync" dir, or "my currently un-installed un-compiled game directory\sync", or...?
Exactly how does the EXE reference it's own install dir?

:-\

AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

Well, I just tested it, and if you have a "sync" folder inside "Compiled", it will be read when you run the game from the editor and reference files in "$INSTALLDIR$/sync". (But you are correct that the directory is not automatically copied into "Compiled/Windows/sync"; you have to do that yourself when you're ready to distribute the game.)

Crimson Wizard

  • AGS Project Tracker Admins
    • Best Innovation Award Winner 2013, for spearheading the AGS 3.3.0 project
    •  
    • Lifetime Achievement Award Winner
    •  
    • Crimson Wizard worked on a game that was nominated for an AGS Award!
      Crimson Wizard worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.5
« Reply #24 on: 07 Jun 2017, 15:05 »
AFAIK when you run the game from the Editor (with F5), game does not look into Compiled at all, it gets files and subfolders that are located right in the project's root folder.

Well, I just tested it, and if you have a "sync" folder inside "Compiled", it will be read when you run the game from the editor and reference files in "$INSTALLDIR$/sync". (But you are correct that the directory is not automatically copied into "Compiled/Windows/sync"; you have to do that yourself when you're ready to distribute the game.)

Oh right... probably I had a moment of amnesia; it was just couple of months ago what I was working around that, because in 3.4.1 Compiled folder is now Compiled/Data.

So, yes, when debugging from Editor the rules are a bit complicated, the "installdir" is actually "made" of three folders:
- working directory, which is project root folder: for example game takes font files from there
- Compiled folder: I think this was made to take speech.vox from there, and maybe something else, like translation files.
- AudioCache folder: it takes audio files from there.

Basically it looks into project root first, and if it did not find needed files, it also checks either Compiled or AudioCache, depending on what kind of material it is looking for.

Why it was made so: I think to increase compilation speeds when testing, because it does not have to gather/package all those files every time.
« Last Edit: 07 Jun 2017, 15:09 by Crimson Wizard »

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #25 on: 07 Jun 2017, 15:22 »
For the record, I think this is a good thing, because it means you can keep all your "to be distributed" files in "Compiled/", and you don't need separate copies for each target platform (with the potential nightmare of keeping all the copies in sync).

(I might split this whole discussion off as a separate thread, since it doesn't really have much to do with this module specifically.)

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #26 on: 13 Jan 2018, 12:50 »
I wrote a little script to help people who use Rhubarb to do lip-syncing.

Rhubarb does the lip-sync tracking automatically by analyzing the audio, but you can also supply the actual text of the dialog to help guide it, improving the results. Using something like this batch file, you can then create the lip-sync tracking for the entire game automatically:

Code: Bash
  1. for %%F in (clips/*.wav) do (
  2.     rhubarb.exe clips/%%~nxF -d guide/%%~nF.txt > sync/%%~nF.tsv
  3. )

There hasn't been a convenient way to create these guide files from AGS, however. This script helps to automate the task.

It works with the voice acting HTML scripts generated by the Speech Center plugin. Once you've created the voice acting scripts, place them in a subfolder (e.g. /VoiceScripts) inside your compiled game folder. Place a call to this function in your game, and run it to extract the guide files, like so:

Code: Adventure Game Studio
  1.   ExtractVoiceScriptLines("$INSTALLDIR$/VoiceScripts", "$APPDATADIR$/guide");

(The output directory must be in $APPDATADIR$ or an existing subfolder of it. Where %APPDATADIR% is pointing can be set in winsetup.exe under Advanced.)

Here's the script (yeah, I know it's ugly, but it seems to work OK):
Code: Adventure Game Studio
  1. #define SCRIPT_TOKEN_LINE_NUMBER "<span class=\"lineNumber\">"
  2. #define SCRIPT_TOKEN_TEXT_START ":</b> "
  3. #define SCRIPT_TOKEN_TEXT_END "</span>"
  4.  
  5. static String ExtractVoiceScriptLines(String sourceDirectory, String targetDirectory)
  6. {
  7.   int i=0;
  8.   String tokenLineNumber = String.Format("%s", SCRIPT_TOKEN_LINE_NUMBER);
  9.   String tokenTextStart = SCRIPT_TOKEN_TEXT_START;
  10.   String tokenTextEnd = SCRIPT_TOKEN_TEXT_END;
  11.   while(i < Game.CharacterCount)
  12.   {
  13.     String prefix = character[i].GetSpeechPrefix();
  14.     String scriptPath = String.Format("%s/char%s.html", sourceDirectory, prefix);
  15.     if(File.Exists(scriptPath))
  16.     {
  17.       File* scriptFile = File.Open(scriptPath, eFileRead);
  18.       String lineNumberPrefix = prefix;
  19.       while(!scriptFile.EOF)
  20.       {
  21.         String scriptLine = scriptFile.ReadRawLineBack();
  22.         int startLineNumberTag = scriptLine.IndexOf(tokenLineNumber);
  23.         if(startLineNumberTag != -1)
  24.         {
  25.           int startLineNumber = scriptLine.IndexOf(lineNumberPrefix);
  26.           int endLineNumber = scriptLine.IndexOf("]");
  27.           if(startLineNumber > 0 && endLineNumber > 0 && endLineNumber > startLineNumber + lineNumberPrefix.Length)
  28.           {
  29.             String strLineNumber = scriptLine.Substring(startLineNumber+lineNumberPrefix.Length, endLineNumber - startLineNumber - lineNumberPrefix.Length);
  30.             int lineNumber = strLineNumber.AsInt;
  31.             //Display("Writing %s%d", prefix, lineNumber);
  32.             if(lineNumber>0)
  33.             {
  34.               String message = "";
  35.               String dialogLine = scriptFile.ReadRawLineBack();
  36.               int startTextTag = dialogLine.IndexOf(tokenTextStart);
  37.               int endTextTag = dialogLine.IndexOf(tokenTextEnd);
  38.               if(startTextTag > 0)
  39.               {
  40.                 while(endTextTag == -1 && !scriptFile.EOF)
  41.                 {
  42.                   dialogLine = dialogLine.AppendChar(' ');
  43.                  dialogLine = dialogLine.Append(scriptFile.ReadRawLineBack());
  44.                   endTextTag = dialogLine.IndexOf(tokenTextEnd);
  45.                 }
  46.                 int textLength = endTextTag - startTextTag - tokenTextStart.Length;
  47.                 if(startTextTag > 0 && endTextTag > 0 && textLength >= 0)
  48.                   message = dialogLine.Substring(startTextTag+tokenTextStart.Length, textLength);
  49.                
  50.                 String linePath = String.Format("%s/%s%d.txt", targetDirectory, prefix, lineNumber);
  51.                 //Display("Writing line \"%s\" to file %s", message, linePath);
  52.                 File* lineFile = File.Open(linePath, eFileWrite);
  53.                 if(lineFile != null)
  54.                 {
  55.                   //Display("Writing line \"%s\" to file %s", message, linePath);
  56.                   lineFile.WriteRawLine(message);
  57.                   lineFile.Close();
  58.                 }
  59.                 else
  60.                   Display("Couldn't write line \"%s\" to file %s", message, linePath);
  61.               }
  62.             }
  63.           }
  64.         }
  65.       } // while(!scriptFile.EOF)
  66.       scriptFile.Close();
  67.     }
  68.     i++;
  69.   }
  70. }
« Last Edit: 13 Jan 2018, 12:54 by Snarky »

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #27 on: 20 Jun 2018, 05:02 »
Hi Snarky,

The speech animation delay seems to be too slow for TotalLipSync - no matter what I set a characters to (1, 5, -25, 30), it doesn't make a difference.
What Speech Animation Delay does TLS use?
ATM I'm using 120fps ('GameSpeed'), so I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.

Code: Adventure Game Studio
  1. cJulius.ChangeView(1);          //julius
  2. cJulius.SpeechView=2;           //julius speech
  3. cJulius.AnimationSpeed=1;       //normal animation speed
  4. cJulius.SetWalkSpeed(7, 6);     //normal walk speed

...though I still use some negatives - for an old man:
Code: Adventure Game Studio
  1. cJulius.AnimationSpeed=0;               //old animation speed
  2. cJulius.SetWalkSpeed(1, 1);     //old walk speed

No negatives have caused problems, but still.
I have 30frames walking for all characters.

So aaaaaanyway, how does TLS do it's timing?
How does Rhubarb do it's timing,
considering it's file is in tenths of a second:

Code: Adventure Game Studio
  1. 0.00    A
  2. 0.50    F
  3. 0.83    C
  4. 0.90    G
  5. 0.97    A
  6. 1.05    B
  7. 1.53    A
  8. 1.61    C
  9. 1.80    B
  10. 1.87    X
  11. 2.38    X

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #28 on: 20 Jun 2018, 07:04 »
Hi Snarky,

The speech animation delay seems to be too slow for TotalLipSync - no matter what I set a characrs to (1, 5, -25, 30), it doesn't make a difference.
What Speech Animation Delay does TLS use?

TotalLipSync doesn't use animation delay: it reads the timing data from the sync files and displays the corresponding frame.

Problems could potentially occur if the computer can't run the game at the game speed you've set. The animation might then fall behind the speech. So...

Quote
ATM I'm using 120fps ('GameSpeed')

Don't.
Film runs at 24 fps. A lot of cheaper animation runs at 12 fps, literally one tenth of the framerate you have set: you're showing ten frames for each one of theirs! AGS by default runs at 40 fps, which is plenty for a point-and-click game, and twitch-based games like FPSs aim for 60 fps, which is also how fast most monitors refresh. Which is all to say that 120 fps is an insane framerate for an adventure game.

More significantly: with a high-res game like yours, many (most) computers won't be able to keep up, and this could affect lip sync (and other sync situations).

Quote
so I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.

No negatives have caused problems, but still.
I have 30frames walking for all characters.

Well, the "cure" is worse than the disease in this case. A better question might be, do you really need 30-frame walkcycles? If you cut it to 15 (every second frame) or even 10 (every third), do you lose noticeable quality?

Quote
How does Rhubarb do it's timing,[/b] considering it's file is in tenths of a second:

Code: Adventure Game Studio
  1. 0.00    A
  2. 0.50    F
  3. 0.83    C
  4. 0.90    G
  5. 0.97    A
  6. 1.05    B
  7. 1.53    A
  8. 1.61    C
  9. 1.80    B
  10. 1.87    X
  11. 2.38    X

Those are hundredths.

Snarky

  • Global Moderator
  • Mittens Earl
  • Private Insultant
    • I can help with proof reading
    •  
    • I can help with translating
    •  
Re: MODULE: TotalLipSync v0.5
« Reply #29 on: 20 Jun 2018, 08:45 »
Two more things...

ATM I'm using 120fps ('GameSpeed'), so I can use a AnimationDelay=1, and not -7 as it was before (on 40fps). I was told using negatives for AnimationSpeed, WalkSpeed, etc. was bad and 'unknowns'.

Code: Adventure Game Studio
  1. cJulius.ChangeView(1);          //julius
  2. cJulius.SpeechView=2;           //julius speech
  3. cJulius.AnimationSpeed=1;       //normal animation speed
  4. cJulius.SetWalkSpeed(7, 6);     //normal walk speed

...though I still use some negatives - for an old man:
Code: Adventure Game Studio
  1. cJulius.AnimationSpeed=0;               //old animation speed
  2. cJulius.SetWalkSpeed(1, 1);     //old walk speed

A. 0 is not a negative number.
B. At 120 fps, a 30-frame walkcycle will run 4 times a second. Since a cycle includes two footsteps, that's 8 footsteps/second. That seems faster than necessary. (Edit: By comparison, the world's fastest runners sprint at a max pace of about 4.4 – 4.8 steps/second.)
« Last Edit: 21 Jun 2018, 12:16 by Snarky »

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #30 on: 31 Jul 2018, 03:26 »
My Inventory window icons dissappear when the character is talking, even if it's just .Say
Is this an AGS thing or a TotalLipSync thing?

abstauber

  • Cavefish
  • Mittens Knight
  • still mowing the lawn
    • abstauber worked on a game that was nominated for an AGS Award!
      abstauber worked on a game that won an AGS Award!
Re: MODULE: TotalLipSync v0.5
« Reply #31 on: 31 Jul 2018, 07:51 »
Just a quick guess, but have a look at 'General Settings' in the AGS Editor. In the category 'Visual', check the setting "When player interface is disabled, GUIs should" and make sure it's set to 'Display normally'.
That could be at least one cause of the icons disappearing.

bx83

  • Get 'Er Doooooone
Re: MODULE: TotalLipSync v0.5
« Reply #32 on: 31 Jul 2018, 07:56 »
That's the one, thankyou :D