Adventure Game Studio

Hello I was wondering if I could have some assistance with the TotalLipSync module.

I've made a game that works well and would like to add lip syncing with voice via the module but I can't seem to get it to work. Its my first time trying a module. I started a brand new game to test it. The game is just one room with one character who says one line. I'm sure I've taken a wrong step or two as I'm getting no lip sync and no voice (the character is just staying on one frame). Here is my process so far (I am using the most up-to-date version of AGS, I downloaded the module from Snarky's 2017 forum post and the latest version of Rhubarb).

1) I used Adobe After Effects to create 9 mouth shapes on a character based on Rhubarb's guidelines. The different mouth shapes are drawn on top of a fixed face in each frame, rather than the mouth as a separate layer.

2) I have nine mouth shapes based on the Rhubarb guidelines so A (frame 1 at 00:00) is P, B and M, B (frame 2 at 00:01) is K, S, T etc. I have used A, B, C, D, E, F, G, H and X as instructed.

3) I put my .wav file into C:/Rhubarb.

I) In After Effects, I selected File > Scripts > Rhubarb Lip Sync.jsx and then animate.

4) Everything looks good so far. I used the Windows Command Prompt (something I'm admittedly a novice with) to export the .dat file like so:
cd C:\Rhubarb
rhubarb JIMM.wav -o JIMM.dat
- perhaps the .dat data needs more fine tuning?

5) This created a .dat file. Opening the .dat file in notebook gives me the following information:

0.00   X
0.13   C
0.31   E
0.66   B
1.08   C
1.15   B
1.43   E
1.64   X
2.03   X

6) The character in my AGS game is called Jimmy (cJimmy) so the files are called JIMM1.wav and JIMM1.dat

7) I downloaded the TotalLipSync module and imported it into Scripts in AGS.

8) I added the following code to my global script:
function game_start()
{
  TotalLipSync.Init(eLipSyncRhubarb);
  TotalLipSync.AutoMapPhonemes();
}

9) I made sure that the lip sync option is disabled in the Lip Sync property.

10) I created a folder called 'sync' and placed it in my project folder (E:/2025/Rhubarb Test/Rhubarb Testing/sync). I placed both the .wav and .dat files in there.
- I'm not totally sure where these files should go so also tried moving these to different locations but none were successful.

11) In Compiler in general settings, I added the following: E:/2025/Rhubarb Test/Rhubarb Testing/sync
- Again, not totally sure if this part is correct.

12) I made sure the character's Speech View was set to View 3 (where I placed all of the frames in order.
- Rhubarb advised on 9 frames, TotalLipSync advises 10. In AGS, I currently have frame 0 set as his neutral mouth shape followed by A (frame 1),B (frame 2) etc.

13) I entered this in the room script:


function room_AfterFadeIn()
{
 cJimmy.SaySync("&1 My text is here.");
}

If anyone knows where I'm going wrong (probably multiple places!), I'd love to know. I'm not an expert and realise that this is quite advanced but I'm a keen learner and, if I can learn how it works in a test room, I think I can apply it to the whole game with some time.

Thank you!

Thanks for the very detailed problem report! It helps a lot in identifying the problem.

Quote from: Ewen on Mon 11/08/2025 16:30:0010) I created a folder called 'sync' and placed it in my project folder (E:/2025/Rhubarb Test/Rhubarb Testing/sync). I placed both the .wav and .dat files in there.
- I'm not totally sure where these files should go so also tried moving these to different locations but none were successful.

This is correct if E:/2025/Rhubarb Test/Rhubarb Testing/ is your game project folder (the folder where the game.agf file is).

Quote from: Ewen on Mon 11/08/2025 16:30:0011) In Compiler in general settings, I added the following: E:/2025/Rhubarb Test/Rhubarb Testing/sync
- Again, not totally sure if this part is correct.

I assume you mean the setting "Package custom data folder(s)"? This is not correct. It should just be "sync" (the path is relative to the project folder). I think this is why it is not working.

It's weird that you're not getting voice, though.

Quote from: Ewen on Mon 11/08/2025 16:30:0012) I made sure the character's Speech View was set to View 3 (where I placed all of the frames in order.
- Rhubarb advised on 9 frames, TotalLipSync advises 10. In AGS, I currently have frame 0 set as his neutral mouth shape followed by A (frame 1),B (frame 2) etc.

The reason for this is that the other common lip sync animation formats (Pamela, Moho Switch, Papagayo) support a tenth frame, and the default mapping is set up to let you use the same speech view regardless of format. In order to get Rhubarb to work correctly, F should be repeated, used for both frame 6 and frame 7 of the view – the important part is that the automapping assumes that G and H are at frame 8 and 9.

Thanks for the response Snarky. Its great that you can try and help! I have implemented your suggestions but it still doesn't work.

Quote from: Snarky on Mon 11/08/2025 22:07:13This is correct if E:/2025/Rhubarb Test/Rhubarb Testing/ is your game project folder (the folder where the game.agf file is).

Yes, the game.agf is in the Rhubarb testing file, where the sync folder now is.

Quote from: Snarky on Mon 11/08/2025 22:07:13I assume you mean the setting "Package custom data folder(s)"? This is not correct. It should just be "sync" (the path is relative to the project folder). I think this is why it is not working.

I have updated this now.

Quote from: Snarky on Mon 11/08/2025 22:07:13the important part is that the automapping assumes that G and H are at frame 8 and 9.

I have updated this in the View in AGS. One small change here: the mouth moves to the A frame and stays there until the text is gone and then it reverts to the idle view. My new pattern, for clarification, is A,B,C,D,E,F,F,G,H,X.

I tried dropping the .wav into the speech folder (also in the Rhubarb Test folder) and the voice clip did play.

Do you have any other thoughts about why its not working? As mentioned, its a blank game other than what I added in specifically for this test.

The only thing I can think of is that I've exported the .dat incorrectly from the command prompt (but the file looks right) or maybe I've downloaded the wrong version of the module. I downloaded version 0.6 from the forum, rather than going into the github links.

Any help to resolve this would be amazing. Thank you!

One thing is in the sync dir there should be only the sync data and not the speech audio files, those should go in the regular speech directory following what's on the AGS manual.

I don't remember seeing games that used TotalLipSync module with LucasArts speech, but I leave the answer if that works or not for @Snarky .

Quote from: eri0o on Yesterday at 00:45:50One thing is in the sync dir there should be only the sync data and not the speech audio files, those should go in the regular speech directory following what's on the AGS manual.

True, but if you've also got them in the speech folder (where they're supposed to be), this should merely make the game file larger than necessary, not lead to it not working.

Quote from: eri0o on Yesterday at 00:45:50I don't remember seeing games that used TotalLipSync module with LucasArts speech, but I leave the answer if that works or not for @Snarky (https://www.adventuregamestudio.co.uk/forums/profile/Snarky/) .

Yes, support for LucasArts speech was one of the main reasons for the module's existence. Are you in fact using LucasArts speech, Ewen? In the other speech modes, there is a need for a dummy view (step 5 in the instructions).

Quote from: Ewen on Mon 11/08/2025 23:27:38I have updated this in the View in AGS. One small change here: the mouth moves to the A frame and stays there until the text is gone and then it reverts to the idle view. My new pattern, for clarification, is A,B,C,D,E,F,F,G,H,X.

It should be X,A,B,C,D,E,F,F,G,H (starting from frame 0 and going to frame 9). See the spoilered table in the instructions in the module release post (https://www.adventuregamestudio.co.uk/forums/modules-plugins-tools/module-totallipsync-v0-5/).

That you were seeing frame A (meant to be the idle frame) perhaps indicates that the module is running properly but not reading the data. Hmm...

Edit: Oh, I think I've got it! Since the module was last updated, Rhubarb changed its default file extension from .tsv to .dat. You just need to tell TLS what the extension is. Use:

function game_start()
{
  TotalLipSync.Init(eLipSyncRhubarb);
  TotalLipSync.SetFileExtension("dat");
  TotalLipSync.AutoMapPhonemes();
}

Some positive news on my end! I altered the code and updated the loop and the mouth is now working perfectly!! However, the speech is not playing.

Quote from: Snarky on Yesterday at 05:22:43Are you in fact using LucasArts speech, Ewen?

Yes I am. As a workaround, I have done what Eri0o said and put the .wav file in the Speech folder. However, I had to then enable the old-style voice clip naming rule for it to work. BUT it does work now! Do you think this will be okay going forward or is there a more streamline way to do it (with the WAV file in the sync folder)?

Thank you! This has been super helpful.

Quote from: Ewen on Yesterday at 08:31:11Yes I am. As a workaround, I have done what Eri0o said and put the .wav file in the Speech folder. However, I had to then enable the old-style voice clip naming rule for it to work. BUT it does work now! Do you think this will be okay going forward or is there a more streamline way to do it (with the WAV file in the sync folder)?

The voice clips have to go in the Speech folder for AGS to recognize them. It might work to use the new filename format for the speech files; you would just have to name them according to that format. However, the module has not been updated to support the new filename format for the sync data files, and having two different styles for the voice clip and the corresponding sync file would probably complicate things considerably.

So I would just stick with this for now.

Another point worth mentioning is that from v3.6.2, AGS supports LucasArts speech for the built-in lip sync, and Rhubarb can now output sync files in Moho dat format (which can be read by the built-in lip sync), so there may not be a need for the module going forward.

Uhm, I guess the manual page on LipSync needs some update then (web (https://adventuregamestudio.github.io/ags-manual/Lipsync.html), source file (https://github.com/adventuregamestudio/ags-manual-source/blob/master/Lipsync.md)), but I haven't actually used lip sync and am not sure on what has to be updated.

Thanks guys! Your help has been exceptional. I'm pretty happy with this solution for now. I'll perhaps dive deeper into what's possible with .dat files in the update but I'll stick with the module as I am able to use it pretty easily. Thanks again.