Jibble

Author Topic: The Custom Speech API proposal  (Read 3103 times)

The Custom Speech API proposal
« on: 17 Dec 2014, 11:53 »
The main problem of custom speech was that it cannot be binded with standard Say commands in game script and speech lines in Dialog script. You cannot make the engine call your custom functions, you have to do that yourself by manualy placing their calls.
Not only that takes more time, but also you cannot take advantage of otherwise automatic features, such as character animation, voice playback and speech skipping: you have to code these yourself.

The most logical (in my opinion) solution would be to add support for speech callbacks, which work in a similar way as custom dialog options rendering: you give your functions predefined names and engine calls them instead of built-in speech processing.

There was already some discussion about this a while ago: http://www.adventuregamestudio.co.uk/forums/index.php?topic=48518.0
Since then I changed my mind a bit though.

My proposal is to support at least two callbacks. We may later extend them, as well as the contents of the callback parameter.

Code: Adventure Game Studio
  1. function speech_start(SpeechRenderingInfo *info);
  2.  
This is called before displaying a speech. User should set up some of the SpeechRenderingInfo properties to initialize the speech overlay

Code: Adventure Game Studio
  1. function speech_render(SpeechRenderingInfo *info);
  2.  
This is called when the speech requires a repaint (default - only once). Here the user should get a drawing surface and do paint operations.

The contents of SpeechRenderingInfo might be:
Code: Adventure Game Studio
  1. struct SpeechRenderingInfo
  2. {
  3.   //
  4.   // the Overlay settings
  5.   //
  6.   attribute int X; // the X co-ordinate of the top-left corner of the dialog options
  7.   attribute int Y; // the Y co-ordinate of the top-left corner of the dialog options
  8.   attribute int Width;  // the width of the dialog options
  9.   attribute int Height; // the height of the dialog options
  10.   attribute bool HasAlphaChannel; // should the drawing surface have alpha channel
  11.   readonly attribute DrawingSurface* Surface; // the surface that the speech is rendered to
  12.  
  13.   //
  14.   // Speech data
  15.   //
  16.   readonly attribute Character *SpeakingChar; // the character who is talking
  17.   readonly attribute String     Text; // the text to display
  18.   readonly attribute int        VoiceID; // the id of the voice file (taken from "&CHARXXXX" part of the speech text)
  19. };
  20.  


I would like to propose game developers to consider if they will be able to implement their actual custom speech using this system. Are there any other callbacks and/or speech data that would be required?
« Last Edit: 17 Dec 2014, 12:02 by Crimson Wizard »

Snarky

  • Global Moderator
  • Global Moderator
  • Mittens Lord
  • Private Insultant
    • Best Innovation Award Winner 2018, for his numerous additions to the AGS open source ecosystem including the new Awards Ceremony client and modules
    • Snarky worked on one or more games that won an AGS Award!
    •  
    • Snarky worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #1 on: 17 Dec 2014, 12:34 »
Nice work!

My only experience with custom speech has been for animated text (primarily to progressively type out the text typewriter-style, but I've also experimented with things like flashing, shaking or changing text). If I understand this API correctly it wouldn't be able to support that, but it might very well be that it falls outside of what it's meant for, and that my requirements not suited to a built-in API.

Re: The Custom Speech API proposal
« Reply #2 on: 17 Dec 2014, 12:59 »
Nice work!
Well, thank you, but it's just the design paper. I have no change to code yet, and frankly it may take some time, because the speech code is pretty messy in AGS.
So first I would like to know what we are doing.

My only experience with custom speech has been for animated text (primarily to progressively type out the text typewriter-style, but I've also experimented with things like flashing, shaking or changing text). If I understand this API correctly it wouldn't be able to support that, but it might very well be that it falls outside of what it's meant for, and that my requirements not suited to a built-in API.
Aha, right, the typewriter effect.
Yes, we might need an extra "repexec" callback for this.

Calin Leafshade

  • Long live King Cat!
    • I can help with making music
    • I can help with voice acting
    • Calin Leafshade worked on one or more games that won an AGS Award!
    •  
    • Calin Leafshade worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #3 on: 17 Dec 2014, 13:43 »
Bravo!

Custom speech is something I have hacked in a few times using an empty font but having some interface for that would be great

Adeel

  • Cavefish
    • Adeel worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #4 on: 17 Dec 2014, 13:49 »
Code: Adventure Game Studio
  1.   // Speech data
  2.   //
  3.   readonly attribute Character *SpeakingChar; // the character who is talking
  4.   readonly attribute String     Text; // the text to display
  5.   readonly attribute int        VoiceID; // the id of the voice file (taken from "&CHARXXXX" part of the speech text)
  6. };
  7.  

Can you please another attribute for the portraits too? The default 'alternative' style doesn't work all the time and often produces sleazy results. AprilSkies can testify to this. She He even told me that he uses a custom speech function just to display alternative portraits correctly.

Scavenger

  • Cavefish
  • In Four Glorious Colours!
    • I can help with animation
    • I can help with backgrounds
    • I can help with characters
    • I can help with scripting
    • Scavenger worked on one or more games that won an AGS Award!
    •  
    • Scavenger worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #5 on: 17 Dec 2014, 13:59 »
While we're talking about rep_exec callbacks, there's no chance that we could get the capability for non-blocking speech in with it, is there? It's the one thing that would make dialogues really shine, since right now we can't really control the pace of the conversation without hacking in fake lines of speech, nor have overlapping speech. The most hacked speech thing for me is definitely background speech, and interrupting. Unifying all the speech types (do we really need a SayBackground anymore?) would reduce the amount of hacking needed to be done with speech. No more speech lines being stored as sound effects, either!

Re: The Custom Speech API proposal
« Reply #6 on: 17 Dec 2014, 14:00 »
Code: Adventure Game Studio
  1.   // Speech data
  2.   //
  3.   readonly attribute Character *SpeakingChar; // the character who is talking
  4.   readonly attribute String     Text; // the text to display
  5.   readonly attribute int        VoiceID; // the id of the voice file (taken from "&CHARXXXX" part of the speech text)
  6. };
  7.  

Can you please another attribute for the portraits too? The default 'alternative' style doesn't work all the time and often produces sleazy results. AprilSkies can testify to this. She He even told me that he uses a custom speech function just to display alternative portraits correctly.

Gurok made some fix for default alternative portraits I think.

I am not sure which attribute do you have in mind?

Calin Leafshade

  • Long live King Cat!
    • I can help with making music
    • I can help with voice acting
    • Calin Leafshade worked on one or more games that won an AGS Award!
    •  
    • Calin Leafshade worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #7 on: 17 Dec 2014, 14:36 »
Instead of callbacks, what about providing an interface instead?

We have managed objects that can inherit other objects, couldnt you just pass an instance of one of those to a function which inherited a base interface struct?

The advantage of that approach is that it's easily modularized and interchangeable. It's also more OOP.

Re: The Custom Speech API proposal
« Reply #8 on: 17 Dec 2014, 15:00 »
Instead of callbacks, what about providing an interface instead?

We have managed objects that can inherit other objects, couldnt you just pass an instance of one of those to a function which inherited a base interface struct?

Because AGS Script has no RTTI, nor virtual table, nor function pointers I cannot make engine call member function of any random object.
Simply put, the AGS does not have late binding, only static binding.

The only way that exists is to call a function of predefined name, because we have a map of function names.
« Last Edit: 17 Dec 2014, 15:19 by Crimson Wizard »

Adeel

  • Cavefish
    • Adeel worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #9 on: 17 Dec 2014, 15:02 »
Gurok made some fix for default alternative portraits I think.
In that case, I missed that fix. I'll make sure to ask him about this.

I am not sure which attribute do you have in mind?
The attribute or property of being able to call the portraits either left or right at any given time, depending on your need.

Re: The Custom Speech API proposal
« Reply #10 on: 17 Dec 2014, 15:11 »
I am not sure which attribute do you have in mind?
The attribute or property of being able to call the portraits either left or right at any given time, depending on your need.
No, you don't seem to understand this idea.
It is you who are supposed to draw portraits in this function. And you may do so the way you like.

Adeel

  • Cavefish
    • Adeel worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #11 on: 17 Dec 2014, 15:18 »
I am not sure which attribute do you have in mind?
The attribute or property of being able to call the portraits either left or right at any given time, depending on your need.
No, you don't seem to understand this idea.
It is you who are supposed to draw portraits in this function. And you may do so the way you like.
My bad, sorry for that...

Scavenger

  • Cavefish
  • In Four Glorious Colours!
    • I can help with animation
    • I can help with backgrounds
    • I can help with characters
    • I can help with scripting
    • Scavenger worked on one or more games that won an AGS Award!
    •  
    • Scavenger worked on one or more games that was nominated for an AGS Award!
Re: The Custom Speech API proposal
« Reply #12 on: 18 Dec 2014, 02:16 »
Oh, I know what attribute is missing! The Pamela lip sync data (like, the current phoneme et al)! That's not opened to the script, but if we're going to do lip sync with our custom speech functions, we'll need it!