Coroutines in AGS

[Crimson Wizard]

I have recently rewrote a engine's script executor in the AGS 4 branch (see PR #2621). While the script format and basic execution remains exactly same, this rewrite changes the way the script execution state is handled. Among other things, it introduced an improved concept of "script thread". AGS had a sort of a "script thread" idea before, there has been 2 threads: normal one which run most script, and non-blocking one which run "repeatedly_execute_always". But this system was very limited, and not convenient to expand. The new system allows to do 2 following things relatively easily:

1. Create new (temporary) "script threads" dynamically. The "script thread" contains its own stack memory (for local variables) and callstack queue (for remembering nested function calls).
2. Saving script thread's state and restoring later. This means that the whole local state of a script's execution is saved at a current point, kept recorded for an undefined duration, and then later may be run from the same point again.

In my opinion this opens a potential for implementing coroutines in AGS. Coroutines are types of functions that can pause, then wait for something (either some event, or a external command) and resume.
I have spoken about my vision of what coroutine may be in AGS earlier here:
https://www.adventuregamestudio.co.uk/forums/editor-development/feature-request-behavior-of-hotspot-that-results-in-a-room-change/msg636665195/#msg636665195
But that was never meant to be a final plan, and when I think about this now I have doubts about few things.
Then I made a dirty experiment here:
https://www.adventuregamestudio.co.uk/forums/editor-development/feature-request-behavior-of-hotspot-that-results-in-a-room-change/msg636668451/#msg636668451
Again, this was merely a technical proof of concept, not a feature suggestion.

I'd like to open a proper conversation about how coroutines may be designed in AGS, both from the point of view of script syntax and internal implementation.

For the reference, following is technically possible to do with "script threads" now:
- script threads may be created anytime, and stored by the engine; they may be identified by something (whether string name or numeric index);
- script threads may be paused at any or almost any instruction in script, and resumed by the engine (resuming means simply that engine runs previously recorded script thread again);
- running a saved script thread may be done either by engine's updating the game (i.e. if there's a list of script threads which are checked on each game update), or by a direct command from a script.
- script threads may be cloned (copied and saved as a different thread object); idk if that is necessary, but it's possible.
- saving or cloning a script thread may be full or partial. For instance, we may record/copy only the latest nested function call and its local data, but ignore everything that preceded it.

What cannot be done:
- as far as I can tell, we cannot save script execution that have nested engine calls in them. What I mean is, suppose there's a engine function that can be run from the script, but also calls another script. Then you have an interleaved nested script and engine calls: engine -> script -> engine -> script -> ...
This kind of nested call was not possible before, it is possible now with the new executor system, but we won't be able to save such callstack nor restore it, because engine own functions cannot be restored by the script executor. I suppose there may be ways around it, but this seems too complicated, so perhaps not worth trying.
Then, perhaps the engine should not make such nested calls on the same script thread, but always create new temporary script thread whenever it has to run another script callback while being inside one script callback already (like what happens when it runs rep-exec-always, while being inside a Wait called by a script).

Above should be kept in mind when designing coroutines feature.

[Crimson Wizard]

I've been thinking about a simpler variant of asynchronous function mechanic.
The "coroutines" idea posted earlier would require designing a new scripting feature, including new syntax. I don't know if i can, and don't know if anyone will help me. Yet there's an alternative, which is simpler both in terms of development, and likely going to be easier for users, because it's based on existing concept.

I am talking about a "WaitAsync" function. Comparing her with the standard Wait, WaitAsync does not block the game script and lets the game run fully normally, with events occuring and player input registering, gui working, and so forth. Back then I did not expect to actually add this function for real, but it may be actually not a bad idea.

My main question about it was: how many use cases will this approach cover?
The WaitAsync is a primitive function that only has a "game ticks number" as a parameter, so all it does it waits for N game updates to pass. But then, it may also be used in a classic loop with Wait(1) when you are waiting for something to complete.

For example:

Code: ags

while (player.Moving) {
    WaitAsync(1);
}

This means that even if we have this 1 function, it may be already enough to wait for any kind of condition. It's just that you will have to write this yourself in script. One can make a collection of custom waiting functions, which wait for character movement, animations, etc.

How does this function work behind the scenes:
* it marks current script thread as suspended;
* script executor interrupts script execution and returns to the engine;
* engine takes a "snapshot" of it and saves in a backup script thread object, paired with a integer counter.
* engine then proceeds running the game as normal, and once in a game update it also decrements this counter.
* when the counter reaches zero, engine restores the "snapshot" of a suspended script and runs it from the same point where it paused last time.

This approach does not create real coroutines, because the "waiting" function does not return back to where it was called from immediately. This looks more like forking a program thread.

In my experiment there was only 1 such snapshot possible at a time, but in theory there could be unlimited number of them. I am currently trying to figure out what would be the requirements for this feature, and if there's a need of anything else to make it usable.


The biggest issue that I've found so far, and not just with WaitAsync idea, but with suspending script in general, is saving and restoring them as a part of the game state. When the script thread resumes, it has to know the execution position, callstack (list of nested function calls), and data stack (values of local variables and function parameters). It's possible to put all that into the saved state. But when restoring, this data has to match the current script. If the script has changed even a little bit since the save was made, then this data can no longer be applied: execution point may refer to a different place, and data may not correspond to local variables. This feature will make the AGS saves significantly more fragile than they already are, as virtually ANY change in a script can lead to breaking older saves. I believe that this is unacceptable.

After thinking about this for a while, I see 3 options:

1. Forbid saving when there are any suspended functions. This is the dumb and simple solution, but will significantly narrow the use case for this script feature, close to making it unnecessary.

2. Don't put them into the save. This is easy for the engine, but will make it very unreliable for the user. Because in such case the user will have to script restarting of these async sequences after the save is restored, and not just restart them, but restart in exact stage. For which reason one would have to keep track at least of a list of stages within each async function as it runs. I think that will mostly defeat the purpose of having this feature, and it will be easier to just script the module that deals with non-blocking sequences.

3. Save, but not just the suspended execution state, but the executed part of the script. This may be quite tricky to do, but theoretically possible. When engine saves the game and knows that there are suspended functions, it will gather a list of code blocks that are involved in those functions, and save these blocks along. When restoring, it will create a "temporary" script (or collection of temporary scripts), where these restored functions will run. As soon as all the "old" function instances end running, this temporary script is discarded.
Note that likely we won't have to save virtually all scripts this way, but only parts of the script where the execution point was suspended, and parts mentioned in the callstack (in case of nested calls). We may not include any other functions that are called from the suspended function. Instead, we may save their references ("import" data), and resolve these upon restoring this suspended function.
From the user's viewpoint it will look like the game runs old variant of a script function.
Any next run of the asynchronous function will of course run the new variant.


Is there anything else I may not be seeing here? Other options, or other problems?