Feature Request: Inspect variable while debugging

Started by JanetC, Tue 10/01/2017 22:11:21

Previous topic - Next topic

JanetC

Most debuggers allow you to hover over a variable while stepping through the code in order to inspect the contents of the variable. This would make my life so much easier!

Crimson Wizard

#1
Quote from: JanetC on Tue 10/01/2017 22:11:21
Most debuggers allow you to hover over a variable while stepping through the code in order to inspect the contents of the variable. This would make my life so much easier!

The HUGE problem of AGS is that compiled code has very limited reflection. In plain language this means that when engine runs the script, it does not always know what exactly it deals with, it does many things blindly according to instructions. For example, variables are saved not as variables, but as a big array of bytes. The engine is not explicitly told what are those variables and where in this array they are located. For the engine there is no "int a", there is, say, address "50" where it must read 4 bytes. It is possible to deduce some things from instructions, as script runs, but still no way to know variable names.

UPDATE: I just remembered that compiled script stores names of imported and exported variables for some reason. That's a good start, but not enough for general case.

So, this is not only a task of making engine tell some information on the script it runs, but making engine being able to gather such information first.
On other hand, if I am right, I believe that it should not be too hard to add such information about global variables into compiled script. I am not that certain about local ones (the way they are dealt with is somewhat different).

Crimson Wizard

There is yet another way this may be solved.
If Editor will write down table of variables when compiling scripts, remembering names of variable and their addresses for the debugging times, then it may ask engine not about certain names, but certain memory addresses, which engine does know.
I cannot tell whether that would be easier or harder to do. Pro is that in such case you do not need to change compiled script format, con is that this probably introduces new kind of temporary output for the project (because Editor needs to keep these tables somewhere). (also engine still won't know what it works with)

RickJ

I would think having a symbol table containing name, type and address would be the way to go (at least for variables with statically assigned memory addresses).  Why not have an option to generate the symbol table and include it in the game file(s)? It would be a small step further to have script commands that access the table so that debug utilities could be implemented in script.

JanetC

Quote from: Crimson Wizard on Tue 10/01/2017 23:06:48
UPDATE: I just remembered that compiled script stores names of imported and exported variables for some reason. That's a good start, but not enough for general case.

Just that would be a big help, because usually the problem variables for me are imported/exported variables (I use them a lot.)

Crimson Wizard

This task includes number of things to consider and research.

From user interface side:
- how the watched variables are displayed? Should there be a pane with list of those, or floating hints.

From script side:
- The compiled script must have a table of variables with offsets and names, or at least only offsets (but in latter case Editor must keep the lookup table to find offset by variable name).

Data transfer:
- There is already a pipe between engine and debugger, to send commands (breakpoints sent to engine, line numbers and callstack sent to the debugger). What protocol would be optimal for this, and when the information is sent. Should each variable value be sent by request from debugger, or should engine send variable values itself, e.g. when they are modified.

RickJ

I don't know if Janet agrees but I would think that if there were a symbol table from which a script command could return a reference to a named variable then people could write their own debug functions.  It could be useful for other things as well.

JanetC

Quote from: Crimson Wizard on Sat 11/02/2017 19:26:45
From user interface side:
- how the watched variables are displayed? Should there be a pane with list of those, or floating hints.

Whichever is easiest to code :) XCode includes both.

Crimson Wizard

This is old... things always progress slow in AGS.

For the reference, after RTTI feature was merged in AGS, it might be possible to similarly generate a table of variables with names and write along the compiled script, as an optional block of data (which may be enabled or disabled).

If this data is accessible by the engine, then the engine may return an information about variable and its current state.



An alternate approach is still viable too (as was years ago):
QuoteThere is yet another way this may be solved.
If Editor will write down table of variables when compiling scripts, remembering names of variable and their addresses for the debugging times, then it may ask engine not about certain names, but certain memory addresses, which engine does know.
The meaning of this is to ask engine to return a data at offset X of size N from a script memory.

Crimson Wizard

#9
The minimal requirements for the variables watch are:

1. Script compiler generates a table that maps name of a variable to offset in script data. Such table must be done per each script, as variables depend on visibility scope, and each script has its own (just like types).
2. This table is saved either in a separate file, or as an extra data in a compiled script file (similar to rtti).
3. This table is read either by the Editor or Engine.
4. If this table is read by the Editor, then on user request it converts variable name to offset, and sends a command to the running engine through the existing communication mechanism. This command asks engine to return a value from the given data offset (there's more than that, but it's a general idea).
5. If this table is read by the Engine, then Editor sends a command with variable's name instead, and Engine is responsible for converting this name to an offset. From this moment p4 and p5 match.
6. After retrieving variable's value, engine passes it back to the Editor, and Editor displays it.
7. The rest here is mostly an issue of GUI.

Details.
   
Variables are identified within certain context (scope of visibility), because there may be multiple variables of same name inside different scripts, and even same script (think local function variables).
This means that the mapping is done in 2 steps:
   
    Context-dependent name -> Global unique name -> Memory offset
   
For global variables the unique name may be formed as "modulename.varname", similar to how global type ids are formed in RTTI.
   
Local variables are more tricky, because they have a limited lifescope, which may be a function, but also a section of a function (anything surrounded by brackets). This means that for them the table of variables should also mention first and last script line of their life scope. If a variable is requested, but there's no such variable found in the given context (script + current line), then such request must be denied.
   
With local variables in mind, the "global unique name" of a variable should also include their scope. Now, I don't remember if AGS compiler supports overriding variable names in the nested scopes, but I think we should assume that eventually it does (IIRC this was discussed on github once). So, I guess the global unique name should be something like "modulename.varname.scope", where scope could be a pair of numbers meaning the first and last line of code of their scope of visibility.
   
A conversion between a "Context-dependent name" and "Global unique name" possibly can be done like this.
We have a table of variable names for the given context (a script module), and for each key in this table we will have not 1 variable, but a list, sorted by the first line this variable is visible on. For a variable request, we find its name in the table and traverse this list until we find an entry which pair of first-last lines matches the location of current breakpoint. This is how we learn the "global unique name". This "global unique name" is passed further to find a memory offset.

Example of a list (only to demonstrate a potential case):
    module.myvar -> module.myvar.10.20 -> module.myvar.30.120 -> module.myvar.50.60
Here we have a global variable "myvar", some local one between lines 10-20, another local between 30-120, and a nested one which overrides previous for the duration of its life scope - between lines 50-60.
   
Noteably, memory offset should be paired with a memory type: either global memory or local memory (stack), so that engine knew where to look for it.
   
After receiving memory offset we need to interpreter the value stored on it, and convert to a string. For that we need RTTI which tells us variable's type.
   
    Editor                                                                                                              Engine
        Context-dependent name -> Global unique name -> Memory offset
        Displayed value <- Data Value <- Memory offset

Handling structs.
   
Reading a struct's member can be done by passing another, nested offset. Getting that offset requires RTTI, which tells which relative offset does a member of certain name has. Same as with variable itself, there are two alternatives here, one where a list of offsets is resolved on Editor's side, and second where it is resolved on Engine's side (and Editor passes just a sequence of names - "variable name :: member name :: member name ...").
   
Regardless, engine must know how to access each nested member. There are two variants here: plain struct and managed struct. A member of a plain struct is accessed simply by adding a relative offset to the struct instance's address. A member of managed struct is accessed by resolving the pointer first.

Handling arrays.

I suppose that arrays are handled like plain structs.

Reading values of attributes (aka properties).
   
Attributes are pairs of get/set functions in AGS. Reading a property would require to call a registered function. In theory it must be possible to do this even outside of a script vm, but the biggest issue is potential side-effects that such call may involve. I'd rather leave this out at least until the variable reading mechanism is developed and proved working.
   
   

eri0o

About the issue of GUI, I think I imagine it would use a new panel for this with a TreeView and it would show each variable in scope as a node in the TreeView. Then it would be possible to expand a struct and view it's entries. Not sure about the arrays if they need expansion or not. My first guess would be to not support this for arrays.

When the node is shown some elements could have an alternative pretty print of them so you could read it more easily (an array of ints could pretty print its values as "{1, 5, 0, 3838}" for instance and a string should pretty print it's contents. A point would pretty print as "(160, 120)".

The node text in my mind is "varname: value".

I don't know if necessary, but it's possible to group the nodes under either Local or Global node.

The TreeView only populates when the script breaks or when it advances only one step.

There's a right click menu for each variable where it's possible to copy its values.

When the game stops running the panel clears.

I don't know if any configuration is necessary to store for this panel. My guess is none now - except layout stuff that AGS already stores somewhere else.

Crimson Wizard

#11
Quote from: eri0o on Yesterday at 15:00:26About the issue of GUI, I think I imagine it would use a new panel for this with a TreeView and it would show each variable in scope as a node in the TreeView.

I must point out that having this for all variables at once will require editor to ask engine for all of their values each step, as editor does not know which ones of them change.
Otherwise there would have to be a mechanism that detects changes to the script memory, which we do not have at the moment.
This would also raise a question of convenience, as user will have to search for wanted variables in this tree view.
For a first iteration, I'd suggest to have a list where user inputs wanted variables by name, similar to how Visual Studio does this.

Crimson Wizard

#12
Hmm, the list of all variables may be presented as a selection of what to add to the "watch" panel.

In other words, there's a watch panel which lists currently watched variables, and a separate panel or a dialog window, where user may choose from a full list of variables. Although this will work reliably for global variables. It may list known local variables too, except not all may be active in particular scope.

EDIT:
well, in any case, I think it should be as simple as possible at start, because the biggest problem is making the variable value extraction, and GUI may always be adjusted later.

SMF spam blocked by CleanTalk