Topic: Help file... here we go again.

[Monsieur OUXX]

Continuing from those threads :
http://www.adventuregamestudio.co.uk/forums/index.php?topic=46223.0
http://www.adventuregamestudio.co.uk/forums/index.php?topic=46220.0
http://www.adventuregamestudio.co.uk/forums/index.php?topic=46471.0

The help file is getting more and more out of sync with the actual Editor. It's no big deal for most script functions, but there are many new features recently (e.g. : multiple platform builds).

The issue with the help file is that it's generated from some old LaTeX dialect, which in turn creates an old-school Windows help file (.chm files).
A few years ago I converted the .chm file to some Html-based online help, using a professional tool. That's great, but I don't think that's a good long-term solution. CHM files are a mess because that's starting to be a really old format (see the kind of trouble I had). Others tried other things along the years.
We need to find a solution.

[Crimson Wizard]

There is one suggested by Alberth: use Sphinx.
Here's his pull request: https://github.com/adventuregamestudio/ags/pull/173

My own wish was to have a single repository of help source, that could be autmatically (using some scripts) converted to multiple outputs: a Windows CHM, some Linux format, and HTML for web site.

I wanted to check that out but could not find time yet. If anyone does that, that would be wonderful.


EDIT: I found the original thread: http://www.adventuregamestudio.co.uk/forums/index.php?topic=50730.0

[abstauber]

I also did some research and Sphinx seems to be the only free alternative. All other tool are ridiculously expensive and/or not suitable for collaboration.
It seems like Alberth already did most of the work and converted everything

[Monsieur OUXX]

Really really cool.

[Crimson Wizard]

Monsieur OUXX, since you seem to be interested in this, can you take Sphinx files created by Alberth and try out what needs to be done to compile them as AGS help file?

[Snarky]

I'm a bit surprised that the scripting reference is entirely separate from the code (and presumably from whatever documentation is used to generate the tooltip help in the editor). Sphinx seems to have a plugin that allows it to pull in doxygen comments from sourcecode files, as long as they're in the right format. Would it make sense to move the api documentation to the engine source?

[abstauber]

In the meantime I compiled the .rst files from Alberth:
http://shatten.sonores.de/wp-content/uploads/2014/12/htmlhelp.7z

I had to install Python 2.7 and Sphinx following this guide:
http://sphinx-doc.org/install.html

After that it were just two steps:
1)make htmlhelp
2)hhc AGSdoc.hhp (inside the build/htmlhelp folder)

This build complains about a missing index file, but I just did a quick compilation test, so this shouldn't be a real issue later on.

[Alberth]

Glad to see new interest in my documentation patch

@Snarky:
It really depends what you want to achieve. Automagic merging of code documentation saves perhaps some typing. You also never miss any documentation.

On the other hand, controlling the order of content may be harder (alphabetical order is easy, keeping things topic-wise or release-wise together is more complicated). You also get the problem of versioning. Imagine a function f, that was added in 1.2, changed in 1.5, changed again in 2.4, and removed in 2.7. How would you handle this with code for say version 2.8? Python.org solves this by making a manual for each version separately, AGS currently has one manual for all versions.

If you keep all text together in some directory, it's relatively easy to find things. If you scatter documentation throughout the code, it becomes harder to find some piece of text, and a documentation change may cause conflicts in code changes and vice versa.

Last but not least there is also the matter of audience. Code documentation is usually not oriented at an end-user. It's brief, full of technical terms, just enough to know how to call the code from another method. An end-user typically needs a few pages explaining which functions exist, how function work together to achieve something in a game, and an example or two with explanation what it does.

@abstauber:
Thanks for trying. I don't have Windows, so I cannot build nor use the Windows help files.

[Snarky]

It really depends what you want to achieve. Automagic merging of code documentation saves perhaps some typing. You also never miss any documentation.

On the other hand, controlling the order of content may be harder (alphabetical order is easy, keeping things topic-wise or release-wise together is more complicated). You also get the problem of versioning. Imagine a function f, that was added in 1.2, changed in 1.5, changed again in 2.4, and removed in 2.7. How would you handle this with code for say version 2.8? Python.org solves this by making a manual for each version separately, AGS currently has one manual for all versions.

If you keep all text together in some directory, it's relatively easy to find things. If you scatter documentation throughout the code, it becomes harder to find some piece of text, and a documentation change may cause conflicts in code changes and vice versa.

Last but not least there is also the matter of audience. Code documentation is usually not oriented at an end-user. It's brief, full of technical terms, just enough to know how to call the code from another method. An end-user typically needs a few pages explaining which functions exist, how function work together to achieve something in a game, and an example or two with explanation what it does.

Well, the current AGS scripting help is sorted alphabetically (by class), and looks like it wouldn't be out of place in source comments, IMO. (But on that last point it's really the developers' opinions that count.) I assume it would go in either the .cpp or .h files here: https://github.com/adventuregamestudio/ags/tree/master/Engine/ac As you can see, there is no documentation in the source code currently, so there wouldn't be any conflict.

It's not correct that AGS currently has "one manual for all versions." Rather, it has a different version of the manual for each different version of AGS (distributed with the build), but only the latest version (at least in theory) is online.

Yes, it might be a drawback in some situations that different parts of the documentation would be split across hand-written articles and snippets extracted from the source code. But it might also have benefits: for example it should make it easier to ensure that the API and the documentation are in sync. I tend to think those benefits would outweigh the much less frequent (?) cases in which having to edit documentation both in the source code and in standalone help files adds a bit of overhead.

I would guess that the drawback is rather that it's a bit more complicated to set up (now we have to learn some basic Doxygen as well, set up that, etc.), and might take more work to convert the current documentation. And by putting reStructuredText comments throughout the code, we're more tied to Sphinx; it would be a little bit harder to switch to a different HAT in the future.

[Alberth]

Well, the current AGS scripting help is sorted alphabetically (by class), and looks like it wouldn't be out of place in source comments, IMO. (But on that last point it's really the developers' opinions that count.) I assume it would go in either the .cpp or .h files

In my experience with C++, near the actual implementation works best, as that gets changed when you change semantics of the call. It also helps in keeping the .h file less cluttered so it is easy to read.

As you can see, there is no documentation in the source code currently, so there wouldn't be any conflict.

I browsed through the code for half an hour to see if I could understand what belonged together but didn't get very far.

As for lack of comments, yeah, I noted. It's the same with almost every open source project. It's unfortunate that people (no offense intended to anyone, it's just an observation) don't take 1/2 a minute to write a few lines about the call interface (in terms that are easy to understand if you don't know the function) when they have it fresh in memory, so everybody now and in the future can read it in 10 seconds and have all relevant details on the first try, instead of having to reverse-engineer the interface in 30+ 1/2 minutes with the additional risks of missing some crucial implicit note or assumption which will surface as a bug 6 months later, which takes an amount of time that is not really usefully  countable in 1/2 minutes.

I started systematically documenting every function/method I wrote since about 8 years ago, and I found it increased my productivity. It takes a lot less mental effort to read 4 lines explaining how to call some function I wrote 3 years ago than it takes to reverse engineer that function without messing up. It makes the difference between "just call that function and get on with the problem you're working on" versus "oh my god, what did I do here again? There was something special here, I need more coffee.". I can also handle larger code bases with less effort now.

Besides the .rst patch, I haven't done anything in the AGS project yet. (Trying to read the manual failed already :p ) I do like the application domain, but I already write compilers/execution engines as day job. AGS doesn't run natively at my linux system, which makes it very hard to do anything useful. If I decide to do more code-oriented things, documentation will be high on my list in this project. My standard tactic in such a case is to start throwing documentation patches at the project (I don't want to reverse engineer anything more than once). However, if such patches are refused or existing devs see no reason to stop adding undocumented code or don't update existing documentation, I'll stop fairly quickly as well. No offense, but there is no point in me doing such work if new work gets created as we speak, so to say.

It's not correct that AGS currently has "one manual for all versions." Rather, it has a different version of the manual for each different version of AGS (distributed with the build), but only the latest version (at least in theory) is online.

Oh sorry, didn't know that. I normally never use builds, I tend to compile everything from source, as sooner or later I want to fix something annoying

by putting reStructuredText comments throughout the code, we're more tied to Sphinx; it would be a little bit harder to switch to a different HAT in the future.

Less than you're now tied to tex2rtf & friends. Unlike the current tex source, there is a free RST parser framework available in Python (docutils project iirc) that you can use if you want to. Writing a simple line-based text conversion script like I did, and pushing all the source code in a somewhat uniform format took a few days. As long as you stick to mostly plain text, it can be converted relatively easily.

[RickJ]

I started systematically documenting every function/method I wrote since about 8 years ago, and I found it increased my productivity. It takes a lot less mental effort to read 4 lines explaining how to call some function I wrote 3 years ago than it takes to reverse engineer that function without messing up.

I agree.  In addition, the exercise of explaining what one has done brings a tremendous amount of scrutiny and focus that tends to reveal inadequacies and outright bugs before they manifest and are identified in the usual time consuming manner.  This also means a tremendous boost to productivity is realized in the long run. 

Also I think Alberth has identified two seemingly conflicting needs with regards to the help file.  First there is a need to accurately document built-in functions and editor/engine features.  The best way to keep these things up to date is to maintain that documentation along with the code.  If both are in the same place then there is little excuse for them being out of sync.

The second need is for 'tutorial' or 'user' type documentation that illustrates how to use a collection of functions and features to accomplish some goal.  This kind of documentation is more easily maintained as a separate document.

I would like to add a third need here that should also be considered.  Third party modules and plug-ins extend AGS by adding additional functions/classes etc that also need documentation. 

It would be nice if all three needs could be satisfied with one coherent mechanism.  Wouldn't it be possible using RST/Sphinx/Python to come up with a modular help system that could be used for all three purposes?  For example  I was thinking along the lines of the help menu having multiple entries, one for each help module? 

[Snarky]

I think discussing this in the abstract is getting us off track. We're talking about the API documentation of the AGS scripting language. Here's the current documentation for Mouse.ChangeModeView() (to pick an example at random):

Changes the view used to animate the specified mouse cursor.

Mouse.ChangeModeView(CursorMode, int view)

Changes the specified mouse cursor mode's animation view to VIEW.

You can pass view as -1 to stop the cursor from animating.

This allows you to dynamically change the view used for the cursor's animation while the game is running.

Example:

mouse.ChangeModeView(eModeLookat, ROLLEYES);

will change the Look cursor's view to ROLLEYES.

See Also: Mouse.ChangeModeGraphic, Mouse.ChangeModeHotspot

How would you want to split this between "API documentation" and "user help/tutorial," and do you really think you're better off maintaining it in separate places? No, I think this should either all be in the source, or all in a separate document.

As for lack of comments, yeah, I noted. It's the same with almost every open source project. It's unfortunate that people (no offense intended to anyone, it's just an observation) don't take 1/2 a minute to write a few lines about the call interface (in terms that are easy to understand if you don't know the function) when they have it fresh in memory, so everybody now and in the future can read it in 10 seconds and have all relevant details on the first try, instead of having to reverse-engineer the interface in 30+ 1/2 minutes with the additional risks of missing some crucial implicit note or assumption which will surface as a bug 6 months later, which takes an amount of time that is not really usefully  countable in 1/2 minutes.

Well, AGS isn't really an open-source project at heart. It's a one-man project that was open-sourced after ten years of development. Sure, better source code commenting is probably a good idea moving forward, but that's really up to the devs, and very much aside from the question of how to best provide documentation for the users. Which documentation has to include the AGS scripting API.

Less than you're now tied to tex2rtf & friends. Unlike the current tex source, there is a free RST parser framework available in Python (docutils project iirc) that you can use if you want to. Writing a simple line-based text conversion script like I did, and pushing all the source code in a somewhat uniform format took a few days. As long as you stick to mostly plain text, it can be converted relatively easily.

That's a completely separate matter. The point is that if the documentation is all in one file/directory, it can fairly easily be converted to some other format. If it's generated from various snippets throughout (one part of) the source code, it gets much more complicated to do so. Going through it by hand would be tedious and error-prone, while trying to write a script to convert RST text embedded in Doxygen format embedded in C++ comments to some other format does not sound like fun, or particularly easy.

I would like to add a third need here that should also be considered.  Third party modules and plug-ins extend AGS by adding additional functions/classes etc that also need documentation. 

It would be nice if all three needs could be satisfied with one coherent mechanism.  Wouldn't it be possible using RST/Sphinx/Python to come up with a modular help system that could be used for all three purposes?  For example  I was thinking along the lines of the help menu having multiple entries, one for each help module?

That would require the help system to generate the help files on the fly depending on which modules/plug-ins were included. Also, AGS is written in C++ while modules are written in AGS script and plugins in whatever language the developer chooses.

In general, this is not how software works. Photoshop help doesn't try to integrate help for any PS plugins I may have installed. Nor does Firefox offer integrated documentation for each of my extensions. All of that is up to the developers of the plugins/modules/extensions. Of course, in AGS there's nothing stopping them from providing a Sphinx help file, though I think most module devs would consider it overkill.

[RickJ]

How would you want to split this between "API documentation" and "user help/tutorial," and do you really think you're better off maintaining it in separate places? No, I think this should either all be in the source, or all in a separate document.

I wouldn't advocate splitting this up at all.  To build upon your example, what could go into a separate document would be a tutorial and/or example using all or some number of Mouse methods to implement a mouse interface for one kind of game or another. It's not the same as API documentation.

That would require the help system to generate the help files on the fly depending on which modules/plug-ins were included.

The module/plugin author would generate the help file as part of their development effort.  When the module/ plugin is installed the help file would just be p[art of the installation.  The game file could contain the help menu entries and filename bindings or the editor could just create the menu entries "on the fly" by doing a help directory list when the editor is first loaded.

Also, AGS is written in C++ while modules are written in AGS script and plugins in whatever language the developer chooses.

Both use remarkably similar syntax; same comment characters, line terminators, etc.  The RST would be the same in either case, embedded in comments.  One wonders how difficult it would be to get Sphinx to read module files? A while ago I made a module document extractor tool using AGS itself.  However, AGS's file writing limitations prevent it from being production ready.  It wasn't a trivial effort but it wasn't a monumental task either. 

[Snarky]

I wouldn't advocate splitting this up at all.  To build upon your example, what could go into a separate document would be a tutorial and/or example using all or some number of Mouse methods to implement a mouse interface for one kind of game or another. It's not the same as API documentation.

I refer you to my first post:

I'm a bit surprised that the scripting reference is entirely separate from the code (and presumably from whatever documentation is used to generate the tooltip help in the editor). Sphinx seems to have a plugin that allows it to pull in doxygen comments from sourcecode files, as long as they're in the right format. Would it make sense to move the api documentation to the engine source?

The discussion is about the best way to maintain the AGS scripting API reference as it exists in the current manual, assuming we move to Sphinx as the HAT. No one has suggested moving the general manual and tutorial articles (either the ones that are already part of the manual or purely imaginary ones) into source code comments; that would be absurd.

The module/plugin author would generate the help file as part of their development effort.  When the module/ plugin is installed the help file would just be p[art of the installation.  The game file could contain the help menu entries and filename bindings or the editor could just create the menu entries "on the fly" by doing a help directory list when the editor is first loaded.

Module authors can already write a (separate) help file in RST, generate docs from it with Sphinx and distribute it with their module. The only benefit you might realize with this suggestion is the ability to launch the help from within AGS. Personally, I think a great benefit of current modules is that they're very simple to make and distribute, and that the file format is simply a plain text file (so I can, for example, examine the code without opening AGS, particularly while I'm on a Mac). You'd lose that simplicity. Also keep in mind that Sphinx just batch-generates documentation. The output (whether PDF, HTML/CHM, LaTeX or whatever) are "dumb" files which won't magically integrate with each other: AGS would need to somehow modify them on the fly, which is much more involved than simply launching the CHM viewer like it currently does.

Both use remarkably similar syntax; same comment characters, line terminators, etc.  The RST would be the same in either case, embedded in comments.  One wonders how difficult it would be to get Sphinx to read module files? A while ago I made a module document extractor tool using AGS itself.  However, AGS's file writing limitations prevent it from being production ready.  It wasn't a trivial effort but it wasn't a monumental task either.

In order to read RST embedded in source code comments, Sphinx uses plugins. For whatever reasons it has different plugins for C++ and Java, so it's not hard to imagine it would need a separate plugin for AGS Script. (I suspect the reasons are more to do with standard commenting practices in each language, e.g. javadoc vs. doxygen, but unless we want to force ALL module authors to write ALL their comments in RST, the problem could still persist.)

Ultimately, I think the current module documentation works pretty well, and there's no need to try to integrate it with AGS documentation, which would just place a higher burden on module authors. Now they would have to learn RST, install and run Sphinx to generate documentation, distribute more files with the module, test the integration of the help file with AGS, etc. All for what? Looking at some of the top modules on the forum, here's what users need to know in order to use the A.S.S Savegame module, for example:

// !MAKE SURE "save screenshots in savegames" IS ENABLED UNDER General Settings IN AGS!
// And don't forget to import gSave.guf and gLoad.guf; this script will not work without them.

// When you want to open the save GUI, all you have to do is call: sl_save();
// And to open the load gui - you guessed it: sl_load();

Would you really gain much from having that in the manual rather than at the top of the header file (where modules are conventionally documented)?

[abstauber]

Maybe we should split this into separate threads: decision and discussion
Otherwise I guess we might never get started - and the help file situation is really not the best.

To sum things up, there's just one usable Help Authoring Tool: it's Sphinx.
Moving over to Sphinx isn't hard, as Alberth already provided a conversion tool and did the conversion.
At the moment the AGS help file is huge and rewriting it from scratch would be a tremendous task. It will need to be split up, but it can wait.
Moving all the documentation into the actual source code (my opinion) would make things even worse and waste countless hours, although I'd like to hear CW's and Guroks take on this.

I've just converted the documentation of the 9Verb-template to Sphinx:
http://shatten.sonores.de/wp-content/uploads/2014/12/9Verbs_sphinx_doc.zip

After having that done I'd like to underline that a rewrite of the AGS help file will take ages. Also I highly doubt that module authors are willing to learn how to write .rst files But for the included templates, we should provide the documentation somehow.

@CW: Since you are building the current chm files, what do you think about moving from the .tex file to .rst ?

[Snarky]

Maybe we should split this into separate threads: decision and discussion
Otherwise I guess we might never get started - and the help file situation is really not the best.

Yes, sorry I brought it up. I thought the decision part was a no-brainer, if as you say:

To sum things up, there's just one usable Help Authoring Tool: it's Sphinx.
Moving over to Sphinx isn't hard, as Alberth already provided a conversion tool and did the conversion.

In fact, it sounded like it was more or less done, and just a matter of fixing some small issue and then committing it to the repo (or whatever the git equivalent). I see now from the pull request and sample HTML output that there are still some rough edges.

At the moment the AGS help file is huge and rewriting it from scratch would be a tremendous task. It will need to be split up, but it can wait.

We wouldn't want to rewrite it from scratch anyway, would we? I think some parts could be better, but it's very much a matter of incremental improvements.

Moving all the documentation into the actual source code (my opinion) would make things even worse and waste countless hours, although I'd like to hear CW's and Guroks take on this.

Whether or not it's better in principle, I'm less and less convinced myself that the benefits outweigh the hassle.

[Monsieur OUXX]

Monsieur OUXX, since you seem to be interested in this, can you take Sphinx files created by Alberth and try out what needs to be done to compile them as AGS help file?

Sorry, I didn't see that. Yes, I definitely should do that.

- Yes, I think .tex should be converted to .rst
- I don't think the documentation should be in the code. I thought it was better in the past, but I realized that it's better to keep the doc easily accessible to people who don't mess with the code. They can do their thing quietly in their corner, without worrying about pulling/committing github code and such and browsing through the Visual Studio solution. All they need is one file and a text editor.

[GarageGothic]

Possibly the wrong thread, but I've been unable to read any of the actual articles in the help file after upgrading to Windows 10 - I can see the index and contents fine, but no articles show up when I click them. Anyone else experiencing this, and do you have a workaround?

Edit: Tried a different chm viewer app called UltraCHM and I get the same result as with the MS viewer.

[Crimson Wizard]

Yes, this is wrong thread, here we discussed different format for manual...


For you problem, try to use this solution: http://www.adventuregamestudio.co.uk/forums/index.php?topic=52398.msg636517886#msg636517886

[GarageGothic]

Thanx Crimson, worked great.