Help file... here we go again.

Started by Monsieur OUXX, Sat 29/11/2014 01:38:02

Previous topic - Next topic

Crimson Wizard

#20
It's sad that this endeavour just ended with nothing. I can make a guess that people were expecting me to do something, but I cannot do everything, and I had other priorities at a time. I actually hoped there would be a community work in this.

My natural question: is this actually still needed? The fact that no one tried to improve situation in about 1.5 years since the last talk looks like either everyone is waiting for someone to start, or no one really cares about this anymore(?).
I'd like to know following:
- is moving to another help system still wanted?
- am I expected to do something about it? Because, frankly, most of the related tasks could be done by anyone, not just me.


My opinion all this time was that manual is simply lacking material. For example, there is almost no structured information on how Editor works (for example, in the form of per-task or per-window reference).
Speaking honestly, is there someone who will be ready to work on such articles? Because it was possible all this time to add them even without changing manual format.



Splitting the overall problem, there are following issues that may in theory be solved separately:

1. Lacking documentation and/or bad manual structure (articles organized in a way that makes it hard to find what you want). For example, Editor is never properly explained, sans the Tutorial, but Tutorial mentions only few things you need to know to make your first demo game.
How to solve? Someone should simply write the text. That's it.

2. Bad source format. The current source format of the manual is an old version of LaTeX, and all manual is written as one huge file. Also, personally I could not find a satisfactory (that would be at least intuitive) tool to edit the source, so I myself do that in a common text editor.
How to solve? Look around for better format (some were already suggested in the past: Sphynx, etc). Tell us how source is edited, are there good utilities for that (provide examples). After the format is decided, make a plan for converting current manual: design guidelines (consistent presentation, styles), write automatic conversion script (latter is done by savvy person).

3. Slightly (?) outdated end format. AGS Editor is distributed with the manual as a CHM file. CHM is outdated, and sometimes does not run properly on latest versions of Windows. For the least we may try to upgrade it to WinHelp 2.0. Since AFAIK both are created from HTML pages, it should not matter which source format was chosen, there will always be a way to compile manual source into what we want (source -> HTML -> CHM/WinHelp2/other).
After end format is confirmed, Editor should be "taught" to work with that format (to keep context help working). Latter will be done by programmers, so don't worry about it.

4. Manual is "hidden" from editing by most community members. What I mean is that its source is located inside the AGS code repository, which may be psychologically complicating (?). Besides, you need to know and use Git to commit any changes there. Is there a way to make editing process easy enough for any regular community member to add/fix material, like they may do with Wiki?

5. Online manual(s) are out of date. There are two online manuals on AGS server, one is this: http://www.adventuregamestudio.co.uk/manual/ , and another is in the Wiki: http://www.adventuregamestudio.co.uk/wiki/Contents_(manual)
First was updated 2 major versions ago (5 years old), second - 3 major versions (7 years old!).
Question one: is there really a need for the one in the Wiki? No one is updating it anyway.
Question two: is there a way to automate primary online manual update? I guess the answer mainly depends on how the manual editing itself is organized (points 2 and 4).


This is to outline the potential tasks.


To be honest, this is probably my last attempt to initiate this work. Seeing no one is interested makes me indifferent myself, and if such activity is not wanted, I will just focus on limited range of tasks I do and never return to this myself.

sonneveld

I can help.  I think it might be a good idea to just throw away all formats and host the documentation on https://readthedocs.org/ .  It automatically builds docs for restructuredtext/markdown. This would mean resurrecting the doc pull request that converts all documentation to rST.

It would be worth finding out if people need documentation. Something that's google-able would be helpful for newcomers.

Crimson Wizard

#22
Nick, this reply of yours is pretty confusing.

Quote from: sonneveld on Sun 03/07/2016 10:50:47
I think it might be a good idea to just throw away all formats and host the documentation on https://readthedocs.org/
<...>
It would be worth finding out if people need documentation.
Do you mean -  throw away help file (offline documentation)? Basically, you suggest to convert manual to RST, and then use it to create online documentation?

I would not like to disband offline docs. Imagine a person goes somewhere with no internet connection (or bad connection), and want to continue working on game.
Unless that site allows to download docs (in which form?)...

selmiak

#23
I have an idea. tell me what you think.

we make a new separate wiki only for the manual on the ags server. everyone with an forum account can edit it, just like the normal AGS wiki. This new wiki is only for the manual, the manual wiki so to say. Or the wiki manual, whatever you prefer and gets used the most I think.
Then with some code magic we strap the complete content, the topics of the articles and the links and convert this content from the wiki to some (html) format the helpdisplay in ags and windows can display. grab what is on that manual wiki and make ags help display this.
At the beginning it will be empty like every wiki so we should first fill it. we should gather some 5-10 or even more people to at first transfer the current manual to the wiki. everyone get one letter, check this letter, then the next one, the more people help converting the old manual content to wiki format the faster this will happen.
Then we can add new topics for new functions over time and it will always be up to date in the wiki manual already. We could even have some cronjob on the server to gather the new wiki pages and pack it to the ags distribution files. I think it is close to impossible to add some malicious code to wiki markup that will break out from the helpdisplay sandbox and screw with users computers.

almost everyone can write wiki markup and if not, it is very easy to pick up, tech savvy people like the forum users that want to help write the technical manual will get into it in no time. The easier the pages are to edit to more people can help and the faster things get done in that scale.

so my conclusion for this idea is to find some code wizard that can or thinks he can convert some wiki markup to some helpfile display content. he or she shall chose the wiki software and the help file display software and get coding. once this code works we only need to fill the wiki and voila.

Crimson Wizard

Quote from: selmiak on Tue 05/07/2016 23:49:21
so my conclusion for this idea is to find some code wizard that can or thinks he can convert some wiki markup to some helpfile display content. he or she shall chose the wiki software and the help file display software and get coding.

There is certainly software that does this. Recently I found one: http://pandoc.org/

Gurok

Quote from: Crimson Wizard on Sun 03/07/2016 23:00:26
I would not like to disband offline docs. Imagine a person goes somewhere with no internet connection (or bad connection), and want to continue working on game.
Unless that site allows to download docs (in which form?)...

Agree completely. This might also be an intentional thing, e.g. Someone might go somewhere remote to get away from the Internet and write their game. Actually, that sounds like a rather nice idea for a holiday, come to think of it.

Quote from: Crimson Wizard on Wed 06/07/2016 02:21:34
Quote from: selmiak on Tue 05/07/2016 23:49:21
so my conclusion for this idea is to find some code wizard that can or thinks he can convert some wiki markup to some helpfile display content. he or she shall chose the wiki software and the help file display software and get coding.

There is certainly software that does this. Recently I found one: http://pandoc.org/

I saw you mention this on GitHub too and I just want to clarify.

Was your suggestion to use Pandoc to go from some Wiki->rST and then Sphinx2 (as currently nominated on GitHub) to go from rST->CHM?

It's unfortunate that CHM output doesn't seem to be a priority for the Pandoc people:
https://github.com/jgm/pandoc/issues/2056
[img]http://7d4iqnx.gif;rWRLUuw.gi

Crimson Wizard

#26
Quote from: Gurok on Wed 06/07/2016 02:36:54This might also be an intentional thing, e.g. Someone might go somewhere remote to get away from the Internet and write their game. Actually, that sounds like a rather nice idea for a holiday, come to think of it.
Haha, my thoughts exactly.
Quote from: Gurok on Wed 06/07/2016 02:36:54
I saw you mention this on GitHub too and I just want to clarify.

Was your suggestion to use Pandoc to go from some Wiki->rST and then Sphinx2 (as currently nominated on GitHub) to go from rST->CHM?

It's unfortunate that CHM output doesn't seem to be a priority for the Pandoc people:
https://github.com/jgm/pandoc/issues/2056

No, my suggestion on github was to use Pandoc to convert LaTeX to rST (both are source formats).

We apparently do not need to use Pandoc to get CHM from rST: http://www.adventuregamestudio.co.uk/forums/index.php?topic=51340.msg636501803#msg636501803
Sphynx builds HTML Help project from rST, and Microsoft's own "hhc" utility compiles CHM.
If Pandoc could convert to HTML Help too, it could be used in the same process (although that is not the main point).

Also, we were wanting to get rid of CHM altogether. It is just that we may have to keep it for some time (if solving the issues step by step).

The remaining question is just which (offline) help viewer to use instead of CHM.

sonneveld

Pandoc's a document converter but it wouldn't be the final step. You'd still need a document tool like Sphinx to create offline documentation for chm.

I'm not sure the wiki is a great idea. You still want to have documentation tied to your particular branch in the AGS repository.  How would the wiki treat documentation that would be different for stable and development branches?

However there is something to be said for making it easier to contribute.

Picking a different markup language for the manual would be a good first step. Despite RestructuredText possibly being better suited for documentation, I think a lot of major projects are using Markdown for documentation because more people are familiar with it and that lowers the barrier for people to contribute. Even Microsoft has started using it for some of their github backed docs.

Github allows people to fork repository, edit files and create pull requests all within the browser. That is almost a wiki-like experience but with better change controls.

Nick.

Alberth

I would suggest to first define what you want from documentation, then consider the format and place of the source, then consider how to get there from what you have now.

"what you want" would include
- online/offline (both I think)
- purpose of the documentation (what does it contain, what can you learn from it)
- up-to-date-ness of the documentation (how bad is it if it runs behind a year or more)
- users of the documentation
- editors of the documentation (who for what parts mostly)
- try to come up with more properties that you desire from the documentation

Crimson Wizard

#29
Ok... I will give this another try.

Quote from: Alberth on Sat 23/07/2016 07:27:43
I would suggest to first define what you want from documentation, then consider the format and place of the source, then consider how to get there from what you have now.

Quote
- online/offline (both I think)
Yes, both, because it would be more comfortable to know that people can use documentation even without internet connection. Of course format could be same (like, you can view html pages from your HDD too).

Quote
- purpose of the documentation (what does it contain, what can you learn from it)
Priority is given to end-user documentation for AGS Editor and engine.
For AGS Editor, ideally, documentation should include sufficient information for a person who first met AGS to create and deploy games in it: feature overviews, explanation of workflow, detailed each window description, basic tutorials, and of course script API reference.
For the engine - at least the basic information on its work from the game-dev and player's perspective: (games) installation requirements, paths and files it uses, manual configuration, command-line arguments.

Quote
- up-to-date-ness of the documentation (how bad is it if it runs behind a year or more)
It should be up-to-date with the latest official final (non-beta) release.

Quote
- users of the documentation
For AGS Editor documentation: game developers,
For engine end-user documentation: game developers and players (because the former should be able to reference something for the latter).

Quote
- editors of the documentation (who for what parts mostly)
Program developers must be able to update documentation when new release is out (script API, features).
Any other volunteer should be given a chance to contribute anytime whenever they think some articles needs improvement of corrections.
In simple case, program developer writes basic articles, and contributors help to fill in the gaps. (Not every programmer can write good tutorial)

Quote
- try to come up with more properties that you desire from the documentation
Offline docs should be browseable on most platforms (not something Windows-only).
Index and Search functionality. Capability to display images. Basic formatting capabilities (tables, lists, etc).


Other points.
There should be single source (repository) shared for the online and offline documentation.
Offline docs may be same format as online one (if applicable), just installed on disk (e.g. html files).
Ideally an automation should be set up: online documentation is updated as soon as source is modified, and offline version is available for download from the host location. Respective version of offline docs are also included with the AGS Editor installer.

How to upgrade from old manual
Setting up simple format conversion, then tidying up manually.

What about organizing documentation parties: announce a "week of improving documentation" - gathering volunteers, then giving them articles to improve.

eri0o

Hey,

I used to use pandoc and sphinx, I was passing on the AGS topics and this is the first thing I thought "I know how to automate this!"

I have a project here (it's slightly abandoned) that has a help file built from Markdown source: https://github.com/ericoporto/fgmk/tree/master/docs

I use pandoc to go from .md to .rst and sphinx to build my help files (QtHelp and html).

You can take a look at the generated html help here: http://ericoporto.github.io/fgmk/

Can someone point me the source files?

Crimson Wizard

#31
Quote from: eri0o on Fri 21/07/2017 23:34:23
Can someone point me the source files?

https://github.com/adventuregamestudio/ags/tree/master/Manual


EDIT: if you run "run.cmd" (you also need to have Python 2.7 installed), the HTML files are created as an intermediate output. These may be easier to use than source TEX file.

morganw

Since the CHM format was reverse engineered you can also get the HTML files from the CHM file, by extracting it with 7-zip.

Crimson Wizard

#33
We really need to do something with this. Any kind of progress would be better than nothing.
Many parts of manual are getting more obsolete with every year, and there are new topics that should be covered.


Quote from: eri0o on Fri 21/07/2017 23:34:23
I used to use pandoc and sphinx, I was passing on the AGS topics and this is the first thing I thought "I know how to automate this!"

I have a project here (it's slightly abandoned) that has a help file built from Markdown source: https://github.com/ericoporto/fgmk/tree/master/docs

I use pandoc to go from .md to .rst and sphinx to build my help files (QtHelp and html).

You can take a look at the generated html help here: http://ericoporto.github.io/fgmk/

Alright then, the resulting html pages look good in general. Some time ago someone told me that nowadays Markdown is becoming popular and may be a good choice.

So, why not start doing something?

The plan:
1) First of all, let's create a new repository for the manual. That would be cleaner, and also easier to give access to various people without worrying they will break engine code by mistake.
2) Let's make several Markdown pages, only to test its abilities. Copy few manual topics, or some topics from Wiki. Then add pandoc/sphynx scripts as in eri0o's example, and try making html documentation and see how it looks and works.
3) If everything works good, decide on how to convert existing manual. We could automate and write scripts, or use one of the tools like pandoc (I recall finding some method to make such convertion before, need to review earlier posts in this thread). Of course, we'd have to fix style manually afterwards.
4) If all above works - move forward. We'd need to invent how the offline manual will work from now on. Could use some converter to create Windows Help file from the markdown/rst sources with sphynx, or even go Unity3D style and distribute html pages.

The new manual will be open for all people to edit (with pull requests, or directly), it may host both official documentation, and tutorials written by users, and articles from Wiki, etc.

eri0o

#34
Ok, I have some ideas on this... First, a way to get the manual from the repo...

update_manual_files.sh
Spoiler
Code: bash

#!/bin/bash

release_branch="release-3.4.1"
function git_sparse_clone() (
  rurl="$1" rbranch="$2" localdir="$3" && shift 3

  mkdir -p "$localdir"
  cd "$localdir"

  git init
  git remote add -f origin "$rurl"

  git config core.sparseCheckout true

  # Loops over remaining args
  for i; do
    echo "$i" >> .git/info/sparse-checkout
  done

  git pull --depth=1 origin "$rbranch"
)

git_sparse_clone "https://github.com/adventuregamestudio/ags.git" "$release_branch" "$release_branch" "Manual/*"

cd "$release_branch"/Manual/
[close]

then the ags.tex has to be converted to Markdown. Pandoc can help there, but it will fail on lines 2845 ... 2851:

Code: latex

...
}\row{{ \verb$%d$ } & { Integer (use to display value of int and short variables) }
}\row{{ \verb$%0Xd$ } & { Integer left-padded with up to X zeros }
}\row{{ \verb$%s$ } & { String (use to display string variables) }
}\row{{ \verb$%c$ } & { Character (displays the ASCII character of the supplied value) }
}\row{{ \verb$%f$ } & { Float (displays a float variable) }
}\row{{ \verb$%.Xf$ } & { Float to X decimal places }
}\row{{ \verb$%%$ } & { Display the percent character (ie. no variable) }
...


I can't deal with percent between a pair of money sign ($), so each percent on those lines has to be escaped.

Once that is done, you can convert.

pandoc -S ags.tex -o ags.md

Some things will break. These are some weird non latex marking used in the AGS documents: LTSSimg , GTSS, ILBRK. This will result in no image links. Once you get to the markdown part, though, that's where I stopped last time... here in this github repo... Note that since then, Sphinx has started to support Markdown, so the whole Md to Rst conversion can now be skipped.

Meanwhile, recently I have been back into using LaTeX through Overleaf, they recently acquired Sharelatex. The problem is the free account is very limited - only 60 files, the ags docs is comprised of 64 files in my best reduction...


Crimson Wizard

Quote from: eri0o on Fri 06/07/2018 13:40:33

pandoc -S ags.tex -o ags.md

Some things will break. These are some weird non latex marking used in the AGS documents: LTSSimg , GTSS, ILBRK. This will result in no image links. Once you get to the markdown part, though, that's where I stopped last time... here in this github repo... Note that since then, Sphinx has started to support Markdown, so the whole Md to Rst conversion can now be skipped.


Mmmm.... this conversion results in 1 huge file? Is there a way to split it?

Will that make things easier if you do not use original TEX file, but HTML pages prepared by the script in AGS repository instead? It already removes ILBRK, GTSS and LTSS tags on its own.

eri0o

#36
Which conversion? I haven't got to the part of using Sphinx yet... I am still at step 1!

Quote
1) First of all, let's create a new repository for the manual. That would be cleaner, and also easier to give access to various people without worrying they will break engine code by mistake.
2) Let's make several Markdown pages, only to test its abilities. Copy few manual topics, or some topics from Wiki. Then add pandoc/sphynx scripts as in eri0o's example, and try making html documentation and see how it looks and works.
3) If everything works good, decide on how to convert existing manual. We could automate and write scripts, or use one of the tools like pandoc (I recall finding some method to make such convertion before, need to review earlier posts in this thread). Of course, we'd have to fix style manually afterwards.
4) If all above works - move forward. We'd need to invent how the offline manual will work from now on. Could use some converter to create Windows Help file from the markdown/rst sources with sphynx, or even go Unity3D style and distribute html pages.

What I am thinking, is there a way to automate step 1 and 2 ? I don't know.

So I recently learned there is no Markdown specification, meaning one must be adopted...

Also, this is a better way to generate headers (atx headers are defined by # character preceding the header text instead of a line of ===== or ----- under the header text)
Code: bash

pandoc -f latex ags.tex --atx-headers --to=markdown -o ags.md


Now with this, I am thinking on how to split the single markdown in multiple files... I have to say that at the end, being able to ctrl+f through the manual is useful.

About batch converting the .htm files, you are talking about Manual/htmlfiles folder in the repo?

convert_htm_to_md.sh
Code: bash

#!/bin/bash
find . -name \*.htm -type f -exec pandoc -o {}.md {} \;


These html files are a bunch of td/tr html tables, so the output is useless...

Oh right, about Sphinx. Why did I talked about it... It's the only thing I know that can generate the Windows help file. :(

Crimson Wizard

#37
Quote from: eri0o on Fri 06/07/2018 14:52:17
Which conversion? I haven't got to the part of using Sphinx yet... I am still at step 1!
I mean from Latex to MD, if MD is supposed to be the new source(?).

Quote from: eri0o on Fri 06/07/2018 14:52:17
What I am thinking, is there a way to automate step 1 and 2 ? I don't know.
I meant simple manual (as in "by hand") test just to see how it works. Writing that plan I assumed that you don't want to participate, since you left this topic one year ago without saying anything, so I thought you are no longer interested. Therefore we'd have to learn how to work with conversion from point zero.

Quote from: eri0o on Fri 06/07/2018 14:52:17I have to say that at the end, being able to ctrl+f through the manual is useful.
But that's crazy... how will people read and edit that monster?

Quote from: eri0o on Fri 06/07/2018 14:52:17
About batch converting the .htm files, you are talking about Manual/htmlfiles folder in the repo?
No, I am talking about html files generated with the scripts in Manual folder. Run run.bat on Windows and it converts tex into 100 htmls.
This is an intermediate step between TEX and actual CHM.

I could upload them somewhere in case you do not have access to Windows -

https://www.dropbox.com/s/snrrhug5ugrc52l/ags-manual-htmlpages.zip?dl=0

If converting these to MD is possible, these could be better start than 1 giant TEX file.


But once again, is MD a good format? I don't know, this is something I was planning to find out.

eri0o

#38
Oh, I was not talking as editing a single file, just having a possible visualization option at the end...

Ok, so there is a compile_documentation_unix.sh, but it only builds the tutorial. But I liked three lines...

Code: bash

sed -i 's/ILBRK/<br>/g' *.htm
sed -i 's/GTSS/>/g' *.htm
sed -i 's/LTSS/</g' *.htm


This is what is doable!

While I understand the md would be the new source, I am too lazy to generate by hand - it's over 22k lines!

Anyway, from the html files you upload to dropbox, I generated this!

My command was the naive bash+pandoc: find . -name \*.htm -type f -exec pandoc -o {}.md {} \;

Edit: Just noticed one has to place the images directory on there to be able to see images in a markdown previewer.
Edit2: All links are broken since they link to an htm instead of md file, but this can be replaced using sed.
Edit3:  find . -type f -iname "*.md" -exec sed -i 's/\.htm/\.htm.md/g' {} \;
Edit4:  very quick very bad conversion repo here.
Edit5: github wiki test. All links are broken except for the right side pane..
Edit6: Fixed links with :  find . -type f -iname "*.md" -exec sed -i 's/\.md//g' {} \;
Edit7: quick example of sphinx output for the markdown docs

morganw

Quote from: Crimson Wizard on Fri 06/07/2018 15:08:04
But once again, is MD a good format? I don't know, this is something I was planning to find out.
It is the easiest option I know of to write the pages, and if someone is making changes they would be able to do it directly on the GitHub web interface, so I think it is a good idea. I believe this is the original spec, and GitHub just add a few extra bits like tables and task lists. Worst case, if it needs to come off GitHub, any GitHub specific content would need to be changed, but I think the online editing is enough of a benefit to outweigh the risk (plus there are lots of tools for bulk processing text).

To begin with, is there anything that can just be deleted? There is still a load of stuff in there about licensing which no longer applies (I've never heard of a 'swapware license'). I am currently trying to make a replacement verb coin template, but even if that doesn't end up getting added I'm not sure why the existing verb coin template has manual entries, whilst the templates that people are likely to actually use have nothing.

SMF spam blocked by CleanTalk