Topic: MODULE: (Unit Tests) tap 0.1.0

[eri0o]

tap version 0.1.0


Get Latest Release tap.scm | GitHub Repo | Demo Windows | Demo Linux | Download project .zip

Test Anything Protocol - AGS Module

This Module implements Test Anything Protocol for Adventure Game Studio. It's useful for unit testing! You can read more about it here: testanything.org

An example test file output by this libary may look like:

Code: Bash

1. 1..4
2. ok 1 - Input file opened
3. not ok 2 - First line of the input valid
4. ok 3 - Read the rest of the file
5. not ok 4 - Summarized correctly # TODO Not written yet

The demo game will execute and exit, leaving a agstest.log file, with the test results, under this game savegame directory (ex: C:\Users\%USERNAME%\SAVEDG~1\tap_demo). This module is more useful to other people building modules and plugins, who wish to unit test in AGS Engine itself.

Motivation
A long time ago, Morgan talked to me about tests and unit testing, and showed me this magical thing that is TAP. Time passed and after experimenting with a bunch of different continuous integration systems I discovered they were able to magically pickup the specific output from tap and understand them when they are output to the CI shell... A bit of fiddling around later I got it working with AGS! This has been laying around in my repository for quite sometime and I noticed I never released it as a module. So here it is!

The plan
The plan tells how many tests will be run, or how many tests have run.  It’s a check that the test file hasn’t stopped prematurely. It must appear  once, whether at the beginning or end of the output. The plan is usually the  first line of TAP output (although in future there may be a version line  before it) and it specifies how many test points are to follow.  For example,

Code: Bash

1. 1..10

means you plan on running 10 tests. This is a safeguard in case your test  file dies silently in the middle of its run. The plan is optional but if  there is a plan before the test points it must be the first non-diagnostic  line output by the test file. In certain instances a test file may not know  how many test points it will ultimately be running. In this case the plan can  be the last non-diagnostic line in the output. The plan cannot appear in the  middle of the output, nor can it appear more than once.

The test line
The core of TAP is the test line. A test file prints one test line per test  point executed. There must be at least one test line in TAP output. Each test  line comprises the following elements:

Code: Bash

1. ok or not ok

This tells whether the test point passed or failed. It must be at the  beginning of the line. /^not ok/ indicates a failed test point. /^ok/ is a  successful test point. This is the only mandatory part of the line. Note that unlike the Directives below, ok and not ok are case-sensitive.

Test number
TAP expects the ok or not ok to be followed by a test point number.

Description
Any text after the test number but before a # is the description of the test  point.

Code: Bash

1. ok 42 this is the description of the test

Directive
The test point may include a directive, after a hash on the test line.  There are currently two directives allowed: TODO and SKIP. Hashes at the  beginning of a line are considered comments.

Executing tap with AGS

Before running any test, remove any leftover files and from a previous run, by running a clean test.

Code: Adventure Game Studio

1. tap.clean_test()

By default, test is printed to a file named agstest.log on the saved games directory for the game under test.

If you have a plan, state it before starting out.

Code: Adventure Game Studio

1. tap.plan(3);

Then run the needed tests.

Code: Adventure Game Studio

1. String null_string;
2. tap.ok(String.IsNullOrEmpty(null_string), "String.IsNullOrEmpty test null string");
3. tap.ok(String.IsNullOrEmpty(""), "String.IsNullOrEmpty test empty string");
4. tap.ok(!String.IsNullOrEmpty("with text"), "String.IsNullOrEmpty test non-empty string");

After you are done testing, state it.

Code: Adventure Game Studio

1. tap.done_testing();

TAP AGS Script API

Spoiler: ShowHide

tap.clean_test

Code: Adventure Game Studio

1. void tap.clean_test()

This command removes previously outputed test log files if they exist.

tap.plan

Code: Adventure Game Studio

1. void tap.plan(int n,  String skip_reason = 0)

This allows setting a plan, so we know how many tests will be run. Just use it once for the group of tests following. If you want to start a new test, after a plan has been set, remember to call tap.done_testing() before.

Alternatively, you can pass NO_PLAN if you have no plan.

If the following tests are to be skipped, pass [SKIP_ALL, so all tests until next plan are skipped. They will still be executed, they just won't output log  or be considered whether they fail or succeed.

tap.ok

Code: Adventure Game Studio

1. void tap.ok(bool condition, String test_name = 0)

This test will succeed if the condition is true, and fail if the condition is  false.

tap.nok

Code: Adventure Game Studio

1. void tap.nok(bool condition, String test_name = 0)

This test will succeed if the condition is false, and fail if the condition is  true.

tap.is

Code: Adventure Game Studio

1. void tap.is(String got,  String expected, String test_name = 0)

This test will succeed if got and expected matches. If this test fails, we can have some more diagnostic information since we know both inputs and the premise they are supposed to match.

tap.isnt

Code: Adventure Game Studio

1. void tap.isnt(String got,  String expected, String test_name = 0)

This test will succeed if got and expected does not matches. If this test  fails, we can have some more diagnostic information since we know both inputs  and the premise they are not supposed to match.

tap.is_int

Code: Adventure Game Studio

1. void tap.is_int(int got,  int expected, String test_name = 0)

This test will succeed if got and expected matches. If this test fails, we can have some more diagnostic information since we know both inputs and the premise they are supposed to match. This is for int values.

tap.isnt_int

Code: Adventure Game Studio

1. void tap.isnt_int(int got,  int expected, String test_name = 0)

This test will succeed if got and expected does not matches. If this test  fails, we can have some more diagnostic information since we know both inputs  and the premise they are not supposed to match. This is for int values.

tap.is_float

Code: Adventure Game Studio

1. void tap.is_float(float got,  float expected, float epsilon, String test_name = 0)

This test will succeed if got and expected matches. If this test fails, we can have some more diagnostic information since we know both inputs and the premise they are supposed to match. This is for float values.

tap.isnt_float

Code: Adventure Game Studio

1. void tap.isnt_float(float got,  float expected, float epsilon, String test_name = 0)

This test will succeed if got and expected does not matches. If this test  fails, we can have some more diagnostic information since we know both inputs  and the premise they are not supposed to match. This is for float values.

tap.done_testing

Code: Adventure Game Studio

1. void tap.done_testing()

After you are done testing, before calling the next plan, state you are done testing.

tap.Harness

Code: Adventure Game Studio

1. void tap.Harness()

This outputs a small summary of currently tests and failures. Don't depend on this output format.

Author
Made by eri0o

License
Distributed under MIT license. See  LICENSE for more information.

[ChamberOfFear]

Pretty cool. Some stretch goals would be to build an editor plugin that gives a test explorer similar to the one in IDEs like Visual Studio, and getting test coverage from the test results.

[eri0o]

Ha! That's a cool goal! Parsing tap output with C# is super easy - and if I am truly lazy someone has probably made a tap consumer in c# already.

So the test panel would basically be picking up the output file and making a pretty panel with green and red things to show what ran and what failed, this should be easy.

For coverage, I think I would do in a boring way, which would be in the tap comment (things after #), verify that the function is mentioned and require the test file (for coverage usage) be named like (modulename)_test.asc, then it's possible to use the editor to pickup the functions in modulename.asc so we can see what's covered and what's not - and I would do a little checkbox to disable coverage in the panel 

[ChamberOfFear]

Code coverage comes in different standard formats which CI systems need to present the data, Cobertura, Lcov, etc.

[eri0o]

I think I would aim at only Function coverage, since things like Line coverage would (at least in my mind) require some static analysis tooling - to be able to check the paths. Function coverage is easy to add, line coverage feels harder.

I can't seem to find a report that CIs expect, the implementations I found kinda depend on the good will of the CI to support your language

Ah, not exactly statically analysis btw, but if you do AGS Script using Atom, with Edmundito's AGS Script package for it, you can install an Atom linter and code AGS with linting and some quick fix options which makes it pretty cool!