SoftwareTesting

The wesnoth project a few different software testing systems. Especially for graphical and layout features, we use the test scenario -- this is a highly decorated test scenario with a ton of graphical features that we can use to quickly spot problems. For non-graphical issues, we also have automated tests. We (loosely) term these unit tests. In the software development industry at large, a unit test normally refers to a small, quick test of a single class or method. Unit tests guarantee that each unit is working individually. Unit tests are particularly valuable when used with continuous integration, because they can then immediately catch regressions when they occur. For continuous integration, we currently use both travis-ci and jenkins.

Overview

The test systems which we currently use are:

0. The test scenario: https://github.com/wesnoth/wesnoth/blob/master/data/scenario-test.cfg This can be launched by running

wesnoth -t

That is, it is the default scenario when wesnoth is run in test mode.

1. C++ unit tests using the boost unit testing framework: https://github.com/wesnoth/wesnoth/tree/master/src/tests

These can be run by compiling the test executable. This is one of the possible targets when compiling with cmake / scons.

scons test
./test

2. WML unit tests (tests of the WML API)

These are wesnoth test scenarios which contain events which run at the start, do some sanity check, and then report victory or defeat immediately depending on the results. They are not meant to be interactive. They are run using the wesnoth executable, by means of a script. On unix systems, this script is run_wml_tests located at the root of the repository. See the forums for alternative scripts / a method to add this to your visual studio project file.

./run_wml_tests

Submitting Unit Tests with Patches

It is not mandatory in the project to accompany any patch with unit tests, however it is very welcome and encouraged. For many kinds of patches unit tests aren't appropriate, but it's always good practice to find ways to test your code. Submitting tests with your code helps to ensure that your contribution will keep working, and makes everything easier to maintain.

Similarly, if you add new graphical features, it might be a good idea to add them to the graphical test scenario.

How to add C++ unit tests

To add a C++ unit test, create a .cpp file in src/tests/. A good template might be this file: https://github.com/wesnoth/wesnoth/blob/master/src/tests/test_config.cpp

In the simplest case, you just have to do the following things:

0. Include the boost unit test framework.

 #include <boost/test/unit_test.hpp>

1. Declare a "boost auto test suite" and give it an appropriate name. An auto test suite is a bundle of tests.

 BOOST_AUTO_TEST_SUITE( test_my_feature )

 ...

 BOOST_AUTO_TEST_SUITE_END()

2. Declare any number of test cases. A test case is BOOST_AUTO_TEST_CASE( ... ) with an identifier as the argument, followed by a C++ method body. The method body should contain statements like BOOST_CHECK_EQUAL(... , ...). All of the checks should pass when the function is executed.

For an example, you might check this file: https://github.com/wesnoth/wesnoth/blob/master/src/tests/test_config.cpp

Please consult boost docs for other styles of checks that you can use.

http://www.boost.org/doc/libs/1_44_0/libs/test/doc/html/tutorials/hello-the-testing-world.html http://www.boost.org/doc/libs/1_44_0/libs/test/doc/html/utf.html

If a check fails, you will generally get the suite, the test case name, and the line number and the exact expression that prompted the fault. BOOST_CHECK_EQUAL will also give you the mismatched values. If the test code segfaults, you might get just the test case name, or only get the test suite name afair.

If you have many test cases which test an object which is very complicated to construct, you should use a "global fixture", which essentially puts the object at file scope where it will be available to all the test cases in the suite when they run. It's better to use the boost fixture system than to just put it at file scope, because then the boost system knows what is going on, and can report problems more easily.

You can see an example of a global fixture being used here: https://github.com/wesnoth/wesnoth/blob/master/src/tests/test_mp_connect.cpp

Finally, add entries in sconscript and cmakelists.txt alongside the other test .cpp files, to ensure that it is compiled as part of the test executable. You do NOT need to register your test cases in any other way, the main test .cpp file will link in all of the tests that it finds.

How to add WML unit tests

To add a WML unit test, make a test scenario .cfg file and place it in data/test/scenarios/. Any wesnoth scenario using the [test] tag might be a valid unit test. However, there are some macros to make writing unit tests simpler. A good example might be: https://github.com/wesnoth/wesnoth/blob/master/data/test/scenarios/two_plus_two.cfg

Test scenarios need to be registered in the test schedule file to be used by the continuous integration systems. The test schedule file also says what is supposed to be the result of the test -- is it supposed to pass? fail? timeout? crash? result in a corrupted replay? The test schedule file is here: https://github.com/wesnoth/wesnoth/blob/master/wml_test_schedule

#
# Sanity checks of the unit test system
#
0 test_return
1 test_return_fail
0 test_assert
1 test_assert_fail
1 test_assert_fail_two
2 empty_test
4 break_replay_with_lua_random
0 fixed_lua_random_replay_with_sync_choice
0 test_end_turn
...

Make a line with your test scenario id. The line should begin with a number, this is the result code.

The schedule includes various kinds of expected failure to ensure that the system is working. Almost surely you want your own tests to pass, and should use code 0 for that.

Consult the help output of the wesnoth executable to see the codes:

 wesnoth --help

 ...
     0 - PASS
     1 - FAIL
     2 - FAIL (TIMEOUT)
     3 - FAIL (INVALID REPLAY)
     4 - FAIL (ERRORED REPLAY)

Please refer to this forum post for more info: http://forums.wesnoth.org/viewtopic.php?f=8&t=40449