The Wesnoth project uses a few different software testing systems. Especially for graphical and layout features, we use the test scenario, a highly decorated test scenario with a ton of graphical features that we can click around in to quickly spot problems. For non-graphical issues, we also have automated tests. We (loosely) term these unit tests. In the software development industry at large, a unit test normally refers to a small, quick test of a single class or method. Unit tests guarantee that each unit is working individually. They are particularly valuable when used with continuous integration, allowing us to catch regressions as they occur. For continuous integration, we currently use Travis CI.
This page is meant to explain how to use the tests and how to add new tests.
The test systems which we currently use are:
The test scenario
This is the default scenario used when
wesnoth is run in test mode. This can be launched by running:
C++ unit tests (game engine and API)
These can be run by compiling the test suite executable, one of the possible targets when compiling with SCons or CMake. You will need to have the Boost.Test library development files installed, and, in CMake's case, pass
-DENABLE_TESTS=ON during configuration.
# Building with SCons: scons boost_unit_tests # Building with CMake: cmake .. -DENABLE_TESTS=ON make boost_unit_tests # Running the test suite from the root dir (one above data): # (Adjust if using a separate build dir) ./boost_unit_tests
WML unit tests (WML/Lua API)
These are Wesnoth test scenarios which contain events which run at the start, perform sanity checks, and then report victory or defeat immediately depending on the results. They are not meant to be interactive. They are run using the main Wesnoth game executable, by means of a script. On Unix-based systems, this script is
run_wml_tests located at the root of the repository. See the forums for alternative scripts/a method to add this to your Visual Studio project file.
Submitting unit tests with patches
It is not mandatory in the project to accompany any patch with unit tests, however it is very welcome and encouraged, especially for patches that includes changes or additions to the engine's internal APIs or the public WML/Lua API. For many kinds of patches unit tests aren't appropriate, but it's always good practice to find ways to test your code. Submitting tests with your code helps to ensure that your contribution will keep working in the future, and makes everything easier to maintain.
Similarly, if you add new graphical features, it might be a good idea to add them to the graphical test scenario.
How to add C++ unit tests
To add a C++ unit test, create a .cpp file in
src/tests/. A suitable template to start with might be
In the simplest case, you just have to do the following things:
0. Include the boost unit test framework.
1. Declare a "boost auto test suite" and give it an appropriate name. An auto test suite is a bundle of tests.
BOOST_AUTO_TEST_SUITE( test_my_feature ) [...] BOOST_AUTO_TEST_SUITE_END()
2. Declare any number of test cases inside the test suite. A test case is
BOOST_AUTO_TEST_CASE( ... ) with an identifier as the argument, followed by a C++ method body. The method body should contain statements like
BOOST_CHECK_EQUAL( ... , ... ). All of the checks should pass when the function is executed.
For an example, you might check
Please consult the Boost.Test docs for other styles of checks that you can use:
3. Finally, add entries for your new .cpp file in
src/CMakeLists.txt alongside the other test .cpp files, to ensure that it is compiled as part of the test executable. You do not need to register your test cases in any other way, as the unit testing framework will execute all of the tests that it finds.
If a check fails, you will generally get the suite, the test case name, and the line number and the exact expression that prompted the fault.
BOOST_CHECK_EQUAL will also give you the mismatched values. If the test code segfaults, you might get just the test case name, or only get the test suite name.
If you have many test cases which test an object which is very complicated to construct, and you don't want to construct and destroy it repeatedly, then you should use a "global fixture", which essentially puts the object at file scope where it will be available to all the test cases in the suite when they run. It's better to use the boost fixture system than to just put it at file scope, because then the boost system knows what is going on, and can report problems more easily. (There are probably other reasons as well.)
You can see an example of a global fixture being used in
How to add WML unit tests
To add a WML unit test, make a test scenario .cfg file and place it in
data/test/scenarios/. Any wesnoth scenario using the
[test] tag can be a valid unit test. However, there are some macros to make writing unit tests simpler. Here's a minimal example.
Generally speaking, a unit test is just a scenario with events which cause it to end in response to the start event, without interaction from the user. If you know how to make a Wesnoth scenario with events, and use the
[endlevel] tag, you know everything you need to know. There are a handful of macros which can make it a little more succinct.
Test scenarios need to be registered in the test schedule file to be used by the continuous integration systems. The test schedule file also says what is supposed to be the result of the test -- is it supposed to pass? fail? timeout? result in a corrupted replay? The test schedule file is
wml_test_schedule at the root of the source tree.
# # Sanity checks of the unit test system # 0 test_return 1 test_return_fail 0 test_assert 1 test_assert_fail 1 test_assert_fail_two 2 empty_test 4 break_replay_with_lua_random 0 fixed_lua_random_replay_with_sync_choice 0 test_end_turn ...
Make a line with your test scenario id. The line should begin with a number, this is the result code.
The schedule includes various kinds of expected failure to ensure that the system is working. Almost surely you want your own tests to pass, and should use code 0 for that.
Consult the help output of the
wesnoth executable (in particular, the description of the
-u switch) to see the codes:
./wesnoth --help [...] 0 - PASS 1 - FAIL 2 - FAIL (TIMEOUT) 3 - FAIL (INVALID REPLAY) 4 - FAIL (ERRORED REPLAY)
run_wml_tests script also supports adding alternate test schedules and various other options, please consult its help text for more info:
For more general info about WML unit tests please refer to this forum thread: http://forums.wesnoth.org/viewtopic.php?t=40449