SoC2011 Student Page Epyon

This page is related to Summer of Code 2011

See the list of Summer of Code 2011 Ideas

This is a Summer of Code 2011 student page

Project: SoC_Ideas_Your_Own_Ideas2011

Description

Kornel Kisielewicz - pure Lua-based scenario definition

The proposal suggests a schema and solution for defining Wesnoth scenarios in pure Lua without the usage of WML. Advantages of such a solution and the additional benefits that will be delivered along it will be described in detail soon.

Project

Abstract

This document proposes an alternative solution to defining Wesonth scenarios without the usage of WML, instead relying on pure Lua files. This suggestion is motivated by many problems that arise in the usage of WML (preprocessor vs tools, validation, sometimes needlessly complex syntax). Commonly encountered arguments against such a replacement are discussed and solutions for a long term plan are included.

Lua is presented as a complete replacement possibility that doesn't sacrifice readability and ease of use, but on the other hand, being a full blown scripting language extremely increases the capabilities of scenario definition without relying on hacks or any preprocessing. Additionally due to Lua's data driven design, "data is code" what allows easy processing, parsing and validation of Lua defined data.

As the topic in question is controversial, much of this document focuses on advocacy, and an approach is suggested that adds functionality without removing and capabilities up to date -- ergo, maintaining backwards compatibility.

I am fully aware that this project dives deeply into Wesnoth's source base with which I am not intimately fammiliar with, so the following document may not take into account many pitfalls, dangers, problems, or may be outright wrong or badly designed in some places. I humbly ask for any possible corrections, hints or information on things that I did not forsee.

Advocacy

Wesnoth already makes heavy use of Lua, and much of WML functionality can be replaced with equivalent and shorter Lua code. As time grows, more advanced options of scenario editing are available only through the usage of Lua. On the other hand, many people expect WML to offer more and more possibilities, what results in the introduction of heavy to maintain features, that are hard to be handled by WML parsing tools -- the notable example here are macros, formulas and the preprocessor in general. It is obvious for anyone writing any complex WML functionality that WML heads towards a scripting language, for what it wasn't initially designed.

Enter Lua. Lua was designed as a data description language, but one also fully suited to scripting. The benefits of Lua are to numerous to list here, and it's benefits are clear to the development team as well.

However, Lua in Wesnoth is still being used mainly as a scripting language, despite it's numerous benefits as a data description language:

1. it offers common readable and newbie friendly syntax for defining data
2. the syntax is well known and well defined, there's a lot of documentation for it, and it sees a lot of usage in the game modding world
3. the parser is readily available, well tested, extremely fast, and comes with a very friendly and well documented API
4. due to Lua's self-reflective capabilities, simple validation is done by the compiler, while more complex validation can be written without much problems
5. no need for preprocessing or macros because the language is a full scripting language
6. structures can be generated on compile time, providing extreme generative capabilities (eg. scenario generators with no changes to host code)
7. syntax can be made very readable and newbie friendly thanks to the built-in customizability of the language
8. syntax shortcuts can be implemented in the code itself (contrary to macros they are fully parsable), and need no change in the hosts code
9. any language that can interface C libraries can compile Lua code, hence has an already written parser, what makes writing tools in any language very simple
10. there are many tools for Lua out there, including but not limited to IDE's, syntax highlighters, debuggers, source analyzers, editor modes
11. use of constructors will allow the dev-team to make enhancements in the format but still support older syntax, by detecting the old usage in the constructor -- engine-side code will not need to be aware of the old format anymore, the conversion will be able to be done by Lua itself

Common critique against:

'Parsing a language is slow!'

Incorrect, Lua's hand written parser is currently faster than libxml [1].

'It's a programming language - modders aren't programmers

While that is true, the pure data definition subset of Lua isn't less readable than WML (and IMHO even more readable), and more conscise (no hacks). In addition, any serious modder will have to touch Lua at some point anyway, meaning he would have to learn Lua anyway.

'No one will use it'

Out of the box Lua-based scenarios will offer a lot more power, and be a lot easier to debug. That in itself is a big advantage. Many existing requests for WML would be available out of the box in the Lua version. Clearly it's only a case of conservatism.

'This is a lot of work, and what we have already works'

True and untrue. The project presented here covers a huge amount of that work in reasonable doable steps. Much of the change is possible due to the well written "config" object usage in the sources.

'There's a lot of samples in WML, none in Lua'

As a part of the project, a fully automated system of conversion both ways (WML -> Lua, Lua -> WML) will be implemented -- which will provide valid Lua Scenario files (which might not benefit from all of Lua's capabilities, but might be hand beautified later at will).

'But WML is not only used in scenarios but also in...'

And in the longer run, those usages could also get the same facelift, enjoying the same benefits! Most of the code needed for that will be written as part of this scenario loading project.

'It's needless work to support two loading mechanisms'

If that is a problem, and once Lua's advantages are noted, there is no problem in removing WML support from the engine, plugging in the written WML -> Lua converter as a frontend to scenario loading.

Target format

Linked below we have an example of a conversion of a simple scenario [2]. This should not be treated as a formal sample of the final format, but rather as a proof of concept that Lua scenario files can be as readable and simple (if not more) then equivalent WML data. As a proof of concept, it may contain mistakes and bad assumptions.

http://pastebin.com/7nKsxtAm

As the first part of the project will be about dealing with a form that is as simple as possible (without beautifications) we need a syntax that has direct translation capabilities. Presented below is a small WML snippet, and the Lua code it would translate to, in three versions -- the latter versions are more complex, but more readable, while still being equivalent (due to the use of constructors, they revert to the first form) -- and while they are superior in terms of readability, they might be more difficult to translate to by the converter.

http://pastebin.com/pP9R4kh2

Of course this doesn't take into account more complex cases, that surely will arise and be problematic. Some solutions are hinted at in later parts of this document.

Deliverables

1. Fully working solution fo loading Wesnoth scenarios without the usage of WML

Includes an implementation of the config class based on Lua tables w/ usage of the lua registry, universal handling for both (so that all code accepts one or the other), additional loading code needed for the substitution and macro/preprocessor removal in the Lua case. The implementation will be designed to avoid any scenario specific hacks, providing a base toolset for loading any WML entities in Lua format in the future in which I'd gladly assist.

2. WML -> Lua, Lua -> WML converter

Depending on how much time I'll have it it will be either simple translation to the most simple valid representation, or already equipped with beautification. The main challenge of this part is handling of macro's and the preprocessor. The former will be dealt with using lua functions and prepopulating the lua state with data from the engine, while the latter might proove more challenging (in case of non-trivial conditionals). Trivial cases like difficulty settings will be dealt with gracefully.

3. Lua-based toolset for beautification and simplification of Scenarios

Functions, constructors, syntax extensions to make the scenarios as human readable and simple as possible. There's no limit to the possibilities here, however the initial version will have only enough functionality to prove that Lua scenarios can be more readable.

4. Complete conversion with beautification of a Wesnoth campaign of choice

As a working goal and final delivery of the project one campaign will be fully converted to the beautified Lua as an example and tease for modders. This is extremely important because it will generate interest in the new (and hopefully considered superior) format, and also a proof that the system is fully mature. Suggested have been "A tale of Two Brothers" (due to small size), and "An Orcish Incursion" (due to simplicity), however, I'd like to get away with the most complex scenario that the new format will manage.

5. Extensive documentation

For both the code changes and the new Lua format. This is extremely important in this project as it is something that hopefully will evolve. Code-side documentation is needed for further extension of the Lua format to other WML applications, while the Lua Scenario format documentation is absolutely required to grab the attention of the modders. Multiple examples of usage (especially focusing on things Lua can do easily that WML has to hack around) will be provided, as well as detailed comments in the converted example campaign.

Implementation details

SimpleWLua

A simplified version of the Lua format needs to be established, taking into account all pitfalls that may happen further in development. Major concerns here is the handling of macros and the preprocessor, however, both problems can be solved at script compile-time. This will be the format to which all the syntax extension will expand to. Care will be taken to take into account possible future development -- ergo the format will take into account all existing WML files, not only the ones used in scenarios.

Converters

Once it is established, one of the hardest parts will be to write the converters -- probably the format will need to be revised during this stage. Lua -> WML converter will surely be written in Lua, for there will be no need for a parser. The harder part will be WML -> Lua conversion, but if possible I might try to also write it in Lua for future ease of use as a frontend (while WML is not a complex format, using an additional library like LPeg [3] might make the process easier.

Engine integration

Core work on the integration is the implementation of a matching "config" class for the compiled Lua table handling, and making it work side by side with the existing wml config class. This is most simply achieved by using the lua registry to store the "root node" of our passed config object. We can pass around tables or values using a lightweight object holding a registry ID and quickly access it when needed (similar to the luabind::object implementation [4]).

For it to work as a side-by-side possibility to WML will by no doubt introduce many problems. While inheritance will solve the most basic ones, the differences between the representation will surely lead to problems in parsing scenario data. These bottlenecks will need to be identified, and separate WML and Lua config treatment might be needed in those cases.

Loading code (especially the multiple file dependency treatment) will of course have to be written separately in the source. However, thanks to that we'll be able to add Lua-specific preparation (for macro/preprocessor removal solution), and enhancements.

All code should take into account the existing Lua toolset to smoothly integrate with it. Probably a lot of code useful for this project is already in place, and just needs to be rigged into the lua_config implementation. However, the timeline (pesimistically) assumes that all code would be written from scratch. As an added benefit all Lua event code would be compiled with the scheme, and be plugged into the Lua tables at the appropriate places.

Toolkits

The Lua side toolkit, loaded by the loading code, will be geared towards readability, ease of use and data validation. The major part of the toolkit will be data constructors, which will serve as basic data validators in itself, however they'll be expanded with data validation against a validator scheme (surprisingly the scheme definitions will be lua tables themselves). A simple example of basic validation for a project I wrote can be seen here:

• items (data) and blueprints (scheme data) : http://pastebin.com/NchWTBrW
• validator function for the above in lua : http://pastebin.com/UGna40Qc

Timeline

I'll be available probably 4/5 time during the coding period (I need some time for my PhD). Also 4-5 days will probably be spent on a short vacation, although I might still be able to do some work remotely.

April 25 - May 22 (pre-planning)

This will be quite a busy time due to my policy of doing research before GSoC coding time. All research on the code will be done in the pre-planning phase, so once GSoC starts I'll be able to jump into coding immediately. Additionaly a basic instance of the SimpleWLua format needs to be prepared, so coding can begin on May 24.

May 23 - June 26 (phase 1 - SimpleWLua)

Objective: Integrate SimpleWLua into Wesnoth

• May 23 - May 29 - creation of the converters based on the pre-designed (in planning stage) SimpleWLua format
• May 30 - June 5 - testing the converters (two-way), testing of the converters on the scenarios and implementation of needed changes, basic testing on non-scenario WML to prepare the format against future expansions
• June 6 - June 12 - refactoring of the wml config class to allow another config type (probably a split into config, wml_config and lua_config), testing for breakage
• June 13 - June 19 - implementation of the lua_config class, initial tests
• June 20 - June 26 - integration of lua_config to serve as a replacement for wml_config in the scenario loading case
• June 27 - July 3 - loader support for macro/preprocessor substitute, then testing the changes on a live game, bugfixing, cleanup before the midterm evaluation
• July 4 - July 10 - buffer week 1 (see below)

July 10 - August 7 (phase 2 - Wesonth Lua)

Objective: Provide the enhancements that will make Lua superior to WML for scenario editing

• July 11 - July 17 - Lua-side loader code, constructors, validation code and basic toolkit
• July 18 - July 24 - initial conversion of the example scenario, validator scheme's for scenarios
• July 25 - July 31 - further extensions to the toolkit based on experience from previous week
• August 1 - August 7 - conversions and bugfixing, detailed documentation for everything (probably on the Wiki and in lua toolkit sources (luadoc) )
• August 7 - August 14 - buffer week 2 (see below)

Buffer weeks

These weeks are purposefully left out as buffer time. I am fully aware that this is a huge task on a codebase that is not well known to me, and expect the unexpected to blow up at the worst possible moment -- time for such emergencies is allocated as buffer and/or if midterm evaluation demands a rescheduling. If unused will be spent on polishing of the toolkit and enhancements.

Future possibilities

While the GSoC scoped project has big merits in itself, once done I think it would be a sin to stop. The project opens up many directions for further development, most of which I would love to be a part of in my free time.

One obvious direction would be to bring Wesnoth Lua support to other WML files -- conversion of unit data has it's immedate benefits, like being able to write Lua hooks to behaviours in the same language, and the validation mechanisms. The existing converters will be able to be used for other WML files without much changes.

The other (in the future), if the devs and the community decides that it would benefit the project, would be a move to Wesnoth Lua fully as the main language. Again, converters would be invaluable here, not only because they'd provide a mechanism to quickly convert the data, but also will allow for legacy scenarios and tools to work even if there would be no WML related code in the sourcebase.

Footnotes

• [1] Lua vs. XML benchmark results -- http://www.ogre3d.org/forums/viewtopic.php?f=16&t=54057
• [2] An Orcish Incursion -- http://svn.gna.org/viewcvs/wesnoth/trunk/data/campaigns/An_Orcish_Incursion/scenarios/01_Defend_the_Forest.cfg?view=markup
• [3] LPeg, Lua parser generatorr-- http://www.inf.puc-rio.br/~roberto/lpeg/lpeg.html
• [4] luabind -- http://www.rasterbar.com/products/luabind.html

IRC

epyon

SoC Application

SoC Application

Questionnaire

Questionnaire will be submitted directly to the Google application due to privacy reasons.