Proposal

My proposal is to create a new addon server from scratch, using a modular design easy to extend and built on top of Boost asio. The objectives of the new server are:

Use a fully documented modular design
Use boost.asio to deliver better performance (you can see a good in-depth look at the problem of handling tons of clients at [1])
- Better performance and scalability are the main objectives of this proposal
Use RAII extensively to provide good support for exceptions and avoid leaks (something essential to any code that is intended to run for ages)
Support multiple login backends
Support multiple database backends (but only one would be implemented in this GSoC)
Better metadata support (extra arguments shouldn't be silently discarded)
Better localization and internationalization support
Store all versions of the addons

You can see a draft (remember: it's a draft and its purpose is to explain what strategies will be used to solve the problems faced; the future system will probably implement some things undocumented here) below:

Support multiple login backends

The aim of this feature is to provide a system ready-to-use on the ship date with its own database of users and a LDAP backend for future integration with the remaining Wesnoth online services. I would provide both backends (self and ldap) in this project.

The users would belong to some groups and the groups should be used to create an acess control. It should exist a group for the translation team with power to upload translations and it should exist a group for the admins (with unrestricted powers). The simple/self login backend wouldn't implement groups.

Support multiple database backends

The server architecture would abstract data CRUD (create, retrieve, update, delete) to allow different servers to be used.

In this project, I'd only implement one concrete backend. This backend would use WML config files and would serve as an example to implement new backends.

A migration script will be provided to convert the add-ons from the previous format to the new one.

Better metadata support

Currently the addon is saved in the disk with few modifications, but the metadata used is fixed and cannot be extended (extra metadata is silently discarded). This project aims to better handle the metadata. Some features would be:

The server would allow the use of custom icons. Currently is required to use an icon that is already installed.
Extra metadata wouldn't be discarded. This feature aims to provide better extensibility.
Allow the use of metadata in query requests (eg. only show addons of type campaign)

Better localization and internationalization support

Currently, there is no integration between the Addon Server and the WesCamp translations. A script is run by hand every time the translations need to be updated.

The translations are developed/maintained at github under multiple repositories of a single organization (but not all repositories are translations).

It should be too difficult for the server to watch for the notifications and the requirement of the use of older tools/dependencies make this challenge still more difficult. To improve this scenario, the proposed addon server should host the translations itself.

The translator should send the translation to the translation team and the translation team could use their users (with some administrative power) to upload the translation to the server through the Wesnoth client, with a system similar to the .pbl files.

Architecture

The server would be built on top of boost.asio and would run a fixed number of threads. Each thread would run its own event loop, maximizing the possible number of concurrent clients.

An early draft of the components is shown below:

To avoid locking a thread for more time than necessary, the EventLoop class would allow to register a delayed action in some queue. The actions that require lock would use try_lock (in conjunction with lock guard, to maintain the RAII idiom) in combination with delayed actions to maintain a lock as little as possible.

Different versions of the same protocol may share a lot of actions in common. To avoid code duplication, the session component will use factories to instantiate the necessary handlers for each protocol and each handler will delegate the handling of every specific action to other classes.

Namespaces will be used to group classes related to protocols/versions.

Config files

The configuration file syntax would be a WML file with the following elements:

umcd (the name is based on Trademark/mordante proposal): Top-level element
- old: The tag containing config related to the old (the current one) server
  - Port: Which port would run
- threads: The number of threads to use. Put 0 or don't put anything to let the server auto-detect through boost
- database: It may contain several child tags to configure what database to use. If simple (the SimpleAddonManager) is chosen, you'd have the option to choose the directory data.
- Port: Which port it would use
- SSL: Settings related to SSL (key file, key file password, server port, ...)

SimpleAddonManager Filesystem layout

The SimpleAddonManager would save the add-ons using the following schema:

/: Top-level add-ons directory:
- add-ons.cfg
- journal.log: This won't be used to implement a full ACID support, but it will provide basic guarantees to avoid severe data loss.
- Add-ons/
  - Add-on name/
    - metadata.cfg
    - Add-on version/: It would contain the add-on binary files related to version.
      - i18n/ (the name can be changed): It would contain the translation files. The version management would also be applied to translations and each translation should apply to a specific version of the add-on.
    - Another add-on version/: It would contain the add-on binary files related to another version.

Protocol

The server would listen on some different ports:

15002: It would accept a subset of the protocol defined in CampaignServerWML. The objective is maintain backward compatibility with older Wesnoth clients. The commands not recognizable would be the ones related to uploading a campaign, changing the passphrase for a campaign, deleting a campaign and developer commands.
15003 (it's subject to change and can be any other): It would recognize the new protocol, as defined below.
15004 (it's subject to change and can be any other): The same as the previous one, but it would run atop of SSL.

The new protocol

The new protocol purpose is to address limitations faced in current protocol. For the new protocol, BinaryWML still will be used, to ease integration with Wesnoth client. The protocol is case-sensitive.

From now on, messages will use error codes (like HTTP and JSON-RPC). This behaviour is useful to build automated tools and to show customized (read translated) message errors to users.
The new messages will use a top-level tag named umc (the name is based on Trademark/mordante proposal) and will have a version attribute. This attribute will extend the protocol in the future without break compatibility with older Wesnoth clients.

Version 0

For version 0 (the one to be implemented during this GSoC), the following type of requests are supported:

Listing supported version: The server should return a list of supported versions and the client can use this info to adapt itself to the server's limits.
Listing available add-ons: It would replace the current CampaignServerWML#Listing_Available_Campaigns. In this command, every tag would be a filter with a syntax similar to MongoDB declarative queries syntax. In version 0, only the following filter types are supported:
- date
- language
- PblWML#title: This attribute should be used if someone wants to receive info about a single add-on.
- PblWML#author
- PblWML#description: This tag will have child tags to allow translatable descriptions.
- PblWML#type
Downloading an add-on: It will be possible to download a specific version of an add-on and its related translations.
Uploading an add-on: It will have an auth child tag. It will have an ownership tag, with the following child tags:
- Type: It can be password, user or group.
- Password: Only if type is password. It should be the campaign password (just like it works now).
- Auth: It must be present if type is different from password. It shall contain two child tags:
  - User: The user's nickname.
  - Pass: The user's password.
- add-on: The add-on itself, with the child metadata and data tags. The data tag would contain the add-on file. The metadata tag could contain any extra child tags and they wouldn't be discarded (this is the behaviour adopted by ODF to increase the interoperability).
Edit an add-on: It could be used to edit an add-on's metadata, ownership or delete the add-on (or a specific version of it).
Request uploading terms
Submit translation: It will allow submit a translation (the description field would also be translated)

Some random notes about the new server/protocol:

Multiple authors per add-on are possible
The "listing available add-ons" request would use the following tags differently:
- Dependencies: The list of dependencies would include a version (allowing the operators >, >=, <, <=, ==).
- Icon: This different tag would contain the following child tags:
  - Type: It can be link or png. Future protocol versions can define different types.
  - Data: The icon data. It can be a path to an image in the standard image directory for Wesnoth or a base64 encoded small PNG icon.
Error messages would use a format similar to the JSON-RPC 2.0 error object. The error message/object would contain three child tags:
- Code: An integer specifying a previous documented value
- Message: An english human-readable string with a description of the error. Clients can use the error code to show their own messages, possibly translated to the user's main language.
- Data (optional): Custom data with further details about the specific error.

Timeline

Week -1 (May 12 to 18): Install a Debian chroot in my system that mimics Wesnoth main server.
Week 0 (May 19 to 25): Create a Wesnoth branch and start coding. Get used to Boost.asio
Week 1-2 (May 26 till June 8): Create the SimpleAddonManager, capable of execute the operations needed to manage add-ons.
Week 3 (June 9 to 15): Reimplement the current protocol in Boost.asio.
Week 4 (June 16 to 22): Start working on the authentication support
Week 5-7 (June 23 till July 13): Implement the new protocol (documentation should follow it progress).
Week 8 (July 14 to 20): Introduce threads and SSL
Week 9-11 (July 21 till August 10): LDAP support
Week 12 (August 11 to 17): Do more tests, fix bugs and integrate the new branch into the Wesnoth main branch
Week 13-16 (August 18 till September 14): Integrate the new protocol in Wesnoth client

Questionnaire

4.1) Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?

I choose the Addon Server project.

4.3) Why did you choose this project?

I like to program code that will use networking capatibilities.

4.6) What do you expect to gain from this project?

I intend to gain a lot of fun developing in C++ for a famous open source game.

4.7) What would make you stay in the Wesnoth community after the conclusion of SOC?

github, C++11, good development infrastructure, generic reusable subprojects, ... (this list is an OR and it's incomplete... a lot of reasons can motivate me)

User:Vinipsmaker/GSoC13/Proposal

Contents