User:Vinipsmaker/GSoC13/Proposal

Proposal

My proposal is to create a new addon server from scratch, using a modular design easy to extend and built on top of Boost asio. The objectives of the new server are:

• Use a fully documented modular design
• Use boost.asio to deliver better performance (you can see a good in-depth look at the problem of handling tons of clients at [1])
  • Better performance and scalability are the main objectives of this proposal
• Use RAII extensively to provide good support for exceptions and avoid leaks (something essential to any code that is intended to run for ages)
• Support multiple login backends
• Support multiple database backends (but only one would be implemented in this GSoC)
• Better metadata support (extra arguments shouldn't be silently discarded)
• Better localization and internationalization support
• Store all versions of the addons

You can see a draft (remember: it's a draft and its purpose is to explain what strategies will be used to solve the problems faced; the future system will probably implement some things undocumented here) below:

Support multiple login backends

The aim of this feature is to provide a system ready-to-use on the ship date with its own database of users and a LDAP backend for future integration with the remaining Wesnoth online services. I would provide both backends (self and ldap) in this project.

The users would belong to some groups and the groups should be used to create an acess control. It should exist a group for the translation team with power to upload translations and it should exist a group for the admins (with unrestricted powers). The simple/self login backend wouldn't implement groups.

Support multiple database backends

The server architecture would abstract data CRUD (create, retrieve, update, delete) to allow different servers to be used.

The WML-based database

In this project, I'd only implement one concrete backend, the WML-based database backend. This backend would use WML config files and would serve as an example to implement new backends.

This database would separate the add-on data from its meta-data. The meta-data would be saved in a child tag of the main add-on's config file. Extra meta-data will be stored there also.

A migration script will be provided to convert the add-ons from the previous format to the new one.

Better metadata support

Currently the addon is saved in the disk with few modifications, but the metadata used is fixed and cannot be extended (extra metadata is silently discarded). This project aims to better handle the metadata. Some features would be:

• The server would allow the use of custom icons. Currently is required to use an icon that is already installed.
• Extra metadata wouldn't be discarded. This feature aims to provide better extensibility.
• Allow the use of metadata in query requests (eg. only show addons of type campaign)

An extra meta-data example is the number of players. Currently there is no field to describe the number of players, but in the future developers may want to add this feature (and players could start using it ASAP).

The support to extra meta-data influence the database and the network protocol. If you want info about how they solve the new problem, see the appropriate section.

Better localization and internationalization support

Currently, there is no integration between the Addon Server and the WesCamp translations. A script is run by hand every time the translations need to be updated.

The translations are developed/maintained at github under multiple repositories of a single organization (but not all repositories are translations).

It should be too difficult for the server to watch for the notifications and the requirement of the use of older tools/dependencies make this challenge still more difficult. To improve this scenario, the proposed addon server should host the translations itself.

The translator should send the translation to the translation team and the translation team could use their users (with some administrative power) to upload the translation to the server through the Wesnoth client, with a system similar to the .pbl files.

Architecture

The server would be built on top of boost.asio and would run a fixed number of threads. Each thread would run its own event loop, maximizing the possible number of concurrent clients.

An early draft of the components is shown below:

Different versions of the same protocol may share a lot of actions in common. To avoid code duplication, the session component will use factories to instantiate the necessary handlers for each protocol and each handler will delegate the handling of every specific action to other classes.

Namespaces will be used to group classes related to protocols/versions.

Event loop != boost::io_service

The boost::io_service object has an option to act as an event loop, but this option may be suboptiomal when we want to schedule our own actions. Suppose we need to acquire some mutex, then we can't use try_lock because we can't schedule our own actions if it's not I/O related. So, I'd implement my own event loop class, allowing anyone to register its own actions and integrate it with io_service.

To avoid locking a thread for more time than necessary, the EventLoop class would allow to register a delayed action in some queue. The actions that require lock would use try_lock (in conjunction with lock guard, to maintain the RAII idiom) in combination with delayed actions to block the current event loop as little as possible.

Use one boost::io_service per thread decreases the contention when locking the io_service internal mutex for queuing and dequeuing requests and is preferred for maximum scalability.

Config files

The configuration file syntax would be a WML file with the following elements:

• umcd (the name is based on Trademark/mordante proposal): Top-level element
  • old: The tag containing config related to the old (the current one) server
    • Port: Which port would run
  • threads: The number of threads to use. Put 0 or don't put anything to let the server auto-detect through boost
  • database: It may contain several child tags to configure what database to use. The database data and configuration are kept in separate.
    • If simple (the SimpleAddonManager) is chosen, you'd have the option to choose the directory data. The SimpleAddonManager (WML-based database), like the config file, store its metadata in WML.
  • Port: Which port it would use
  • SSL: Settings related to SSL (key file, key file password, server port, ...)

SimpleAddonManager (WML-based database) Filesystem layout

The SimpleAddonManager would save the add-ons using the following schema:

• /: Top-level add-ons directory:
  • add-ons.cfg
  • journal.log: It will provide basic guarantees to avoid severe data loss (see below).
  • Add-ons/
    • Add-on name/
      • metadata.cfg
      • Add-on version/: It would contain the add-on binary files related to version.
        • i18n/ (the name can be changed): It would contain the translation files. The version management would also be applied to translations and each translation should apply to a specific version of the add-on.
      • Another add-on version/: It would contain the add-on binary files related to another version.

journal.log

When some update is requested, the add-on server may update data in several different places. If a crash happens in the middle of the update, the data may corrupt. This database is not meant to support full ACID semantics, but it'll provide atomicity to prevent severe data loss, while a new backend is not developed.

The main implementation technique is to use a journal file, storing when an operation starts and when it ends. Implement atomicity in this database is a requirement of this proposal, but the journal.log is optional and I might choose a different technique.

Protocol

The server would listen on some different ports:

• 15002: It would accept a subset of the protocol defined in CampaignServerWML. The objective is maintain backward compatibility with older Wesnoth clients. The commands not recognizable would be the ones related to uploading a campaign, changing the passphrase for a campaign, deleting a campaign and developer commands.
• 15003 (it's subject to change and can be any other): It would recognize the new protocol, as defined below.
• 15004 (it's subject to change and can be any other): The same as the previous one, but it would run atop of SSL.

The new protocol

The new protocol purpose is to address limitations faced in current protocol. For the new protocol, BinaryWML still will be used, to ease integration with Wesnoth client. The protocol is case-sensitive.

• From now on, messages will use error codes (similar, but not equal, to HTTP and JSON-RPC). This behaviour is useful to build automated tools and to show customized (read translated) message errors to users. The message errors will still be WML-based.
• The new messages will use a top-level tag named umc (the name is based on Trademark/mordante proposal) and will have a version attribute. This attribute will extend the protocol in the future without break compatibility with older Wesnoth clients.

Version 0

For version 0 (the one to be implemented during this GSoC), the following type of requests are supported:

• Listing supported version: The server should return a list of supported versions and the client can use this info to adapt itself to the server's limits.
• Listing available add-ons: It would replace the current CampaignServerWML#Listing_Available_Campaigns. In this command, every tag would be a filter with a syntax similar to MongoDB declarative queries syntax. In version 0, only the following filter types are supported:
  • date
  • language
  • PblWML#title: This attribute should be used if someone wants to receive info about a single add-on.
  • PblWML#author
  • PblWML#description: This tag will have child tags to allow translatable descriptions.
  • PblWML#type
• Downloading an add-on: It will be possible to download a specific version of an add-on and its related translations.
• Uploading an add-on: It will have an auth child tag. It will have an ownership tag, with the following child tags:
  • Type: It can be password, user or group.
  • Password: Only if type is password. It should be the campaign password (just like it works now).
  • Auth: It must be present if type is different from password. It shall contain two child tags:
    • User: The user's nickname.
    • Pass: The user's password.
  • add-on: The add-on itself, with the child metadata and data tags. The data tag would contain the add-on file. The metadata tag could contain any extra child tags and they wouldn't be discarded (this is the behaviour adopted by ODF to increase the interoperability).
• Edit an add-on: It could be used to edit an add-on's metadata, ownership or delete the add-on (or a specific version of it).
• Request uploading terms
• Submit translation: It will allow submit a translation (the description field would also be translated)

Some random notes about the new server/protocol:

• Multiple authors per add-on are possible
• The "listing available add-ons" request would use the following tags differently:
  • Dependencies: The list of dependencies would include a version (allowing the operators >, >=, <, <=, ==).
  • Icon: This different tag would contain the following child tags:
    • Type: It can be link or png. Future protocol versions can define different types.
    • Data: The icon data. It can be a path to an image in the standard image directory for Wesnoth or a base64 encoded small PNG icon.
• Error messages would use a format similar to the JSON-RPC 2.0 error object. The error message/object would contain three child tags:
  • Code: An integer specifying a previous documented value
  • Message: An english human-readable string with a description of the error. Clients can use the error code to show their own messages, possibly translated to the user's main language.
  • Data (optional): Custom data with further details about the specific error.

SSL

SSL is the standard protocol atop of sockets used to securely comunicate between two entities. It doesn't provide much advantage when you only intends to download an add-on, but it's very useful when you want to protect your password from eavesdroppers and alike.

The Wesnoth client could add a configuration to choose secure connection and use unsecure connection by default (decreasing this overhead on the server).

The add-on server would provide both (unencrypted and encrypted) connections. It'd listen to two ports to differentiate the connection type.

Boost::asio already provides support for SSL and add it in the add-on server would be easy.

Timeline

Period	Task	Status	Notes
Week -1 (May 12 to 18)	Install a Debian chroot in my system that mimics Wesnoth main server.	DONE	Debian 6.0 (squeeze) with all Wesnoth build deps (tested)
Week 0 (May 19 to 25)	Create a Wesnoth branch and start coding. Get used to Boost.asio.	Mandatory
Week 1-2 (May 26 till June 8)	Create the SimpleAddonManager, capable of execute the operations needed to manage add-ons.	Mandatory	The base to better meta-data and i18n support.
Week 3 (June 9 to 15)	Reimplement a subset of the current protocol in Boost.asio.	Optional	This feature will depend on the event loop It'll delegate the read operations to the database
Week 4 (June 16 to 22)	Start working on the authentication support.	Mandatory	The standalone login backend won't support access control groups
Week 5-7 (June 23 till July 13)	Implement the new protocol	Mandatory	It'll delegate read and write operations to the database. It'll be integrated with the authentication support. Documentation should follow it progress.
Week 8 (July 14 to 20)	Introduce threads	Mandatory
Week 8 (July 14 to 20)	Introduce SSL	Optional
Week 9-11 (July 21 till August 10)	LDAP support	Mandatory	It may take less time, but I'm overestimating because I have little experience with LDAP (only as a sysadmin at LCCV for a while)
Week 12 (August 11 to 17)	Do more tests, fix bugs and integrate the new branch into the Wesnoth main branch	Mandatory
Week 13-16 (August 18 till September 14)	Integrate the new protocol in Wesnoth client.	Mandatory

I overestimated all tasks, so I may finish some tasks a bit sooner and use this time to compensate the wrongly estimated tasks, but this made the timeline lose some precision.

Threads in wiki 8

Usually it's advised to introduce threads in the initial design of the application if someone wants to support multithread design. A rule of dumb is to take a step further and make threads part of the initial code.

In this proposal, I suggest to introduce threads only at week 8, but I already have experience with multi-threaded programs and I'd make the design ready to be extended right from the week 1. The reason why I want to introduce threads later is because I want to deliver something usable sooner and work incrementally till deliver the full product.

I like to work incrementally, because you catch specifications errors sooner (users not usually know what they want till you deliver the product and they say "Hey! This is not what I asked for!").

Questionnaire

4.1) Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?

I choose the Addon Server project.

4.3) Why did you choose this project?

I like to program code that will use networking capatibilities.

4.6) What do you expect to gain from this project?

I intend to gain a lot of fun developing in C++ for a famous open source game.

4.7) What would make you stay in the Wesnoth community after the conclusion of SOC?

github, C++11, good development infrastructure, generic reusable subprojects, ... (this list is an OR and it's incomplete... a lot of reasons can motivate me)