View source for User:Trademark/GSoC 2013/Addon Server: Create a new and shiny one

{{SoC2013Student}}
__TOC__
==Caution==
This proposal does '''not''' reflect the work realized, even if some parts are still correct I suggest you to take only for reference the specific [[UMCD|UMCD Wiki page]].

==Latest update==
'''14th May'''

* Add a dependency section.

==Proposal Part 1 — Addon Server==
===Description===
This proposal aims to rewrite and improve the addon server with Boost.Asio. The current server becomes old and has few drawbacks:

* The code is monolithic, resumed in a single file which made it hard to extend and maintain.
* The responses are in english only, support for gettext should be added.
* The translation of the addons are not synchronized with Wescamp.
* Native handling of the different types of add-on (campaigns, factions, eras, …).

The hope is to design a new and robust server with the help of Boost.Asio, make it extensible with every layer of code clearly defined. Moreover the wire has been completely reviewed to integrate translatable features. It'll add meta-information on the different types of add-on.

===Server configuration===
Most of it is described in the [https://gna.org/patch/?3848 patch #3848]. You can also find a PDF version [http://hyc.io/wesnoth/umcd.pdf here]. The following is a summary of the chapter "Server configuration" of the umcd.pdf document.

====Options====
The file umcd.cfg resumes the main characteristics of the server:

* Port: The TCP/IP port to listen to for incoming requests.
* Threads: The number of working threads to start.
* Dir: The directory where the add-on will be stored.

===Wire format===
The wire format has been extensively described in the [https://gna.org/patch/?3848 patch #3848]. You can find a PDF version of the wire at this link [http://hyc.io/wesnoth/umcd.pdf umcd.pdf]. For documentation purpose, the following is a list of the major features.

====Special packets====
These packets inform the user that abnormal packet has been sent in the request. For example, it warns if a field is missing in the request.

* Warning packet
* Error packet

====User request packets====
These packets can be sent by users of Wesnoth having any of the following requests:

* Request the campaign list in a favorite language.
* Request UMC download in a specific language.
* Request UMC change password.
* Request a new password.
* Request UMC upload.
* Request UMC delete.
* Request license.

====Admin request packets====
Should be discussed with mordante before further analysis.

===Integration===
The add-on server is used among others services such as the translation service. The following diagrams show how the add-on server is currently integrated into the overall system. Firstly from the translator point of view:

http://www.hyc.io/wesnoth/translator-user-architecture.png

We see that the translator doesn't interact with the add-on server, so this system part should stay as it is. The UMC author interact with the add-on server:

http://www.hyc.io/wesnoth/author-user-architecture.png

We can made some observations on this diagram:

* The add-on server query the script wescamp.py to get the translations from github.
* A synchronization is made by hand between the add-on server and github via wescamp.py.

In a first time, we would keep the manual synchronization. An automatic support will be added later in the GSoC period or not. The add-on and translation will be locally store on the add-on server. A translation check out with the help of the github notifications will be implemented. It means that the add-on server will not communicate with wesnoth.py anymore. You can see below a diagram of the expected architecture (still from the author point of view).

http://www.hyc.io/wesnoth/author-user-new-architecture.png

The changes are concentrated around the add-on server, it's a step toward the merging of the translation server and the add-on server. The dependences of the add-on server will be:

* The game Wesnoth where queries come from.
* The back-up server github.

The diagram doesn't show the wescamp.py script which interact with the add-on server for the back-up. In fact, we hope having the time to do it in the time frame of the GSoC, if not, it should be done after the GSoC.

===Front-end Architecture===
The front-end architecture is all about the controller and the business logic. This architecture is inspired from the [http://static.springsource.org/spring/docs/2.0.x/reference/mvc.html spring framework], [http://struts.apache.org/ struts framework], [http://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller MVC architecture] and [http://en.wikipedia.org/wiki/Service-oriented_architecture service-oriented architecture]. 

====Flow diagram====
The following diagram shows briefly the data flow and the components involved.

http://www.hyc.io/wesnoth/flow-diagram.png

Here a description of these components:

* <strong>Request handler</strong>: Thread listening the income request, the download and management of the request is asynchronous.
* <strong>Request dispatcher</strong>: Read the request and select the good action to apply on it.
* <strong>Actions</strong>: These are the different actions that the server can do. For instance, it can be uploading an addon. Their behaviors are very specific, note that most of them will need to access the back-end.
* <strong>Request checker</strong>: This class could be a part of the actions, it will check that the files (or any data) sent along with the request are well-formed.

====Class diagram====
The following schema is a simplify class diagram. Note that trivial class are not represented, it's here to give you an overall idea of the architecture.

http://www.hyc.io/wesnoth/class-diagram.png

We can describe a little further some of the classes:

* '''ActionFactory''': Production of action executors with a look-up name.
* '''Action''': All the action such as the list add-on campaign must inherit from this class and override the abstract method 'execute'. It takes a request in parameter and return a reply. Note that the reply is actually sent by the RequestDispatcher.
* '''AddonData''': The data of the request/reply. The metadata are mainly the protocol that has been described in [https://gna.org/patch/?3848 patch #3848]. The binary data are the files of the add-on or other data such as the license.

Finally the class '''AddonDatabase''' is implemented as a singleton. The purpose is to avoid multiple connections (issues with the thread safety will be considered).

====Back-end Architecture====
The back-end architecture is all about how to store the data. I present different solution with pros and cons and discuss about it.

=====Discussion=====
An add-on database is required for this project. Here are the characteristics of the system we need:

* Storing an addon package along with meta-information.
* Create dependencies between package.
* Easy operation for adding and removing package.

Indeed, such a system has been well-studied and is known as [http://en.wikipedia.org/wiki/Package_management_system package manager] such as RPM or dpkg. They can help us to be aware of some problems such as the [http://en.wikipedia.org/wiki/Dependency_Hell dependency hell], however they are system-oriented, and much more complicated than what we actually need.

====Option 1: Flat file hierarchy====
The flat file hierarchy would consist to store the add-ons and meta-information in one repositories per add-on.

<strong>Pros</strong>

* Simple.
* Would exactly fit our needs.

<strong>Cons</strong>

* We would reinvent the wheel.
* Doesn't respect the [http://en.wikipedia.org/wiki/ACID ACID] properties.

====Option 2: Relational database====
A relational database respects the ACID properties. One of the other advantages is to be able to filter the binary data with the help of the meta-data. For example, we could request the add-ons of a specific author.

A simple relational diagram would be as follow:

http://www.hyc.io/wesnoth/addon-database.png

Few observations:

* The historic of a add-on is recorded. Only the most recent add-on could be saved and the older would go on the back-up server.
* The dependencies are between a specific add-on version and the main add-on which it depends upon. Note the version mask in the dependency table.
* We would avoid to store blobs in the database by specifying the path to the data. A [http://programmers.stackexchange.com/questions/150669/is-it-a-bad-practice-to-store-large-files-10-mb-in-a-database post] on stackexchange explains why we should avoid this.
* The language and AddonType tables are enumeration-like tables.

'''Caution''': This database schema has not been reviewed yet. Feel free to ask clarification and share comments.

About the database server, we could use:

* [http://www.mysql.com/ MySQL]: The Wesnoth server already runs a MySQL server so we wouldn't add dependencies on the server.
* [http://www.sqlite.org/ SQLite]: It's a light and easy way to use a database. Everything is inlined in the code and so, no communication with external service is required.

However the final choice is let at the appreciation of the server administrators. Note that both respects the [http://www.sqlite.org/transactional.html ACID properties], but note also that having files store outside the database can be problematic, some transactions must be handle in the code.

Finally we would use the [http://soci.sourceforge.net/index.html SOCI] interface to communicate with the database. It's a C++ project that abstract the communication with database. It's quite stable, the proof is that the CERN use it. It's really useful if we want to change the database one days, just a little of the code would be rewritten.

===Front-end to back-end bridge===
The front and back-end must be independent, so some high levels choices must be made. Given the wire, we have an idea of the operations on the database and thus we can establish the database interface. Note that all names can change.

  class AddonDatabase
  {
    public:
      virtual Reply request_campaign_list(const Request& request) =0;
      virtual Reply request_campaign(const Request& request) =0;
      virtual Reply request_password_update(const Request& request)=0;
      virtual Reply request_new_password(const Request& request)=0;
      virtual Reply request_campaign_upload(const Request& request)=0;
      virtual Reply request_campaign_delete(const Request& request)=0;
      virtual Reply request_license(const Request& request)=0;
  };

The database dependent code will inherits from this interface and implements the methods needed.

===Others===
====Signal====
The signals are handle via std::signal (or Boost.Asio, but using standardize tools when possible seem best). The default behavior for every signal is:

* Notice all current tasks that a problem occurs and they must stop.
* If some tasks communicate with any user, send a packet describing the interruption.
* Waiting (with timed out) that all threads have been stopped.
* Restart the server.

Specific actions are applied for SIGTERM and SIGINT (in addition of the 3 first):

* Launch a maintenance server which respond to users with an informative message.
* Close the server.

For the other signals, we mail the server owner (in addition of the default behavior described above. If the server has restarted more than 3 times in half a hour, we do the same as with SIGTERM/SIGINT.

====Log====
We log several informations:

* User connection with request type (and other relevant information like timestamp).
* Add-on added.
* Add-on erased.
* New password and password change.
* And all actions that actually change/access the database.

But we would also log the errors from the code, unexpected behavior, system error, signal catch,

A home-made implementation would be used. We'd also consider the state of the [http://boost-log.sourceforge.net/libs/log/doc/html/index.html Boost.Log] implementation.

===Resources===
I will start by studying the different [http://www.boost.org/doc/libs/1_53_0/doc/html/boost_asio/examples.html#boost_asio.examples.http_server Boost.Asio http servers].

===Dependencies===
{| style="border:1px dashed #AAA;border-collapse:collapse;"
|-
!style="border:1px dotted #AAA;padding:5px;"|Dependency
!style="border:1px dotted #AAA;padding:5px;"|Optional
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|[http://www.boost.org/ Boost C++ library]
|style="border:1px dotted #AAA;padding:5px;text-align:center;"|Required
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|[http://www.openssl.org/ OpenSSL] for secure transmission of the password (Boost.Asio.SSL need it).
|style="border:1px dotted #AAA;padding:5px;text-align:center;"|Required
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|[http://soci.sourceforge.net/ SOCI] – The C++ Database Access Library for communicates with any kind of underlining databases
|style="border:1px dotted #AAA;padding:5px;text-align:center;"|Advisable
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|The back-end specific to the database:
* [http://dev.mysql.com/doc/refman/5.6/en/c.html MySQL client] (if MySQL database).
* [http://www.sqlite.org/ SQLite 3 library] (if SQLite database).
|style="border:1px dotted #AAA;padding:5px;text-align:center;"|Required
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|[http://boost-log.sourceforge.net/libs/log/doc/html/index.html Boost.Log] (which will be officially added in the next release)
|style="border:1px dotted #AAA;padding:5px;text-align:center;"|Optional
|}

==Timeline==
{| style="border:1px dashed #AAA;border-collapse:collapse;"
|-
!style="border:1px dotted #AAA;padding:5px;"|Date
!style="border:1px dotted #AAA;padding:5px;"|Description
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|May-June
|style="border:1px dotted #AAA;padding:5px;"|Refinement of the analysis.
|-
|style="border:2px dotted #AAA;padding:5px;color:#444;"|17/06
|style="border:2px dotted #AAA;padding:5px;text-align:center;"|GSoC start
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|22/06 to 01/07
|style="border:1px dotted #AAA;padding:5px;"|Set up the database and related back-end architecture code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|01/07 to 05/07
|style="border:1px dotted #AAA;padding:5px;"|Implements the UMC download back-end code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|06/07 to 15/07
|style="border:1px dotted #AAA;padding:5px;"|Implements the overall front-end architecture along with the download action code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|15/07 to 25/07
|style="border:1px dotted #AAA;padding:5px;"|Test, document and correct bugs. Some code refactoring could occur if needed.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|26/07 to 29/07
|style="border:1px dotted #AAA;padding:5px;"|Movable days. Can be use for extent anything above. The server should work now.
|-
|style="border:2px dotted #AAA;padding:5px;color:#444;"|29/07
|style="border:2px dotted #AAA;padding:5px;text-align:center;"|Mid-term evaluation 
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|30/07 to 05/08
|style="border:1px dotted #AAA;padding:5px;"|Implements the UMC upload code (back and front-end).
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|06/08 to 11/08
|style="border:1px dotted #AAA;padding:5px;"|Implements the UMC list code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|12/08 to 15/08
|style="border:1px dotted #AAA;padding:5px;"|Movable days.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|16/08 to 22/08
|style="border:1px dotted #AAA;padding:5px;"|Implements the translations automatic check-out with github.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|23/08 to 28/08
|style="border:1px dotted #AAA;padding:5px;"|Implements the UMC deletion code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|29/07 to 02/09
|style="border:1px dotted #AAA;padding:5px;"|Test, document and refactor.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|03/09 to 10/09
|style="border:1px dotted #AAA;padding:5px;"|Implements the password related actions.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|11/09
|style="border:1px dotted #AAA;padding:5px;"|Implements the UMC license code.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|12/09 to 16/09
|style="border:1px dotted #AAA;padding:5px;"|Movable days.
|-
|style="border:1px dotted #AAA;padding:5px;color:#444;"|16/09 to 27/09
|style="border:1px dotted #AAA;padding:5px;"|Final tests, documentation polishing and correction of latest bugs.
|-
|style="border:2px dotted #AAA;padding:5px;color:#444;"|23/09
|style="border:2px dotted #AAA;padding:5px;text-align:center;"|Final evaluation 
|}

==IRC==
Trademark

==Questionnaire==
===Basics===
====Information====

<strong>Email :</strong> ptalbot [at] mopong [dot] net

<strong>Nickname :</strong> Trademark

<strong>Study :</strong> Master degree in computer science at the University of Lyon 1

<strong>Country :</strong> France

<strong>Availability :</strong> Mostly during the evening (4pm to 9pm UTC)

====Write a small introduction to yourself.====
My name is Pierre Talbot, I currently follow a Master Degree (first year) in computer science in Lyon (France). After being a hardcore gamer for few years during high school, I choose to study computer science. So I begun my studies in Belgium and after 2 years I joined the University of Lyon 1 in France. Now, more than anything I like to program and to learn more on the various fields of computer science.

====Why do you want to participate in summer of code?====
Because I already participated in the GSoC, I know that is an invaluable program to excel your programming level. Furthermore being paid to learn from experienced people is just an opportunity.

====Do you have other commitments for the summer period? Do you plan to take any vacations? If yes, when?====
I will restart school 2 weeks before the ending. During these weeks I will work more during the week-end instead of balancing the works all over the week. But I'll try to finish most of the work before this date. I didn't planned any vacation but if I take some, it will be at most 3 days.

===Experience===
====What programs/software have you worked on before?====
I mainly worked on numerous school projects which involves various skills, the relevants one's for this project were:

* FTP-like server in C.
* Network game programming in Java and C.
* MVC web application (team of 4 persons).

I also worked on many C++ projects (data structures, Boost.Check, garbage collector, …) without a networking layer.

====Have you developed software in a team environment before? (As opposed to hacking on something on your own)====

Yes, the most of my school projects are in team of 2,3 or 4 students. We are using source versionning such as Mercurial, Git. Depending on the project, we use other team tools such as the ones provided by Redmine.

I used SVN during my previous GSoC.

====Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?====

I participated in 2011 as a student in the Boost organization. I worked on a new library that is not yet accepted into Boost, however I succeed the GSoC. [http://boost-sandbox.sourceforge.net/libs/checks/doc/html/index.html Boost.Check] is a library helping users to validate and create check digits of various numbers (such as credit card numbers, ISBN, IBAN, …). One of the concern is to design it to be easily expendable.

====Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.====

My long term support for an open-source project is [http://boost-sandbox.sourceforge.net/libs/checks/doc/html/index.html Boost.Check]. After nearly two years, I still work on it.

I also worked on a CGAL library during a two months research work in my university. I've upgraded the [http://www.cgal.org/Manual/latest/doc_html/cgal_manual/Combinatorial_map/Chapter_main.html Combinatorial Map] library to be compatible with Boost.Graph concepts and specific CGAL concepts. Now the Combinatorial map can work with generic algorithms that request these concepts (such as the [http://www.cgal.org/Manual/latest/doc_html/cgal_manual/Surface_mesh_simplification/Chapter_main.html Triangulated Surface Mesh Simplification] algorithm).

====Gaming experience - Are you a gamer?====
Not so much since few years, mainly because the time is lacking.

=====What type of gamer are you?=====
I was a hardcore gamer during several years. I played a lot for the competition side. Now I mainly play during holidays or free times.

=====What type of games?=====
I played strategies games, FPS and MMORPG. I spent a hundreds of hours on Warcraft 3 and other hundreds on Guild Wars.

=====What type of opponents do you prefer?=====
I prefer the opponents that are stronger enough to play beautiful games. The one that can help me to excel my skills. However, sometimes I just play for fun with friends, and then, the opponents doesn't matters.

=====Are you more interested in story or gameplay?=====
I like both. Games with great story and poor gameplay are not interesting. I prefer a good gameplay than a great story, however I often expect a little more than just the gameplay.

=====Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.=====
I mainly played the solo game but not for a long time (few hours). For this project, I'd need to play in multiplayer, especially with the add-on. It'll be easier to understand the needed/requested/hoped features.

====Patches====
This year patch:

* [https://gna.org/patch/?3848 Patch #3848].

2012 patches:

* [https://gna.org/bugs/?19538 Bug #19538].
* [https://gna.org/patch/?3236 Patch #3236].
* [https://gna.org/patch/?3319 Patch #3319].

===Communication skills===
====Though most of our developers are not native English speakers, English is the project's working language.  Describe your fluency level in written English.====
I can easily understand people, technical documents and text in English. My writing is good enough to get understood by people. My overall level is medium.

====What spoken languages are you fluent in?====
My native language is the French. I can speak in English if I know well the speaking topic.

====Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.====
I know how gamers can be when the game doesn't fit their expectations. We will need to ask them feedback about the add-on server. I think I can handle the dissatisfied uploader, I'd try to find compromises and quick solutions.

====Do you give constructive advice?====
I'm helping on a french [http://www.developpez.com/ programming forum]. I try to explain why and how-to instead of just giving a solution. So I hope my advices are constructive.

====Do you receive advice well?====
As a programmer, you must receive advices well because it's a field where ideas evolve really fast. Exchanging ideas and receiving/giving advices is a day-to-day task. However I prefer receiving advices from experienced people.

====Are you good at sorting useful criticisms from useless ones?====
The useless are often a kind of judgment and should be easy to sort out.

====How autonomous are you when developing? Would you rather discuss intensively changes and not start coding until you know what you want to do or would you rather code a proof of concept to "see how it turn out", taking the risk of having it thrown away if it doesn't match what the project want====
Most of the modern programming methodologies show that you can't just discuss and analyse. You must continue the analysis through out all the process, it requires a lot of refactoring. Resuming, a bit of analysis is always required, but you can't loose to much times on it, otherwise nothing will be never done.

===Project===
====Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?====
I choose the add-on server.

====If you have invented your own project, please describe the project and the scope.====
Not yet.

====Why did you choose this project?====
To learn the Boost.Asio library. I like to have the flexibility to code from scratch a server. I like to think about good software architecture, and this project helps me to do so.

====Include an estimated timeline for your work on the project. Don't forget to mention special things like "I booked holidays between A and B" and "I got an exam at ABC and won't be doing much then".====
Please, give a look at the timeline section.

====Include as much technical detail about your implementation as you can====
The proposal is described in the proposal section.

====What do you expect to gain from this project?====
Learning a new library, validate a home-made architecture on a world-wide use server. Of course, it's also to earn money from my passion.

====What would make you stay in the Wesnoth community after the conclusion of SOC?====
A successful project, I'd like to maintain the server after the SOC (or finish/polish it, if needed).

===Practical considerations===
====Are you familiar with any of the following tools or languages?====
* '''Sub&shy;&shy;version/Git''': Good knowledge, I already used both for other projects (subversion for the Boost.Check project and git for all my team project).
* '''C++''': Good knowledge even if I learn more each days on the idioms and features of this language.
* '''STL, Boost, Sdl''': I used a lot the STL and Boost but not really the SDL.
* '''Python''': I coded a SAT solver in Python, but I don't know much of this language.
* '''build environments (eg cmake/scons)''': Already used cmake, but I haven't a good knowledge of how it works.
* '''WML''': I started to study it the last year for my previous proposal. I know pretty well the common rules.
* '''Lua''': Never used.

====Which tools do you normally use for development? Why do you use them?====
I tried many tools such as Visual Studio, VIM, Eclipse,… But I'd say that my favorite is Sublime Text 2 for all the nice features it offers. However it's not an IDE, so the compilation/debugging must be made in the console. However the VS debugger is quite exceptional and I wouldn't be against using it only for this feature.

====What programming languages are you fluent in?====
The language I know the most is the C++. However I often programmed in C and Java too.

====Would you mind talking with your mentor on telephone / internet phone?====
Sure. I'm not used to speak with people in English but I think I can do it !
[[Category:SoC Ideas Addon Server 2013]]