SoC 2014 Spoffy Finishing and Enhancing the UMCD

This page is related to Summer of Code 2014

See the list of Summer of Code 2014 Ideas

This is a Summer of Code 2014 student page

Description

Spoffy - UMCD Extension Proposal

I intend to complete and extend the User Made Content Daemon produced by Trademark during GSoC 2013. This means implementing the basic functionality (Uploading, downloading addons, etc) in a secure manner. After this, I would continue on to extend it by adding administration tools, refactoring code, adding build scripts, adding cross-platform support (in principle), and other things (probably!). Outlined on this page is an in-depth proposal

More to come as I fill this out.

IRC

spoffy

Design Specification

NOTE: HEAVY WORK IN PROGRESS

This is the current design proposal for continuing work with the UMCD. This is likely to be a continous work in progress up until the Summer of Code deadline. Firstly, I intend to outline the series of steps involved. After this, I will outline a timetable before going into more information on each step. Hopefully, this means the document will get more detailed as you read on, allowing for a logical progression.

Steps

1. Bring the existing UMCD up to date with the latest Wesnoth build.
2. Refactor the existing server for Neev.
3. Basic Testing Client implementation.
4. Update the WML Protocol schemas.
5. Implement requests - Core functionality

Timetable

19/05/14	1. Updating the UMCD	None	Uni Exams 19 May - 7th Jun My Exams: 3 Days in this period
24/05/14	2. Refactoring to Neev	Optional Step, see description	None
01/06/14	3. Standalone Test Client	None	None
08/06/14	4. WML Schema Completion	None	None
15/06/14	5. Core Functionality	Should produce working server	None
01/07/14	6. Security and Encryption	Basic production ready server	None
10/07/14	7. Refactoring	Cleaning up produced code.	None
17/07/14	8. Server Administration	(For those with server access)	None
24/07/14	9. Client integration	Adding client to Wesnoth	None

NOTE: The Timetable is an approximation. Some scenarios are worst case, some are exact expected length. Hopefully it's an overestimation, and I can accomplish more than is listed.

NOTE2: During my exam period (19th May - 7th June) I will likely be quite slow with producing code. It may be my exams fall at the very beginning, in which case it would not affect SoC at all. On the timetable, I have considered this entire period as 'slow code'.

Details

Step 1 - Bringing the UMCD up to date.

The UMCD was, I believe, primarily coded during Summer of Code last year. At the time of writing, the asio_umcd branch is around 2100 commits behind Wesnoth master. The first step would be to rebase the UMCD onto master, avoiding merges where possible. Having briefly attempted this, there are a number of conflicts, so this would be the first task.

Step 2 - Refactoring the existing server for Neev

(Optional Step - Depending upon discussion with Trademark/Mentor) The existing server uses the boost.asio library for networking. Trademark has abstracted much of this into a simple networking library called Neev (available at http://github.com/ptal/Neev, or my fork at Spoffy/Neev. After discussion, it seems like it would be a better idea to use Neev than a custom abstraction over asio, as it will hopefully make the server more maintable in the long run. This would ultimately require more dicussion, and can easily be skipped if it's decided to not be required. Refactor is early to make it easier in the long run.

Step 3 - Basic standalone client implementation

DISCLAIMER: At the current point of writing, I am unsure of how the client will integrate with Wesnoth. I presume it will interface with Wesnoth's GUI OR be a standalone tool. I will attempt to clarify this on IRC. Until then, take this section lightly. Consider it a work in progress.

UPDATE: On studying the existing server, the client appears to be integrated into Wesnoth. This client will be a standalone tool for testing with reusable code.

A basic client is necessary to test the functionality of the server. I propose creating a test client (using Neev or existing asio) early so that it can be develop in parallel with the server, allowing for constant testing. The code would consist of a basic method for interfacing with the console, as well as a class of methods designed to be reused in Wesnoth itself.

At this point, the test client would simply be created and brought into parellel with the current state of the server. It would be a basic command-line client on Linux, however it would naturally be buildable on Windows as well.

Step 4 - Complete UML Protocol Schema

The current UML Protocol Schemas are incomplete, offering a small number of the schemas required to implement a functioning UMCD. Naturally, these need to implemented. Implementation details based on Trademark's existing work to follow.

Step 5 - Adding remaining server functionality

At this point, the prep for adding the core functionality (Adding, updating, deleting, downloading addons) has been completed. This functionality can thus now be implemented and develop in conjunction with the client library. This means that real-functionality can be tested as it's produced, rather than engaging in creating very fake scenarios Implementation details soon to follow.

Step 6 - Basic Security and Encryption

The server is now functional, capable of basic addon management. Huzzah! However, it needs securing to ensure no passwords get leaked onto the internet! Firstly, this ,means laying down SSL encryption over the existing network framework (Neev or Asio). This is one point where Neev would be an advantage, as we're guaranteeing modular and reusable code, though at the extra cost of time due to implementing SSL in Neev using Asio, then tweaking existing code to specify it wants an SSL connection.

Once SSL encryption is active, we can implement logging in with passwords. While implementation details will follow, I'm initially proposing sending the password over the SSL connection, then applying SHA512 and matching against a hash in the database. Further security could be added, such as encrypting the password, then sending the encrypted password over SSL and descrypting it on the other side. I need to research if this would be required.

Further implementation details to follow.

Step 7 - Refactor Refactor Refactor

The server is now, in theory, ready to see some basic use. Users can upload and download securely, as well as add and delete addons from the server. I will also have gotten, hopefully, a very good feel for the code by this point. That makes it a good time to refactor any bits of code I feel could be improved before continuing on. This will hopefully save time later.

No further information, as the implementation relies on knowledge I will gain during the project, although an example might be the environment code mentioned on the UMCD proposals page.

Step 8 - Console Administration

Allow the server host to locally view all registered modders, all available addons, last update, etc. and be able to delete, move, add and otherwise modify stored addons, probably in a way that doesn't modify the addon itself, in order to maintain integrity and try to prevent evil server owners from adding malicious files, or whatever it may be. More details to follow as I think of them. Emphasis at this point is on local administration (i.e, by people with server access).

Step 9 - Client Integration

The WML based communication protocol should be well tested at this point, given it was implemented in Step-4. The server should also have all the core functionality it needs for a release and it should be stable in terms of its interface with the client. Therefore this is a good time to integrate the client into the Wesnoth client itself ready for release.

I would initially propose maintaining the existing GUI interface and simply retying it to the client library that has been developing seperately. Whether the GUI was changed would depend on the functionality needed at this point in time, though I don't, at the point of writing, predict any changes, except perhaps with how addons are uploaded? A few more details to follow, as usual, though it depends heavily on the library at this point.

Other Possibilities

Remove UMCD from Wesnoth Repo/lower coupling - Instantiate as a submodule.

The UMCD is itself an independant entity, which communicates over the network with other entities (clients). It is not integrated into Wesnoth, but shares some dependencies.

Having the UMCD contained within the Wesnoth repo is, possibly, adding unnecessary coupling between the UMCD and Wesnoth itself. I would propose having the UMCD in its own repository, linked via Git submodules to the main repository (Either with the UMCD a submodule of Wesnoth, or vice versa). This would make the UMCD a standalone entity, and remove the need for it to be constantly rebased alongside the main code, the majority of which is unrealted the the UMCD itself. It would also allow for the UMCD to have its own build script, which makes sense, again for coupling reasons.

Downsides include having the client in a seperate repository to the server, although with submodules this should be transparent to the person building wesnoth, and the use of submodules allow

Implementation details to follow.

Questionnaire

1) Basics

1.01) Preamble I may tend to rant on a little bit in this. I hope it gives a better, more honest taste of who I am. If not, then let me know and I can make an abridged version!

1.1) Write a small introduction to yourself. Heyho! I'm Callum, a 1st year Software Engineering student in Southern England! (Well aware this is a public wiki. This will be updated in the actual application!). I'm an avid hacker and open source supporter, going to FOSDEM in 2014 and trying to use as much free software as possible! I also love Hackathons, having gone to 3 between February and March and organising another for late march! All of the code produced in these is available as free software on my Github page (which I encourage people to look at. Especially the Musical Packets hack, it's awesome IMO!).

1.2) State your preferred email address.

spoffeh@gmail.com

1.3) If you have chosen a nick for IRC and Wesnoth forums, what is it?

Spoffy - This will always be my nickname where I can get it! (See my email for what happens when I can't :( )

1.4) Why do you want to participate in summer of code?

I want to hack on more open source projects. While I do enjoy making my own from scratch, it'd be nice to actually chip in and get involved with larger projects. Sadly, the entry barrier to some of these can be quite high, between mailing lists, IRC, coding styles, learning the source, etc, and I rarely seem to have time between the other things I'm hacking to justify it! For me, Google Summer of Code really does help make that initial apparent investment seem more than worthwhile! The money also means I can spend my summer hacking awesome open source things, rather than working at a shop or taking an Internship at somewhere boring and/or propietary. That wouldn't be nearly as cool!

1.5) What are you studying, subject, level and school?

Software Engineer MEng, 1st year. I'll update the school on the actual application. Again, I'm well aware of the internet's scope.

1.6) What country are you from, at what time are you most likely to be able to join IRC?

England! I try to be on IRC whenever possible, but naturally I sleep. I aim for at least 7pm - 11pm GMT each day, and I'm usually on and off at various points between 10am and 12am. Soon (Or perhaps already, if I haven't changed this by the time its submitted) I plan on just rigging one up to run permanently on a remote server to log ALL the messages. :)

1.7) Do you have other commitments for the summer period ? Do you plan to take any vacations ? If yes, when.

I have a planned vacation abroad for two weeks around June time, however I plan on taking my laptop and a Pi, with the full intention of continuing coding regardless.

2) Experience

2.1) What programs/software have you worked on before?

I spent 2 years or so prior to University working on a game called Space Station 13 on the BYOND platform. I'm the head programmer for one of the game communities there, although I'm trying to phase out my involvement somewhat due to other commitments (such as SoC and Uni :) ).

Since coming to University, I've played with a variety of projects.. if played is the right word. Since Java is the main language used, I've obviously done a lot of work on Courseworks and extra projects in it, though these are minor things, often with quickly hacked code due to one-week deadlines. Unfortunately, this has also meant University has taken up much of my time, and due to the nature of the projects, left me with little to show for it! (I'll note some exceptions later).

I've also been to a number of hackathons and hacked together projects there, including Musical Packets and a few others (with Musical Packets being one I'm working on refining and looking to continue on and off). These are often made up of very quick prototype code, but are almost always working systems.

2.2) Have you developed software in a team environment before? (As opposed to hacking on something on your own)

As mentioned above, I've been a part of/head of the SS13 programming team for a good two or three years, which has varied between 2 and 5 active members in that time. I've also had group projects at University, and a number of hackathons, all of which have involved teams of two or more!

2.3) Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?

Sadly no, this is my first year! Though it should be an exciting one!

2.4) Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.

As above, I'm involved with Musical Packets (ish, it's only done by 3 people. Does that count as involved?), and SS13, with the latter being phased out. I usually only contribute to these when I have free time, with other things such as Uni taking priority.

2.5) Gaming experience - Are you a gamer?

Very much so! I have a rather large game collection, though I'm struggling to find time to play them between all the hacking!

2.5.1) What type of gamer are you?

I'm tempted to say casual? I have lots of games, but I don't often play them at the moment. Except Tetris. That's always strangely addicting. Can one be a casual hardcore tetris player?

2.5.2) What type of games?

TETRI... oh, other games? Various, I enjoy almost every variety of game. Dating sims are a notable exception... Anyway. Puzzle games are always awesome because of how intellectually challenging they are. I've spent long stints playing Tetris and Puzzle Pirates. I also once spent 12 hours or so optimising a SpaceChem puzzle. I've basically refused to play since, given I know I'll get sucked in for another puzzle. I quite enjoy Turn based strategy as well, although I find many games fail to get it quite right, often requiring super in-depth knowledge of vast tech trees (Space Empires for example.). Wesnoth is one exception to that, more on that later I suspect!

I do also enjoy sandbox games. Anything that lets you stretch yourselve creatively while also posing a challenge. I find Minecraft often fails in the latter, although Space Engineers and other ship-building or colony-building games like Hazeron are often entertaining.

Artemis Bridge Sim is also a game that I rarely get to play, but stands out as being decidedly awesome. How would this even by categorised.

Also, space games. Always.

2.5.3) What type of opponents do you prefer?

I'm guessing AI or Human? Depends on the game. Definitely human in an FPS. Puzzle games, I like to challenge myself, or another human player. AIs tend to feel too strong or too weak. In terms of the type of person I like to play with, those who are competitive, but also equally in it just for fun!

2.5.4) Are you more interested in story or gameplay?

Depends. Usually gameplay, as that also tends to add replayability to a game. Stories have though, when they're good, kept me hooked for an entire game session though. Finding the story/gameplay balance is always difficult though.

2.5.5) Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.

Briefly, I remember playing it a month or so back. I remember playing a few missions and the tutorials and contemplating how balanced and strategy focused the game felt. It's admirable how well it does in that regard. When I find some free time when I'm not hacking, it's definitely on my play again list.

2.6) If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for GSoC. If you have gained commit access to our repository (during the evaluation period or earlier) please state so.

Not as of yet, though I'm attempting to contribute and refine the NEEV library that is intended to replace ASIO in the UMCD. This seems like a more productive use of time, as much as anything.

3) Communication skills

3.1) Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.

Perfectly fluent? I'm a native English speaker, and I love creative writing! Hopefully this questionnaire as well as IRC interactions give a good feel for this.

3.2) What spoken languages are you fluent in?

English... that's about it. I know a little Spanish and French, technically? Oh, I also speak Python.

3.3) Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.

My first role on SS13 was an administrator. It was that path that actually led me to coding. We've also had a number of DOS attacks from players that have been... disagreeable, as well. I'm used to dealing politely with that sort of person. Occasionally, there's a genuine issue behind what they're saying as well, so it's important to get that out.

3.4) Do you give constructive advice?

I do my best to! Though sometimes, I also feel the best advice is to not give much advice. People don't learn if you give them everything, though equally, they sometimes need advice to help them learn. I do my best, but it's a fine balance.

3.5) Do you receive advice well?

Again, I do my best! I like to think I do, and I always welcome ways to improve, though occasionally this does lead to (fun) debates.

3.6) Are you good at sorting useful criticisms from useless ones?

I'd like to think so, at any rate. I'm used to having non-coders try to criticise code or ways of working (For example, saying 'It's easy, it shouldn't take so long!' etc). I'm not sure how to elaborate more here, so onwards!

3.7) How autonomous are you when developing ? Would you rather discuss intensively changes and not start coding until you know what you want to do or would you rather code a proof of concept to "see how it turn out", taking the risk of having it thrown away if it doesn't match what the project want

I prefer to dicuss changes as much as possible, though naturally it depends on the scale. If it's something I feel I can sort out myself and doesn't concern people much at all, I'm perfectly happy to go away and come back with a proof of concept or

4) Project

4.1) Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?

I selected the User Made Content Daemon! I want to focus on... all of it? I'm not sure. It's an interesting project that looks fun, but more under that for 'Why'

4.2) If you have invented your own project, please describe the project and the scope.

[Left Empty]

4.3) Why did you choose this project?

Two major reasons. One: It's a standalone project. It's not tied into Wesnoth itself, which means a much smaller code base to learn and more autonomy when contributing, as well as control/flexibility with where it goes. This really, really appeals to me.

Two: Networking. This is one of my primary interests in terms of Software, as well as overall software architecture (Modular, flexible designs, etc). I saw the opportunity to engage is a project that is significantly about network communication I jumped at it. It was for a game too! I love developing for games! As much because of the community as anything, I always find game communities to be passionate and interesting. The project here is certainly cool enough I'm only making a single application to focus on it!

4.4) Include an estimated timeline for your work on the project. Don't forget to mention special things like "I booked holidays between A and B" and "I got an exam at ABC and won't be doing much then".

Naturally. Wait, hey, this isn't a question! It'll be below, along with other project details.

4.5) Include as much technical detail about your implementation as you can

This will also be below. It's naturally deserving of its own section!

4.6) What do you expect to gain from this project?

A fully working UMCD. Experience. An epic time. Maybe not in that order. Maybe 'A fully working UMCD' seems like a weird thing to put, but I always gain satisfaction from a cool, working hack. The UMCD fits that category.

4.7) What would make you stay in the Wesnoth community after the conclusion of SOC?

Firstly, and this is important to me, that lack of a 'need' to contribute. I like to be able to choose when and where I do, and I think Wesnoth offers that. Secondly, if I don't finish the UMCD. Maybe finish is the wrong word, but I hate incomplete things. I like to have things functioning, and functioning well. Thirdly, the people. If I get to know awesome people here, naturally I'll want to stick around!

5) Practical considerations

5.1) Are you familiar with any of the following tools or languages?

Git (used for all commits)

Git is amazing. Yeah, I used it all the time, and have all my University work under Git version control.

I also use it on all my open source hacks.

C++ (language used for all the normal source code)

I've never done a large project with C++, but I know the language and I intend of adding example code and practicing between now and summer of code.

STL, Boost, Sdl (C++ libraries used by Wesnoth)

I'm fairly familiar with the STL, certainly with abstract data structures in general. I know a bit about Boost Asio (Though this is a focus of mine with respect to working with C++ at the moment). I'm not familiar with Sdl at all.

Python (optional, mainly used for tools)

I'm fluent in Python, it's my main Hackathon language. And such a nice language to write in too.

build environments (eg cmake/scons)

I know a bit about cmake, enough to make basic make files. This is again something I'm actively working on improving at the moment. Scons I know little to nothing of.

WML (the wesnoth specific scenario language)

I know nothing of WML really, except that from the examples I've seen it looks like XML with square brackets, or at least very similar.

Lua (used in combination with WML to create scenarios)

Fluent in Lua, at least I was last time I checked. Haven't written in it in a while, but it was also the first language I ever learnt!

5.2) Which tools do you normally use for development? Why do you use them? Git firstly. Its flexibility with respect to branching and checking out different commits, as well as its ability to quickly pull and push to other repositories make it awesome. Also the fact it has a modular client!

Vim is my go-to text editor for smaller projects, with a number of plugins for Syntax checking, etc. I can't do without Vim keybindings, they make navigating the code so much easier and faster, as you never need to touch a mouse!

Usually however, for larger projects, I like an IDE for code completion and real-time error checking. When I start a new project, there's usually a small internal debate on using an IDE vs Vim. I usually settle for an IDE with a vim keybindings plugin. Hence IntelliJ or PyCharm tend to be gotos, as they have an easy to install but powerful vim plugin.

If Linux is a tool, I always use Linux, as command-line linux is incredibly flexible and powerful.. plus free software!

Otherwise, it really does depend on the project.

5.3) What programming languages are you fluent in?

I'm fluent in Python, Java and Lua. Python and Lua tend to be my gotos for most projects. Also DreamMaker, but it's BYOND specific. I'm competant in C++, C, HTML, Javascript and Awk, though I don't use them often. I have a knowledge of D (Which I love, I just need a project to use it on!), Bash and probably a couple of others I've forgotten.

5.4) Would you mind talking with your mentor on telephone / internet phone? Not at all, I'm perfectly happy talking over phone. IP is preferred though, as I like to have my hands free to type.