SummerofCode Orfest

This page is related to Summer of Code 2010

See the list of Summer of Code 2010 Ideas

This is a Summer of Code 2010 student page

Project: SoC Ideas Network Stack Rewrite

Description

Orfest - Rewriting network stack

gna.org : orfest
irc : orfest

I am really interested in rewriting network stack using boost::asio. This task will not only improve the current network stack implementation but will also bring new features such as proxy support, configurable timeout values for connections, connection multiplexing on the client.

Please see more details on my page.

IRC

orfest

SoC Application

Rewriting network stack using boost::asio

Questionnaire

March, 22. Committed a patch fixing the statistics uploader. April, 7. Commited another path adding proxy support with no and basic authentication to the statistics uploader.

Please, see answers to the questions about the network stack project at the bottom of this page.

1.1) Write a small introduction to yourself.

My name is Nikolay Kurtov, I'm 21 years old, young, enthusiastic and experienced enough. This year I've represented my university at the ACM ICPC World Finals. Unfortunately, we didn't get any medals. I'm looking forward to get a lot of fun working with the Battle of Wesnoth developers this hot Summer of Code :)

1.2) State your preferred email address:

nkurtov@gmail.com

1.3) If you have chosen a nick for IRC and Wesnoth forums, what is it?

orfest

1.4) Why do you want to participate in summer of code?

I'm getting more and more engaged into Open Source. Summer of code is a great opportunity for me to significantly contribute to Open Source and gain a lot of experience in programming and working with community.

1.5) What are you studying, subject, level and school?

I'm a Master student at Novosibirsk State University, Faculty of Information Technologies. My bachelor diploma is about hardcore optimization of a C++ library for shared-memory parallelism.

1.6) What country are you from, at what time are you most likely to be able to join IRC?

I'm living in Novosibirsk, Russia. My time zone is UTC+7. Every day I'm online for 4-8 hours, mainly at night.

1.7) Do you have other commitments for the summer period ? Do you plan to take any vacations ? If yes, when.

I'm going to have an exam session at the beginning of June. The exam session will consist of 2 exams. Also I have a part time job, I'm an intern software engineer at Intel corp.

2) Experience

2.1) What programs/software have you worked on before?

• FlexyPool - a summer school on the first year of studying. There were four young and enthusiastic of us, who wanted to make a very original game. Personally I was responsible for the Physics Engine. The project was not released and only developers have its beta version somewhere.
• Santa's Quest - a casual Christmas game developed by the same team as FlexyPool. I was responsible for level generation, balance, high scores etc. We even were firm to publish this game, but didn't succeed.
• Eclipse DLTK - for half a year I worked at xored Inc. as QA, performing GUI testing, writing unit tests, etc. Submitted a lot of bugs and several patches.
• TSP Flaming - a tiny academic project made for studying the method of Simulated Annealing. At that moment my interest to Open Source was growing and I wanted to try the process of managing a project at the SourceForge, that's why I made a full path to host it there.
• Apache Harmony - a student project of bringing Edge Profiler to Jitrino.JET compiler. As a result I developed and submitted a patch. Results were published at the International Student Scientific Conference in Novosibirsk and were awarded 3rd degree diploma.
• Intel(R) Concurrent Collections for C++ - an intern software engineer researching high-level performance optimizations.

2.2) Have you developed software in a team environment before? (As opposed to hacking on something on your own)

• Both FlexyPool and Santa's Quest were developed by a team of 4 developers, we called our team MadmileGames :)
• xored Inc. is a hardcore outsourcing team. At that time it was a team of about 20 developers.
• Software Group of Intel corp. is just huge :) I'm constantly working together with people from USA and Germany.

2.3) Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?

I haven't participated to the Google Summer of Code before. Last year I really wanted to, but missed the deadline :)

2.4) Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.

None

2.5) Gaming experience - Are you a gamer?

Yeeah, since yearly childhood I had a Spectrum (ZX Delta) and spent a lot of time gaming. Then I got Pentium 100 and I've got into WarCraft 2, Duke Nukem 3D, Worms Reinforcements, etc.. Until recently I haven't got a good internet connection, so most of my experience is Single Player. So, I've been playing in LAN only: Unreal Tournament, HoMM3, Disciples 1, Teeworld, Allods 1,2,3. I've never been playing MMORPG.

2.5.1) What type of gamer are you?

I'm easily engaged into a game and can just forget to go to sleep.
I like games where every your step should be perfect just not to perish.
Also I rarely think about defense, I'm all in offense.

2.5.2) What type of games?

• Strategies: HoMM 3,4, King's Bounty
• Shooters: Unreal Tournament, Teeworlds
• RPG: Fallout 1,2, Allods, Might&Magic 3,4,5,6,7,8
• and I don't like fightings, but there is just 1 that I love. I've spent 1.5 years playing Guilty Gear XX #Reload, because it is amazingly beautiful, dynamic and hardcore.

2.5.3) What type of opponents do you prefer?

I prefer aggressive opponents, I do not like campers.

2.5.4) Are you more interested in story or gameplay?

For me the most interesting is gameplay.
I'm always playing at maximum difficulty level and can spend a huge amount of time trying to complete the game on this difficulty.
My last achievement was completing 2 episods in DOOM at Nightmare %)

2.5.5) Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.

Frankly speaking, I've been playing Wesnoth just for several hours and haven't tried multiplayer. But I'm looking forward to complete a campaign and then go playing on the internet.

2.6) If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for GSoC. If you have gained commit access to our S­­V­­N (during the evaluation period or earlier) please state so.

March, 22. Committed a patch fixing the statistics uploader. April, 7. Commited another path adding proxy support with no and basic authentication to the statistics uploader.

3) Communication skills

3.1) Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.

Recently I've got a certificate about completing Advanced English course.
I'm constantly communicating with American colleagues both by e-mail and by phone and we understand each other perfectly.

3.2) What spoken languages are you fluent in?

English, Russian, Tokipona:)

3.3) Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.

Yes, that's ok. I'm well-tempered.

3.4) Do you give constructive advice?

Yes, I always try to use logical arguments to support my ideas.

3.5) Do you receive advice well?

Yes, I'm paying attention to each advice received and I'm thankful to a people who share their knowledge with me.

3.6) Are you good at sorting useful criticisms from useless ones?

Yes

3.7) How autonomous are you when developing ? Would you rather discuss intensively changes and not start coding until you know what you want to do or would you rather code a proof of concept to "see how it turn out", taking the risk of having it thrown away if it doesn't match what the project want.

I'm rather autonomous but it'd still be better if mentor would monitor my progress from time to time.
I tend to search for answers in the sources first and only then to ask a person who knows the answer. About coding I prefer writing 1 or 2 tiny prototypes, it helps me to really get into sources and tools used and also to find problems I haven't noticed yet.

4) Project

4.1) Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?

Yes, I really want to rewrite the networking stack to boost::asio. I think one of the main aspects is to make a good Object-Oriented design and to make it easily extendable. I understand that rewriting itself is a huge task and there is a risk that I will not have time to implement other features declared.

4.2) If you have invented your own project, please describe the project and the scope.

[At the google proposal this point is removed because I'm applying for a project proposed by wesnoth developers] I like things working fast and stable that's why I'd like to increase wesnoth's performance, doing profiling, refactoring "hot spots", analyzing memory consumption, making it even more accessible for old computers. Currently, htop tells me Wesnoth is consuming 300MB of Virtual Memory. It's a very good number but I believe it still can be reduced :) Also I'm rather good in shared-memory parallelism and maybe when I become very much acquainted with the wesnoth's design, I'll bring parallelism to other components except networking.

4.3) Why did you choose this project?

The task about networking stack is really interesting for me, I can experience the full power of boost::asio, understand details how the game is designed. I still want to develop games and maybe I will sometimes join Innova Systems or so, this experience will be highly valuable.

4.4) Include an estimated timeline for your work on the project. Don't forget to mention special things like "I booked holidays between A and B" and "I got an exam at ABC and won't be doing much then".

Let's number all the summer weeks w1-w13
w1 - installing VMs for different OSes, configuring authorizing proxies

w2-w6 - modifying client:

• w2-w3 - rewriting network code to boost::asio, adding proxy support (I assume, the proxy support is implemented beforehand)
• w4-w5 - work on moving network code to a separate library and testing the client
• w6 - documenting all the client changes

At the end of w6 I'll be ready to present the rewritten client

w7-w11 - modifying server:

• w7-w8 rewriting network code to boost::asio (doesn't even require proxy support :)
• w9-w10 - testing the server, improving performance, throughput
• w11 - documenting all the server changes

At the end of w11 I'll be ready to present the rewritten server

w12 - Google Summer of Code wrap-up

4.5) Include as much technical detail about your implementation as you can

Please see the section of answers below.

4.6) What do you expect to gain from this project?

As I mentioned in "Why did you choose this project", I going to gain a lot of practical game developing experience and network programming experience.

4.7) What would make you stay in the Wesnoth community after the conclusion of SOC?

More challenging tasks :)

5) Practical considerations

5.1) Are you familiar with any of the following tools or languages?

   * Sub­­version (used for all commits)

Yes, we are using S­­V­­N in our project at Intel

   * C++ (language used for all the normal source code)

Yes, I've been coding on C++ for 5 years. All ACM competitions I also write using C++.

   * STL, Boost, Sdl (C++ libraries used by Wesnoth)

I know STL perfectly, I'm using it everyday, especially on ACM competitions
I've been several boost features in academic projects: threads, bind, any, asio, graph. So, I'm not proficient at it, but I really want to experience more of it :)
Unfortunately, I know nothing about Sdl

   * Python (optional, mainly used for tools)

I'm always writing small scripts using python, but I never go beyond defining 1-2 classes without behaviour.

   * build environments (eg cmake/autotools/scons)

I like cmake. Have never worked with autotools and scons.

   * WML (the wesnoth specific scenario language)

No

   * Lua (used in combination with WML to create scenarios) 

No

5.2) Which tools do you normally use for development? Why do you use them?

In Intel I'm using Windows and MSVS, because it's a corporate-wide used environment
at home I'm using Gentoo, Vim, Qt, Cmake, Valgrind. I really like Open Source and at home I'm really Open :) Vim with ctags is incredibly powerful, Qt is very well-designed and easy to use, Cmake is a tool used by KDE, Valgrind allows me to find all kinds of memory errors.

5.3) What programming languages are you fluent in?

C++, Java, C#, C

5.5) Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail. If you are willing to do so, please do list a phone number (including international code) so that we are able to contact you. You should probably *only* add this number in the application for you submit to google since the info in the wiki is available in public. We will *not* make any use of your number unless some case of "there is no way to contact you" does arise!

I don't really think there would be such a situation. Also I don't have a stationary phone, mobile only. Of course I'll add this to my SoC application

6) Answers

6.1) what proxies (authentication types) we can support, if we code a network stack using boost::asio, and how hard it'll be ?

As it was discussed previously at #wesnoth-dev, it is reasonable to support basic and digest proxies.
NTLM proxies are also widespread but NTLM support will cause extra coding efforts and can be avoided. It's possible to use an external wrapper like cntlm that makes the NTLM proxy visible as a Basic proxy.
Basic and Digest authorizations are described at the RFC2617

Also, I'm considering that any proxy-server implements CONNECT method, that is introduced in the RFC2616
Proxy support requires additional actions only at the moment of connecting to the server. If CONNECT is accepted then the proxy-server works as a simple tunnel and read/write operations require no modifications.
The most easy way to implement the described approach is introducing own socket-classes, aggregating boost::asio::tcp::ip::sockets and implementing own connect() methods.
This approach will not add coding overhead except defining own socket classes themselves.

6.2)what general architecture for the client you're going to use ?

I'd like to keep client's architecture simple.
A separate thread sends and receives data from the server. Data can be stored in incoming/outgoing data queues. As soon as a message arrives, the thread notifies the application about a new server message.
I'd like to notice that the described architecture naturally fits into boost::asio::io_service capabilities which encapsulates queues mentioned.

6.3)what general architecture for the server you're going to use ?

I propose using a widespread and well-documented pattern Proactor that is effectively implemented by the Boost.Asio library. As I mentioned, the pattern is widely spread and that's why I'm not expecting any problems with implementing it. Also this pattern applies several serious benefits highlighted at the page mentioned above. I'm answering this short, please ask me if you want more details.

6.4)how you'd allow the wesnoth client to multiplex connections to wesnoth server ?

A client owns separate Games objects and each of them naturally has its own network-layer-data: sets of sockets, connections, etc.
The ability to multiplex connections will come for free.

6.5)how you'd make it possible to use wesnoth network client with a different c++ backend?

The whole network layer will be encapsulated in a separate module.
All network data objects are to implement defined interfaces and to be instantiated by calls to factory methods/factories. So, first of all, network API should be defined. And then the network module should implement the API defined.

6.6)what problems and potential pitfalls do you see in the task?

The design is critical in this task. A not-good-enough design can neglect results of all the task. Another risk is unsufficient network code testing. I desperately believe that boost is absolutely portable, but it doesn't make testing unnecessary. Rewriting such a low-level code is dangerous to introduce regressions.

Functional requirements:

0. Support existing functionality
1. Proxy
2. Timeouts
3. Encapsulated in a single object
4. Network layer to be moved to a separate library

Obstacles:

0.
a. Complicated connect/reconnect/pending logic
b. server is multithreaded
c. excessive use of low-level network API like send_buffer throughout the code.
d. testing is complicated and involves different OSes

1.
a. host name, port to be specified separately
b. proxy authorization may be required
c. proxy authorization method is not known in advance
d. authorization credentials to be specified separately
e. additional manipulations required on connect, disconnect, reconnects
f. testing is complicated and involves configuring proxy servers

2.
a. No idea about desired timeouts flexibility - should they differ for different operations, should they differ between connections?
b. each socket operation to be wrapped in a call to asynchronous timer. timers used should be disarmed on successful completion - races are possible (in case if the operation finishes just before the timer deadline)
c. are they really required? socket.cancel() functionality solves many problems. will it be of any help?

3.
a. network.cpp employes poor programming techniques:
i. static variables, variables defined in the scope of network.cpp and these are key variables like connections, sockets, etc.
ii. network layer API is accessed throughout the code, especially low-level functions like send_data() and nconnections().

4.
a. motivation is not clear (no other wesnoth-client-network libraries known)
b. same as 3a
c. windows has its own pitfalls regarding dynamic libraries

Solution:

0.
a. ask author about details which are not obvious
b. don't change server logic except of fixing bugs and synchronization issues
c. avoid changing logic
d. use VirtualBox to run wesnoth on different OSes

1.
a. modify settings window, use environment as a temprorary solution
b. implement basic and digest authorization schemes. document solution for wrapping ntlm into basic/digest scheme
c.
i. implement simple http-proxy-server response parser to determine desired authorization scheme
ii. ask user
iii. try all
d. use the same proxy settings window mentioned in 1a.
e. employ virtual polymorphism - define an abstract ISocket, inherit DirectConnectionSocket and ProxySocket. ProxySocket overloads connect/disconnect methods to establish CONNECT to the game server. Using such sockets will add no coding overhead.
f. configure Squid3 on localhost or another computer for the sake of testing.

2.
a. continue asking developers about the idea
b. study boost samples about the race problem
c. continue asking developers about the timeouts idea

3.
a. redesign existing network code introducing network object that will passed around together with such an omnipresent object as game_config

4
a. continue asking developers about the idea
b. Do encapsulation (3.) first, then adopt to be a library
c. Remember troubles experienced during dlls development

The following code will be changed:

0.
* Code dealing with SDLNet library:
- network.cpp
- network_worker.cpp
* Code invoking funtions of namespace network. e.g.:
if (network::nconnections() > 0)
network::disconnect();
After network is encapsulated in an object, such an invokation won't be possible. It will have to be changed to something like game->get_network()->disconnect()
- addon_management.cpp
- dialogs.cpp
- game_preferences.cpp
- menu_events.cpp
- multiplayer_connect.cpp
- multiplayer.cpp
- multiplayer_ui.cpp
- playcampaign.cpp
- play_controller.cpp
- playmp_controller.cpp
- playsingle_controller.cpp
- playturn.cpp
- random.cpp
- replay.cpp
- server/campaign_server.cpp
- server/game.cpp
- server/player_network.cpp
- server/proxy.cpp
- server/server.cpp
- tests/test_network_worker.cpp

1. Modify preferences window:
- game_preferences.hpp
- game_preferences.cpp
Use md5.cpp and sha1.cpp to implement digest proxy authorization
Basic proxy authorization requires base64 algorithm, already implemented in http_file_uploader.cpp. Move the algorithm to a separate source file.
Provide a verbose description of using ntlm-wrappers such as cntlm at wiki
Define socket interface duplicating boost::asio::ip::tcp::socket interface in
- abstract_socket.hpp
Provide 2 socket implentations:
- network/direct_connection_socket.Xpp
- network/proxy_connection_socket.Xpp
both socket implementations aggregates boost socket that performs the actual work.
proxy_connection_socket will stand for dealing with proxies and all authorization schemas
For the sake of ability to move network layer in a separate library, provide socket_factory interface and implementation:
- abstract_socket_factory.hpp
- network/socket_factory.Xpp

2. To be discussed

3. redesign network.cpp introducing a network class, containing all kinds of variables currently defined inside network.cpp.
passing this object around will require a number of changes in classes

4. Major buildfiles changes to build a dynamic library (wesnoth_client_network.so/.dll) and link client against it