SummerOfCodeProposal corn

Introduction

Hi, I am Greg Shikhman and I'd like to work on and improving stats.wesnoth.org.

Preferred Email

cornmander@cornmander.com

Nicknames

IRC - corn
Wesnoth Forums - cornmander

Why I want to participate

Working on Wesnoth for GSoC would be my first experience working on an large and public open source project. GSoC is a structured way (similar to an internship at a company) for me to get into the development of this project.

Studies

I am a high school senior who will be attending an accredited university this fall. This section will be more definite when I decide where I want to go :)

Patches

https://gna.org/patch/?1153 https://gna.org/patch/?1149

1 or 2 more patches to come related to bug #13094.

Experience

Last summer I had a full time summer internship at Morgan Stanley, the financial company. My primary accomplishment was the creation and maintenance of a statistics interface for their networking hardware. I created a system of scripts that handled statistics acquisition, and I wrote a java servlet that showed these statistics, with nice looking graphs via RRDTool. The tool was put into production shortly after my intership ended.

A smaller task that I also handled was the creation of a real-time packet rewrite/editing script on the network hardware, to support and upgrade legacy software.

Reference is available on request.

As part of a class, I have also created a real time rasterizer, with model loading, animation, and texture mapping. A scripting language for inputting animation sequences was also developed as part of the project. Source code, along with instructions for compiling are available at http://cornmander.com/dragonballx.tar.gz .

I also run my own server at http://cornmander.com . It is a colocated Gentoo Linux box. I have working sysadmin experience.

Programs / Software

python
LaTeX
GNU make
gdb
vi(m)
linux
subversion
gcc
grep, sed, awk, other standard shell hacker tools :)
bison, flex
libraries: opengl, sdl, opencv
languages: c, python, php, java, latex, html, tcl

Team Environment Experience

My rasterizer was created as part of a team project. I managed and handed out tasks to my teammates, and tried to keep them updating their TODOs.

My internship at Morgan Stanley was done under the supervision of my manager. Although I was working on my project alone, I took suggestions and feature requests from the end users.

Open Source

Involvement

I've reported bugs to the Gentoo Linux project, and I am credited for reporting a minor security bug: http://www.derkeiler.com/Mailing-Lists/Full-Disclosure/2007-08/msg00364.html .

I follow the development of ioquake and mplayer, but I haven't had a chance to contribute to either project. I may try to enter the ffmpeg (an integral part of mplayer) GSoC process next year.

Gaming Experience

I have been a PC gamer since late elementary school. I am a hardcore gamer primarily interested in FPS games - this is what sparked my interest in working on cutting edge graphics tech like rasterization. My current addiction is TF2. I also appreciate RPG games. I played Wesnoth for a few weeks with my friends after the Wesnoth 1.5 release, and I will start playing again now.

What type of Gamer

I used to play video games for a few hours per week, but school has cut it down to 2-4 per week. Nevertheless, I'd categorise myself as a hardcore gamer.

Preferred Opponents

I like playing versus friends, or at the very least versus people. AI isn't good enough to adapt and get better as you play, and there is always the joy of rubbing in your victory to the face of a human opponent.

Story or Gameplay?

Gameplay is king. I am used to playing multiplayer games, and the game mechanics are much more important than plot are backstory. Nevertheless, I do enjoy reading a good story or being a part of it in a single player campaign. Co-op support is an interesting idea.

Have I played Wesnoth

I played 1.5 multiplayer for a few weeks, and played through the Heir to the Throne campaign.

Communications

English

I am a native english speaker, and I am extremely fluent in Russian as well, although I can't read or write it.

Player Interaction

I ignore trolls, and I haven't been involved in player communities for a while. I lurk on forums, but I talk on IRC.

Constructive Advice

I take criticism well, but sometimes I am impatient when I don't see the point to a suggestion/requirement.

Receive Advice?

My priority is getting the job done. I don't take criticism personally.

Sorting out Criticism

The people I work with are typically more experienced and smarter than I am. Although I have the urge to question a suggestion, I'll never dismiss a suggestion without giving it some consideration.

Project

Which Project

I want to improve the stats server.

Why

I have existing experience with displaying stats, and the freedom of tools and programming languages should make it easy for me to hit the ground running.

Timeline

Technical Details

Frontend will be written in python with the turbogears application platform (for the nice templating language support). Databases for stats data and any user/configuration settings for the frontend will be in MySQL. Charting will be done using Google Charting API.

Changes will be made to the wesnoth client so that it keeps an array or a map where a list of dead units is kept for each tile. This data will be sent along with stats so that killgraphs can be generated. A program will also be written to take wesnoth maps, generate PNG files of them (if there isn't already such functionality) and color in each tile according to how many units were killed on it. This will be generated nightly and used for the killgraphs.

The stats interface will scale up to millions of rows by operating on a randomly taken subset of the data grabbed nightly. If the subset turns out not to have data for a particular set of filters, the interface will revert back to the bigger database.

Goals

PRIORITY	SUBPROJECT	RESULT	PROGRESS
MUST	Basic stats.wesnoth.org website	Create a better formatted version of my prototype at http://cornmander.com:9090	Completely rewrote the code for generating graph pages. Everything is driven by a database and new graphs take less than a minute to create. I still need to create a page for adding new graphs. <- (edit 8/17/09): still haven't thought of new pie/line/bar graphs, but I have two ideas for hexmaps (aka killmaps) that would be useful: most captured villages, most traveled tiles.
MUST	Add new stat windows to the website	Created a recruited units window, a dead units window, and a unit by level breakdown in addition to improving the things currently in the prototype. Add date ranges to all the charts on the site.	Incomplete. Added a killmap statistics page but I haven't added any more pie charts or line graphs. Bar graphs are still unsupported.
MUST	Improve filter behavior	Make the filters automatically remove choices with 0 entries based on currently selected filters	Not being done. This may be too performance-intensive to accomplish.
MUST	Remove outlier data	Give statistics for the median 80%, leaving out the outlying 10% below and 10% above.	Still not done. A discussion that came up on IRC was to allow developers to set a wml variable designating their campaign as beta - not to be tracked. This should get rid of some outrageous values (-9000 gold at start turn, for example). A statistical approach to getting rid of outliers still needs to be created, however.
MUST	Add scalability	Make the statistics presentation scalable to the millions of rows currently on stats.wesnoth.org. I plan to do this by creating a 10k random-row table subset of the data in a nightly cronjob and using this by default for all queries. If a query gives under CONSTANT_NUMBER results, the site will use the main database instead.	Done. I created a script that generates subset tables of 10k, 100k, and 1000k rows. I wrote a wrapper around my SQL queries that tries these tables first and checks the size of the result (using a library of evaluators). If the result is too small, it queries the next largest size table. Queries take less than a second on average.
MUST	Kill maps	Create a section of stats that gives a breakdown by tile on a scenario of the units killed on that tile. The presentation will be a jscript-enhanced map where a user can hover over a tile and see a percentage breakdown of the units killed on that tile. In addition to website coding, significant changes must be made to stat collection.	Done. Added the code to the wesnoth client to send gzipped maps and kill event data. Also added code for a '--screenshot' parameter that allows a server without X11 to generate map screenshots using the wesnoth client. I implemented an extremely tiny WML parser to parse upload logs on my server, wrote a script to generate google maps tilesets from map screenshots (generated on server), and wrote a frontend that has filterable killmaps.
MUST	CSS and Design	Integrate the look of the stats page with the rest of wesnoth.org . Make the website user-friendly	Done. It was pretty simple - I just named my div tags according to the wesnoth.org site and imported the stylesheet.
MUST	Email subscription to charts	Create a login system so that users can subscribe to particular charts, and receive daily emails for comparison. This will be useful for basic AI regression testing, ex: run 10 different campaigns with an AI during a particular day and compare with the charts of the previous day	Not done.
GOOD	Geographic data	Not done.
GOOD	Get MP data	Partially complete. MP logging is enabled for AI vs AI matches but the log format doesn't have particularly useful stats. I still need to add more log data so that useful graphs can be made (faction win percentage, average end turn).
GOOD	GUI or scripting interface for hooking in new stat pages	Create an interface for looking at new stats. User will have a choice of several graph types and what data sets to use for x and y axes.	Done. This is the database driven graph stuff.

Future work that I will do on this project will be focused on improving statistical/graphical analysis of AI vs. AI games.

What I expect to get from this Project

I want to have done a substantial open source contribution. This project will be my first coordinated team effort and hopefully help me get started on other OSS projects. It will also expose me to the process of maintaining a large, polished product.

Would I stay with Wesnoth after GSoC?

I would maintain my code for Wesnoth. I think that I will continue to contribute code in unrelated areas to the project after I am done the project as well.

Practical considerations

Familiarity with Tools

Subversion - I know how to use svn, and I run it on my own server as well.
C++ - I know C and Java very well, but I don't know C++. I think I can learn it over the course of a week or two, however.
Python - I know python very well.
Build Environments - My development environment is linux with GNU make. However, I have used MSVS before but I am not familiar with their solution files. I have a pretty good idea of how to use SCons now.

Development Tools

I use vi, xterm, svn, and GDB. I dislike working in IDEs because of the overhead and effort of getting your project set up properly in them.

Fluent Programming Languages

Java
C
Python
PHP

I also know LaTeX and HTML, markup languages.

Fluent Spoken Languages

I am a native english speaker. I also am fluent in Russian, but can't read or write it.

Hours of Availability

Until the end of June, I am available from 6 PM EST (22:00 UTC) to 3 AM EST (7:00 UTC). I will either be available during the entire day of 9 PM EST (1:00 UTC) to 10 PM EST (21:00 UTC). I expect to put in full work days of 8 hours on this project.

Phone / Internet Phone Conversations

I would prefer written electronic communication like IRC or email, but I am OK with talking over the phone. It's a bit frustrating to explain your code over the phone, however.