The Battle for Wesnoth Wiki - User contributions [en]

Description

2011-09-01T19:25:50Z

Cjhopman: /* What is Battle for Wesnoth? */

{{Description/Translations}}
== What is Battle for Wesnoth? ==

The Battle for Wesnoth is a turn-based tactical strategy game with a high fantasy theme.

<div class="thumb tright"><div>
[http://www.wesnoth.org/images/sshots/wesnoth-1.6-4.jpg http://www.wesnoth.org/images/sshots/wesnoth-1.6-4-175.jpg]
<div class="thumbcaption">Delfador's magical lightning strike</div></div>
</div>

<div class="thumb tright"><div>
[http://www.wesnoth.org/images/sshots/wesnoth-1.6-5.jpg http://www.wesnoth.org/images/sshots/wesnoth-1.6-5-175.jpg]
<div class="thumbcaption">An ambush in a campaign</div></div>
</div>

Build up a great army, gradually turning raw recruits into hardened veterans. In later games, recall your toughest warriors and form a deadly host against whom none can stand! Choose units from a large pool of specialists, and hand-pick a force with the right strengths to fight well on different terrains against all manner of opposition.

Wesnoth has many different sagas waiting to be played out. Fight to regain the throne of Wesnoth, of which you are the legitimate heir... step into the boots of a young officer sent to guard a not-so-sleepy frontier outpost... lead a brutal quest to unite the orcish tribes... vanquish a horde of undead warriors unleashed by a foul necromancer, who also happens to have taken your brother hostage... guide a band of elvish survivors in an epic quest to find a new home... get revenge on the orcs by utilising the foul art of necromancy...

200+ unit types. 16 races. 6 major factions. Hundreds of years of history. The world of Wesnoth is absolutely huge and limited only by your creativity - make your own custom units, compose your own maps, and write your own scenarios or even full-blown campaigns. You can also challenge up to 8 friends - or strangers - and fight in epic ''multi-player'' fantasy battles.

== Features ==

<div class="thumb tright"><div>
[http://www.wesnoth.org/images/sshots/wesnoth-1.6-2.jpg http://www.wesnoth.org/images/sshots/wesnoth-1.6-2-175.jpg]
<div class="thumbcaption">The map editor</div></div>
</div>

* Build up a formidable fighting force, starting from a single leader and a small amount of gold.
* Over 200 unit types in six major factions, all with distinctive abilities, weapons and spells.
* Experienced units gain powerful new abilities as they advance.
* Several multi-player options available, including internet play.
* Scores of different custom-designed maps, and unlimited random maps.
* Hundreds of campaign scenarios available for download via a simple in-game procedure.
* 'Fog of war' feature available for a true test of generalship.
* Sophisticated mark-up language lets advanced users make their own maps, factions or campaigns.
* Excellent language support – 35 different languages currently available.
* Windows, Mac OS X, GNU/Linux, RISC OS, BeOS, Solaris, FreeBSD, OpenBSD, NetBSD, DragonFly BSD, AmigaOS4, OS/2 & eComStation compatible.

{{Play}}

CodingStandards

2009-05-07T16:58:23Z

Cjhopman: remove /* Use the function templates minimum and maximum */ as we no longer support msvc 6

Wesnoth uses modern/advanced C++ that is portable to modern C++ compilers.

== Formatting ==

When working on C++ for Wesnoth, indent your code with a tab character. After fully indenting, if you still need to line up the text with a specific character on the line above, you may further align it using space characters.

You may use long lines.

== Evil Evil Things To Avoid ==

=== Avoid implicit conversions ===

Make all constructors which only take one argument that is of a different type to the class 'explicit'.

Do not use operator T() where T is a type to allow an implicit conversion to a different type.

Example:

t_string(const std::string&);

This is very evil! It can cause many situations where a temporary t_string is implicitly created and then gets destroyed unexpectedly.

=== Do not use non-private data members of classes ===

It's okay to have a struct with all-public members, if that's what you want.

However, once something is a class, with private data members, do not add public (or even protected) data members to the class. Doing this breaks encapsulation and can cause all kinds of confusing and evil things to happen.

== Naming ==

=== End Non-Public Members of Classes with an Underscore ===

All non-public data members of classes should have their names terminated with an underscore, to show that they are a
class member. This makes for more readable code, once one is familiar with the convention.

== Idioms ==

=== Use References when a value may not be NULL ===

If a value passed to a function can never be NULL, use a reference instead of a pointer. I.e.

void myfunction(Object& obj);

rather than

void myfunction(Object* obj);

This more clearly shows the user of the function that obj may never be NULL,
without them having to consult documentation or the implementation of the function.

=== Use Const ===

The 'const' feature of C++ allows interfaces to more clearly specify how they treat objects.
Always use const when you are not going to modify an object.

I.e.

void myfunction(const Object& obj);

demonstrates to the caller of myfunction() that obj will not be modified.
If myfunction may modify obj, then use

void myfunction(Object& obj);

likewise, if a variable is not changed after initialization, make it const.

=== Know the behaviour of const references when types differ ===
If you assign something to a const reference of a different type, if necessary (if the type is different but there is a conversion) the compiler will create a temporary and guarantee it lasts for the lifetime of the reference. So

char c = 0; const int& i = c; c = 5;

will result in c == 5 and i == 0 which may not be what you expect.

=== Write Exception-Safe Code ===

Wesnoth code should be exception-safe, even if you do not use exceptions directly.
That is, you should be able to assume that an exception is thrown almost anywhere
from within the code, with well-defined results (i.e. no resource leaks).

Code that uses a pattern like,

{
SDL_Surface* image = IMG_Load("image.bmp");
...some code, which uses 'image'...
SDL_FreeSurface(image);
}

is bad, because the code may throw an exception, and 'image' will never be freed.
Instead, use wrapper objects which free the object in their destructor.

For SDL_Surface objects, use the <tt>surface</tt> class.
So you could rewrite the above code,

{
surface image(IMG_Load("image.bmp"));
...some code, which uses 'image'...
} ''the image is automatically freed here when 'image' is destroyed

Instead of allocating memory directly using new[] or malloc(),
use language-provided containers, such as vector.

=== Do not use sprintf ===

Sprintf does not check whether or not it is writing past the end of the space allocated.
This is a security problem if someone other than the person running the game
can cause sprintf to write very long strings.
In Wesnoth this untrusted data could come potentially from other players
in a multiplayer game or from downloaded campaigns.
Instead you should use snprintf with the second argument being sizeof of the buffer
that will hold the result.

== Standard C++ to avoid ==

=== Do not use wstring ===

The standard C++ wstring class, defined as a basic_string< wchar_t >,
does not exist in some platforms supported by Wesnoth.
Use wide_string, defined in language.hpp, instead.
wide_string is actually defined as a vector< wchar_t >

=== Do not use 0 when you mean NULL ===

Several Wesnoth developers, including Dave, find the number 0 to be very ambiguous when used in a non-numeric context. In keeping with the precedent that has already been established in the Wesnoth source code, you should avoid using literal zero for initializing and/or comparing null pointers.

== C legacy to be avoided ==

=== Use util::array instead of C-style Arrays ===

C-style arrays are very efficient, but their interface is ugly.
Use util::array defined in array.hpp.
It is a wrapper for an array which has a C++ container-style interface.
If you need to, extend it to make it fit your needs.

=== Do not use C-style casts ===

The following code,

if(i->second.side() == (size_t)player_number_) {

is considered bad practice in C++ since a C-style cast is overpowered -- if types change around it could end up casting away constness, or performing an implementation-defined data reinterpretation (basically a C-style cast is a compiler generated combination of static_cast, reinterpret_cast, and const_cast).

Good programming style is to use the least powerful tool available that does what you want.
For example,

if(i->second.side() == static_cast<size_t>(player_number_)) {

Alternatively, a constructor call may be used for non-builtin types.

Note: there may be some obscure cases where a C-style cast is desirable,
such as converting a pointer to an integer type of unspecified size.

=== Do not use #define for constant variable ===

<nowiki>#</nowiki>define foo X is not typesafe and you can use a const foo = X;
(in an anonymous namespace) to achieve the same but typesafe.

== Documentation ==

=== Document "config" preconditions and postconditions ===

In the Wesnoth code you will commonly encounter a data container known as the "config",
which contains heirarchical string data (such as WML contents or game settings).
The tagged "children" of the config and their string "attributes" are arranged
in an ordered and mapped format internally using STL.

Because config data is utilized in so many ways and places, it can be difficult to track across the scope of the entire program. You should document all public functions that take/return a config, specifying config content expectations (and updating any related entries in the [[ReferenceWML]] wiki pages).
In particular, if your function requires a config parameter, specify where/how the config should be created. This will be a great help to any future coders who need to call or modify your function.

=== Doxygen ===
See [[Doxygen]] for tips on how to comment the code,
so that doxygen can nicely document it.

== See also ==
* [[HackingWesnoth]]

[[Category:Development]]

SummerOfCodeProposal cjhopman

2009-04-08T16:38:08Z

Cjhopman: /* WML Profiling */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that insertions can move elements to different buckets and so our "indices" will need to be updateable. This likely means that they would need an extra level of indirection so they can point to something that doesn't move. Also, with blocked cuckoo hashing the table has a fixed constant size, if it gets too large we would have to resize it which can be quite slow. A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). We would likely hold a large block of listnodes in a vector or deque so that they don't all require a small memory allocation. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 9/4/09 00:40 GMT 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
EDIT: I have started doing some of this [[cjhopman_wml|here]]. 
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-08T16:37:10Z

Cjhopman: /* WML Profiling */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that insertions can move elements to different buckets and so our "indices" will need to be updateable. This likely means that they would need an extra level of indirection so they can point to something that doesn't move. Also, with blocked cuckoo hashing the table has a fixed constant size, if it gets too large we would have to resize it which can be quite slow. A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). We would likely hold a large block of listnodes in a vector or deque so that they don't all require a small memory allocation. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 9/4/09 00:40 GMT 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
EDIT: I have started doing some of this [[#cjhopman_wml|here]]. 
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-04T02:53:48Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that insertions can move elements to different buckets and so our "indices" will need to be updateable. This likely means that they would need an extra level of indirection so they can point to something that doesn't move. Also, with blocked cuckoo hashing the table has a fixed constant size, if it gets too large we would have to resize it which can be quite slow. A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). We would likely hold a large block of listnodes in a vector or deque so that they don't all require a small memory allocation. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 9/4/09 00:40 GMT 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-04T02:40:57Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that insertions can move elements to different buckets and so our "indices" will need to be updateable. This likely means that they would need an extra level of indirection so they can point to something that doesn't move. Also, with blocked cuckoo hashing the table has a fixed constant size, if it gets too large we would have to resize it which can be quite slow. A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 9/4/09 00:40 GMT 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-04T00:40:24Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 9/4/09 00:40 GMT 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-04T00:39:44Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

'''EDIT''' 
Lazy optimizations is not really a good title for these as I actually intend for config objects to move back and forth between these states. That is, we should be able to, when constructing a config object, tell it which of the three states to be in. And we should be able to tell it at any time to go into any of the three states. 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-04T00:30:16Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we have these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. This would allow us to aggressively unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:46:42Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and so is not very memory-efficient if we want decent performance. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distinct string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:43:59Z

Cjhopman: /* Timeline */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9 here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:41:39Z

Cjhopman: /* Technical details */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:38:56Z

Cjhopman: /* Technical details */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can leads to even worse memory-efficiency. There are several different ways that we can improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:36:58Z

Cjhopman: /* Larger patches and other substantial contributions */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up because being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:31:40Z

Cjhopman: /* Technical details */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if certain assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:30:55Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The problem is that a basic linear-probing hash table performs very poorly at high load factors and is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:29:01Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). As we will be allocating a lot of these listnodes, this is a nearly perfect opportunity to use Boost.Pool so that we don't actually do a bunch of small memory allocations. 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:26:39Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an extra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:25:19Z

Cjhopman: /* Config Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (that is, basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:24:48Z

Cjhopman: /* Config Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This is a significant amount of overhead per child config and attribute. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:22:32Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each tag stored with its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:21:54Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store some more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:21:00Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell how long a WML tag/attribute is. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:17:16Z

Cjhopman: /* WML Profiling */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but investigating them is one reason for me to do some extensive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage patterns (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T21:10:37Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading''' 
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction''' 
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T20:43:27Z

Cjhopman: /* Basics */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the computer science PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T20:42:50Z

Cjhopman: /* Basics */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T19:44:56Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T19:25:22Z

Cjhopman: /* Project */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* I already know and like the Wesnoth community. I will stay regardless of anything that happens for SOC.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T18:49:32Z

Cjhopman: /* Lazy Work */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload a lot of WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T18:45:00Z

Cjhopman: /* WML Profiling */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeIdeas

2009-04-03T18:44:06Z

Cjhopman: /* Optimize implementation of WML for memory usage */

This is a compilation of ideas from ML. Needs to be refined (more detailed description, deliverables, workload estimation?):

== I want to be one of your Google Summer of Code students, what should I do... ==

Here is a quick list of things to do to get you started
* Create an account on gna.org
* Create an account on the wesnoth forum, and tell an admin on the IRC channel to mark is as a GSoC Student account (Admins are boucman, Ivanovic, mordante, Shadow_Master, Sirp and Turuk)
* Join the irc channel (#wesnoth-dev on irc.freenode.net) and introduce yourself. We will not give formal interviews, but we will clearly favor people we have learned to know during the selection process (basically communication via IRC is mandatory for our project! it is the main way of "every day communication" for Wesnoth. For the same reason, it's also a good idea to regularly read the [http://wesnoth.debian.net/?C=M;O=A IRC logs].).

* Start a wiki page about your idea, add a link on the bottom of this page and add this information on it:
** List your account names (gna, forum, irc nick) so that we can recognize you
** Fill the questionnaire on this page: [[SoC_Information_for_Google#Does_your_organization_have_an_application_template_you_would_like_to_see_students_use.3F_If_so.2C_please_provide_it_now.| List of questions to answer]]
** Detail your idea as much as possible, look at other students pages, and please give milestones and studies you've done
** Add a link to the page at the bottom of this page

* Though not mandatory, it is highly advisable to go to the [[EasyCoding]] and [[NotSoEasyCoding]] pages and implement one of these ideas (or any idea of similar scope) so we have an idea how you work. Be sure to use your gna account when submitting these patches so we know who it is coming from. You can also implement some features from our feature request database at gna. When you implement something, also list it on your own page with a reference to the patch.

* For working on Wesnoth you have to be able to compile trunk. To do so you should have a look at the [[WesnothSVN|page about svn]] and afterwards [[CompilingWesnoth|compile Wesnoth svn]].

* Once you have everything done here and think your idea is okay, go to [http://groups.google.com/group/google-summer-of-code-announce/web/guide-to-the-gsoc-web-app-for-student-applicants page at google] to submit your application. You have to submit it before '''April 3rd at 19:00 UTC (12:00 PST)''' or you have no chance to get in!

== List of Ideas for the Project (Suggestions from the wesnoth developers) ==

Here is only a short description of possible Ideas we have, each has a page of its own with a more detailed version on it.

=== Optimize implementation of WML for memory usage ===

Based on this idea: [http://dave.wesnoth.org/?p=9] optimize WML to minimize its memory usage. High memory usage has been a problem for Wesnoth, and this project will aim to reduce it.

=== Implement campaign statistics reports on stats.wesnoth.org ===

Wesnoth has an infrastructure which records details of campaigns that players play into a centralized MySQL database. However, we only have rudimentary reports based on this MySQL database available at this time, at [http://stats.wesnoth.org].

This project would involve writing a stats reporting web site which would take the data from the MySQL database and produce reports in chart and table form. Campaign designers would be able to use these reports to gather feedback on their campaigns and get ideas for improvements.

A student could largely make their choice of infrastructure for creating the Website -- whether they prefer Python, Perl, Ruby, PHP, etc. This is a great opportunity for someone who doesn't want to dive into hardcore C++ to make a valuable contribution to Wesnoth.

[[SoC Ideas Stats Server]] - Full Version of the idea, with detailed information

=== Extending the Multiplayer server ===

Our multiplayer community is generally strong and healthy, but we believe its growth is limited by some problems in the interface of the multiplayer lobby.

[[SoC Ideas Multiplayer server]] - Full version of the idea, with detailed information

=== Addon server ===
Wesnoth has an addon server which offers users to upload user
made content (UMC). This allows all other users of Wesnoth
to easily download and install this content. The server was
originally written for user-made campaigns but contains a lot
more types of addons nowadays. Both the server side and the
client side need to be improved.

[[SoC Ideas Addon Server]] - Full version of the idea, with detailed information

=== WML validation schemes ===
Wesnoth uses WML as basic data structure. Over the years
this language has evolved and got more complex. At the
moment the WML is validated at runtime and in case of a
problem the engine stops. With schemes these problems can
be validated when loading the WML, making it easier to find
problems before running into them.

[[SoC Ideas Schemes]] - Full version of the idea, with detailed information

=== Write a primitive library for Formula AI ===

Wesnoth has always had a simple C++ based AI. David (our lead developer) has been working on a simple language to write AI in Wesnoth: [[FormulaAI]]

The Wesnoth AI is used as an opponent in most campaigns, and as such is an important piece of code for the Wesnoth project. Unfortunately, because the skills required to understand and modify it are rather arcane, it is also one of the most neglected parts of the Wesnoth code. This is a place where a lot of research and useful work could be done. But keep in mind that [[WhyWritingAWesnothAIIsHard|writing an AI for Wesnoth is difficult]].

Writing a whole AI is so complicated that we believe it can't be done in a single Summer of code. All proposals should keep that in mind and try to identify an interesting subset that would be workable in the limited time of a summer of code

[[SoC Ideas FormulaAI]] - Full version of the idea, with detailed information

=== Savegame reorganization ===
The savegame formats of Wesnoth for single player campaigns
and multiplayer differ from each other. And they are processed
differently as well. Now there is an additional request coming
up: Multiplayer campaigns. The task will be to unify the savegames
for all types of scenarios in order to provide a maintainable code
again.

[[SoC Ideas Savegame]] - Full version of the idea, with detailed information

=== Other possible ideas to be fleshed out ===
A MapGenerator rewrite - better scalable for outdoor maps, plus the possibility to define areas (similar to the caverns in the cave generator) etc.

=== Make your own ideas ===
If you have your own idea the best thing is to join IRC wesnoth-dev at irc.freenode.net and discuss the idea with the developers there. If the developers think your idea is interesting and like the feature you can start to turn it into a full proposal. Once done discuss it again on IRC so the developers can accept your idea.

== Information about our Project ==
The information we provided google with about our project can be looked up at the site [[SoC Information for Google]].

Also see the [[DeveloperResources]] link (from the [[Project]] page).

== People to bug on IRC ==
We have prepared a list of people with their "area of competence". This is to give you an idea on which areas those people can be of help for you. Of course you should always just ask in the IRC chan, but those are the most likely ones to answer questions in the respective area. And here is the list:

[[SoC People to bug on IRC]]

== GSoC Student pages ==

Please add a link to your wiki page below

==== AI ====

[[SummerOfCodeProposal_Velory| Velory - SoC Proposal]]

[[SummerOfCodeProposal_AI_Improvement_Crab| Crab - SoC Proposal - AI Improvement]]

[[SummerOfCodeProposal_Sparksteel | Sparksteel - Improving the AI engine design]]

[[FormulaAISoC2009Dragonking | Dragonking - Improving the formula AI ]]

[[GSoC_Proposal_-_Position_Evaluation | henning - Position Evaluation ]]

[[GSoC_Proposal_Evolutionary_AI | heineckm - Evolutionary AI]]

==== Savegame reorganization ====

[[SummerOfCodeProposal_Euschn | Euschn - Savegame reorganization]]

[[SummerOfCodeProposal_lmg| lmg - Savegame reorganization]]

[[SummerOfCodeProposal_grantwu| grantwu - Savegame reorganization]]

==== Extending the Multiplayer server ====

[[SummerOfCodeProposal_rubend | rubend - Extending the Multiplayer server]]

[[MP Server Ilor | Ilor - Extending the Multiplayer server]]

==== Addon server ====

[[SummerOfCodeProposal_Ryochan7| Ryochan7 - Addon server]]

[[SummerOfCodeProposal_iyonius| iyonius - Addon server]]

==== Optimize implementation of WML for memory usage ====

[[SummerOfCodeProposal_res| res - Optimize implementation of WML for memory usage ]]

[[SummerOfCodeProposal_jdmunro| jdmunro - Optimize implementation of WML for memory usage ]]

[[SummerOfCodeProposal_Ryanc| ryanc - Optimize implementation of WML for memory usage ]]

[[SummerOfCodeProposal_cjhopman| cjhopman - Optimize implementation of WML for memory usage ]]

==== Implement campaign statistics reports on stats.wesnoth.org ====

[[SummerOfCodeProposal_Elbin| Elbin - New campaign statistics page]]

[[SummerOfCodeProposal_Munk| Munk - New campaign stats page]]

[[SummerOfCodeProposal_carlestyle| carlestyle - New campaign stats page]]

[[SummerOfCodeProposal_nerwa| NeRwa - New campaign stats page]]

[[SummerOfCodeProposal_mrfinch| mrfinch - New Campaign Statistics Page]]

[[SummerOfCodeProposal_corn| corn - New Campaign Statistics Page]]

[[SummerOfCodeProposal_csaunders | csaunders - SoC Proposal - Stats Server]]

[[SummerOfCodeProposal_rohita | Rohita - New campaign stats page ]]

[[Category:Summer of Code|*]]

SummerOfCodeProposal cjhopman

2009-04-03T18:43:09Z

Cjhopman:

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev] 

[[Category:Summer of Code]]

SummerOfCodeProposal cjhopman

2009-04-03T18:41:54Z

Cjhopman: /* String Representation */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will likely have more overhead than the hashing options. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T18:40:35Z

Cjhopman: /* note to mentors */

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T18:40:05Z

Cjhopman: /* Timeline */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. Start with string optimizations. 

June 20th 
String optimizations finished, tested, and documented. 
Begin wml representation optimizations. 

June 27th 
WML representation optimizations finished, tested, and documented. 
Begin working on lazy loading/construction. 

July 6th 
Submission of midterm evaluations begins. 

July 27th 
Framework for lazy loading/construction finished, tested, and documented. 
Begin pass of intensive debugging and documentation on all three parts. Each of the three parts have already been tested and documented so this is just a sort of final cleaning pass. 

August 3rd 
Begin migration of current code to use lazy loading/construction where appropriate. 

August 10th 
"Pencils down" date. 
Everything should actually be completed by this time, and this final week is a sort of buffer to ensure that everything is very well-tested and documented. 

August 17th 
End of GSOC. 
Final evaluation submission begins. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T18:19:01Z

Cjhopman: /* Timeline */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th 
Convince Wesnoth mentors to accept my proposal. 
Finish up some cpu optimizations that I am working on for Wesnoth. 
Begin more in-depth profiling of loaded WML. 

April 17th - April 23rd 
I get to go to Stockholm for a week. 

April 20th 
Accepted proposals announced on GSOC site. 

April 23rd - May 23rd 
Finish up in-depth profiling of loaded WML in preparation for coding to begin. 
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here]. 

May 10th - May 16th 
I have a couple of finals. I will likely actually have more time this week as I won't have class every day. 

May 23rd 
Coding begins. 

July 4th 
Have string optimizations and wml representation optimizations complete. Begin working on lazy loading/construction. 

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T18:18:07Z

Cjhopman: /* Timeline */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

April 3rd - April 15th
Convince Wesnoth mentors to accept my proposal.
Finish up some cpu optimizations that I am working on for Wesnoth.
Begin more in-depth profiling of loaded WML.

April 17th - April 23rd
I get to go to Stockholm for a week.

April 20th
Accepted proposals announced on GSOC site.

April 23rd - May 23rd
Finish up in-depth profiling of loaded WML in preparation for coding to begin.
Make changes to config class to hide internals as described [http://dave.wesnoth.org/?p=9|here].

May 10th - May 16th
I have a couple of finals. I will likely actually have more time this week as I won't have class every day.

May 23rd
Coding begins.

July 4th
Have string optimizations and wml representation optimizations complete. Begin working on lazy loading/construction.

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T18:04:20Z

Cjhopman: /* WML Profiling */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===
Before doing any optimizations, we need to know whether or not it has a possibility of making an improvement. Basically the assumptions that we have made are that there a lot of strings, a lot of these strings are not unique, and there are a lot of config objects. These are reasonable assumptions, but it is one reason for me to do some intrusive profiling. 
The second reason to do profiling is to determine the benefit of optimizations that we do write. For this reason I have already set up a heap profiler so that I can see some higher level memory usage profiles (see [[Cjhopman Profiling]]). I also plan to do some more intrusive programming, basically I want to be able to see the structure of the graph of config objects (with some info like size of current, size of all children, etc.). Also, I will want to see the distribution of strings in tag names, attribute keys and attribute values. This will be the first thing that I do for this project, currently scheduled to be done before the actual start of coding time.

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T17:53:09Z

Cjhopman: /* String Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
''More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper).'' 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
''More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf].'' 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T17:52:08Z

Cjhopman: /* Lazy Work */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. 
While lazy loading will cut memory usage of a config object by, basically, 100%, I expect lazy construction to cut it by as much as 50%. The benefit of lazy construction would be that construction of the object once it is needed will not have to go to disk and so will be faster. 

If we wanted to get really aggressive, we could have either of these lazy schemes actually store a bit more information so that construction would not require full parsing. In particular, I am thinking that we could have each element store its length, then when we need to build a thing we can do it much more quickly. One benefit of this is that it would allow us to reasonably unload WML that we think won't be needed. 
This optimization actually has the potential to greatly reduce memory usage. Implementing this will likely hit several parts of the code (config, config_cache, parser, ...). 

===WML Profiling===

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T17:30:24Z

Cjhopman: /* String Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation is that we can optimize the low level representation of the string. I think the best option for this would be to use or adapt an already written lightweight string. A more important aspect of the low-level string representation is that it does a lot of small memory allocations. To solve this we can use some type of pool allocation (possibly Boost.Pool) for our lightweight strings. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. This is, I'd say, a more inter

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T17:16:02Z

Cjhopman: /* Lazy Work */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. In fact, the associated type identifiers could be packed into the high bits of a 32 bit index into the string hash table. This is, I'd say, a more inter

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:34:05Z

Cjhopman: /* Lazy Work */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===
Finally, we load a lot of WML (and build the corresponding config objects) that we don't need to. I intend to implement a framework that will allow us to lazily load WML from disk, and that will possibly allow us to lazily construct config objects from the loaded WML. 

'''Lazy Loading'''
The most aggressive lazy optimization would be to leave WML on disk until needed. This could save us a lot of memory, but could require us to minimally parse the WML multiple times as there is no way to tell when a WML block ends. One other similar possibility is to load and parse the WML, and then write it to a temporary cache. This would be simpler to do and its possible that cached WML is already in a format that would allow for easy lazy loading (I am not sure of the format of cached WML). 

'''Lazy Construction'''
A slightly less aggressive, but still useful, optimization would be to not fully construct some config objects. With the previously discussed string representation, a config object becomes simply a stream of ints and associated types. That is, everything

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:14:03Z

Cjhopman: /* Config Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). This should cut the overhead per config object in half, and there should be almost no overhead per attribute. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:04:31Z

Cjhopman: /* Technical details */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy Work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). I believe that, after a config object is built, most operations are lookups and there are only a few insertions and deletions. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:04:17Z

Cjhopman: /* Technical details */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). I believe that, after a config object is built, most operations are lookups and there are only a few insertions and deletions. Without further profiling, I believe that the sorted vectors is the best approach. This change would have less memory overhead and would allocate the memory that it does use in larger blocks than the current method. 

===Lazy Work===

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:03:00Z

Cjhopman: /* Config Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (basically, vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). I believe that, after a config object is built, most operations are lookups and there are only a few insertions and deletions. Without further profiling, I believe that the sorted vectors is the best approach.

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:02:18Z

Cjhopman: /* Config Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (vector<pair<string_index, t_string> > and vector<pair<string_index, config*> >, that is). I believe that, after a config object is built, most operations are lookups and there are only a few insertions and deletions. Without further profiling, I believe that the sorted vectors is the best approach.

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]

SummerOfCodeProposal cjhopman

2009-04-03T09:01:55Z

Cjhopman: /* Config Representation */

===note to mentors===
this is a work in progress, feel free to not review it yet.

==Questionnaire==
===Basics===

# Write a small introduction to yourself.
#* My name is Chris Hopman, I'm a student of computer science and mathematics. I'm graduating from the University of Wisconsin - Madison in May and will be attending the computer science PhD program here in the fall.
# State your preferred email address.
#* cjhopman@gmail.com
# If you have chosen a nick for IRC and Wesnoth forums, what is it?
#* My nick is cjhopman pretty much everywhere.
# Why do you want to participate in summer of code?
#* Participating in summer of code will give me an opportunity to spend a lot of time doing something that I enjoy. I will get to work on an interesting project and will get to make a major contribution to a great project.
# What are you studying, subject, level and school?
#* I am currently an undergraduate studying mathematics and computer science at the University of Wisconsin - Madison. I will be graduating in May and will be attending the PhD program here in the fall.
# If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for Wesnoth. If you have gained commit access to our SVN (during the evaluation period or earlier) please state so.
#* I received commit access to the Wesnoth SVN early in 2008. There is a list of some of my [[#My contributions to Wesnoth|contributions]] below.

===Experience===

# What programs/software have you worked on before?
#* The largest program that I have worked on is definitely Wesnoth. Other than that, and other than projects just for classes, I have worked on two other projects. The first is [http://code.google.com/p/autoscanner autoscanner]. The goal of this program is to do automatic 3d reconstruction of a statue from a short video clip. I did this project with another student for a professor that we worked for. The second project is more of a library of various functions. It can be found [http://code.google.com/p/wrong-answer-library here]. It is meant primarily to be useful for algorithm competitions (TopCoder, ICPC, etc.) and so is mostly graph theory, computational geometry, linear algebra, and other similar stuff.
# Have you developed software in a team environment before? (As opposed to hacking on something on your own)
#* Other than Wesnoth, the two projects I just mentioned were both done in groups of 2-3. So, primarily my experience of developing software in a team environment is just that from my contributions to Wesnoth.
# Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?
#* I have not participated in Google Summer of Code before.

===Open Source===

# Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.
#* Wesnoth is the only open source project that I am involved in (actually both of my projects on code.google.com are also open source but it's a whole different level). Most of my contribution to Wesnoth has been bug fixes and other minor work, though I have had several larger contributions. There is a list of these [[#My contributions to Wesnoth|below]].

===Gaming experience - Are you a gamer?===

# What type of gamer are you?
#* I am a diverse gamer. I've been playing games for more than 80% of my life (wow, just realized how long it has been)
# What type of games?
#* I like pretty much all genres. Yet, my favorites tend to be strategy games or rpgs. And those that find a good blend of the two are great. For example, Battle for Wesnoth (some others, too--Final Fantasy Tactics comes to mind).
# What type of opponents do you prefer?
#* Smart ones. I love the challenge of trying to outplay a smart player.
# Are you more interested in story or gameplay?
#* It depends. Generally when I am playing single-player games story and gameplay are both important though I am more likely to accept below average gameplay for an above average story than vice-versa. Playing multiplayer, particularly competitive multiplayer, gameplay is much more important.
# Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.
#* I have played Wesnoth for a bit over a year. I had focused on single player campaigns but in the last two months have shifted to almost only multiplayer.

===Communication skills===

# Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.
#* I am fluent in written English. I had better be as I am definitely not in any others.
# Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.
#* Yes, I am good at interacting with other players and with other people in general.
# Do you give constructive advice?
#* I think that I give constructive advice. I have been a tutor in math and computer science for a couple years and a sailing instructor for longer. Both of these have definitely improved my ability to give constructive advice.
# Do you receive advice well?
#* Yes, I do. I feel that a factor in this is that I am always interested in learning more and advice often offers an opportunity to do that.
# Are you good at sorting useful criticisms from useless ones?
#* Yes.

===Project===

# Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?
#* I chose to work on the optimization of wml for memory usage problem.
# Why did you choose this project?
#* I chose this project because I enjoy working with algorithms and data structures, and, in particular, I enjoy the challenge of finding better ways of doing things. I think that this project will be a chance for me to do that.
# Include an estimated timeline for your work on the project.
#* See [[#Timeline|timeline]] below.
# Include as much technical detail about your implementation as you can
#* See [[#Technical details|technical details]] below.
# What do you expect to gain from this project?
#* I expect to gain experience working with and redesigning a complex data structure. Also, I will get to be mentored by a person with more experience programming (and working on larger projects) which can only improve my own skills.
# What would make you stay in the Wesnoth community after the conclusion of SOC?
#* A cookie, chocolate chip preferably.

===Practical considerations===

# Are you familiar with any of the following tools or languages? Subversion, C++, Python, build environments
#* I am very familiar with C++. I have used subversion enough that I can do the basics, I am not very familiar with creating and merging branches. I only have a little experience with both Python and build environments. With either of these, I currently can only do the most basic tasks.
# Which tools do you normally use for development? Why do you use them?
#* Currently, I do almost all of my development in Linux, with a text editor, gdb and other command-line tools. Linux in general just makes development so much easier and the tools it provides are very powerful. In Windows (which I haven't really developed in in almost a year) I use Visual Studio 2005/2008. For debugging I feel it is even better than the tools available in Linux, and it has a few other nice features that may improve my productivity.
# What programming languages are you fluent in?
#* I am very fluent in C++, and significantly less so in Java. I have some experience with C and Scheme.
# What spoken languages are you fluent in?
#* English.
# At what hours are you awake and when will you be able to be in IRC (please specify in UTC)
#* I am generally awake from 1:00pm to 5:00 am UTC and will be available most of that time.
# Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail.
#* I would not mind that at all.

==Technical details==
The in-memory storage of WML currently uses a significant amount of memory. Also, the various data structures used often make small memory allocations which can lead to heap fragmentation and even worse memory-efficiency. There are several different ways that we can significantly improve the memory-efficiency of loaded WML. 
There are basically three areas that I intend to optimize for this gsoc project. First is the [[#String Representation|representation of strings]] in loaded WML. Second, the [[#Config Representation|representation of the config class]]. And third, implement a [[#Lazy work|framework]] for both lazy loading and lazy construction of config objects. These three things should all improve the memory-efficiency of loaded WML, but only if some assumptions hold true. For this reason I will also do some in-depth [[#WML Profiling|WML profiling]]. 

===String Representation===
A large part of the memory usage comes from strings(including t_strings). It's highly likely that a lot of these strings are the same (for example, "Id", "Name", etc). If enough of these are shared, then a lot of memory could be saved by sharing the string representations. The basic idea here is to have all strings in some global area accessible by an index or pointer. For example, a naive approach would be to just have an unsorted vector of strings and then the actual string representation in the config class would be an index into the vector. The obvious problem with that is that lookup for new strings is slow. Basically, our needs are low memory overhead, fast lookup of new (unknown) strings, fast lookup by "index", and fast insertion. 
I have a couple ideas of how to do this. 
My primary idea is to use a hash table to store the strings. The difficulty with this is that the basic linear-probing hash table suffers poor performance as the load factor gets high and requires rebuilding with more buckets. The real problem is that a basic hash table is not very memory-efficient. There are a couple of other implementations that would be more memory-efficient. Each of these implementations are efficient at high load factors. 
'''Blocked cuckoo hashing''' 
One variant that would be good for this project is blocked cuckoo hashing, a variant of [http://en.wikipedia.org/wiki/Cuckoo_hashing| cuckoo hashing] where each position in the table can hold some fixed amount of keys. There are several benefits to this implementation. At a very high ( > 99.9% ) load factor this implementation is still efficient and has very little memory overhead. The storage needed for blocked cuckoo hashing can be allocated as one large contiguous block. One downside is that the table has a fixed size and resizing the hash table would be difficult as it requires updating all the indices (the easiest way to do this adds an actra pointer per distinct string). A better option than resizing the table would be to have a backup stash. This could be a simple vector, but it would likely be better for it to be a smaller hash table that more gracefully handles dynamic size requirements. 
More info available [http://www.sciencedirect.com/science?_ob=ArticleURL&_udi=B6V1G-4N56BWF-5&_user=10&_rdoc=1&_fmt=&_orig=search&_sort=d&view=c&_acct=C000050221&_version=1&_urlVersion=0&_userid=10&md5=067aef2a68139cd8694dcdab5d847db3 here] (I was unable to find a free version of this paper). 
'''Move-to-front chaining''' 
In this variant the hash table is an array of linked lists. When we do lookups, the node that we find is moved to the front of its respective list. This move-to-front heuristic makes the hash table very efficient when there is a skew in the distribution of looked up words (as is likely the case with WML). The downside to this method is that it has more memory overhead per distint string (due to the linked list pointer). We could allocate a deque or vector of listnodes so that they aren't each a small allocation (though this makes removing them and reusing that space a bit more complex). 
More info on efficiency of this implementation [http://goanna.cs.rmit.edu.au/~jz/fulltext/ipl01.pdf]. 
'''Chaining with dynamic array''' 
Here we replace the linked list with a dynamically sized array. This has a bit less memory overhead but the allocated memory will not be in a single contiguous allocation. Also, in this method is it more difficult to use the move-to-front optimization as it would require updating the indices. Without that optimization, this is likely the least cpu-efficient of the three implementations. 
I believe that the best for this project will be one of the first two (though once one is working it will be quite simple to drop in another as the interfaces are the same). 
For this project, if profiling shows that there is a lot of non-distinct strings as is expected, I will implement at least one of the first two variants. The first has an overhead of at least 32 bits per distinct string(assuming 32 bit size_t) and the second at least 64. Again, profiling will show if it will be worth it. 

Note: Another option for this would be a B-tree with large branching. This will have a bit more overhead than the blocked cuckoo hashing but possible less than the MtF chaining. I do not currently plan to implement this during gsoc, though again it would have the same (or very similar) interface as the hash tables and should be easy to drop in and profile. 

The last thing with string representation that I will mention is that we can optimize the low level representation of the string. This is currently outside the scope of my gsoc project, though if profiling shows that there is a lot to gain in this area then that could change. I think the best option for this would be to use or adapt an already written lightweight string. 

===Config Representation===
Moving a level up from strings we get to the representation of the config class in memory. Currently it is basically 
struct config {
map<string, t_string> attributes;
map<string, vector<config*> > children;
vector<pair<map<..>::iterator, size_t> > ordered_children;
};
This looks like the equivalent of approximately 5 pointers overhead for each child_config, 2 for each attribute. Also, each config has overhead of about 3 pointers and 3 ints for the maps and vector. On my system, I estimate that is about 70 bytes of overhead per config object and about 15-20 per attribute. This is just a quick estimate, I have not done in-depth profiling yet to determine this. 
We can probably just replace this with two sorted vectors (vector<pair<string_index, t_string> > and vector<string_index, config*>, that is). I believe that, after a config object is built, most operations are lookups and there are only a few insertions and deletions. Without further profiling, I believe that the sorted vectors is the best approach.

==Timeline==

==My contributions to Wesnoth==

====Larger patches and other substantial contributions====

'''unit_map''' 
Originally redesigned this in early 2008 to have iterators that would be updated as a unit is moved and that would know when they become invalidated. Also added some more powerful iterators/accessors.[https://gna.org/patch/?1020][https://gna.org/patch/index.php?1046]
Recently I have refactored this a bit, templatizing the iterators. Even better, I greatly improved the documentation of the interface. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''gui2 text box history''' 
I added support for text box history to gui2. [http://svn.gna.org/viewcvs/wesnoth?rev=33868&dir_pagestart=250&view=rev] 

'''Profiling and some optimization''' 
I have [http://code.google.com/p/google-perftools/ google performance tools] working with Wesnoth to do some profiling. Originally I set this up as being able to profile memory usage and cpu usage will be important to this project. I have some results of this up at another [[cjhopman_profiling|page]]. 
While doing some profiling I found a couple of bottlenecks in the code. 
First, image::locator::locator() was using ~8-10% of the time in-game. I rewrote its lookup to use a hash-based map and cut the time the function used in half. [http://svn.gna.org/viewcvs/wesnoth?rev=34388&view=rev] 
Second, I changed the algorithm that we used to determine what rectangles to redraw. This shows significant gains in some graphics-intensive situations (as much as 3x fps).[https://gna.org/patch/?1141] 
Third, I am doing some work on the tokenizer used for parsing wml that shows a 10-20% speedup of WML loading at the cost of significant complexity. I'm not sure yet that this is worth it without first optimizing the underlying preprocessor stream that the tokenizer gets its input from.

===Smaller changes and bug fixes===
Report an error when a macro is not resolved and is not a filename.[https://gna.org/patch/?1035] 
Very simple bug fix.[https://gna.org/patch/?1041] 
Prevent duplicate advances_from entries.[http://svn.gna.org/viewcvs/wesnoth?rev=25960&view=rev] 
Fix bug #13003.[http://svn.gna.org/viewcvs/wesnoth?rev=32849&view=rev] 
Fix bug #12990.[http://svn.gna.org/viewcvs/wesnoth?rev=32854&view=rev] 
Began to refactor the attack class.[http://svn.gna.org/viewcvs/wesnoth?rev=32952&view=rev] 
Fix bug #11031.[http://svn.gna.org/viewcvs/wesnoth?rev=33360&view=rev] 
Fix bug with KNOCKBACK and similar wml.[http://svn.gna.org/viewcvs/wesnoth?rev=33652&view=rev] 
Simplified unit_map lookup by id.[http://svn.gna.org/viewcvs/wesnoth?rev=33956&view=rev] 
Improved error message when attempting to dereference invalid iterator.[http://svn.gna.org/viewcvs/wesnoth?rev=34224&view=rev]