SummerOfCodeProposal JodyNorthup

From The Battle for Wesnoth Wiki

This page is related to Summer of Code 2010
See the list of Summer of Code 2010 Ideas

This is a Summer of Code 2010 student page
Project: SoC Ideas Persistent Gameworld



Jody Northup - Implement gameworld data persistence

This is the Persistent Gameworld idea.
If accepted, I would communicate extensively with the community to determine a solution that provides gameworld persistence with the greatest ease of access and implementation for the established community of world and campaign builders for Wesnoth.
Currently, I expect that this will involve extending WML, and creation of some system of communication about the extended WML for multiplayer games.


Upthorn, Upth

SoC Application

Upthorn - Data saving for persistent game worlds



Write a small introduction to yourself.

My name is Jody Northup. I am a 25 year old college student from the United States. Under normal circumstances, I should have graduated by now, but there have been a number of obstacles on my route to academic success.

I have been programming since I was introduced to C++ in the summer of 1999.

State your preferred email address.

If you have chosen a nick for IRC and Wesnoth forums, what is it?

  • irc: Upthorn
  • irc alternate: Upth
  • Wesnoth forums: Upthorn
  • Upthorn

Why do you want to participate in summer of code?

It is a good opportunity to gain experience in aspects of programming that aren't covered in classes, it is a great way to contribute to the open source community on projects that I believe in or even make use of myself, being able to contribute usefully to something gives me a great sense of accomplishment, and it is much more fun than any other summer job opportunity I know of.

What are you studying, subject, level and school?

I'm an undergrad with an undeclared major at Sacramento City College. In terms of credit earned, I am in my third year, in terms of time spent I am in my 5th year, and in terms of the length of time remaining until I graduate, I might as well just be starting my 2nd year. In another year or two I hope to transfer to the University of Maryland, where I plan to double-major in Computer Science and Linguistics, getting a simultaneous degree in both if possible.

What country are you from, at what time are you most likely to be able to join IRC?

I live on the west coast of the United States (GMT-7 during the summer), but my sleep schedule is very flexible. I am generally available on between around 19:00 and 11:00 GMT, but I can change this during the coding period if it is inconvenient to my mentor. Until my class period ends, I will be completely unavailable on Wednesdays.

Do you have other commitments for the summer period ? Do you plan to take any vacations ? If yes, when.

I may be unavailable from 23:00 Wednesday to 19:00 or later on Thursdays (GMT), but this commitment can be cancelled if there is urgent work to be done. Additionally, I would like to take up an additional part-time job during the summer if I can fit one into the schedule, but there are no definite plans around this at the moment.


What programs/software have you worked on before?

  • A variety of programming projects for high school and college classes
  • I have hacked around quite a bit on open source emulators. Primarily the rerecording branch of gens (see introduction and these forum threads for more information, with some work also being done on snes9x and mupen64.
  • I have done a great deal of feature programming and debugging work on the Sonic 1 MegaMix romhacking project for the sega genesis.
  • I have done some simple reverse engineering work to examine a simple archive file format and help create a translation patch for the Japanese indie game Bunny Must Die.
  • I have done some minor testing and debugging work on a project to encapsulate windows games in a deterministic environment, allowing input to be recorded and replayed without concern for RNG input from the system clock, or other sources.
  • I worked on ScummVM for Summer of Code last year, implementing an API to allow engine modules to negotiate with backend modules to determine color depth.

Have you developed software in a team environment before? (As opposed to hacking on something on your own)

Yes, I frequently collaborate with other developers in my development of gens, and Sonic 1 Megamix, as well as having worked in pair and group projects for classes. And, of course, I contributed to ScummVM for last year's Summer of Code.

Have you participated to the Google Summer of Code before? As a mentor or a student? In what project? Were you successful? If not, why?

I participated in last year's summer of code as a student for ScummVM. My project was providing support for RGB color games. I was successful, but had a great snag towards the end as I completed my project as proposed before the midterm, went on a pre-arranged 2-week vacation, and had a great deal of difficulty with the project I was assigned to work on for the remainder of the summer of code period.

Are you already involved with any open source development projects? If yes, please describe the project and the scope of your involvement.

I have been the project maintainer and coordinator of the Gens-rerecording project since about mid-2006, having produced many new features, created a google code project page, and organized the development of many more during that time.

Gaming experience - Are you a gamer?

Yes, I enjoy games, although lately I haven't been playing them as much as I used to.

What type of gamer are you?

I am an obsessive perfectionist when it comes to gaming. I always try again and again until I get the best possible outcome, and I've collected all the items. Sometimes the frustration from this will drive me to save-scum, or edit my saves outright.

What type of games?

I love all types of games, from real time strategy (command & conquer, starcraft, total annihilation) and turn-based strategy (wesnoth, civilization) to RPGs (Planescape: Torment, Skies of Arcadia), adventures (King's Quest, Space Quest, The Void/Тургор), platformers (Sonic the Hedgehog, Mario) and racing games (Virtua Racing, Burnout) and even to First person shooters (Deus Ex, Unreal Tournament) and simulations (Escape Velocity, Dwarf Fortress)

What type of opponents do you prefer?

I prefer opponents that are on the same level or slightly better than me (so, generally, ones that are not very good).

Are you more interested in story or gameplay?

I cannot choose between story and gameplay. In the best games, there is no separation between the two, but either can hold up a good game on its own.

Have you played Wesnoth? If so, tell us roughly for how long and whether you lean towards single player or multiplayer.

Yes, I got hooked on wesnoth during my vacation last summer, and the Heir to the Throne campaign became one of the greatest difficulties I had in accomplishing additional work for ScummVM. I am not very good at strategy games, so I have not even attempted multiplayer yet.

If you have contributed any patches to Wesnoth, please list them below. You can also list patches that have been submitted but not committed yet and patches that have not been specifically written for GSoC. If you have gained commit access to our S­­V­­N (during the evaluation period or earlier) please state so.

3) Communication skills

Though most of our developers are not native English speakers, English is the project's working language. Describe your fluency level in written English.

I am a native English speaker, and my proficiency with written English is better than average.

What spoken languages are you fluent in?

I am a fluent native speaker of English, can communicate alright in Japanese, and am at a basic level in Mandarin and Korean. I hope to learn Russian next year.

Are you good at interacting with other players? Our developer community is friendly, but the player community can be a bit rough.

I do not know about wesnoth players specifically, but I am fairly good at interacting with players of other games. I tend to be friendly and open, and I understand that the other players' insults and other such things are just for fun and don't mean anything.

Do you give constructive advice?

I do give constructive advice whenever I can, although sometimes I think I cause offense by mistake. I always try to think my criticism through so that there is a clear indication of how the aspect could be improved.

Do you receive advice well?

I believe so, as long as it is clear and helpful.

Are you good at sorting useful criticisms from useless ones?

I think I am pretty good at sorting helpful criticism from bikeshed painting, yes.

How autonomous are you when developing ? Would you rather discuss intensively changes and not start coding until you know what you want to do or would you rather code a proof of concept to "see how it turn out", taking the risk of having it thrown away if it doesn't match what the project want

I like to keep a good balance between discussing changes and coding proof of concepts. In last year's summer of code, I had to interact extensively with the development community in order to design a useful API for them, but at times when it wasn't clear what the community preference was, I would make a proof of concept to demonstrate the features and drawbacks of a given method. Some of the work would be thrown out, but I did my best to maximize the amount of work that could be reused on a different method.


Did you select a project from our list? If that is the case, what project did you select? What do you want to especially concentrate on?

I did select a project from your list. Specifically, I selected the persistent gameworld project. I would like to concentrate on the necessary modifications to WML to make it work in single player, and add multiplayer support as time permits.

Why did you choose this project?

It seemed like something that I am farely likely to be able to accomplish at my current skill level, but is large enough that I shouldn't have to worry about completing it too early as happened last summer.

Include an estimated timeline for your work on the project. Don't forget to mention special things like "I booked holidays between A and B" and "I got an exam at ABC and won't be doing much then".


  • Now - May 23: Research the internal workings of WML, and scenario/campaign design, discuss with mentors and UMC developers to determine needs.
    • Milestone: By May 23rd, preliminary specification for persistence handling is complete for single player purposes.
  • May 24 - June 2: Implement prototype solution for single player context.
    • Milestone: By June 2, SP mode handles persistence data accurately to preliminary spec.
  • June 3 - July 4:
    1. Refine solution through iterative design and implementation with continuous discussion with community members.
    2. Design and implement persistence data management features while awaiting feedback during discussion periods.
    • Milestone: By July 4th, specification for persistence handling in SP contexts is finalized and implemented accurately. There is a persistence data management interface allowing users to backup, clear, and restore their persistence data.
  • July 5 - July 12:
    1. Discuss with player, software dev, and UMC dev community to determine details about needs for world persistence in multiplayer context.
    2. Document existing changes on wiki while awaiting feedback during discussion periods.
    • Milestone: By July 12th (Midterm), there is a complete specification addressing all concerns for standard multiplayer, additionally Single Player implementation and persistence data management are documented on the wiki.
  • July 13 - July 27:
    1. Implement proof of concept solution for addressing multiplayer needs in persistence data saving/loading.
    2. Present finalized persistence WML capabilities to UMC developers on forums.
    • Milestone: By July 27th, persistence data works and syncs in standard MP mode to preliminary spec.
  • July 28 - August 16: Refine solution through iterative design and implementation with continued discussion.
    • Milestone: By August 16th, multiplayer handling of persistence data is satisfactory to majority of players and developers.

Optimistic: TODO: Account for all mandatory goals in optimistic timeline.

  • Now - May 23: Design and implement preliminary specification while researching the internal workings of scenario/campaign design, and discussing with mentors and UMC developers to determine needs.
    • Milestone: By May 23rd, SP mode handles persistence data accurately to preliminary spec.
  • May 24 - June 7: Refine solution through iterative design and implementation with continuous discussion with community members.
    • Milestone: By June 7th, specification for persistence handling in SP contexts is finalized and implemented accurately.
  • June 8 - June 13: Discuss with player, software dev, and UMC dev community to determine details about needs for world persistence in multiplayer context.
  • June 14 - June 28: Implement proof of concept solution for addressing multiplayer needs in persistence data saving/loading.
    • Milestone: By June 28th, persistence data works and syncs in standard MP mode to preliminary spec.
  • June 29 - July 12: Refine solution through iterative design and implementation with continued discussion.
    • Milestone: By July 12th (Midterm), specification for standard MP handling of persistence data is finalized and implemented.
  • July 13 - July 27: Discuss with player, software dev, and UMC dev community to determine details about needs for gamemaster mode.
  • July 28 - August 11: Implement proof of concept gamemaster mode.
    • Milestone: By August 11th, game master mode is functioning.
  • August 11 - August 16: Refine game master mode iteratively via mentor, player, and UMC dev feedback.
    • Milestone: By August 16th (final), game master mode is satisfactory to player needs.

Include as much technical detail about your implementation as you can

I have relocated this content to Project Details for ease of organization.

What do you expect to gain from this project?

Experience with custom scripting language implementations, understanding of how to process XML-style languages efficiently, a sense of having been able to do something useful for someone besides myself, and a nifty t-shirt from google.

What would make you stay in the Wesnoth community after the conclusion of SOC?

My interest in the project itself, and any relationships I might forge with community members during the coding period.

Practical considerations

Are you familiar with any of the following tools or languages?

   * Sub­­version (used for all commits)

Yes, I have used it extensively for gens development, megamix development, and my work on ScummVM last summer.

   * C++ (language used for all the normal source code)

Yes, I have been working with C++ for nearly 11 years now.

   * STL, Boost, Sdl (C++ libraries used by Wesnoth)

I am fairly familiar with the STL, and worked almost exclusively with SDL on ScummVM last summer. I am new to Boost, but in the past I have been able to catch on to these things fairly quickly with a little bit of practice.

   * Python (optional, mainly used for tools)

I have worked with python somewhat, contributing a few minor bugfixes to the Civilization 4 mod "Fall From Heaven II"

   * build environments (eg cmake/autotools/scons)

I do not have any experience with cmake, autotools, or scons.

   * WML (the wesnoth specific scenario language)

I have some minor experience with WML, from editing my own wesnoth savegames last summer.

   * Lua (used in combination with WML to create scenarios)

I have experience both in scripting Lua, and in working with the Lua C API to provide hooks and bindings.

Which tools do you normally use for development? Why do you use them?

  • C/C++: I generally use Microsoft Visual C/C++, because I am familiar with it from classes, and because it provides a very useful debugger that allows me to trace problems to their source quite quickly. I have used Dev-C++ in the past, though.
  • Java: Eclipse, because it was recommended to me and I do not know anything about the others, and also because it provides a very useful debugger facility.

What programming languages are you fluent in?

C, C++, Lua, Java, Javascript.

Would you mind talking with your mentor on telephone / internet phone? We would like to have a backup way for communications for the case that somehow emails and IRC do fail. If you are willing to do so, please do list a phone number (including international code) so that we are able to contact you. You should probably *only* add this number in the application for you submit to google since the info in the wiki is available in public. We will *not* make any use of your number unless some case of "there is no way to contact you" does arise!

I have provided a telephone contact in my SoC application.

Project Details


I will extend WML to provide for events that can save persistent data about what has occured in a scenario or campaign. This data will be accessible to the scenario and campaign in all future plays from the same machine (or related ones that ask for it by name), even if all normal savegames have been deleted. If I have time remaining once this is completed, I will attempt to extend this functionality into mult-player contexts in a meaningful way.



  1. WML extension to provide events for persistent variables. -- Completed by Aug 26
    • events and syntax -- Completed by June 2
      • bulletproof singleplayer implementation of WML persistence events -- Completed by July 4
      • multiplayer implementation of WML persistence events -- Completed by July 27
        • error recovery in MP in case players drop/join. -- Completed by Aug 16
        • transaction blocks for global variable operations. -- Completed by July 27
      • Wiki documentation of events and syntax -- Completed by July 12
    • present new WML capabilities to UMC authors on wesnoth's forums. -- Completed by July 27
  2. User ability to manage persistence data -- Completed by July 12
    • Backup -- Completed by July 4
    • Clear -- Completed by July 4
    • Restore -- Completed by July 4
    • User Interface (probably modified from "save game" or "load game" dialog. -- Completed by July 4
    • wiki documentation/manual page -- Completed by July 12


  1. gamemaster mode multiplayer
    • UI
      • button, checkbox, or list option somewhere in the multiplayer interface
      •  ???
    • Backend
      • Separate gamemaster persistence from standard SP and MP persistence
        • SQL storage backend?
        • player profiles?
    • Wiki documentation/manual page
  2. Player profiles
    • UI
      • a menu option to select profile (perhaps a "tab" in preferences, along with graphics, sound, etc)
      • a select profile dialog
        • a list of profiles that currently exist
        • an option to create a new profile
      • dialog allowing the user to specify a profile name
        • option to migrate previous (non-profiled) data into new profile
          • dialog allowing player to select which savegames to migrate
    • Backend implementation
      • Append profile name as subdirectory to user_data directory path.
      • Save selected profile to config on exit (or profile selection)
      • Load selected profile from config on startup
    • Wiki documentation/manual page
  3. Engine or built-in macro support for default namespace
  4. SQL backend for global variable storage
  5. First massively multiplayer campaign
    • Collaborate with UMC authors and playtesters from Wesnoth's MP community
    •  ???

WML Syntax

Basic Syntax

Crab has suggested





Keyword: global

Allows wml author to specify that the variable is not associated with a specific side, and is to be loaded from host during multiplayer.

Default namespace

May be placed in addon cfg files to specify a default namespace for global variables set by this campaign. This namespace will be used if a "namespace=" line is not specified in [*_global_variable] tags.

General Thoughts

Crab's syntax for WML events seems reasonable, so I will be using it as the base on which to build. Currently, to set a persistent variable with a constant value, it is recommended to make a temporary variable with that value, and then set the global variable from that. However, syntax may be defined later to directly set a global variable with a constant value.
In general, any deviation I make from crab's suggested syntax will be done in order to make persistent "global variables" work more like regular WML variables from the standpoint of a WML author.

Namespace Format


Tentatively, I am suggesting that namespaces be of the format (addon-name)_(author-handle), to maximize uniqueness. For example, (judging by info in its _main.cfg) the "Love to Death" UMC campaign would use the namespace "Love_to_Death_Geos_PP2_etc". However, this is extremely open to suggestion and revision.

Importance of Uniqueness

Multiple addons can possibly have the same name if at least one does not come from the addon server. So the addon name alone is insufficient. It is important that a namespace is globally unique, because even if it is not possible for two addons with the same name to be installed and operating at the same time, a player may wish to retain persistence data when uninstalling an addon. If a player uninstalls an addon, retains persistence data, and then installs a different addon with the same name, that addon should probably not be loading the old one's persistence data, and almost certainly should not be overwriting it.

Consider this cautionary tale:

Example A

Addon "foo" by author "bar" (henceforth referred to as "bar::foo") has persistence data. Addon "foo" by author "plank" (henceforth referred to as "plank::foo") also has persistence data. bar::foo and plank::foo both use "foo" as the namespace for global variables, because it is the name of the addon, and two addons with the same name cannot be installed at the same time. Adam, a user, has played bar::foo extensively, likes it well, and has unlocked a number of special units in it, but has heard that plank::foo is well made, and would like to try it out. Adam knows that two addons with the same name cannot be installed happily at the same time, and so he uninstalls bar::foo. Adam fully intends to return to bar::foo at some point, and so leaves the persistence data intact. Furthermore, he does not realize that both addons use the namespace "foo", or does not understand how the namespace is relevant to the file choice, and does not think to back up his persistence data from bar::foo. Adam installs plank::foo, and starts a game with it. plank::foo unintentionally loads the persistence data from bar::foo, and because some of plank::foo's persistent variables coincidentally have the same names as some of bar::foo's persistent variables, right from the beginning, Adam gets confusing messages referring to events in plank::foo that haven't happened yet. Adam decides he does not like plank::foo, and so uninstalls it and reinstalls bar::foo. Adam starts a game with bar::foo, and discovers that, because of corruption from plank::foo, he has lost access to several of the unlockable units.

Example B

Addon "foo" by author "bar" (henceforth referred to as "bar::foo") has persistence data. Addon "foo" by author "plank" (henceforth referred to as "plank::foo") also has persistence data. bar::foo uses the namespace "foo_bar", because it is recommended that wml authors use their own handle in addition to the addon title, to avoid collision. plank::foo violates spec and simply uses "foo". Brian, a user, has played bar::foo extensively, likes it well, and has unlocked a number of special units in it, but has heard that plank::foo is well made, and would like to try it out. Brian knows that two addons with the same name cannot be installed happily at the same time, and so he uninstalls bar::foo. Brian fully intends to return to bar::foo at some point, and so leaves the persistence data intact. Brian installs plank::foo, and starts a game with it. Brian discovers that plank::foo is a lovingly rendered campaign with a cohesive story and an engaging metanarrative thread tha reveals itself on subsequent playthroughs. After many happy hours with plank::foo, Brian has done all there is to do in it, and seen all there is to see in it, and decides to return to bar::foo to finish rounding out his unlockables/achievements. Brian uninstalls plank::foo and reinstalls bar::foo, and starts a game with bar::foo. Brian sees that all the data is intact and happily resumes his progress from just where he left it.


Because (the first section) of a namespace will be the cfg file name, all characters in a namespace must be filename safe. That is, they cannot include '*', '?', ' ', '/', '\', '<', '>', '|', or '"'. Additionally, '.' will be interpreted specially, so WML authors should be aware of that.


Single Player

Persistent data will be stored in "(user data directory)/persist/(namespace).cfg".

Standard Multiplayer

Persistent data will be stored locally to each player. Persistent variables with "side=global" during [get_global_variable] will be loaded from the client of the initial host player. If the initial host player has been dropped from the connection, they will read from the current host. Persistent variables with "side=global" during [set_global_variable] will be stored to all players. Persistent variables that are side-specific will be stored and loaded from the client of the player that has taken the specific side.

Dedicated Game Master Multiplayer

In the case of a dedicated game master, persistent variables from all sides will be stored and loaded from the game master. However, due to the nature of game master mode, it is unlikely that the standard single-player storage location will be sufficient. TODO: Define storage location for game master.

Persistence Config Format


Internally, persistence cfg data will primarily be collections a [variables][/variables] block surrounding valid variable definitions. I believe it would be useful to provide facilities for further structure, allowing [variables] blocks to be enclosed within further tags.



Proposed Syntax

In order to allow WML authors to structure their persistence file in an arbitrarily defined manner, I suggest an extension to the parsing of the namespace attribute in global_variable tags: . should be used to denote sub-namespace access in the same way that, in variables, . denotes access to member variables of a container. This will allow WML authors to keep their variables organized in specific ways that make sense to them, as well as allowing for reuse of variable names in different contexts.

An example
Special Namespaces

Obviously, "variables" and any other automatically generated tags would have to be reserved, and therefore invalid as sections in the namespace attribute. However, we could also provide keyword namespaces that have specific effects on their contained variables. For instance, as a an example the [single_player] namespace could be provided to specify variables that are not to be used during multiplayer.

Handling Of Eras

Era add-ons will also be allowed to save persistence data, but it has not yet been determined how era persistence and scenario persistence will interact.

See below for detailed consideration of this:

Addon interaction

If a user runs a campaign that saves persistence data with an era active, which of the following options should be the result?

Continue as normal

The campaign doesn't know or care that the era is active, and loads and saves persistence data as usual.

This is the easiest to implement, but seems like it has potentially undesirable implications for players and UMC developers.

Disable persistence functionality

Campaigns have their persistence functionality disabled while the era is active.

This option preserves data integrity, but seems overly limiting. Additionally, this seems to shut eras out of the whole world persistence scene. Not a good solution.

Second set of data

Campaign loads and saves a different second set of persistence data.

This preserves data integrity while maintaining flexibility, but comes with a whole host of additional considerations:

On a campaign's first run under the active era, what should happen?

  1. Campaign copies existing persistence data into second set.
  2. Campaign saves and loads an entirely new version of persistence data for use with this era.

My gut instinct says that 2 is the better option, but I can easily imagine a significant set of players being upset with either decision. Perhaps we should prompt the player and allow them to decide.

Where do we store the second set of data?


This is a simple solution, but it may cause tons of clutter in the user's persistence data directory.

The same persistence .cfg file as normal, but enclosed in an [era][/era] block.

I like this option better, but it brings its own set of questions -- what about persistence data that the era wants to save, does it get enclosed into [campaign][/campaign] blocks? If so, how do we know what files to put [era][/era] in and what ones to put [campaign][/campaign]? I propose that persistence .cfg files don't care what type of addon made them, and may have any number of [campaign][/campaign] blocks in them. If an era is active, it should make an [era][/era] block at the level between [campaign][/campaign] and [scenario][/scenario].

What if era is irrelevant to the purpose of a persistent variable?

For example, a variable that just keeps track of how many times this campaign has been played total?

User specifies interaction

Provide some method allowing campaign authors to specify whether any given variable should be duplicated when eras are active. This would require syntax to be defined for [*_global_variable] blocks. I suggest an optional "era=" line, with the following effects:


There is only one copy made of this variable, all eras use it.


This is the default if the line is not provided. A copy of this variable is made for each era that this campaign is used with.

Additionally, this can be extended to allow for


which would allow campaign authors to specify global variables that only exist in certain eras, thereby allowing an easy path for UMC developers to build in era/campaign interactions.

Of course, there are plenty of other options we could take to allow user-specified era/campaign interaction, and this option would practically necessitate the addition of a "campaign=" for era addons to use.

Don't worry about it, it's not that important

Allow this one instance of eras interacting poorly with campaign persistence data, since it isn't ever going to be that big a deal if an era-campaign config keeps a separate play count from the campaign alone. And players can always edit the persistence files without much difficulty if it troubles them.

Optional Subtasks

User profile

Add user profiles, so that multiple users can share the same wesnoth installation without having to share persistence data and savegames.

This page was last edited on 21 March 2013, at 00:03.