WIP

Addon server

After collecting the info in this thread http://www.wesnoth.org/forum/viewtopic.php?t=15664 this is a first proposal to all changes for the addon server. Personally I'm most interested to do the server side changes and not the client side changes so I hope to find a victim for that part ;)

The changes are splitted in 3 parts, the general changes, the changes specific to improve the handling of translations and last the automatic test for some broken content.

General changes

Client side changes

For the client side some extra fields will be available to show, some are set by the author and some by the admin/devs. Also note I think it might be a good idea to give some non-devs dev access to the server, that would be people the devs think should have that access. I don't want to do the voting system at the moment since that will be quite some work (due to possible abuses) and adds little benefit. (The main reason for starting this project is to allow asynchronous updates of the translations.)

icon        like current icon
name        like current name
title       like current title
description like current description (but hopefully shown in the client)
image       larger image to show (as shown in the addon menu when you select
            an addon)
type*       type of the content see below (the client might want to split this
            displayed info per type)
author      name of the author
version     version number of the addon (as set by the author)
date        timestamp of last upload, this way 2 updates on one day can be seen
            as different. (ideally the client should notice a newer version by
            this value and give a visible clue)
size        download size of the addon excluding languages. Languages show
            this value separately.
length      number of scenarios, mainly for campaigns
difficulty  difficulties available as comma separated list
version_a   string with versions it should work with eg (a better tag name is to
            found)
            - 1.2.* all 1.2 versions
            - 1.2.3+ versions equal or newer as 1.2.3
            - 1.2.0 - 1.2.2 versions from (including) 1.2.0 to (including) 1.2.2
version_b   string with the version used for the upload to be able to block
            specific versions that are known not to work on client side or to at
            least be able to show a warning to the user
dependency  other content this addon depends on, comma separated list of names
            (This might allow the client to allow automatic downloads of
            dependencies)
status      The status is split in 3 parts
            - author status, the development status of the addon**
            - dev status, the status the devs give to the addon***
            - macroscope status, the status macroscope gives****

* type is one of the following
SP campaign
MP campaign
MP map(s)
era
faction
mod
theme (not yet possible, but might add support for it later)
other

** author status is one of the following
under construction      addon is still in its early stages and is incomplete
alpha                   addon is pretty much complete
beta                    addon is complete but needs more polishing/debugging
complete                addon is complete and hopes to get a star soon

*** dev status is one of the following
Unknow      default status
broken      the addon doesn't work as expected
buggy       the addon works almost as expected but has some bugs
works       the addon works as expected but uses deprecated WML
certified   the addon works and triggers no deprecated WML messages
            (I don't like the word certified, so a better name is welcomed)
star        same as certified but the devs enjoy to playing with this addon

**** Macroscope status
Macroscope is a tool which generates an automatic status, described in more
detail later, it has 2 possible statuses
broken      a red cross, the addon is broken due to unresolved references
not broken  a green V, the addon has no unresolved references (this does
            not mean the addon can't be broken in other ways.)

Server

On the server is some more info stored

email   email address of author mandatory so it's possible to notify about status
        changes

Translations

The goal is to separate translations from the addon so it's easier to allow translators and the author update independently. Also the author no longer needs to work with Wescamp the update is automatically send to Wescamp. (If the author wants that of course.)

Note the specifics about the Wescamp are not discussed here only the general changes. The specifics will be discussed when the time is there and probably on irc.

client side changes

In the client a user can select 0 or more languages (s)he's interested in, this is a general preference it's used at the client as a hint. No need to show 30 languages with their status if the user isn't interested in it. The untranslated addon is always downloaded directly so English speakers might not be in translations at all.

The client can show the following info about the translation

date        date of the last update
size        the size of the download
revision    revision number of the last update (Since the info is pulled from
            Wescamp the updater can use this info to test for changes. The
            client can use this info to determine whether the downloaded
            translation is the most recent version and notify the user if not.)
status      the percentage of strings translated. (Since fuzzy strings are
            shown in English the count as not translated.)

Upload

The pbl file will get a entry to tell whether an addon should or shouldn't be translated. If translated it will need the svn path to Wescamp for that addon. If marked for translation the addon is copied to Wescamp overwriting the stuff there. After the upload the version at Wescamp is directly updated.

This part will be updated in a later phase for now only the marking for translation is important, the other details will depend on the implementation.

Updating translations

The will be a cronjob on the addon server that will check all translations in Wescamp whether there's a newer revision as the last copied revision if so the newer revision will be copied to the addon server. (It's undetermined whether this will be a real cron job or a cron like facility in the addon server.)

wmllint

Esr wrote a tool, named wmllint, which can validate whether WML references are broken or not (it can do more but for the server only the broken references are used) Broken references are - calls to non-existing macros - references to non-existing images/sound/music files

Wmllint will run on the server, the problems with running on the clients are - Python is required - It's possible to upload broken content by modifying wmllint - it's required anyway due to new versions

Upload

On upload an addon is checked whether it has no broken references. If it's broken it's not allowed on the server, the output of wmllint is send back to the uploader so (s)he can see what the problems are.

After the upload is successful there will be a preprocessed file made which can be used by the dependent addons.

NOTE this means if addon B depends on macro A in addon A and A uploads a new version without macro A then addon B is broken. A can't be blamed that B uses some of it's stuff so the upload of A will not be blocked. This means the server needs to "remember" the reverse dependencies and test them after a new upload, so B gets marked broken.

New version

If a new version of Wesnoth is released a preprocessed file uploaded and all current available content is checked. The wmllint status is set in the addon, upon change of status the author is notified per email (if possible)

Since stable versions should be compatible this shouldn't change the status. For the development version it can be that a content gets broken.

Misc

Directory structure

At the moment there needs to be the following structure

$datadir/foo.cfg
$datadir/foo.pbl
$datadir/foo

with the changes in the directory parsing the following structure would also be possible and maybe should be used to make it easier to copy unpack etc stuff

$datadir/foo/_main
$datadir/foo/addon.pbl
$datadir/foo

Upload

At the moment it seems to be quite confusion for uploading to get things properly installed, maybe allow to browse to a directory and upload that. This directory then needs to have a file named addon.pbl. With the wmllint interface this also can allow to save things at a different location as the upload to avoid overwriting stuff.

Terrain system

terrain files

Eleazar would like to move the terrains in subfolders, this seems to be possible with the current system. The required modifications can be found in the following changelog http://svn.gna.org/viewcvs/wesnoth?rev=14069&view=rev

I've no idea what the best possible directory layout will be. I'll ask Eleazar to do this change when I'm ready for it. At the moment I've some patches in my queue which will conflict with these changes.

starting positions

At the moment Wesnoth supports 9 starting positions, this proposal allows for more positions but the engine can't use the extra positions. This limit will be removed in the near future.

aliases

The aliases will also be a comma separated list, where the same discussion about the + and - can be started as in the types.

encounter terrains

This list maintains the encountered terrains by the user this list is used in the help function to decide what to show and what not. This is now a comma seperated list. The new version will also be a comma separated list but with the new terrain strings instead of letters. Backwards compability will be dumped, the new list will be called encountered_terrains_list and the user will be required to rediscover every terrain. (The conversion is harder as expected and the benefit too small)

This way the 1.2 branch will use the old list and the 1.3 branch the new list.

AnimationWML

These also need an update here the * will also be allowed. The current system look at the aliasses (which is not documentated) the new system will look at the string and as said before allow the * to match.

editor

The editor also will need an overhaul due to the changes, since the terrain consists of a maximum of 4 characters I propose it writes the terrain as a string of 6 characters; the terrains string followed by a comma and padded with spaces. eg the following terrain string

Vh,Vha,Vhaa,Vha,Vh

will be written as

123456123456123456123456123456 (numbers won't be written.)
Vh,   Vha,  Vhaa, Vha,  Vh

Discussion: Should the editor be able to write the old format terrain or not.

custom terrains

Zookeeper told me there's no option to add custom terrains at the moment. I want to add a button to add a custom terrain, once clicked it will prompt for the terrain string and insert the tile to the list of available tiles and allow to use this tile. The tile will be 'painted' with the terrain string. When loading a map with unknown strings these will be added to the list of terrains. The only problem will be the location of the terrains but I'll worry about that later.

There should also be an option to add fog/shroud tiles to the map. See http://www.wesnoth.org/forum/viewtopic.php?p=192326#192326

Discussion: Would this solve the unkown tile problem?

themes

In my opinion the list with terrains to browse is pretty long and this will only grow in the future. I want to add an extra grid of buttons with theme's the buttons will have an icon for their theme. When a button is clicked the list of terrains is filtered per theme. eg when the theme villages is selected only village tiles are shown. Every terrain gets one or more themes for example a snow forest can be in the theme forest and the theme snow, so with either theme this tile is shown. There will be 2 special theme's favorites and map. Favorites are your favorite tiles saved in preferences and can contain unknown terrains. Map are all tiles already used on this map, including the unknown.

Discussion: Is this handy?

MP dialog

See UseMapSettings

Todo

Things I plan to do after I finished my WIP's.

team sides

After the terrain system has been modified it's possible to define more teams, EdB asked this in irc at 20061027.

It would be nice to support 12 sides in multiplayer with their own tcolour. The engine should support more sides for the AI to use. These will use and scenario WML defined colour. It's not sure whether WML supports this if not the support should be made. EdB will look for an possible algo to define nice auto colours.

terrain archtypes

Eleazar had an idea for terrain archtypes in the terrain thread. It looks sane and would be nice to implant. http://www.wesnoth.org/forum/viewtopic.php?p=191410#191410

Misc

Various things I want to look at at some time.

version checks for WML, needed or not?
modulo WML, needed or not? Done, might cleanup similair code the same way.
continue to work on the Python AI

Savegame format

It would be nice to have the format of the savegame and the data send by the MP server to be a bit more the same. The other idea is that the MP server holds a minimal part in a config object and the rest of the data is stored in a gzipped config. This gzipped config is seen as a blob and not unpacked by the server but simply forwarded to the clients. This is a rough draft used to discuss about on the FOSDEM.

The format will get the following sections

header basic info which is at the moment at the root of the file.
- kind this key contains SP/MP value.
- state this key is like the current key, but can also be start.
carry_over this section is only used in campaigns and contains the gold carry over info. This info is the same for all players.
dependencies 0 or more sections containing the id's of the dependencies. (Use sections since we might want to add more info later.)
summery this section isn't really needed for a savegame but it contains the info the addon server needs to know about (not sure what's needed.)
start_of_scenario section with all info regarding the start of scenario (a bit like replay_start atm).
- player the player data stored from the previous scenario, always saved.
- scenario this section always exists in MP but not in SP (in SP there's a key scenario_id) and contains the scenario data.
- era this section contains the era data for MP, section doesn't exist in SP.
snapshot section wit the info about the current state of the game.
- player current state of the players
- scenario current state of the scenario
- era current state of the are
statictics same as atm.
replay same as atm.

What to do with unit info? I want to remove some info from the unit like movement cost, defence stats, animations etc. Should this info be saved or not?

BTW it might be a nice project to have a something like a DOM viewer but then for WML.

User:SkeletonCrew

Contents

WIP

Addon server

General changes

Client side changes

Server

Translations

client side changes

Upload

Updating translations

wmllint

Upload

New version

Misc

Directory structure

Upload

Terrain system

terrain files

starting positions

aliases

encounter terrains

AnimationWML

editor

custom terrains

themes

MP dialog

Todo

team sides

terrain archtypes

Misc

Savegame format