Revision as of 05:47, 28 June 2011

This page is related to Summer of Code 2011

See the list of Summer of Code 2011 Ideas

This is a Summer of Code 2011 student page

Project: SoC_Ideas_Eclipse_Plugin2011

Description

timotei - Improving the Eclipse Plugin

I aim at improving the plugin, so it will become the tool used by all UMC Authors. That is, UMC Authors will get rid of the lot of different tools they use for different tasks (for example the Emacs WMLMode/other plugins for different text editors), and instead use one single unified Tool that will speed up the development. My target is to make this plugin a complete and very stable suite for all umc tasks that don't need specialized software - like a graphics editor, so I'd like to have this oportunity to finish it in the good sense.

Contacts

IRC

timotei

Other

Forum: timotei
gna.org: timotei

SoC Application

Battle for Wesnoth - Google Summer of Code Application

Proposal Summary

Page Last Updated: 2011-06-28

In the following lines I will give details on how I'm planning to implement some of the required tasks, and also add some new ones.

The Notes sections are aimed for personal guideliness. They are took from my todo list.

Multiple Wesnoth installations

Having multiple wesnoth setups, and working with them at the same time would be nice to be done from inside the plugin. For that, there will be a new preferences page. The page will look something like this:

There will be buttons that add/remove/edit a certain wesnoth installation. When adding/editing an installation, a new page like the current one (in which we set the paths), will be shown. The user will be able to name it and specify the version (I could automatically get the version by invoking wesnoth --version). The current page for setting the paths will act as a default installation.

After we have configured the paths, the user can chose the installation to which the (new) projects are bound, by opening the project properties. There will be a wesnoth-related page, where the user will be able to chose to what installation the project binds. With that, every command that uses the paths, won't use the single current paths, but the ones assigned to each project. This project properties page will open new ways for adding new

There would be a problem in case the project will be migrated between 2 different plugin's versions. For example, in one place we would have defined "1.9.4svn" but on the other there would be "trunk".

We can solve this in 3 ways:

Alert the user that the installation doesn't bind, and let him chose the current matching installation for his plugin.
Default automatically to the current plugin's default installation.
"Hardcode" the version, so that there may be defined just installations like: 1.8.0, 1.9.3. With this we lose some flexibility, but we could use this as base version for new installations, so for example, a user can have two 1.9.3 installations. Provided this, we have again a possible problem, which can be solved again by 1. or 2., but this time, trying to match the base version first.

With this feature, the user won't need to restart the plugin at all, switching workspaces.

Regarding the technical PoV, all paths will be stored using eclipse's preferences system, by appending an unique id of each installation to each path in that set.

Note:

Add the NO_TERRAIN_GFX preprocess option checkbox to the project properties rather than being a global one.

Enhance the Content Assist(CA) feature

The CA framework is very extensible but it lacks proper contexts for providing the content. I was thinking of trying to accomplish the generating of CA list either by using a config file, in case something like that can be used or writing directly the code that does that.

Another factor that is involved in a better auto-complete, is the schema.cfg the plugin bases on. Depending on whether the schema is worked on during this summer of code, I'll need to redo a bit the parser of the schema, in case the format/semantics change.

Project files dependecy tree

In order for the CA to work better - and maybe other areas would benefit from this - we will need to build a project files tree. With that tree, we can see how a certain declared macro/variable will affect subsequent related files and also their scope. Basically, we can see the relation from a file to anothers.

This tree will be (re)built in 2 situations:

A file is added/removed/changed it's name
A full rebuild is triggered (for example, the project is added for the first time in the workspace)

The tree will be built by the WML parser's rules:

files are processed in the alphabetically order
_initial.cfg is the first processed before anything else.
_final.cfg is the last processed after anything else.
if there is a _main.cfg in the directory, it is the only one processed.

Since the current WesnothProjectBuilder doesn't take in account the previous rules, I will create my own method of visiting each folder's children in the correct order. This method will look close to wesnoth's one, namely get_files_from_in_dir() found in src/filesystem.cpp:93.

For a very fast lookup, as a data structure we will use a Hashtable, which has as the key the local project path to the file, and as a value a custom structure, which has the previous/next/parent/son properties, used to traverse the tree. The previous/next go on the same tree level, while the parent/son go up/down in the tree. I will use this instead of a tree, because this will give us access to info about a file, faster, in O(1) compared to a breadth-first search which has O(V+E) - where V is number of vertices and E number of edges.

Talking with Crab about this, he alerted me about a little possible problem with current aproach:

<Crab_> a note about 'The tree will be built by the WML parser's rules: ' - don't forget that you can (conditionally) include files/dirs from a file.
<Crab_> so, if we have 'ai/ais/zzz' which, say, defines a macro, it might get included by campaigns/bbb/scenarios/some_scenario.cfg , even if it is not inside /campaigns/bbb/scenarios

For that we have 2 cases:

{~file/dir path}

or

{file/dir path}

The paths are simply discarded if the current project's directory name is not found in the path.

Currently the so called "Tree" will be actually just a List or a single path tree. For the first iteration of this project, I won't tackle the advanced macro usage that might direct the flow based on defines (this will be optional if there will be time in GSoC, or if not, after GSoC, since I plan to remain active developer to maintain the plugin/enhance it furthermore). So, for example (taken from LoW):

#ifdef CAMPAIGN_LOW

{campaigns/Legend_of_Wesmere/utils}
{campaigns/Legend_of_Wesmere/scenarios}
#endif

#ifdef MULTIPLAYER

#define EASY
#enddef
{campaigns/Legend_of_Wesmere/utils}
{campaigns/Legend_of_Wesmere/scenarios}
#undef EASY
...
#endif

With current way to do this we will just take "blindly" all included directories/files without taking in consideration the #define flow control.

If we are at LoW, the following tree(list) will be created on its project:

_main.cfg -> utils/*.cfg -> scenarios/*.cfg -> units/*.cfg

Each file in the dependency tree, will have associated a number. For more details on why this and how it's done, read the Scoping section.

Scoping

In order for the plugin to know more about the semnatics, it should have the posibility to represent scopes. The scopes will be used mainly for variables and macros.

The Scoping is a delicate problem, because there are some tricky situations that might interfere with the scoping. I choosed to represent scoping in the following way:

Scoping is everything just about order relations. That is, we will compare numbers when dealing with scoping. With that we move the "problem" upper, to the "Dependency tree". We won't use filenames/paths because:

The filename might change, but the order index may not. So, not having to update the index helps us.
Number comparison is faster than string comparison.

So, from now on we'll talk in terms of indexes instead of filenames for scoping.

So, we are working with numbers. But there is still a lil' problem. What happens, when we remove/need to add a new file in the current tree, in the middle of existing 2? We would need to update all subsequent files's indexes. That would not be so ok performance wise.

To counter this problem we will not use consecutive numbers, but rather have a certain step between numbers. The step might be configured inside the plugin based on real statistics (for example, trying to see the number of cfg files per project. Taking a big number, like 1000 won't really hurt). For example, we could take the step 50. That means we would have the following tree for LoW:

   0              50                   100 
_main.cfg -> utils/_main.cfg -> utils/journey.cfg -> ....

instead of:

   0              1                    2 
_main.cfg -> utils/_main.cfg -> utils/journey.cfg -> ....

So, when it's the case of insertion, we insert the new node at the middle of the current "distance". That way we may insert/delete files *without* breaking the order, and our scoping will still work.

There is another problem it rised in my head: how do we know when to put the number as the next step (the previous number + step) or add it in the middle? Well, since every project will surely do a Full Build, we'll use that oportunity to create the initial tree. Afterwards we will just insert files based on the halving of the distance between 2 indices.

The scoping will be represented by a simple structure, consisting of start index and offset and end index and offset.

Content Assist Config(CAC) Files

The CAC files will provide values that are default/available to config files from the wesnoth engine itself. There will be several files that define the values based on their context. This will allow a very great flexibility editing the CA list.

A CAC file will be just a simple text file, that has on each line one (or more) value(s) depending on the context.

The cac/variables.txt will contain variables available. The list is currently available at [1]. For example the file could look like:

$side_number
$turn_number
...

The cac/events.txt will contain the game events. The list is availabe at: [2]. Since the events that have spaces in their names can replace that with an underscore '_', the file will contain the underscore instead space variants also.

Variables

The variables are either defined by user or defined by default by the wesnoth engine. User's variables will be parsed from each processed file by the WesnothProjectBuilder, using the WMLSaxHandler. The default variables will be parsed and added from the CAC file. (cac/variables.txt)

Variables definitions can be triggered using the VARIABLE macro:

{VARIABLE var_name "value"}

or the [set_variable] tag:

[set_variable]
	name=my_var
	value="value"
[/set_variable]

The ConfigFile class in the org.wesnoth.wml.core already has a list of variables. The Variable class has some fields that allow its identification. We will build that list when we parse each file, adding each variable found by the previous defined triggers. When the content assist finds the '$' character, it will supply the user + default variables gathered so far.

Since the variables can be cleared, thus they have scope, the Variable class needs to be refactorted, so it can represent the scope aswell. The scoping was settled before.

Name Collisions

I've considered that we should store the variables per project, like we currently do with macros. This will minimize the memory we need, rather than storing variables on each file - like we do currently. With this, I've discovered it's innevitable to have collisions. That is, have the same variable name on multiple files.

To solve this problem, each variable/macro structure, will not have just a single scope, but rather a list of scopes.

Events

Besides the default event names, supplied by the wesnoth engine, there can be created custom ones, wich will be triggered with [fire_event]. As usual the default events names will be available in the cac/events.txt file.

The events defined by the user will be parsed from the name attribute of the [event] tag.

Attributes

All attributes are currently got from the schema.cfg

Attributes Values

All attributes values should be parsed based on the schema.cfg.

Macros

The support for macros is already built in, but it needs a way to let them have scope. That is, when an #undef is found, the certain macro would not be available to subsequent files unless redeclared. This will be done with the aid of the dependency tree and scoping. The collisions will be solved like I said at the variables section

Support for custom luaWML tags

The lua can create new WML Tags. It would be nice that the plugin suggest those aswell besides the schema.cfg. The lua will be easily parsed, since there is a simple pattern for creating new tags:

wesnoth.wml_actions.<tag name>(cfg)

The parser will just run over the lua code/file, over each line, trying to match the 'function *wesnoth.wml_actions.*' pattern. If it finds something, it will add the tag name to the list of tags. There is also a small thing we might have missed. The user could use a shorthand notation for the actions namespace, so when searching for the pattern, I will add to the 'pattern to match' list, the respective variable when searching for new WML Tags.

Thus, like the wml-tags.lua from the data/lua uses:

local wml_actions = wesnoth.wml_actions

the 'to match' list will contain: 'function *wesnoth.wml_actions.*' and 'function *wml_actions.*', thus finding other tags too.

Going further more, for an optional enhancement, I could also search that function to see if it needs/has some extra attributes. The attributes can be found by:

function's parameter (usually cfg) is asked for a property. For example:

function wml_actions.terrain(cfg)
	local terrain = cfg.terrain or helper.wml_error("[terrain] missing required terrain= attribute")
	cfg = helper.shallow_literal(cfg)
	cfg.terrain = nil
	for i, loc in ipairs(wesnoth.get_locations(cfg)) do
		wesnoth.set_terrain(loc[1], loc[2], terrain, cfg.layer, cfg.replace_if_failed)
	end
end

Here we need the attributes terrain, layer and replace_if_failed.

get_child call:

function wml_actions.message(cfg)
	local show_if = helper.get_child(cfg, "show_if")
	if not show_if or wesnoth.eval_conditional(show_if) then
		engine_message(cfg)
	end
end

Here we need to have the attribute show_if.

Automatic/headless building

The eclipse plugins can be automatically built from command line, provided there is the eclipse sdk and the eclipse delta pack on the builder's machine. The plugin can also use Buckminster tool to automate the building of the plugin.

Notes

Add all plugin's files *inside* a folder to prevent cluttering the directory where it's extracted

More info at:

Standalone app auto-update

The current standalone application doesn't offer the same eclipse's update system. I will add it, so anytime there is a new version of the plugin, it can be updated with a single click by the user, without the needing to download a new standalone app again.

Documentation

External documentation

The current existing readme will be split in 2 manuals: User Manual and Developer Manual (Don't forget to add images where relevant).

The user manual will contain:

Prerequisites and Installing the prerequisites

Note: don't forget to mention the MINIMUM eclipse/java version needed to run

Installing the plugin / standalone version
Using the plugin
Plugin features
Step-by-step use cases
- Setupping the workspace
- Creating a new project and running it in the game
- Importing existing projects
- Using the WML Editor
- Updating the plugin
Frequently Asked Questions

The developer manual will contain:

Prerequisites and Installing the prerequisites
Adding the plugin's projects in the
Compiling and Running the plugin
Deploying the plugin and standalone app

Internal documentation

The Eclipse infrastructure has features for letting the developer embed help into the application, depending on the current context. After I will write the external documentation, I will work on adding internal documentation aswell. Some of it already comes from the external one.

Plugin welcome page
Eclipse Help (When pressing Help->Help contents)
Wizards help

Plugin polishing

This are some tasks needed to be done in order to get the plugin more stable and more usable than it's currently. Some of the tasks are taken from my todo list. New tasks will be added through the project advancing when improvements to actual code /user interaction can be done, so this list is not final.

Don't copy a project that belongs to data/campaigns to the user addons's directory. This can be done by looking at the path when creating a new project.
Enhance the logging so it can help debugging other's problem much easier. I could split the current log in 2:
- Commands executed by the plugin
- Other informations like parsed information/statuses
Enhance the integration with the eclipse platform by adding the "Wesnoth" related wizards directly to the "New" menu.
Allow users to import a directory as a wesnoth project. Under the hood it will either open the wesnoth project (if there is any .project files in it) or create a new empty project on it.
Allow the user to clear it's current plugin's settings just with a simple button.
Wesnoth Project Explorer - Add error markers just like the Java Explorer has
Refactor the current way of getting the macro list. Currently if there is no _main.cfg practicaly the _MACROS_.cfg file is not built resulting in no MACRO suggestions.
Create a view filter to hide the Project _AutoLinked_CFGExternalFiles_ which is created for opening the external config files (external meaning outside the current workspace).
Fix the F3 navigation when the map's location is specified withing " ", as a string.

New parsing method (optional)

The current parsing method involves a lot of Wesnoth preprocessor/wmlparser.py. This is doing somehow the parsing job double. This is because the xtext framework already does 1 pass over the files when added to the project. My plan is to see if I can reuse the already built AST(Abstract Syntax Tree) and the parse tree, to use those values instead of additional preprocessing/wmlparsing, and use the latter only when really needed.

Using the xtext's model will greately fasten the "build" time of each project and it will be *much* simpler to work with, since the framework already provides exceptions recovering (that is, when the code is not well formed according to the grammar) and based on the defined grammar the respective tokens. So, for example instead using 2 another external processes, we could just iterate over each WMLTag and get it's WMLKeys and parse their individual values.

If this idea works, I will add it on the Plugin polishing tasks.

Other ideas

Addons management

There will be a new Addons management view. It will behave much like the in-game addon manager, but it will be more handy since the user doesn't need to start the game for that (and also wait till the cache refreshes after downloading it). This view will be based on the python script from data/tools/wesnoth_addon_manager, like it does currently for uploading the project as an addon.

The view will let the user see the list/details for each of the addons currently on the server. There will be also a filter based on what addon server to use (trunk/1.8/1.9/etc)

The user will be able to just right click on an addon and download it to it's user data folder. Optionally a new project will be created, based on the just downloaded addon.

Unit testing

Grammar

For a better wml handling with in the editor, I will create a Testing class for the current grammar. The class will do testing on all *.cfgs in the data directory. That way after modifing something in the grammar we will be able to see how many errors we have introduced/eliminated. This will greatly help the developing of a better grammar that can hold all current WML formats.

Notes:

optionally rework the grammar and try to use a custom lexer/parser in order to benefit from the preprocessor, injecting the valid tokens. This would help the cases when the current grammar doesn't work:
  {name} = value
  or unclosed tags

Parsing and getting information from files

I will create some basic (or taken from existing campaigns) config files that will be parsed in order to check if the correct behaviour - correct information is extracted from the files.

This will check for extracting the:

scenario name
campaign name
next_scenario
first_scenario
variables
macros

Timeline

The GSOC period is between 24th May and 20th August. But since I have exams for 3 weeks (during 30.05-19.06.2011), I will start working on the plugin before the 24th of May, to recover some of the lost time. Also, my ImagineCup team made it to the national phase - that is in 5,6,7 May. That means till that time I will not be able to do so much work, so I'll make the timeline according with the starting of work on 9 May.

Here is a table with key periods of the time spent on this project.

Period	Actions
26 April	Get accepted
26 April - 8 May	Busy working on ImagineCup project for the national phase. Little or no work done.
9 May - 15 May	Allow headless build of the plugin
16 May - 22 May	Add Support for multiple installations
23 May - 28 May	Add the auto-update feature for the standalone app
30 May - 19 June	Exams
20 June - 24 June	Create the external documentation
25 June - 16 July	Enhance Content Assist
15 July	Mid-term evaluations deadline
17 July - 24 July	Addons management view
25 July - 30 July	Unit tests
1 August - 14 August	Do the Plugin polishing tasks
15 August	Suggested 'pencils down' date. Take a week to scrub code, write tests, improve documentation, etc.
15 August - 17 August	Create the internal documentation.
18 August - 22 August	Buffer half-week. This will be used in case some tasks fell a bit outside their proposed time.
22 August	Firm 'pencils down' date. Mentors, students and organization administrators can begin submitting final evaluations to Google.

Tasks

This is a list of the "atomic" tasks that need to be done, based on the project part. Since the big tasks are split into smaller task, the progress will be seen better and any problems that arise with the timeline will be seen faster. I will update this table based on my current progress.

Project part	Task	Status (Legend: None, Done, Not done, In Progress)
Headless build	Research about what system to use for automatic building of the plugin	Done
Headless build	Implement the chosen system for the wesnoth plugin	Done
Multiple installations	Implement the preferences pages	Done
Multiple installations	Implement the logic for saving and loading the installation's paths on the disk	Done
Multiple installations	Implement the project preferences page	Done
Multiple installations	Rewrite the current mechanisms that use the paths to use the ones based on the project.	In progress
Auto-update	Implement the auto-update feature.	In progress
Documentation	Write the User Manual	Done
Documentation	Write the Developer Manual	Done
Documentation	Write the internal documentation	In progress
Enhance Content Assist	Project dependency tree	None
Enhance Content Assist	Content assist files	None
Enhance Content Assist	Variables	None
Enhance Content Assist	Events	None
Enhance Content Assist	Lua tags	None
Addon management	Create the GUI of the view	None
Addon management	Create the view's logic	None
Unit testing	Tests for the grammar	None
Unit testing	Tests for parsing the config files	None

Optional tasks will be dealt with if the time allows it:

Project part	Task	Status (Legend: None, Done, Not done, In Progress)
Enhance Content Assist	Project dependency tree with multiple paths based on defines	None
Enhance Content Assist	Lua tags attributes	None
Grammar	Enhance the grammar to support the 2 current unsupported syntax constructs	None
Documentation	Implement a simple WML project from start to end using Help's built-in cheat sheets tutorial	None

Questionnaire

The questionnaire can be found here: http://wiki.wesnoth.org/User:Timotei21/Questionaire

@@ Line 385: / Line 385: @@
 | Multiple installations || Implement the project preferences page || <span style="color: green;">Done</span>
 |-
-| Multiple installations || Rewrite the current mechanisms that use the paths to use the ones based on the  project. || <span style="color: green;">Done</span>
+| Multiple installations || Rewrite the current mechanisms that use the paths to use the ones based on the  project. || <span style="color: gold;">In progress</span>
 |-
 | Auto-update || Implement the auto-update feature. || <span style="color: gold;">In progress</span>
 |-
-| Documentation || Write the User Manual ||  <span style="color: gold;">In Progress</span>
+| Documentation || Write the User Manual || <span style="color: green;">Done</span>
 |-
 | Documentation || Write the Developer Manual || <span style="color: green;">Done</span>
 |-
-| Documentation || Write the internal documentation || None
+| Documentation || Write the internal documentation || <span style="color: gold;">In progress</span>
 |-
 | Enhance Content Assist || Project dependency tree || None

Difference between revisions of "SummerofCode2011 Timotei21"