Difference between revisions of "Maintenance tools"

From The Battle for Wesnoth Wiki
m (wmlindent)
(wmlscope: Update with info from the big block comment at the top of the script)
 
(67 intermediate revisions by 21 users not shown)
Line 1: Line 1:
 +
<div class="floatright"> __TOC__ </div>
 +
 
The Wesnoth source code distribution includes a couple of tools intended to help authors maintain campaigns, faction & unit packs, and other WML resources. These
 
The Wesnoth source code distribution includes a couple of tools intended to help authors maintain campaigns, faction & unit packs, and other WML resources. These
 
are:
 
are:
Line 4: Line 6:
 
; wmlscope: a cross-reference lister, useful for finding unresolved macro and resource-file references.
 
; wmlscope: a cross-reference lister, useful for finding unresolved macro and resource-file references.
  
; wmllint: a utility for porting your old WML to the current version of WML.   
+
; wmllint: a utility for sanity-checking WML syntax and porting your old WML to the current version of WML.   
  
 
; wmlindent: a utility for reindenting WML to a uniform style.
 
; wmlindent: a utility for reindenting WML to a uniform style.
  
You will need a Python interpreter on your system to use these tools. Linux, *BSD, and Mac OS/X should already have Python installed; for Windows it's a free download
+
; GUI.pyw: a graphical interface
from http://www.python.org.  You will also need to know how to run command-line tools
 
on your system.
 
  
Both tools will require you to supply a <i>directory list</i>.  This is a set of directories containing the WML files you want to work on.
+
== General Information ==
 +
 
 +
You will need a Python 3 interpreter on your system to use these tools.  Linux, *BSD, and Mac OS/X should already have Python 3 installed; for Windows it's a free download
 +
from http://www.python.org.  You will also need to know how to run command-line tools on your system.
 +
 
 +
If you're working with Debian or Ubuntu you might have to install the package wesnoth-1.16-tools (or the convenient version).
 +
sudo apt install wesnoth-1.16-tools
 +
 
 +
 
 +
All three tools will require you to supply a <i>directory list</i>.  This is a set of directories containing the WML files you want to work on.
 +
 
 +
This page is intended as documentation for users.
 +
 
 +
<u>Note to Windows Users:</u> This means you have to run it from the '''Command Line'''. The command line may be reached by hitting Start, then Run, then "cmd" or "command" depending on your version of Windows.
 +
 
 +
Example uses:
 +
python wmllint path\to\files
 +
python wmlindent path\to\files
 +
 
 +
Another example:
 +
"C:\Program Files\Python3.7\python.exe" data\tools\wmllint --dryrun data\core data\{multiplayer,themes} data\campaigns
 +
(You have to specify the full directory path to the executable if you don't have your environment variables set up correctly).
 +
The first thing you type is the path to your python executable, followed by a space. The second thing you type is the path to the desired script to run, followed by a space. The third thing you type is the path to the folder (or file) to be processed.
 +
 
 +
 
 +
'''A convenient way of running wmllint''' on Linux (Debian or Ubuntu) and Windows in comparison, '''Linux''':
 +
 
 +
Assuming we're working with wesnoth 1.16 or more advanced versions.
 +
python3 /usr/share/games/wesnoth/1.16/data/tools/wmllint --dryrun /usr/share/games/wesnoth/1.16/data/core ~/.local/share/wesnoth/1.16/data/add-ons/A_Simple_Campaign 1>wmllint-run.log 2>wmllint-err.log
 +
I have these commands inside of a file named
 +
wmllint_dryrun_ASC.sh
 +
and execute it by opening a shell (=terminal, console, command window, bash,...), navigating into the directory with that file and typing
 +
bash wmllint_dryrun_ASC.sh
 +
The python3 command should be automatically known on Debian. The path to the script tells the python interpreter what to execute. --dryrun: A wmllint option, see below. The path to the core files is needed to let wmllint know about e.g. defined core units, followed by the path to the add-on that shall be checked; the last two commands cause the result of the wmllint usage to be written into those files in the same directory as the script.
 +
'''Windows''', this is logically exactly the same as the Linux shell script above, so if you are on a Mac you can probably conclude how you need to adapt the paths:
 +
E:\Python37\python.exe E:\Programme\Wesnoth_1.16_git\data\tools\wmllint --dryrun E:\Programme\Wesnoth_1.16_git\data\core E:\Programme\Wesnoth_1.16_git\userdata\data\add-ons\A_Simple_Campaign 1>wmllint-run.log 2>wmllint-err.log
 +
This is the content of a .txt file, whose extension I rename to .bat and double-click onto it. Opening a command window is not needed this way.
 +
Since Python isn't natively installed on windows and I don't have environment variables set, the full path to python.exe is given. If your directories contain spaces it may help to include the path in quotes:
 +
"C:\Programs\Battle for Wesnoth 1.16\data\tools\wmllint"
 +
Remember that you do not need to enter all of the commands/paths at once. If it doesn't work, start with only "python" or "C:\Python37\python.exe" or the like and interpret the error messages that you get. If you get an "unknown command", python isn't installed or environment variables aren't set correctly. After that, you can add the later commands one by one.
  
 
== wmlscope ==
 
== wmlscope ==
  
The main use for <tt>wmlscope</tt> is to find WML macro references without definitions and references to resource files (sounds and images) that don't exist.  These are difficult to spot from in-game because they usually result in silence or a missing image rather than actual broken game logic.  They may happen because of typos in your WML, or because the name of a macro or the location of a resource file changed betweeen versions of the game.
+
The main use for <tt>wmlscope</tt> is to find WML macro references without definitions and references to resource files (sounds and images) that don't exist.  These are difficult to spot from in-game because they usually result in silence or a missing image rather than actual broken game logic (see [https://github.com/wesnoth/wesnoth/issues/5332 issue 5332] for more info).  They may happen because of typos in your WML, or because the name of a macro or the location of a resource file changed between versions of the game.
 +
 
 +
<tt>wmlscope</tt> also checks macro invocations for consistency.  It will complain
 +
if a macro is called with the wrong number of arguments.  In most cases it can deduce information about the type of the literal expected to be passed to a given macro argument by looking at the name of the formal.
 +
 
 +
<table class="wikitable"><tr>
 +
<th>Type</th>
 +
<th>Meaning</th>
 +
<th>Formals requiring this type</th>
 +
<th>Literals of this type</th>
 +
</tr>
 +
<tr>
 +
<td>side</td>
 +
<td>a single side number</td>
 +
<td>SIDE, *_SIDE, SIDE[0-9]</td>
 +
<td>a numeric or "global"</td>
 +
</tr>
 +
<tr>
 +
<td>numeric</td>
 +
<td>a numeric integer literal</td>
 +
<td>SIDE, X, Y, RED, GREEN, BLUE, TURN, PROB, LAYER, TIME, *_SIDE, *NUMBER, *AMOUNT, *COST, *RADIUS, *_X, *_Y, *_INCREMENT, *_FACTOR, *_TIME, *_SIZE, DURATION</td>
 +
<td>\-?[0-9]+</td>
 +
</tr>
 +
<tr>
 +
<td>percentage</td>
 +
<td>a percentage</td>
 +
<td>*PERCENTAGE</td>
 +
<td>a numeric or 0\.[0-9]+</td>
 +
</tr>
 +
<tr>
 +
<td>position</td>
 +
<td>a single x,y coordinate</td>
 +
<td>POSITION, *_POSITION, BASE</td>
 +
<td>-?[0-9]+,-?[0-9]+</td>
 +
</tr>
 +
<tr>
 +
<td>span</td>
 +
<td>a set of coordinates or coordinate ranges</td>
 +
<td>*_SPAN</td>
 +
<td>a numeric, position or ([0-9]+\-[0-9]+,?|[0-9]+,?)+</td>
 +
</tr>
 +
<tr>
 +
<td>alliance</td>
 +
<td>a set of side numbers</td>
 +
<td>SIDES, *_SIDES</td>
 +
<td>a span, or the empty string</td>
 +
</tr>
 +
<tr>
 +
<td>range</td>
 +
<td>an attack range</td>
 +
<td>RANGE</td>
 +
<td>"melee" or "ranged"</td>
 +
</tr>
 +
<tr>
 +
<td>alignment</td>
 +
<td>an alignment keyword</td>
 +
<td>ALIGN</td>
 +
<td>"lawful" or "neutral" or "chaotic"</td>
 +
</tr>
 +
<tr>
 +
<td>types</td>
 +
<td>a set of unit types</td>
 +
<td>TYPES</td>
 +
<td>a shortname, name, or anything that contains spaces and matches no other type</td>
 +
</tr>
 +
<tr>
 +
<td>terrain_pattern</td>
 +
<td>a set of terrain codes to filter</td>
 +
<td>ADJACENT*, TERRAINLIST*, *TERRAIN_PATTERN, RESTRICTING</td>
 +
<td>a terrain_code or name</td>
 +
</tr>
 +
<tr>
 +
<td>terrain_code</td>
 +
<td>a single terrain code, perhaps with overlay</td>
 +
<td>TERRAIN*, *TERRAIN</td>
 +
<td>a shortname or (\*|[A-Z][a-z]+)\^([A-Z][a-z\\|/]+\Z)?</td>
 +
</tr>
 +
<tr>
 +
<td>shortname</td>
 +
<td>a terrain code or a short, capitalized variable name</td>
 +
<td></td>
 +
<td>[A-Z][a-z][a-z]?</td>
 +
<tr>
 +
<td>name</td>
 +
<td>a name or identifier</td>
 +
<td>NAME, VAR, IMAGESTEM, ID, FLAG, *_NAME, *_ID, NAMESPACE, BUILDER, *_VAR</td>
 +
<td>anything without spaces that matches no other type</td>
 +
</tr>
 +
<tr>
 +
<td>optional_string</td>
 +
<td>a string value (may be empty)</td>
 +
<td>ID_STRING, NAME_STRING, DESCRIPTION, IPF</td>
 +
<td>a string, or the empty string</td>
 +
</tr>
 +
<tr>
 +
<td>string</td>
 +
<td>a nonempty string not matching any of the preceding types</td>
 +
<td>STRING, TYPE, TEXT, *_STRING, *_TYPE, *_TEXT</td>
 +
<td>a shortname, a name, a stringliteral, or anything that contains spaces and matches no other type</td>
 +
</tr>
 +
<tr>
 +
<td>stringliteral</td>
 +
<td>a string in doublequotes or a translated string</td>
 +
<td></td>
 +
<td>".*" or _.* but not _[a-z].*</td>
 +
</tr>
 +
<tr>
 +
<td>image</td>
 +
<td>an image path, perhaps with [[ImagePathFunctionWML|image path functions]]</td>
 +
<td>*IMAGE, PROFILE</td>
 +
<td>[A-Za-z0-9{}.][A-Za-z0-9_/+{}.-]*\.(png|jpg)(?=(~.*)?)</td>
 +
</tr>
 +
<tr>
 +
<td>sound</td>
 +
<td>a music or sound filename</td>
 +
<td>MUSIC, SOUND</td>
 +
<td>string ending with ".wav" or ".ogg"</td>
 +
</tr>
 +
<tr>
 +
<td>filter</td>
 +
<td>[[FilterWML|WML filter]]</td>
 +
<td>FILTER</td>
 +
<td>any non-quoted string containing "="</td>
 +
</tr>
 +
<tr>
 +
<td>WML</td>
 +
<td>arbitrary WML fragment</td>
 +
<td>WML, *_WML</td>
 +
<td>any non-quoted string containing "=", or the empty string</td>
 +
</tr>
 +
<tr>
 +
<td>affix</td>
 +
<td>a prefix, suffix, or infix for a variable name</td>
 +
<td>AFFIX, *AFFIX, POSTFIX, ROTATION</td>
 +
<td>a shortname or name, or the empty string</td>
 +
</tr>
 +
<tr>
 +
<td>any</td>
 +
<td>anything</td>
 +
<td>*VALUE, [ARS][0-9]</td>
 +
<td>anything</td>
 +
</tr>
 +
</table>
 +
 
 +
If the actual argument is a macro call {.*}, then it matches any formal.  Otherwise, if the formal has an identifiable type, <tt>wmlscope</tt> will complain if the actual literal does not match it.
 +
 
 +
The argument type check only works in macro calls that fit on a single line.
  
 
<tt>wmlscope</tt> has many options for changing the reports it generates; the more advanced ones are intended for Wesnoth developers.  Invocations for the most commonly useful reports it generates are included in <i>data/tools/Makefile</i> of the source distribution. Here are some of those reports:
 
<tt>wmlscope</tt> has many options for changing the reports it generates; the more advanced ones are intended for Wesnoth developers.  Invocations for the most commonly useful reports it generates are included in <i>data/tools/Makefile</i> of the source distribution. Here are some of those reports:
  
; make unresolved: Report on unresolved macro calls and resource references.  (This is what you are most likely to want to do).  Note that in the current version this report works for mainline but still contains some false positives (warnings about bad references that aren't actually bad) when you are checking UMC.  The path-resolution logic is being worked on.
+
; make unresolved: Report on unresolved macro calls and resource references; also report macro argument-type mismatches.  (This is what you are most likely to want to do).  
  
 
; make all: Report all macro and resource file references, not just unresolved ones.
 
; make all: Report all macro and resource file references, not just unresolved ones.
Line 30: Line 215:
  
 
; -h, --help:                Emit a help message and quit
 
; -h, --help:                Emit a help message and quit
; -c, --crossreference:      Report resolved macro references (implies -w 1)
+
; -c, --crossreference:      Report resolved macro references (implies <tt>-w 1</tt>)
 
; -C, --collisions:          Report duplicate resource files   
 
; -C, --collisions:          Report duplicate resource files   
 
; -d, --deflist:              Make definition list.  (This one is for campaign server maintainers.)
 
; -d, --deflist:              Make definition list.  (This one is for campaign server maintainers.)
Line 37: Line 222:
 
; -l, --listfiles:            List files that will be processed
 
; -l, --listfiles:            List files that will be processed
 
; -r <i>ddd</i>, --refcount=<i>ddd</i>:    Report only on macros with references in exactly <i>ddd</i> files.
 
; -r <i>ddd</i>, --refcount=<i>ddd</i>:    Report only on macros with references in exactly <i>ddd</i> files.
 +
; -t <i>TYPELIST</i>, --typelist <i>TYPELIST</i>: List actual & formal argtypes for calls in fname
 
; -u, --unresolved:          Report unresolved macro references
 
; -u, --unresolved:          Report unresolved macro references
 
; -w, --warnlevel:            Set to 1 to warn of duplicate macro definitions
 
; -w, --warnlevel:            Set to 1 to warn of duplicate macro definitions
; --forced-used reg:         Ignore reference count 0 on names matching regexp
+
; -p, --progress:            Show progress
 +
; --force-used reg:           Ignore reference count 0 on names matching regexp
 
; --extracthelp:              Extract help from macro definition comments.
 
; --extracthelp:              Extract help from macro definition comments.
 +
; --unchecked:                Report all macros with untyped formals.
 +
; --version:                  show program's version number and exit
 +
 +
These options are used with a list of directories as arguments; if none is given,
 +
<tt>wmlscope</tt> behaves as though the current directory had been specified as a
 +
single argument.  Each directory is treated as a separate domain for
 +
macro and resource visibility purposes.
 +
 +
<tt>wmlscope</tt> recognizes two kinds of namespace, exporting and non-exporting.
 +
Exporting namespaces make all their resources and macro names
 +
globally visible.  You can make a namespace exporting by embedding
 +
a comment like this in it:
 +
 +
    # wmlscope: export=yes
 +
 +
Wesnoth core data is an exporting namespace.  Campaigns are non-exporting;
 +
they should contain the declaration
 +
 +
    # wmlscope: export=no
 +
 +
somewhere.  <tt>wmlscope</tt> will complain when it sees a namespace with no export
 +
property, then treat it as non-exporting.
 +
 +
You can tell <tt>wmlscope</tt> to ignore stretches of config files
 +
with the following magic comments:
 +
 +
    # wmlscope: start ignoring
 +
    # wmlscope: stop ignoring
 +
 +
Similarly, you can tell <tt>wmlscope</tt> to ignore multiple or duplicate macro
 +
definitions in a range of lines with the following magic comments:
 +
 +
    # wmlscope: start conditionals
 +
    # wmlscope: stop conditionals
 +
 +
The following magic comment:
 +
 +
    # wmlscope: prune FOOBAR
 +
 +
will cause <tt>wmlscope</tt> to forget about all but one of the definitions of
 +
<tt>FOOBAR</tt> it has seen.  This will be useful mainly for symbols that have
 +
different definitions enabled by an <tt>#ifdef</tt>.
 +
 +
Due to a preprocessor limitation, inline macros cannot contain a documentation
 +
string. If you need to document these macros in the HTML macro reference, you
 +
can use the following directive:
 +
 +
    # wmlscope: docstring FOOBAR
 +
 +
The docstring for the FOOBAR macro will be collected until a non-comment line,
 +
a <tt>#define</tt> or another <tt># wmlscope: docstring</tt> are found. External
 +
docstrings '''''must''''' be defined before the macro to which they refer; defining
 +
two or more external docstrings keeps only the most recent one, but having both an
 +
external and an internal docstring is allowed (in this case, the internal one
 +
will be appended to the external one in the macro reference).
 +
 +
This tool does catch one kind of implicit reference: if an attack name
 +
is specified but no icon is given, the attack icon will default to
 +
a name generated from the attack name.  This behavior can be suppressed
 +
by adding a magic comment containing the string "no-icon" to the <tt>name=</tt>
 +
line.
 +
 +
The checking done by this tool has a couple of flaws:
  
Note: in 1.3.2 and earlier versions, this tool was named '''macroscope'''.
+
(1) It doesn't actually evaluate file inclusions.  Instead, any
 +
macro definition satisfies any macro call made under the same
 +
directory.  Exception: when an <tt>#undef</tt> is detected, the macro is
 +
tagged local and not visible outside the span of lines where it was
 +
defined.
 +
 
 +
(2) It doesn't read <tt>[binary_path]</tt> tags, as this would require
 +
implementing a WML parser.  Instead, it assumes that a resource-file
 +
reference can be satisfied by any matching image file from anywhere
 +
in the same directory it came from.  The resources under the '''''first'''''
 +
directory argument (only) are visible everywhere.
 +
 
 +
(3) A reference with embedded {}s in a macro will have the macro's
 +
formal args substituted in at WML evaluation time. Instead, this
 +
tool treats each {} as a .* wildcard and considers the reference to
 +
match '''''every''''' resource filename that matches that pattern.
 +
Under appropriate circumstances this might report a resource filename
 +
statically matching the pattern as having been referenced even
 +
though none of the actual macro calls would actually generate it.
 +
 
 +
Problems (1) and (2) imply that this tool might conceivably report
 +
that a reference has been satisfied when under actual
 +
WML-interpreter rules it has not.
 +
 
 +
The reporting format is compatible with GNU Emacs compile mode.
 +
 
 +
For debugging purposes, an in-line comment of the form
 +
 
 +
    # wmlscope: warnlevel NNN
 +
 
 +
sets the warning level.
  
 
== wmllint ==
 
== wmllint ==
Line 50: Line 330:
 
* Resource files and macro names may change between versions of the game. <tt>wmllint</tt> knows about these changes and will tweak your WML to fit where it can.
 
* Resource files and macro names may change between versions of the game. <tt>wmllint</tt> knows about these changes and will tweak your WML to fit where it can.
  
* Between 1.2.x and 1.3.1 the terrain-coding system used in map files underwent a major change. It changed again in a minor way between 1.3.1 and 1.3.2. <tt>wmllint</tt> will translate your maps for you, unless you use custom terrains in which case you will have to do it by hand.
+
* Between 1.2.x and 1.3.1 the terrain-coding system used in map files underwent a major change. It changed again in a minor way between 1.3.1 and 1.3.2. If you port such old code, use <tt>wmllint-1.4</tt>, which is located in the same directory as <tt>wmllint</tt>. It will translate your maps for you, unless you use custom terrains in which case you will have to do it by hand.
 +
 
 +
<tt>wmllint</tt> also performs various sanity-checking operations, reporting:
 +
 
 +
* unbalanced tags
 +
* strings that need a translation mark and do not have them
 +
* strings that have a translation mark and should not
 +
* translatable strings containing macro references
 +
* filter references by description= (id= in 1.5) not matched by an actual unit
 +
* abilities or traits without matching special notes, or vice-versa
 +
* consistency between recruit= and recruitment_pattern= instances
 +
* double space after punctuation in translatable strings.
 +
* unknown races or movement types in units
  
 
<tt>wmllint</tt> takes a directory-path argument specifying the WML directories to work on.  It will modify any cfg and map files under those directories that need to be changed.  Here is a summary of its options:
 
<tt>wmllint</tt> takes a directory-path argument specifying the WML directories to work on.  It will modify any cfg and map files under those directories that need to be changed.  Here is a summary of its options:
  
 
; -h, --help:                Emit a help message and quit.
 
; -h, --help:                Emit a help message and quit.
; -d, --dryrun:              List changes but don't perform them.
 
; -o, --oldversion:          Specify version to begin with (not usually needed).
 
; -v, --verbose:              Set verbosity; more details below.
 
 
; -c, --clean:                Clean up -bak files.
 
; -c, --clean:                Clean up -bak files.
; -D, --diff:                 Display diffs between unconverted and unconverted files (only on Linux, *BSD or Mac).
+
; -D, --diffs:               Display diffs between converted and unconverted files.
 +
; -d, --dryrun:              List changes (-v) but don't perform them.
 
; -r, --revert:              Revert the conversion from the -bak files.
 
; -r, --revert:              Revert the conversion from the -bak files.
 +
; -m, --missing:              Don't warn about tags without side= keys now applying to all sides.
 +
; -s, --stripcr:              Convert DOS-style CR/LF to Unix-style LF.
 +
; -v, --verbose:              Set verbosity; more details below.
 +
; -K, --known:                Suppress check for unknown unit types, recruits, races, scenarios, etc.
 +
; -S, --nospellcheck:        Suppress spellchecking
 +
; --version:                  show program's version number and exit
  
 
The verbosity option works like this:
 
The verbosity option works like this:
Line 78: Line 374:
 
# Use either --clean to remove the -bak files or --revert to undo the conversion.
 
# Use either --clean to remove the -bak files or --revert to undo the conversion.
  
Note: in 1.3.2 and earlier versions this tool was named '''upconvert'''.
+
Additionally, wmllint tries to locate a spell checker on your system and spell-checks storyline and message strings.  It will work automatically with any of [https://github.com/AbiWord/enchant enchant]'s spellchecking backends (including aspell, myspell, ispell, applespell, hunspell, nuspell, and so on, depending on the version of enchant), provided you have the <tt>enchant.py</tt> Python library installed. Note that the spellchecking results can vary depending on which backend enchant decides to use.
 +
 
 +
wmllint supports a number of magic comments to customize its behaviour and avoid false positives. All magic wmllint comments begin with the string <tt>wmllint:</tt>, followed by some additional keyword and potentially some arguments. In the below explanations, a string of the form <code>[a|b]</code> means you may use either <code>a</code> or <code>b</code> at that location, while a string of the form <code><arg></code> indicates free text substitution.
 +
 
 +
* <code>ignore</code>: Disables checking of terrains and translation marks on the current line only.
 +
* <code>noconvert</code>: Disables conversion of terrains and image/sound filenames on the current line only.
 +
* <code>markcheck [on|off]</code>: Enables or disables translation mark checking from the current (next?) line onward.
 +
* <code>no-icon</code>: Prevents description insertions (what does that mean?)
 +
* <code>recognize <name></code>: Indicates that the character with the given name exists even though a declaration (ie a <code>[unit]</code> or <code>[side]</code> tag) is not visible.
 +
* <code>whofield <macro> <number></code>: Indicates that the specified macro declares a character whose name is given by the specified argument to the macro.
 +
* <code>whofield clear <macro></code>: Removes the <code>whofield</code> definition for the specified macro.
 +
* <code>who <macro> is <name> <name> ...</code>: Indicates that the specified macro declares one or more characters whose names are given as a space-separated list. This can also be used for macros that auto-recall a list of characters, some of which join later or may die at some point. If a definition for the macro already exists, then the specified names are appended to it. If a name is preceded by a double minus (<code>-- <name></code>) then it is removed from the definition.
 +
* <code>unwho all|<name></code>: Removes the <code>who</code> definition for the specified macro, or all <code>who</code> definitions.
 +
* <code>usage of "<unit>" is <class></code>: Declares the usage of the specified unit (the <code>usage=</code> key in the <code>[unit_type]</code> tag). Useful if you are using macros to generate several similar unit types.
 +
* <code>usagetype <class></code>: Declares a valid usage type for units. <code>usagetypes</code> is also recognized, and a comma-separated list of usage types can be specified.
 +
* <code>validate-[on|off]</code>: Enables or disables stack-based validation checks. Use when you have unbalanced tags in macros.
 +
* <code>unbalanced-[on|off]</code>: Similar to above, the precise difference is unclear.
 +
* <code>no translatables</code>: Suppresses warnings about a missing textdomain declaration in the current file. Make sure the file really does have no translatable strings!
 +
* <code>display [on|off]</code>: Enable or disable warnings about newlines in messages.
 +
* <code>notecheck [on|off]</code>: Enable or disable note consistency checks for unit descriptions.
 +
* <code>deathcheck [on|off]</code>: Enable or disable the check for units speaking in their death events.
 +
* <code>[general|directory|local] spellings <word> <word> ...</code>: Declares the specified space-separated list of words to be valid spellings in the specified context. The context <code>local</code> indicates the current file only, while <code>directory</code> means the current file and any siblings in the same directory or subdirectories. The <code>global</code> context indicates the spellings are valid anywhere.
 +
* <code>no spellcheck</code>: Disables spell checking on the current line.
 +
* <code>skip-side</code>: Indicates that there is a missing side declaration at this location that will be provided by a macro expansion.
 +
* <code>match <string> with <notes_macro></code>: Indicates that uses of the specified string in a unit definition (usually a macro, including the curly braces) should be matched up with the specified special notes macro (which is also specified as the full macro with curly braces).
 +
* <code>no ellipsecheck</code>: Disables checking of unit ellipses on the current line.
 +
 
 +
=== Explanations of <tt>wmllint</tt> diagnostics ===
 +
Some <tt>wmllint</tt> diagnostics may require further explanation for UMC authors to understand; this section will be for providing such explanations, and descriptions of how to solve and/or silence them.
 +
 
 +
In these, <code>%s</code> will be replaced by a string. When the recommended solution also includes a <code>%s</code>, it's a suggestion to copy the string from the error message into your code.
 +
 
 +
* <code>nonstandard word-wrap style within message</code>: This message meant there was an unexpected newline character within a <code>[message]</code> tag. However, this check was removed in the 1.15.10 release. Pre-1.15.10, add-on developers could silence by putting a <code># wmllint: display on</code> comment before the string and a <code># wmllint: display off</code> comment after the string, however, post-1.15.10, this is no longer necessary.
 +
* <code>%s is not a known unit type</code> (in cases where you'd think the unit type ''would'' be known): This means the unit has a type that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can load the mainline unit types. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>unknown speaker '%s' of [message]</code>: use a <code># wmllint: recognize %s</code> magic comment, or, alternatively, if the speaker is created by a macro, use a magic comment of the form of either <code># wmllint: who MACRO is SPEAKER</code> or <code># wmllint: whofield MACRO NUMBER</code>, depending on whether the macro takes an argument for the unit's name or not.
 +
* <code>unknown '%s' referred to by id</code>: use a <code># wmllint: recognize %s</code> magic comment, or, alternatively, if the unit is created by a macro, use a magic comment of the form of either <code># wmllint: who MACRO is UNIT</code> or <code># wmllint: whofield MACRO NUMBER</code>, depending on whether the macro takes an argument for the unit's name or not.
 +
* <code>%s has unknown advancements</code> (in cases where you'd think the advancement ''would'' be known): This means the unit has an advancement that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can load mainline units for checking advancements. Note that it's also possible that you just made typo, too, so be sure to check your spelling. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>.description may need hand fixup</code>: This one comes from mucking around with the <code>.description</code> field of unit data manually in a hackish fashion. There isn't really much of a way to work around it, besides just the "don't do that" solution.
 +
* <code>tag stack nonempty (%s) at end of file.</code>: This means that you have unbalanced tags somewhere in the file, e.g. an opener without a closer, or vice versa. This can often be seen when defining macros for unit abilities. A way to fix this warning is to wrap the section with unbalanced tags with a <code># wmllint: unbalanced-on</code> magic comment beforehand and a <code># wmllint: unbalanced-off</code> magic comment afterwards. As having unbalanced tags will also cause issues for other WML maintenance tools, such as <tt>wmlindent</tt> and <tt>wmlxgettext</tt>, you may also want to add separate magic comments for each of them (see their documentation for the form they take).
 +
* <code>unit declaration without side attribute</code>: the default side for a unit declaration when left implicit is side 1. Specify your unit sides explicitly to solve this.
 +
* <code>no %s units recruitable at difficulty %s</code> (even when there are such units recruitable): This diagnostic has to do with matching the <code>usage</code> key of units recruitable by an AI side with their <code>recruitment_pattern</code>. It means the unit has a usage that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can know which mainline units are recruitable. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>%s has unknown movement type</code> (even when you'd think that that movement type ''would'' actually be known): This means the unit has a movetype that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can load the mainline movement types. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>%s has unknown race</code> (even when you'd think that that race ''would'' actually be known): This means the unit has a race that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can load the mainline races. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>derivation of %s from %s does not resolve</code> (even when you'd think it would): This means that a unit using the [[UnitTypeWML#Other_tags|[base_unit]]] tag specifies a unit ID in that tag that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using <tt>wmllint</tt> (which is something that the GUI version of it will do automatically for you), so that it can load the core units for its derivation checks. (You can also silence this warning by passing the <tt>-K</tt> flag to <tt>wmllint</tt>.)
 +
* <code>[advancefrom] needs to be manually updated to [modify_unit_type] and moved into the _main.cfg file</code>: This one is pretty self-explanatory: [[UnitTypeWML#Unit_Type|[advancefrom]]] was deprecated in [https://github.com/wesnoth/wesnoth/commit/3950f40f3f0483032bc70b3e57166bd355acd9fc commit 3950f40] due to [https://github.com/wesnoth/wesnoth/issues/3955 issue #3955], and in fact doesn't even work anymore (in 1.16) as per [https://github.com/wesnoth/wesnoth/issues/6204 issue #6204]. The main reason <tt>wmllint</tt> can't fix this automatically is because it could end up being too complicated for it to figure out which files to edit if there are multiple uses of <code>[advancefrom]</code>, and it also doesn't want to assume where to put the [[ModificationWML|[modify_unit_type]]] tag in <tt>_main.cfg</tt>.
  
 
== wmlindent ==
 
== wmlindent ==
Line 84: Line 424:
 
Call with no arguments to filter WML on standard input to reindented WML on
 
Call with no arguments to filter WML on standard input to reindented WML on
 
standard output.  If arguments are specified, they are taken to be files to be
 
standard output.  If arguments are specified, they are taken to be files to be
re-indented in place; interrupting will be safe, as each reindenting
+
re-indented in place; a directory name causes reindenting on all WML
will be done to a copy that is atomically renamed when it's done. This
+
beneath it.
code never modifies anything but leading whitespace on lines.
 
  
 
The indent unit is four spaces.  Absence of an option to change this is
 
The indent unit is four spaces.  Absence of an option to change this is
deliberate; the purpose of this tool is to prevent style wars, not encourage
+
deliberate; the purpose of this tool is to ''prevent'' style wars, not encourage
 
them.
 
them.
 +
 +
On non-empty lines, this code never modifies anything but leading and
 +
trailing whitespace. Leading whitespace will be regularized to the
 +
current indent; trailing whitespace will be stripped.  After processing
 +
all lines will end with a Unix-style <code>\n</code> end-of-line marker.
 +
 +
Runs of entirely blank lines will be reduced to one blank line, except
 +
in two cases where they will be discarded: (a) before WML closing
 +
tags, and (b) after WML opening tags.
 +
 +
It is possible to wrap a section of lines in special comments so that
 +
<tt>wmlindent</tt> will ignore them.  You may need to do this for unbalanced
 +
macros (it's better, though, to get rid of those where possible).
 +
Use '<code>wmlindent: {start,stop} ignoring</code>' anywhere in a comment.
 +
 +
It is also possible to declare custom openers an closers, e.g for macros
 +
that are actually control constructs.  To do this, use declarations
 +
 +
    # wmlindent: opener "{EXCEPTIONAL_OPENER "
 +
    # wmlindent: closer "{EXCEPTIONAL_CLOSER "
 +
 +
The lines after an opener will be indented an extra level; a closer
 +
and lines following will be indented one level less. Note that these
 +
declare prefixes; any prefix match to the non-whitespace text of a line
 +
will be recognized.
 +
 +
The public utility macros "<code>{FOREACH</code>" and "<code>{NEXT</code>" come as wired-in exceptions,
 +
because it is not guaranteed that their indent declarations will be processed
 +
before the macro library is reached.
 +
 +
Interrupting <tt>wmlindent</tt> ought to be safe, as each reindenting will be done to a copy
 +
that is atomically renamed when it's done.  If the output file is identical
 +
to the input, the output file will simply be deleted, so the timestamp
 +
on the input file won't be touched.
 +
 +
The <tt>--dryrun</tt> option detects and reports files that would be changed
 +
without changing them. The <tt>--verbose</tt> or <tt>-v</tt> option enables reporting
 +
of files that are (or would be, under <tt>--dryrun</tt>) changed.  With <tt>-v -v</tt>,
 +
unchanged files are also reported.  The <tt>--exclude</tt> option takes a regexp
 +
and excludes files matching it.
 +
 +
If you don't apply this tool to your own WML that you wish to submit, the
 +
mainline-campaign maintainers will do it when and if your code is accepted into the tree.
  
 
Note: This tool does not include a parser.  It will produce bad results on WML
 
Note: This tool does not include a parser.  It will produce bad results on WML
 
that is syntactically unbalanced.  Unbalanced double quotes that aren't part
 
that is syntactically unbalanced.  Unbalanced double quotes that aren't part
 
of a multiline literal will also confuse it.  You will receive warnings
 
of a multiline literal will also confuse it.  You will receive warnings
oiif there's an indent open at end of file or if a closer occurs with
+
if there's an indent open at end of file or if a closer occurs with
 
indent already zero; these two conditions strongly suggest unbalanced WML.
 
indent already zero; these two conditions strongly suggest unbalanced WML.
 +
 +
== GUI.pyw ==
 +
 +
Starting from version 1.11.15 and 1.13.0, a GUI (written in Tkinter, plus the themed widgets ttk) is available in the same directory as the other tools. To use it, you need to have a version of Python equal to or greater than 3.1.0 (the 3.0.x series doesn't include the ttk widgets, and as such is unsuitable for this script).
 +
 +
If you're on Linux, be sure to have installed the ''python3-tk'' module, '''or the application won't run at all'''. To install it in a Debian-based distro (like Ubuntu), type this line in a Terminal:
 +
sudo apt install python3-tk
 +
 +
To start it, just double click on the GUI.pyw file. The interface is pretty much self-explanatory, and allows you to run wmllint, wmlscope, wmlindent and wmlxgettext, modify their options, select an add-on and save the tools' output as a text file.
 +
 +
== See Also ==
 +
* [[Translation Maintenance Commands]] (for <tt>wmlxgettext</tt>)
 +
 +
[[Category:Create]]
 +
[[Category:Tools]]

Latest revision as of 23:19, 17 August 2024

The Wesnoth source code distribution includes a couple of tools intended to help authors maintain campaigns, faction & unit packs, and other WML resources. These are:

wmlscope
a cross-reference lister, useful for finding unresolved macro and resource-file references.
wmllint
a utility for sanity-checking WML syntax and porting your old WML to the current version of WML.
wmlindent
a utility for reindenting WML to a uniform style.
GUI.pyw
a graphical interface

General Information

You will need a Python 3 interpreter on your system to use these tools. Linux, *BSD, and Mac OS/X should already have Python 3 installed; for Windows it's a free download from http://www.python.org. You will also need to know how to run command-line tools on your system.

If you're working with Debian or Ubuntu you might have to install the package wesnoth-1.16-tools (or the convenient version).

sudo apt install wesnoth-1.16-tools


All three tools will require you to supply a directory list. This is a set of directories containing the WML files you want to work on.

This page is intended as documentation for users.

Note to Windows Users: This means you have to run it from the Command Line. The command line may be reached by hitting Start, then Run, then "cmd" or "command" depending on your version of Windows.

Example uses:

python wmllint path\to\files
python wmlindent path\to\files

Another example:

"C:\Program Files\Python3.7\python.exe" data\tools\wmllint --dryrun data\core data\{multiplayer,themes} data\campaigns 

(You have to specify the full directory path to the executable if you don't have your environment variables set up correctly). The first thing you type is the path to your python executable, followed by a space. The second thing you type is the path to the desired script to run, followed by a space. The third thing you type is the path to the folder (or file) to be processed.


A convenient way of running wmllint on Linux (Debian or Ubuntu) and Windows in comparison, Linux:

Assuming we're working with wesnoth 1.16 or more advanced versions.

python3 /usr/share/games/wesnoth/1.16/data/tools/wmllint --dryrun /usr/share/games/wesnoth/1.16/data/core ~/.local/share/wesnoth/1.16/data/add-ons/A_Simple_Campaign 1>wmllint-run.log 2>wmllint-err.log

I have these commands inside of a file named

wmllint_dryrun_ASC.sh

and execute it by opening a shell (=terminal, console, command window, bash,...), navigating into the directory with that file and typing

bash wmllint_dryrun_ASC.sh

The python3 command should be automatically known on Debian. The path to the script tells the python interpreter what to execute. --dryrun: A wmllint option, see below. The path to the core files is needed to let wmllint know about e.g. defined core units, followed by the path to the add-on that shall be checked; the last two commands cause the result of the wmllint usage to be written into those files in the same directory as the script. Windows, this is logically exactly the same as the Linux shell script above, so if you are on a Mac you can probably conclude how you need to adapt the paths:

E:\Python37\python.exe E:\Programme\Wesnoth_1.16_git\data\tools\wmllint --dryrun E:\Programme\Wesnoth_1.16_git\data\core E:\Programme\Wesnoth_1.16_git\userdata\data\add-ons\A_Simple_Campaign 1>wmllint-run.log 2>wmllint-err.log

This is the content of a .txt file, whose extension I rename to .bat and double-click onto it. Opening a command window is not needed this way. Since Python isn't natively installed on windows and I don't have environment variables set, the full path to python.exe is given. If your directories contain spaces it may help to include the path in quotes:

"C:\Programs\Battle for Wesnoth 1.16\data\tools\wmllint"

Remember that you do not need to enter all of the commands/paths at once. If it doesn't work, start with only "python" or "C:\Python37\python.exe" or the like and interpret the error messages that you get. If you get an "unknown command", python isn't installed or environment variables aren't set correctly. After that, you can add the later commands one by one.

wmlscope

The main use for wmlscope is to find WML macro references without definitions and references to resource files (sounds and images) that don't exist. These are difficult to spot from in-game because they usually result in silence or a missing image rather than actual broken game logic (see issue 5332 for more info). They may happen because of typos in your WML, or because the name of a macro or the location of a resource file changed between versions of the game.

wmlscope also checks macro invocations for consistency. It will complain if a macro is called with the wrong number of arguments. In most cases it can deduce information about the type of the literal expected to be passed to a given macro argument by looking at the name of the formal.

Type Meaning Formals requiring this type Literals of this type
side a single side number SIDE, *_SIDE, SIDE[0-9] a numeric or "global"
numeric a numeric integer literal SIDE, X, Y, RED, GREEN, BLUE, TURN, PROB, LAYER, TIME, *_SIDE, *NUMBER, *AMOUNT, *COST, *RADIUS, *_X, *_Y, *_INCREMENT, *_FACTOR, *_TIME, *_SIZE, DURATION \-?[0-9]+
percentage a percentage *PERCENTAGE a numeric or 0\.[0-9]+
position a single x,y coordinate POSITION, *_POSITION, BASE -?[0-9]+,-?[0-9]+
span a set of coordinates or coordinate ranges *_SPAN a numeric, position or ([0-9]+\-[0-9]+,?|[0-9]+,?)+
alliance a set of side numbers SIDES, *_SIDES a span, or the empty string
range an attack range RANGE "melee" or "ranged"
alignment an alignment keyword ALIGN "lawful" or "neutral" or "chaotic"
types a set of unit types TYPES a shortname, name, or anything that contains spaces and matches no other type
terrain_pattern a set of terrain codes to filter ADJACENT*, TERRAINLIST*, *TERRAIN_PATTERN, RESTRICTING a terrain_code or name
terrain_code a single terrain code, perhaps with overlay TERRAIN*, *TERRAIN a shortname or (\*|[A-Z][a-z]+)\^([A-Z][a-z\\|/]+\Z)?
shortname a terrain code or a short, capitalized variable name [A-Z][a-z][a-z]?
name a name or identifier NAME, VAR, IMAGESTEM, ID, FLAG, *_NAME, *_ID, NAMESPACE, BUILDER, *_VAR anything without spaces that matches no other type
optional_string a string value (may be empty) ID_STRING, NAME_STRING, DESCRIPTION, IPF a string, or the empty string
string a nonempty string not matching any of the preceding types STRING, TYPE, TEXT, *_STRING, *_TYPE, *_TEXT a shortname, a name, a stringliteral, or anything that contains spaces and matches no other type
stringliteral a string in doublequotes or a translated string ".*" or _.* but not _[a-z].*
image an image path, perhaps with image path functions *IMAGE, PROFILE [A-Za-z0-9{}.][A-Za-z0-9_/+{}.-]*\.(png|jpg)(?=(~.*)?)
sound a music or sound filename MUSIC, SOUND string ending with ".wav" or ".ogg"
filter WML filter FILTER any non-quoted string containing "="
WML arbitrary WML fragment WML, *_WML any non-quoted string containing "=", or the empty string
affix a prefix, suffix, or infix for a variable name AFFIX, *AFFIX, POSTFIX, ROTATION a shortname or name, or the empty string
any anything *VALUE, [ARS][0-9] anything

If the actual argument is a macro call {.*}, then it matches any formal. Otherwise, if the formal has an identifiable type, wmlscope will complain if the actual literal does not match it.

The argument type check only works in macro calls that fit on a single line.

wmlscope has many options for changing the reports it generates; the more advanced ones are intended for Wesnoth developers. Invocations for the most commonly useful reports it generates are included in data/tools/Makefile of the source distribution. Here are some of those reports:

make unresolved
Report on unresolved macro calls and resource references; also report macro argument-type mismatches. (This is what you are most likely to want to do).
make all
Report all macro and resource file references, not just unresolved ones.
make collisions
Report on duplicate resource files.

For more advanced users, or those who want to understand what the canned Makefile invocations are doing, here is a summary of wmlscope's options. Some of the more advanced options will require you to understand Python regular expressions.

-h, --help
Emit a help message and quit
-c, --crossreference
Report resolved macro references (implies -w 1)
-C, --collisions
Report duplicate resource files
-d, --deflist
Make definition list. (This one is for campaign server maintainers.)
-e regexp, --exclude regexp
Ignore files matching the specified regular expression.
-f dir, --from dir
Report only on macros defined under dir
-l, --listfiles
List files that will be processed
-r ddd, --refcount=ddd
Report only on macros with references in exactly ddd files.
-t TYPELIST, --typelist TYPELIST
List actual & formal argtypes for calls in fname
-u, --unresolved
Report unresolved macro references
-w, --warnlevel
Set to 1 to warn of duplicate macro definitions
-p, --progress
Show progress
--force-used reg
Ignore reference count 0 on names matching regexp
--extracthelp
Extract help from macro definition comments.
--unchecked
Report all macros with untyped formals.
--version
show program's version number and exit

These options are used with a list of directories as arguments; if none is given, wmlscope behaves as though the current directory had been specified as a single argument. Each directory is treated as a separate domain for macro and resource visibility purposes.

wmlscope recognizes two kinds of namespace, exporting and non-exporting. Exporting namespaces make all their resources and macro names globally visible. You can make a namespace exporting by embedding a comment like this in it:

   # wmlscope: export=yes

Wesnoth core data is an exporting namespace. Campaigns are non-exporting; they should contain the declaration

   # wmlscope: export=no

somewhere. wmlscope will complain when it sees a namespace with no export property, then treat it as non-exporting.

You can tell wmlscope to ignore stretches of config files with the following magic comments:

   # wmlscope: start ignoring
   # wmlscope: stop ignoring

Similarly, you can tell wmlscope to ignore multiple or duplicate macro definitions in a range of lines with the following magic comments:

   # wmlscope: start conditionals
   # wmlscope: stop conditionals

The following magic comment:

   # wmlscope: prune FOOBAR

will cause wmlscope to forget about all but one of the definitions of FOOBAR it has seen. This will be useful mainly for symbols that have different definitions enabled by an #ifdef.

Due to a preprocessor limitation, inline macros cannot contain a documentation string. If you need to document these macros in the HTML macro reference, you can use the following directive:

   # wmlscope: docstring FOOBAR

The docstring for the FOOBAR macro will be collected until a non-comment line, a #define or another # wmlscope: docstring are found. External docstrings must be defined before the macro to which they refer; defining two or more external docstrings keeps only the most recent one, but having both an external and an internal docstring is allowed (in this case, the internal one will be appended to the external one in the macro reference).

This tool does catch one kind of implicit reference: if an attack name is specified but no icon is given, the attack icon will default to a name generated from the attack name. This behavior can be suppressed by adding a magic comment containing the string "no-icon" to the name= line.

The checking done by this tool has a couple of flaws:

(1) It doesn't actually evaluate file inclusions. Instead, any macro definition satisfies any macro call made under the same directory. Exception: when an #undef is detected, the macro is tagged local and not visible outside the span of lines where it was defined.

(2) It doesn't read [binary_path] tags, as this would require implementing a WML parser. Instead, it assumes that a resource-file reference can be satisfied by any matching image file from anywhere in the same directory it came from. The resources under the first directory argument (only) are visible everywhere.

(3) A reference with embedded {}s in a macro will have the macro's formal args substituted in at WML evaluation time. Instead, this tool treats each {} as a .* wildcard and considers the reference to match every resource filename that matches that pattern. Under appropriate circumstances this might report a resource filename statically matching the pattern as having been referenced even though none of the actual macro calls would actually generate it.

Problems (1) and (2) imply that this tool might conceivably report that a reference has been satisfied when under actual WML-interpreter rules it has not.

The reporting format is compatible with GNU Emacs compile mode.

For debugging purposes, an in-line comment of the form

   # wmlscope: warnlevel NNN

sets the warning level.

wmllint

wmllint is a tool for migrating your WML to the current version. It handles two problems:

  • Resource files and macro names may change between versions of the game. wmllint knows about these changes and will tweak your WML to fit where it can.
  • Between 1.2.x and 1.3.1 the terrain-coding system used in map files underwent a major change. It changed again in a minor way between 1.3.1 and 1.3.2. If you port such old code, use wmllint-1.4, which is located in the same directory as wmllint. It will translate your maps for you, unless you use custom terrains in which case you will have to do it by hand.

wmllint also performs various sanity-checking operations, reporting:

  • unbalanced tags
  • strings that need a translation mark and do not have them
  • strings that have a translation mark and should not
  • translatable strings containing macro references
  • filter references by description= (id= in 1.5) not matched by an actual unit
  • abilities or traits without matching special notes, or vice-versa
  • consistency between recruit= and recruitment_pattern= instances
  • double space after punctuation in translatable strings.
  • unknown races or movement types in units

wmllint takes a directory-path argument specifying the WML directories to work on. It will modify any cfg and map files under those directories that need to be changed. Here is a summary of its options:

-h, --help
Emit a help message and quit.
-c, --clean
Clean up -bak files.
-D, --diffs
Display diffs between converted and unconverted files.
-d, --dryrun
List changes (-v) but don't perform them.
-r, --revert
Revert the conversion from the -bak files.
-m, --missing
Don't warn about tags without side= keys now applying to all sides.
-s, --stripcr
Convert DOS-style CR/LF to Unix-style LF.
-v, --verbose
Set verbosity; more details below.
-K, --known
Suppress check for unknown unit types, recruits, races, scenarios, etc.
-S, --nospellcheck
Suppress spellchecking
--version
show program's version number and exit

The verbosity option works like this:

-v
lists changes.
-v -v
warns of maps already converted.
-v -v -v
names each file before it's processed.
-v -v -v -v
shows verbose parse details (developers only).

The recommended procedure is this:

  1. Run it with --dryrun first to see what it will do.
  2. If the messages look good, run without --dryrun; the old content will be left in backup files with a -bak extension.
  3. Eyeball the changes with the --diff option.
  4. Use wmlscope, with a directory path including the Wesnoth mainline WML, to check that you have no unresolved references.
  5. Test the conversion.
  6. Use either --clean to remove the -bak files or --revert to undo the conversion.

Additionally, wmllint tries to locate a spell checker on your system and spell-checks storyline and message strings. It will work automatically with any of enchant's spellchecking backends (including aspell, myspell, ispell, applespell, hunspell, nuspell, and so on, depending on the version of enchant), provided you have the enchant.py Python library installed. Note that the spellchecking results can vary depending on which backend enchant decides to use.

wmllint supports a number of magic comments to customize its behaviour and avoid false positives. All magic wmllint comments begin with the string wmllint:, followed by some additional keyword and potentially some arguments. In the below explanations, a string of the form [a|b] means you may use either a or b at that location, while a string of the form <arg> indicates free text substitution.

  • ignore: Disables checking of terrains and translation marks on the current line only.
  • noconvert: Disables conversion of terrains and image/sound filenames on the current line only.
  • markcheck [on|off]: Enables or disables translation mark checking from the current (next?) line onward.
  • no-icon: Prevents description insertions (what does that mean?)
  • recognize <name>: Indicates that the character with the given name exists even though a declaration (ie a [unit] or [side] tag) is not visible.
  • whofield <macro> <number>: Indicates that the specified macro declares a character whose name is given by the specified argument to the macro.
  • whofield clear <macro>: Removes the whofield definition for the specified macro.
  • who <macro> is <name> <name> ...: Indicates that the specified macro declares one or more characters whose names are given as a space-separated list. This can also be used for macros that auto-recall a list of characters, some of which join later or may die at some point. If a definition for the macro already exists, then the specified names are appended to it. If a name is preceded by a double minus (-- <name>) then it is removed from the definition.
  • unwho all|<name>: Removes the who definition for the specified macro, or all who definitions.
  • usage of "<unit>" is <class>: Declares the usage of the specified unit (the usage= key in the [unit_type] tag). Useful if you are using macros to generate several similar unit types.
  • usagetype <class>: Declares a valid usage type for units. usagetypes is also recognized, and a comma-separated list of usage types can be specified.
  • validate-[on|off]: Enables or disables stack-based validation checks. Use when you have unbalanced tags in macros.
  • unbalanced-[on|off]: Similar to above, the precise difference is unclear.
  • no translatables: Suppresses warnings about a missing textdomain declaration in the current file. Make sure the file really does have no translatable strings!
  • display [on|off]: Enable or disable warnings about newlines in messages.
  • notecheck [on|off]: Enable or disable note consistency checks for unit descriptions.
  • deathcheck [on|off]: Enable or disable the check for units speaking in their death events.
  • [general|directory|local] spellings <word> <word> ...: Declares the specified space-separated list of words to be valid spellings in the specified context. The context local indicates the current file only, while directory means the current file and any siblings in the same directory or subdirectories. The global context indicates the spellings are valid anywhere.
  • no spellcheck: Disables spell checking on the current line.
  • skip-side: Indicates that there is a missing side declaration at this location that will be provided by a macro expansion.
  • match <string> with <notes_macro>: Indicates that uses of the specified string in a unit definition (usually a macro, including the curly braces) should be matched up with the specified special notes macro (which is also specified as the full macro with curly braces).
  • no ellipsecheck: Disables checking of unit ellipses on the current line.

Explanations of wmllint diagnostics

Some wmllint diagnostics may require further explanation for UMC authors to understand; this section will be for providing such explanations, and descriptions of how to solve and/or silence them.

In these, %s will be replaced by a string. When the recommended solution also includes a %s, it's a suggestion to copy the string from the error message into your code.

  • nonstandard word-wrap style within message: This message meant there was an unexpected newline character within a [message] tag. However, this check was removed in the 1.15.10 release. Pre-1.15.10, add-on developers could silence by putting a # wmllint: display on comment before the string and a # wmllint: display off comment after the string, however, post-1.15.10, this is no longer necessary.
  • %s is not a known unit type (in cases where you'd think the unit type would be known): This means the unit has a type that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can load the mainline unit types. (You can also silence this warning by passing the -K flag to wmllint.)
  • unknown speaker '%s' of [message]: use a # wmllint: recognize %s magic comment, or, alternatively, if the speaker is created by a macro, use a magic comment of the form of either # wmllint: who MACRO is SPEAKER or # wmllint: whofield MACRO NUMBER, depending on whether the macro takes an argument for the unit's name or not.
  • unknown '%s' referred to by id: use a # wmllint: recognize %s magic comment, or, alternatively, if the unit is created by a macro, use a magic comment of the form of either # wmllint: who MACRO is UNIT or # wmllint: whofield MACRO NUMBER, depending on whether the macro takes an argument for the unit's name or not.
  • %s has unknown advancements (in cases where you'd think the advancement would be known): This means the unit has an advancement that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can load mainline units for checking advancements. Note that it's also possible that you just made typo, too, so be sure to check your spelling. (You can also silence this warning by passing the -K flag to wmllint.)
  • .description may need hand fixup: This one comes from mucking around with the .description field of unit data manually in a hackish fashion. There isn't really much of a way to work around it, besides just the "don't do that" solution.
  • tag stack nonempty (%s) at end of file.: This means that you have unbalanced tags somewhere in the file, e.g. an opener without a closer, or vice versa. This can often be seen when defining macros for unit abilities. A way to fix this warning is to wrap the section with unbalanced tags with a # wmllint: unbalanced-on magic comment beforehand and a # wmllint: unbalanced-off magic comment afterwards. As having unbalanced tags will also cause issues for other WML maintenance tools, such as wmlindent and wmlxgettext, you may also want to add separate magic comments for each of them (see their documentation for the form they take).
  • unit declaration without side attribute: the default side for a unit declaration when left implicit is side 1. Specify your unit sides explicitly to solve this.
  • no %s units recruitable at difficulty %s (even when there are such units recruitable): This diagnostic has to do with matching the usage key of units recruitable by an AI side with their recruitment_pattern. It means the unit has a usage that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can know which mainline units are recruitable. (You can also silence this warning by passing the -K flag to wmllint.)
  • %s has unknown movement type (even when you'd think that that movement type would actually be known): This means the unit has a movetype that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can load the mainline movement types. (You can also silence this warning by passing the -K flag to wmllint.)
  • %s has unknown race (even when you'd think that that race would actually be known): This means the unit has a race that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can load the mainline races. (You can also silence this warning by passing the -K flag to wmllint.)
  • derivation of %s from %s does not resolve (even when you'd think it would): This means that a unit using the [base_unit] tag specifies a unit ID in that tag that was never defined in mainline or in the add-on. This is why you're supposed to always add Wesnoth's core directory as the first item to be checked when using wmllint (which is something that the GUI version of it will do automatically for you), so that it can load the core units for its derivation checks. (You can also silence this warning by passing the -K flag to wmllint.)
  • [advancefrom] needs to be manually updated to [modify_unit_type] and moved into the _main.cfg file: This one is pretty self-explanatory: [advancefrom] was deprecated in commit 3950f40 due to issue #3955, and in fact doesn't even work anymore (in 1.16) as per issue #6204. The main reason wmllint can't fix this automatically is because it could end up being too complicated for it to figure out which files to edit if there are multiple uses of [advancefrom], and it also doesn't want to assume where to put the [modify_unit_type] tag in _main.cfg.

wmlindent

Call with no arguments to filter WML on standard input to reindented WML on standard output. If arguments are specified, they are taken to be files to be re-indented in place; a directory name causes reindenting on all WML beneath it.

The indent unit is four spaces. Absence of an option to change this is deliberate; the purpose of this tool is to prevent style wars, not encourage them.

On non-empty lines, this code never modifies anything but leading and trailing whitespace. Leading whitespace will be regularized to the current indent; trailing whitespace will be stripped. After processing all lines will end with a Unix-style \n end-of-line marker.

Runs of entirely blank lines will be reduced to one blank line, except in two cases where they will be discarded: (a) before WML closing tags, and (b) after WML opening tags.

It is possible to wrap a section of lines in special comments so that wmlindent will ignore them. You may need to do this for unbalanced macros (it's better, though, to get rid of those where possible). Use 'wmlindent: {start,stop} ignoring' anywhere in a comment.

It is also possible to declare custom openers an closers, e.g for macros that are actually control constructs. To do this, use declarations

   # wmlindent: opener "{EXCEPTIONAL_OPENER "
   # wmlindent: closer "{EXCEPTIONAL_CLOSER "

The lines after an opener will be indented an extra level; a closer and lines following will be indented one level less. Note that these declare prefixes; any prefix match to the non-whitespace text of a line will be recognized.

The public utility macros "{FOREACH" and "{NEXT" come as wired-in exceptions, because it is not guaranteed that their indent declarations will be processed before the macro library is reached.

Interrupting wmlindent ought to be safe, as each reindenting will be done to a copy that is atomically renamed when it's done. If the output file is identical to the input, the output file will simply be deleted, so the timestamp on the input file won't be touched.

The --dryrun option detects and reports files that would be changed without changing them. The --verbose or -v option enables reporting of files that are (or would be, under --dryrun) changed. With -v -v, unchanged files are also reported. The --exclude option takes a regexp and excludes files matching it.

If you don't apply this tool to your own WML that you wish to submit, the mainline-campaign maintainers will do it when and if your code is accepted into the tree.

Note: This tool does not include a parser. It will produce bad results on WML that is syntactically unbalanced. Unbalanced double quotes that aren't part of a multiline literal will also confuse it. You will receive warnings if there's an indent open at end of file or if a closer occurs with indent already zero; these two conditions strongly suggest unbalanced WML.

GUI.pyw

Starting from version 1.11.15 and 1.13.0, a GUI (written in Tkinter, plus the themed widgets ttk) is available in the same directory as the other tools. To use it, you need to have a version of Python equal to or greater than 3.1.0 (the 3.0.x series doesn't include the ttk widgets, and as such is unsuitable for this script).

If you're on Linux, be sure to have installed the python3-tk module, or the application won't run at all. To install it in a Debian-based distro (like Ubuntu), type this line in a Terminal:

sudo apt install python3-tk

To start it, just double click on the GUI.pyw file. The interface is pretty much self-explanatory, and allows you to run wmllint, wmlscope, wmlindent and wmlxgettext, modify their options, select an add-on and save the tools' output as a text file.

See Also

This page was last edited on 17 August 2024, at 23:19.