Difference between revisions of "GettextForWesnothDevelopers"

Revision as of 07:38, 31 July 2020

This page is used to help Wesnoth developers and UMC authors to work with the internationalization (i18n) system, based on GNU gettext.

General design of gettext use

Gettextized programs usually contain the English strings within the source code, with calls like puts (_("Hello world."));, so that the binary can work (in English) when the system does not support i18n.

Some strings look the same in English but should not necessarily look identical in translations. To handle this, those strings can be prefixed with any descriptive string and a ^ character. (Version 1.15.2 and later only) if the string contains more than one ^, the descriptive string ends at the first ^, everything following the first ^ will be shown to the user.

Textdomains

Gettext splits translations in to domains. For Wesnoth, the general idea is to use distinct textdomains for each campaign or add-on, so that UMC authors can easily ship translations together with their campaigns. These domains are covered in more depth in GettextForTranslators.

The convention is to name each domain using the name of the add-on, or just its initials. For example, wesnoth-utbs or wesnoth-Son_of_Haldric. For UMC, it probably makes sense to use the full name to ensure that it doesn't clash with another add-on.

UTF-8

For translation, all C++, WML and Lua files should be in UTF-8. As noted in the Typography_Style_Guide, some punctuation should be used that's outside of the ASCII subset.

Marking up strings in C++

In C++, you can mark up strings for translations using the _("A translation") and _n("Translation", "Translations", int) macros. The _n macro is to be used if the string has a singular and plural form.

If the string contains any placeholders, do not use snprintf. Use vgettext instead, or vngettext for any int placeholders.

You can also add comments for translators directly above the string - use the keyword TRANSLATORS: for that. The comment must be placed in the line immediately above the translateable string, like this:

int handfuls = 2;
const std::string translated_text = vngettext(
    // TRANSLATORS: Yum!
    "$handfuls handful of $taste potatoes",
    "$handfuls handfuls of $taste potatoes",
    handfuls,
    utils::string_map({ {"handfuls", handfuls}, {"taste", "yummy"} }));

The following code will not work for including the comment:

int handfuls = 2;
// TRANSLATORS: Yuck!
const std::string translated_text = vngettext(
    "$handfuls handful of $taste potatoes",
    "$handfuls handfuls of $taste potatoes",
    handfuls,
    utils::string_map({ {"handfuls", handfuls}, {"taste", "yucky"} }));

You can also use multiline comments:

int handfuls = 2;
const std::string translated_text = vngettext(
    /* TRANSLATORS: Yum!
       Best potatoes ever! */
    "$handfuls handful of $taste potatoes",
    "$handfuls handfuls of $taste potatoes",
    handfuls,
    utils::string_map({ {"handfuls", handfuls}, {"taste", "yummy"} }));

By default, all strings in C++ belong to the "wesnoth" textdomain. If a different textdomain is required, you can add a textdomain binding at the top of the source file, before any include statements. A textdomain binding looks like this:

#define GETTEXT_DOMAIN "wesnoth-lib"

You should avoid placing translatable strings in C++ headers if at all possible. Though there are a few places where it may be unavoidable, such as if templates are in use, it creates the risk of the strings sometimes being looked up in the wrong textdomain if the header is included in multiple files with different textdomains. If possible, always factor the translatable strings out into a source file.

Marking up strings in WML

The textdomain declaration

First, your add-on must declare a textdomain. To do this, make sure something like the following is inside of your _main.cfg. This is a top-level tag, so should be outside the [campaign] or [modification] tag, it should probably start on the second line of the file (and the next section of this page says what should be on the first line).

[textdomain]
    name="wesnoth-Son_of_Haldric"
    path="data/add-ons/Son_of_Haldric/translations"
[/textdomain]

For choosing the name, see the #Textdomains section.

The .po (or .mo) files will be loaded from a subdirectory of the translations directory.

The textdomain bindings

All files with translatable strings must declare which textdomain they use, which is normally done by putting #textdomain on the first line of each .wml file. See the example below:

#textdomain wesnoth-Son_of_Haldric

[unit_type]
    id=Mu
    name= _ "Mu"
    # ...
[/unit_type]

Note that it is highly recommended that the first textdomain binding be on the first line of the file. Otherwise, odd stuff may happen.

The translatable strings

To mark a string as translatable, just put an underscore ( _ ) in front of the string you wish to be marked as translatable, like the example below:

name= _ "Mu"

Note that there are certain things you should never do. For example, never mark an empty string as translatable, for wmlxgettext (the tool that extracts strings from WML) will abort upon detecting one. Therefore, what is seen below should never be done:

name= _ ""

Also, never put macro arguments in a translatable string, for it will not work. The reason for this is that the preprocessor does its job before gettext, thus gettext will try to replace a string that does not exist. Therefore, what is shown below should not be done:

name= _ "{TYPE} Mu"

To show why it will not work:

#define UNIT_NAME TYPE
    name= _ "{TYPE} Mu"
#enddef

{UNIT_NAME ( _ "Sword")}
{UNIT_NAME ( _ "Bow")}

Translation catalogues would have this: "{TYPE} Mu", therefore gettext will look for it even though it will not exist because we, in fact, have these after the preprocessor is done:

name= _ "Sword Mu"
name= _ "Bow Mu"

Since those are not in the catalogues, they will not get translated.

If you think a translatable string needs additional guidance to be translated properly, you can provide a special comment that will be seen by the translators. Just begin the comment with '#po:' above the string in question:

#po: "northern marches" is *not* a typo for "northern marshes" here.
#po: In archaic English, "march" means "border country".
story=_ "The orcs were first sighted from the north marches of the great forest of Wesmere."

Gender-specific strings

Several tags, including [message], [abilities] and [trait], can choose different strings based on the gender of the unit. In English the two versions are likely to be the same, but other languages may have gender-specific words for 'I' or 'me'.

[message]
    speaker=student
    message= _ "Have you found an orc for me to fight, huh? A troll?"
    female_message= _ "female^Have you found an orc for me to fight, huh? A troll?"
[/message]

The convention in WML is, as above, to use message= and female_message=, with the latter string including the prefix female^. The mechanism also supports male_message=, but all units will fall back to using the plain message= value if there isn't gender-specific version that matches their gender.

The message is chosen based on the gender of the speaking unit. To change the message based on the gender of another unit requires separate [message] tags:

[if]
    [have_unit]
        id=student
        gender=male
    [/have_unit]
    [then]
        [message]
            speaker=Delfador
            message= _ "Young man, you have $student_hp hitpoints and a sword. I’m fairly sure you’ll win."
        [/message]
    [/then]
    [else]
        [message]
            speaker=Delfador
            message= _ "female^Young lady, you have $student_hp hitpoints and a sword. I’m fairly sure you’ll win."
        [/message]
    [/else]
[/if]

Using a macro to encapsulate most of that can be useful. The example above is from the tutorial, after expanding the GENDER macro which is defined in data/campaigns/tutorial/utils/utils.cfg.

Reusing mainline translations

You can reuse translations for strings in mainline domains by using multiple textdomain bindings:

# textdomain wesnoth-Son_of_Haldric

[unit_type]
    id=Mu
    name= _ "Mu"
    # ...

    [attack]
        id=sword
        #textdomain wesnoth-units
        description= _ "sword"
        # ...
    [/attack]
   
    #textdomain wesnoth-Son_of_Haldric
    # ...
[/unit_type]

Of course, if you use bindings for multiple textdomains, make sure the right parts of the file are bound to the right domains. Also, never try to use the mainline campaigns’ domains, for there is no guarantee that the mainline campaigns will be available on all setups. So, only use the core domains: wesnoth, wesnoth-editor, wesnoth-lib, wesnoth-help, wesnoth-test, and wesnoth-units.

The gettext helper file

A gettext helper file is a lovely file that makes reusing mainline translations nice and easy, by having all strings that should use a specific textdomain in a single file. It is also more wmllint-friendly.

Here is an example of a gettext helper file. The macro names start with 'SOH_' to ensure that they don't clash with another add-on's macros (assuming that this add-on is Son_of_Haldric).

#textdomain wesnoth-lib

#define SOH_STR_ICE
_"Ice" #enddef

#textdomain wesnoth-units

#define SOH_STR_SWORD
_"sword" #enddef

A typical name for gettext helper files is mainline-strings.cfg.

To use it, just wire it into your add-on and use the macros:

[attack]
    id=sword
    name={SOH_STR_SWORD}
    # ...
[/attack]

[terrain_type]
    id=ice2
    name={SOH_STR_ICE}
    # ...
[/terrain_type]

Unbalanced WML macros

WML macros can be unbalanced, meaning that they either include a [tag] without the corresponding [/tag] or a [/tag] before the corresponding [+tag]. These macros are expected to be used in a place where the [tag] is already open. Writing new macros using this isn't recommended; instead please ask in the WML Workshop forum about better ways to do it.

When generating the .pot files for translation, wmlxgettext may stop with one of the errors

error: Son_Of_Haldric/utils/abilities.cfg:29: unexpected closing tag '[/abilities]' outside any scope.
error: Son_Of_Haldric/utils/abilities.cfg:300: End of WML file reached, but some tags were not properly closed. (nearest unclosed tag is: [abilities])

Suppose abilities.cfg line 29 is in the definition of SOH_ABILITY_BLITZ. To get the .pot file generated, the simplest change is to use # wmlxgettext comments to add the missing opening or closing tags:

# wmllint: unbalanced-on
# wmlxgettext: [abilities]
#define SOH_ABILITY_BLITZ
    [dummy]
        id=soh_blitz

... several lines of code, none of which are an #enddef ...

[+abilities]
#enddef
# wmlxgettext: [/abilities]
# wmllint: unbalanced-off

Marking up strings in Lua

In Lua code, textdomains are a callable object that looks up a string. This has support for both singular and plural strings. By convention, the name _ is usually used for the textdomain object.

The following sample code demonstrates how to fetch translatable strings in Lua:

local _ = wesnoth.textdomain "wesnoth"

-- Look up a normal string:
local win_condition = _ "Defeat enemy leader(s)"

-- Look up a plural string, using the preferred style (as of Wesnoth 1.15.3):
local turn_count = 5
turn_counter = _("(this turn left)", "($remaining_turns turns left)", turn_count)
turn_counter = turn_counter:vformat{remaining_turns = turn_count}

-- Example of looking up a plural string, but using deprecated "%d" style strings:
local n = 5
local turns_left = _("(this turn left)", "(%d turns left)", n)

Generating the .pot and .po files for UMC

For each language, Wesnoth will search for a .po file containing the translations. How to create that file will be explained below, but first the overview of where it should go.

Continuing with the Son of Haldric example, the Swedish translation would be in the file data/add-ons/Son_of_Haldric/translations/wesnoth-Son_of_Haldric/sv.po.

data/add-ons/Son_of_Haldric/translations comes from the [textdomain] tag's path
wesnoth-Son_of_Haldric is the textdomain's name
sv is the language code for Swedish. The codes for each language are given in the big table on https://www.wesnoth.org/gettext/ .

Wesnoth 1.14 (but not 1.12) supports reading .po files directly, so when you add the .po file and the new translation should appear as soon as you refresh the cache.

Generating the .pot file

The template (.pot) file contains all of the strings that need to be translated in the .po files, but without the translations.

The .pot is generated from WML and Lua files using a tool called wmlxgettext. With Wesnoth 1.14.5 and later, this is shipped with Wesnoth itself as part of the Maintenance_tools and can be used from the Maintenance Tools' GUI. At the moment it's not documented on that page, but if you follow the instructions to get GUI.pyw running then you'll see there's a wmlxgettext tab.

Pre-1.13 instructions on how to get and use it are in Nobun's forum posting.

Error messages from wmlxgettext

If you get the error from wmlxgettext of "UTF-8 Format error. Can't decode byte 0x91 (invalid start byte).", and the line in question has a curly quotation mark, that likely means that your text editor is using the Windows-1252 character set, and you need to replace the Windows quotes with their Unicode equivalents, see Typography_Style_Guide and your editor's documentation for more info. The same applies if the error message says 0x92, 0x93 or 0x94.

If you get either "unexpected closing tag '[/something]' outside any scope" or "End of WML file reached, but some tags were not properly closed. (nearest unclosed tag is: [something])" then see #Unbalanced_WML_macros above.

Generating the .po files for each language

Each .po file can start as a simple copy of the .pot file. Either the author or the translator copies the template to the language-specific filename, and then the work of GettextForTranslators happens on those copies.

Some .po editors, for example poedit, will recognise that the .pot is a template, and automatically suggest saving to a different filename. The poedit editor can also update a .po file based on changes to the .pot file.

Generating the .mo files for UMC

For Wesnoth 1.14, it's generally not necessary to compile the .po files to .mo files. The mainline translations still use .mo files for better performance, but UMC authors can skip the .mo compilation stage.

Possibly obsolete parts of this page

How to move strings from one textdomain to another

Warning: this section is very outdated (but I'd like someone else to comment on whether it's still useful).

run make -C po update-po and commit, to be sure to only commit your own changes
move the file into the corect po/*/POTFILES.in
add or change #define GETTEXT_DOMAIN "wesnoth-lib" at top of the file, before the includes
update the target POT file to include the new strings in its template (eg. make -C po/wesnoth-editor

wesnoth-editor.pot-update)

copy the translations using utils/po2po (eg. ./utils/po2po wesnoth wesnoth-editor)
update the source POT file to get rid of the old strings (eg. make -C po/wesnoth update-po), then preferably

remove the translation from obsolete strings in all languages, to make sure, in case the strings have to move back, that any translation update gets used instead of the current one)

check cvs diff and commit

How to prepare translation updates for being committed

To ensure that the diffs the version-control system generates are usable and that the po files are actually compilable, it is recommended that i18n and translation managers follow these steps before committing translation updates.

Note that this guide assumes that you are using a Unix-like system and the CMake build system.

1. Run dos2unix on all of the updated po files.

2. Run "make po-update-<locale>" to ensure the updated po files are in sync with their corresponding pot files and to fix the line wrapping.

3. Run "make mo-update-<locale>" to ensure that the updated po files are compilable.

How to update the translation catalogs

Running a .pot update with CMake is documented in Releasing Wesnoth -> General Maintenance.

If you are using scons, run scons pot-update instead.