Machine Learning Recruiter
This page documents the new machine learning recruiter submitted as a patch for Wesnoth 1.11. We describe how to run it, discuss experiment showing that the ML Recruiter achieves dramatically better performance than the RCA AI recruiter, describe known issues and suggest a development road map.
Note that the ML Recruiter is a work in progress. We welcome feedback on it. Please discuss it on the thread "Machine Learning Recruiter" at http://forums.wesnoth.org/viewtopic.php?f=10&t=36642.
Contents
Why include ML Recruiter in Wesnoth?
The ML Recruiter makes use of a small subset of the Waffles Machine Learning toolkit adding 13 pairs of .cpp/.h files to Wesnoth. In addition, the neural nets used by ML Recruiter are serialized as .json files, which is a format Wesnoth has not yet contained. So why is this patch worthwhile?
Why the ML Recruiter will be great for Wesnoth
- Performance is great. ML Recruiter defeats RCA AI 70 - 71% of the time. Although we don't have data, this should translate into better performance against humans
- This superior performance is achieved with comparable "fun factor"
- Variety of units recruited by recommended ML Recruiter is comparable to or better than RCA AI
- Don't need to eliminate RCA AI recruiter. Campaign designers can choose to use one or the other
- ML Recruiter should be easier to customize than RCA AI because
- All core logic is in Lua, which is easier to modify than existing C++
- Performance "out of the box" on known units likely to be strong
- When new recruitable units are introduced by campaign designers, it can be trained by running c. 600 games in two hours. The new model is included as a .json file with the campaign data
- Plug and play architecture of machine learning "features" easily allows minor modifications to mainline recruiter or to campaign-specific recruiters
- Easy way to adjust campaign difficulty: Adjusting ML AI for more/less randomness makes is easier/harder to defeat
- Inclusion of ML Recruiter could lead to greater publicity and more contributors to Wesnoth
- SeattleDad plans to submit this work as a scientific paper to a conference such as Computational Intelligence in Games
- Others might later build on this work by, for instance, trying ML algorithms other than neural nets, adding new features, further generalizing the algorithm, etc.
- The machine learning infrastructure is not specific to recruiting and could be repurposed for, for instance, attack planning, weapon selection, and making "retreat and heal" vs. "attack" decisions
- All of the above is potentially publishable research, so Wesnoth could attract contributions from computer science graduate students
- Note that, having established the basic framework with this patch, future work on machine learning will be much easier
Using ML Recruiter
Playing against the ML Recruiter
- From the main menu, choose "Multiplayer"
- Choose "Local Game"
- Pick a map and adjust settings as desired. ML Recruiter has been trained with the default setting for village gold and support, but it should work fine on other settings
- Hit Okay
- For one side, Choose Player/Type-->Computer Player and then either ML AI (Recommended) or ML AI (Pure)
- For the opponent, either play against it yourself (pick your name) or watch it play the default AI (Computer Player-->RCA AI)
Testing the ML Recruiter in batch mode
Testing in batch mode is easy. After applying the ML Recruiter patch, copy utils/ai_test/ai_test2.cfg to the directory in which you want to run the experiment. Then edit the first line of the .cfg file, "path_to_wesnoth_binary" to point to your Wesnoth executable. Then adjust faction1 and faction2 to point to the factions you want to experiment with and point ai_config1 at the ML configuration file you want to try out. Finally, to make everything easier, add the following to your path:
[Wesnoth_Install]/utils/ai_test/
Now you can test Wesnoth in batch as follows:
ai_test2.py ai_test2.cfg
Experiments: ML Recruiter vs. RCA AI Recruiter
Chart showing relative winning percentages
- RCA AI: The Default AI. Wins 50% of the time against itself (of course)
- Random: Recruiting units are chosen completely at random. Wins 45.5% of the time overall
- Recommended ML Recruiter: Units are chosen at random weighted by their relative value. Wins 70% overall.
- Pure ML Recruiter: ML AI always chooses the unit it thinks is best. Wins 71.5% of the time overall, but is might be seen as boring since it can produce armies which are overwhelmingly one or two units.
Chart showing winning percentage of RCA AI, Random, Recommended, and pure ML Recruiters
Winning percentages for Recommended ML Recruiter
$ analyze_log.py *.log Overall Stats Win % Wins AI 30.0% 1076 "default_ai_with_recruit_log 70.0% 2506 "ml_ai_faction Total: 3582 Win Lose Win % Drakes vs Undead 37 43 46.2% Drakes vs Northerners 44 75 37.0% Drakes vs Loyalists 69 17 80.2% Drakes vs Knalgan Alliance 52 54 49.1% Drakes vs Drakes 66 40 62.3% Drakes vs Rebels 77 17 81.9% Total Drakes 345 246 58.4%
Knalgan Alliance vs Undead 74 19 79.6% Knalgan Alliance vs Northerners 29 73 28.4% Knalgan Alliance vs Loyalists 93 16 85.3% Knalgan Alliance vs Knalgan Alliance 67 37 64.4% Knalgan Alliance vs Drakes 83 39 68.0% Knalgan Alliance vs Rebels 75 18 80.6% Total Knalgan Alliance 421 202 67.6%
Loyalists vs Undead 25 73 25.5% Loyalists vs Northerners 24 67 26.4% Loyalists vs Loyalists 73 20 78.5% Loyalists vs Knalgan Alliance 57 50 53.3% Loyalists vs Drakes 61 27 69.3% Loyalists vs Rebels 55 48 53.4% Total Loyalists 295 285 50.9%
Northerners vs Undead 91 5 94.8% Northerners vs Northerners 72 26 73.5% Northerners vs Loyalists 91 2 97.8% Northerners vs Knalgan Alliance 83 2 97.6% Northerners vs Drakes 67 12 84.8% Northerners vs Rebels 107 6 94.7% Total Northerners 511 53 90.6%
Rebels vs Undead 81 19 81.0% Rebels vs Northerners 34 64 34.7% Rebels vs Loyalists 97 6 94.2% Rebels vs Knalgan Alliance 92 23 80.0% Rebels vs Drakes 68 32 68.0% Rebels vs Rebels 84 14 85.7% Total Rebels 456 158 74.3%
Undead vs Undead 65 29 69.1% Undead vs Northerners 39 42 48.1% Undead vs Loyalists 108 9 92.3% Undead vs Knalgan Alliance 94 9 91.3% Undead vs Drakes 90 19 82.6% Undead vs Rebels 82 24 77.4% Total Undead 478 132 78.4%
Units recruited by Recommended ML Recruiter
Unit recruitment statistics for Drakes % Number Unit 18.5% 2256 Drake Burner 6.9% 841 Drake Clasher 15.2% 1853 Drake Fighter 19.8% 2416 Drake Glider 16.4% 1994 Saurian Augur 23.2% 2821 Saurian Skirmisher Total: 12181
Unit recruitment statistics for Knalgan Alliance % Number Unit 16.1% 2484 Dwarvish Fighter 4.7% 725 Dwarvish Guardsman 19.2% 2963 Dwarvish Thunderer 5.1% 782 Dwarvish Ulfserker 26.1% 4020 Footpad 2.5% 379 Gryphon Rider 12.1% 1862 Poacher 14.1% 2178 Thief Total: 15393
Unit recruitment statistics for Loyalists % Number Unit 25.7% 3814 Bowman 4.3% 646 Cavalryman 13.0% 1929 Fencer 6.8% 1011 Heavy Infantryman 7.4% 1093 Horseman 2.1% 308 Mage 10.4% 1546 Merman Fighter 30.4% 4522 Spearman Total: 14869
Unit recruitment statistics for Northerners % Number Unit 9.3% 1394 Goblin Spearman 1.6% 242 Naga Fighter 23.7% 3544 Orcish Archer 1.1% 168 Orcish Assassin 20.2% 3026 Orcish Grunt 39.4% 5901 Troll Whelp 4.6% 692 Wolf Rider Total: 14967
Unit recruitment statistics for Rebels % Number Unit 15.4% 2151 Elvish Archer 40.9% 5730 Elvish Fighter 11.3% 1576 Elvish Scout 2.1% 294 Elvish Shaman 3.5% 496 Mage 3.7% 523 Merman Hunter 23.1% 3236 Wose Total: 14006
Unit recruitment statistics for Undead % Number Unit 16.7% 2696 Dark Adept 9.2% 1475 Ghost 1.1% 180 Ghoul 43.2% 6966 Skeleton Archer 13.7% 2206 Skeleton 9.2% 1475 Vampire Bat 6.9% 1113 Walking Corpse Total: 16111
How the ML Recruiter works
When it's deciding what to recruit, the ML Recruiter works by predicting the sum of the following quantities for each unit:
- Experience points at the end of the game or when the unit is killed
- Number of villages captured by the unit
Note that the ML Recruiter is blind to other ways a unit can help you (in particular, it doesn't know about poison, healing, and slowing).
This sum, which we'll call the "metric" is then divided by the unit cost to get metric/cost (think of this as goodness per unit of gold). You can see this in the debugging output that the ML Recruiter current prints to stdout:
unit type metric cost wt cost weighted metric Elvish Shaman 8.58 15 15.00 0.57 Elvish Fighter 10.42 14 14.00 0.74 Elvish Scout 8.47 18 18.00 0.47 Wose 17.07 20 20.00 0.85 Mage 10.37 20 20.00 0.52 Elvish Archer 8.46 17 17.00 0.50 Merman Hunter 8.55 15 15.00 0.57
This is from the first turn of a game between the Rebels and the Undead. The ML Recruiter is predicting that if it recruits a Wose now, it will end with 17.07 XP + Village Captures. 17.07/20 = 0.85, which is the highest weighted metric at this time, so it picks a Wose as it's top choice.
How does it know to pick a Wose? It looks at the "features" which describe the current situation. Here's another chart from the same game:
unit type metric cost wt cost weighted metric Elvish Shaman 7.18 15 15.00 0.48 Elvish Fighter 11.82 14 14.00 0.84 Elvish Scout 7.91 18 18.00 0.44 Wose 15.53 20 20.00 0.78 Mage 9.11 20 20.00 0.46 Elvish Archer 9.38 17 17.00 0.55 Merman Hunter 8.36 15 15.00 0.56 Side: 1 Gold: 21 Unit we want: Elvish Fighter PRERECRUIT:, enemy Dark Adept:1 , enemy Deathblade:1 , enemy Ghost:2 , enemy Skeleton:3 , enemy faction:Undead , enemy gold:10 , enemy level3+:0 , enemy total-gold:139 , enemy unit-gold:129 , friendly Elvish Captain:1 , friendly Elvish Fighter:1 , friendly Wose:4 , friendly faction:Rebels , friendly gold:21 , friendly level3+:0 , friendly total-gold:161 , friendly unit-gold:140 , side:1 , terrain-forest:0.082 , terrain-mountain-hill:0.113 , terrain-water-swamp:0.164 , total-gold-ratio:0.537 , turn:4 , village-control-margin:-2 , village-control-ratio:0.417 , village-enemy:7 , village-friendly:5 , village-neutral:4 ,
The "features" that it sees are the values following "PRERECRUIT". The ML AI sees that the enemy faction is the Undead and that they have one Deathblade, two ghosts, three Skeletons. The Rebels currently have 4 Wose, 1 Elvish Fighter, and 1 Elvish Captain, so in this situation, although it still sees that the Wose is likely to score higher on the XP + village capture metric (15.5 vs. 11.8), this isn't enough to overcome the price differential, so it chooses an Elvish Fighter as it's best choice with a weighted metric of 0.84
Retraining the ML Recruiter
Known issues
Bugs
- Haven't added new Waffles files to Visual C++, so it won't compile under VC++. I need some help with this.
- ML Recruiter runs fine vs. RCA AI and vs. human, but can't run against itself (ML Recruiter vs. ML Recruiter). This appears to be an issue with Lua in Wesnoth 1.11 and not with ML Recruiter
Current Limitations
- Only tested on two-player multiplayer games. Doesn't work when there are more than two leaders on the map.
- Works optimally on the following two-player maps (trained on these)
- Weldyn Channel
- The Freelands
- Den of Onis
- Fallenstar Lake
- Tested on all other two-player maps and runs nearly as well except it crashes on the following:
- Aethermaw, Hornshark Island, Dark Forecast, Sablestone Delta, Elensefar Courtyard, Silverhead Crossing
- Currently writes log messages as "print" statements to stdout. I need some advice on this. I've added a new method called "ai_log_message" to core.cpp to allow Lua to write to the "ai/engine/lua" log domain, but would like some advice on whether this is a good idea