Machine Learning Recruiter

This page documents the new machine learning recruiter submitted as a patch for Wesnoth 1.11. We describe how to run it, discuss experiment showing that the ML Recruiter achieves dramatically better performance than the RCA AI recruiter, describe known issues and suggest a development road map.

Note that the ML Recruiter is a work in progress. We welcome feedback on it. Please discuss it on the thread "Machine Learning Recruiter" at http://forums.wesnoth.org/viewtopic.php?f=10&t=36642.

Why include ML Recruiter in Wesnoth?

The ML Recruiter makes use of a small subset of the Waffles Machine Learning toolkit adding 13 pairs of .cpp/.h files to Wesnoth. In addition, the neural nets used by ML Recruiter are serialized as .json files, which is a format Wesnoth has not yet contained. So why is this patch worthwhile?

Why the ML Recruiter will be great for Wesnoth

1. Performance is great. ML Recruiter defeats RCA AI 70 - 71% of the time. Although we don't have data, this should translate into better performance against humans
2. This superior performance is achieved with comparable "fun factor"
   • Variety of units recruited by recommended ML Recruiter is comparable to or better than RCA AI
3. Don't need to eliminate RCA AI recruiter. Campaign designers can choose to use one or the other
4. ML Recruiter should be easier to customize than RCA AI because
   • All core logic is in Lua, which is easier to modify than existing C++
   • Performance "out of the box" on known units likely to be strong
   • When new recruitable units are introduced by campaign designers, it can be trained by running c. 600 games in two hours. The new model is included as a .json file with the campaign data
   • Plug and play architecture of machine learning "features" easily allows minor modifications to mainline recruiter or to campaign-specific recruiters
5. Easy way to adjust campaign difficulty: Adjusting ML AI for more/less randomness makes is easier/harder to defeat
6. Inclusion of ML Recruiter could lead to greater publicity and more contributors to Wesnoth
   1. SeattleDad plans to submit this work as a scientific paper to a conference such as Computational Intelligence in Games
   2. Others might later build on this work by, for instance, trying ML algorithms other than neural nets, adding new features, further generalizing the algorithm, etc.
   3. The machine learning infrastructure is not specific to recruiting and could be repurposed for, for instance, attack planning, weapon selection, and making "retreat and heal" vs. "attack" decisions
   4. All of the above is potentially publishable research, so Wesnoth could attract contributions from computer science graduate students
      1. Note that, having established the basic framework with this patch, future work on machine learning will be much easier

Using ML Recruiter

Playing against the ML Recruiter

1. From the main menu, choose "Multiplayer"
2. Choose "Local Game"
3. Pick a map and adjust settings as desired. ML Recruiter has been trained with the default setting for village gold and support, but it should work fine on other settings
   1. Hit Okay
4. For one side, Choose Player/Type-->Computer Player and then either ML AI (Recommended) or ML AI (Pure)
   1. For the opponent, either play against it yourself (pick your name) or watch it play the default AI (Computer Player-->RCA AI)

Testing the ML Recruiter in batch mode

Testing in batch mode is easy. After applying the ML Recruiter patch, copy utils/ai_test/ai_test2.cfg to the directory in which you want to run the experiment. Then edit the first line of the .cfg file, "path_to_wesnoth_binary" to point to your Wesnoth executable. Then adjust faction1 and faction2 to point to the factions you want to experiment with and point ai_config1 at the ML configuration file you want to try out. Finally, to make everything easier, add the following to your path:

[Wesnoth_Install]/utils/ai_test/

Now you can test Wesnoth in batch as follows:

ai_test2.py ai_test2.cfg

Experiments: ML Recruiter vs. RCA AI Recruiter

Chart showing relative winning percentages

• RCA AI: The Default AI. Wins 50% of the time against itself (of course)
• Random: Recruiting units are chosen completely at random. Wins 45.5% of the time overall
• Recommended ML Recruiter: Units are chosen at random weighted by their relative value. Wins 70% overall.
• Pure ML Recruiter: ML AI always chooses the unit it thinks is best. Wins 71.5% of the time overall, but is might be seen as boring since it can produce armies which are overwhelmingly one or two units.

Chart showing winning percentage of RCA AI, Random, Recommended, and pure ML Recruiters

Winning percentages for Recommended ML Recruiter

$ analyze_log.py *.log 
Overall Stats 
Win %	Wins	AI
30.0%	1076	"default_ai_with_recruit_log
70.0%	2506	"ml_ai_faction
Total:	3582
                                        Win     Lose    Win %
Drakes vs Undead                 	37	43	46.2%
Drakes vs Northerners            	44	75	37.0%
Drakes vs Loyalists              	69	17	80.2%
Drakes vs Knalgan Alliance       	52	54	49.1%
Drakes vs Drakes                 	66	40	62.3%
Drakes vs Rebels                 	77	17	81.9%
Total Drakes                     	345	246	58.4%

Knalgan Alliance vs Undead       	74	19	79.6%
Knalgan Alliance vs Northerners  	29	73	28.4%
Knalgan Alliance vs Loyalists    	93	16	85.3%
Knalgan Alliance vs Knalgan Alliance	67	37	64.4%
Knalgan Alliance vs Drakes       	83	39	68.0%
Knalgan Alliance vs Rebels       	75	18	80.6%
Total Knalgan Alliance           	421	202	67.6%

Loyalists vs Undead              	25	73	25.5%
Loyalists vs Northerners         	24	67	26.4%
Loyalists vs Loyalists           	73	20	78.5%
Loyalists vs Knalgan Alliance    	57	50	53.3%
Loyalists vs Drakes              	61	27	69.3%
Loyalists vs Rebels              	55	48	53.4%
Total Loyalists                  	295	285	50.9%

Northerners vs Undead            	91	5	94.8%
Northerners vs Northerners       	72	26	73.5%
Northerners vs Loyalists         	91	2	97.8%
Northerners vs Knalgan Alliance  	83	2	97.6%
Northerners vs Drakes            	67	12	84.8%
Northerners vs Rebels            	107	6	94.7%
Total Northerners                	511	53	90.6%

Rebels vs Undead                 	81	19	81.0%
Rebels vs Northerners            	34	64	34.7%
Rebels vs Loyalists              	97	6	94.2%
Rebels vs Knalgan Alliance       	92	23	80.0%
Rebels vs Drakes                 	68	32	68.0%
Rebels vs Rebels                 	84	14	85.7%
Total Rebels                     	456	158	74.3%

Undead vs Undead                 	65	29	69.1%
Undead vs Northerners            	39	42	48.1%
Undead vs Loyalists              	108	9	92.3%
Undead vs Knalgan Alliance       	94	9	91.3%
Undead vs Drakes                 	90	19	82.6%
Undead vs Rebels                 	82	24	77.4%
Total Undead                     	478	132	78.4%

Units recruited by Recommended ML Recruiter

Unit recruitment statistics for Drakes
%	Number	Unit		
18.5%	2256	Drake Burner	
6.9%	841	Drake Clasher	
15.2%	1853	Drake Fighter	
19.8%	2416	Drake Glider	
16.4%	1994	Saurian Augur	
23.2%	2821	Saurian Skirmisher	
Total:	12181

Unit recruitment statistics for Knalgan Alliance
%	Number	Unit		        
16.1%	2484	Dwarvish Fighter	
4.7%	725	Dwarvish Guardsman	
19.2%	2963	Dwarvish Thunderer	
5.1%	782	Dwarvish Ulfserker	
26.1%	4020	Footpad	                
2.5%	379	Gryphon Rider	        
12.1%	1862	Poacher	                
14.1%	2178	Thief	                
Total:	15393

Unit recruitment statistics for Loyalists
%	Number	Unit		
25.7%	3814	Bowman	
4.3%	646	Cavalryman	
13.0%	1929	Fencer	
6.8%	1011	Heavy Infantryman	
7.4%	1093	Horseman	
2.1%	308	Mage	
10.4%	1546	Merman Fighter	
30.4%	4522	Spearman	
Total:	14869

Unit recruitment statistics for Northerners
%	Number	Unit		
9.3%	1394	Goblin Spearman	
1.6%	242	Naga Fighter	
23.7%	3544	Orcish Archer	
1.1%	168	Orcish Assassin	
20.2%	3026	Orcish Grunt	
39.4%	5901	Troll Whelp	
4.6%	692	Wolf Rider	
Total:	14967

Unit recruitment statistics for Rebels
%	Number	Unit		
15.4%	2151	Elvish Archer	
40.9%	5730	Elvish Fighter	
11.3%	1576	Elvish Scout	
2.1%	294	Elvish Shaman	
3.5%	496	Mage	
3.7%	523	Merman Hunter	
23.1%	3236	Wose	
Total:	14006

Unit recruitment statistics for Undead
%	Number	Unit		
16.7%	2696	Dark Adept	
9.2%	1475	Ghost
1.1%	180	Ghoul	
43.2%	6966	Skeleton Archer	
13.7%	2206	Skeleton	
9.2%	1475	Vampire Bat	
6.9%	1113	Walking Corpse	
Total:	16111

How the ML Recruiter works

When it's deciding what to recruit, the ML Recruiter works by predicting the sum of the following quantities for each unit:

1. Experience points at the end of the game or when the unit is killed
2. Number of villages captured by the unit

Note that the ML Recruiter is blind to other ways a unit can help you (in particular, it doesn't know about poison, healing, and slowing).

This sum, which we'll call the "metric" is then divided by the unit cost to get metric/cost (think of this as goodness per unit of gold). You can see this in the debugging output that the ML Recruiter current prints to stdout:

unit type               metric  cost    wt cost weighted metric
Elvish Shaman           8.58    15      15.00   0.57
Elvish Fighter          10.42   14      14.00   0.74
Elvish Scout            8.47    18      18.00   0.47
Wose                    17.07   20      20.00   0.85
Mage                    10.37   20      20.00   0.52
Elvish Archer           8.46    17      17.00   0.50 
Merman Hunter           8.55    15      15.00   0.57

This is from the first turn of a game between the Rebels and the Undead. The ML Recruiter is predicting that if it recruits a Wose now, it will end with 17.07 XP + Village Captures. 17.07/20 = 0.85, which is the highest weighted metric at this time, so it picks a Wose as it's top choice.

How does it know to pick a Wose? It looks at the "features" which describe the current situation. Here's another chart from the same game:

unit type               metric  cost    wt cost weighted metric
Elvish Shaman           7.18    15      15.00   0.48
Elvish Fighter          11.82   14      14.00   0.84
Elvish Scout            7.91    18      18.00   0.44
Wose                    15.53   20      20.00   0.78
Mage                    9.11    20      20.00   0.46
Elvish Archer           9.38    17      17.00   0.55
Merman Hunter           8.36    15      15.00   0.56
Side: 1 Gold: 21 Unit we want: Elvish Fighter
PRERECRUIT:, enemy Dark Adept:1 , enemy Deathblade:1 , enemy Ghost:2 , enemy Skeleton:3 , enemy faction:Undead , 
enemy gold:10 , enemy level3+:0 , enemy total-gold:139 , enemy unit-gold:129 , friendly Elvish Captain:1 , 
friendly Elvish Fighter:1 , friendly Wose:4 , friendly faction:Rebels , friendly gold:21 , friendly level3+:0 , 
friendly total-gold:161 , friendly unit-gold:140 , side:1 , terrain-forest:0.082 , terrain-mountain-hill:0.113 , 
terrain-water-swamp:0.164 , total-gold-ratio:0.537 , turn:4 , village-control-margin:-2 , village-control-ratio:0.417 , village-enemy:7 , 
village-friendly:5 , village-neutral:4 ,

The "features" that it sees are the values following "PRERECRUIT". The ML AI sees that the enemy faction is the Undead and that they have one Deathblade, two ghosts, three Skeletons. The Rebels currently have 4 Wose, 1 Elvish Fighter, and 1 Elvish Captain. It also sees a number of other features like how much gold it and its opponent have, what percentage of each the map is covered by different terrain, and how many friendly, neutral, and enemy villages there are. In this situation, although it still sees that the Wose is likely to score higher on the XP + village capture metric (15.5 vs. 11.8), this isn't enough to overcome the price differential, so it chooses an Elvish Fighter as it's best choice with a weighted metric of 0.84.

Note that these predictions of 15.5 vs. 11.8 are computed by the neural net based on a model built from what the algorithm has seen has happened in similar situations during training.

The Weighted Random (Recommended) Recruiter

The recommended recruiter is defined in ai/ais/ml_ai_faction_specific_weighted_random.cfg. It is called "Recommended" in the user interface. Although we are currently measuring it as performing slightly worse than "Pure" ML AI (ai/ais/ml_ai_faction_specific.cfg), (70% win percentage vs. 71.5% for the Pure version), we recommend it because it allows the player to see a greater variety of opposing units.

The weighted random printout looks like the following:

Turn 13:
unit type               metric  cost    metric/ weighted        %
                                       cost    m/c             of total
Merman Hunter           1.10    15      0.07    0.0000          0.0%
Wose                    1.72    20      0.09    0.0000          0.1%
Elvish Shaman           1.66    15      0.11    0.0000          0.4%
Elvish Archer           2.71    17      0.16    0.0000          4.0%
Elvish Fighter          2.54    14      0.18    0.0000          8.5%
Mage                    4.18    20      0.21    0.0001          20.1%
Elvish Scout            4.60    18      0.26    0.0003          66.8%
Random Number chosen was        376
Side: 1 Gold: 27 Unit we want: Elvish Scout
PRERECRUIT:, enemy Dark Adept:2 , enemy Revenant:1 , enemy Skeleton:1 , enemy faction:Undead , enemy gold:8 , 
enemy level3+:0 , enemy total-gold:83 , enemy unit-gold:75 , friendly Elder Wose:1 , friendly Elvish Fighter:2 , 
friendly Elvish Ranger:1 , friendly Mage:1 , friendly Wose:3 , friendly faction:Rebels , friendly gold:27 , 
friendly level3+:0 , friendly total-gold:225 , friendly unit-gold:198 , side:1 , terrain-forest:0.082 , 
terrain-mountain-hill:0.113 , terrain-water-swamp:0.164 , total-gold-ratio:0.731 , turn:13 , 
village-control-margin:0 , village-control-ratio:0.5 , village-enemy:8 , village-friendly:8 , village-neutral:0 ,

Known issues

Bugs

1. Haven't added new Waffles files to Visual C++, so it won't compile under VC++. I need some help with this.
2. ML Recruiter runs fine vs. RCA AI and vs. human, but can't run against itself (ML Recruiter vs. ML Recruiter). This appears to be an issue with Lua in Wesnoth 1.11 and not with ML Recruiter

Current Limitations

1. Only tested on two-player multiplayer games. Doesn't work when there are more than two leaders on the map.
2. Works optimally on the following two-player maps (trained on these)
   1. Weldyn Channel
   2. The Freelands
   3. Den of Onis
   4. Fallenstar Lake
3. Tested on all other two-player maps and runs nearly as well except it crashes on the following:
   • Aethermaw, Hornshark Island, Dark Forecast, Sablestone Delta, Elensefar Courtyard, Silverhead Crossing
4. Currently writes log messages as "print" statements to stdout. I need some advice on this. I've added a new method called "ai_log_message" to core.cpp to allow Lua to write to the "ai/engine/lua" log domain, but would like some advice on whether this is a good idea
5. The ML Recruiter is currently blind to certain special abilities. It doesn't see the benefits of poisoning or slowing an enemy or healing a friendly unit, consequently it undervalues units that poison, slow, or heal