Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  

Author Topic: A Year of Military Training [Science/Case Study]  (Read 1301 times)

blue sam3

  • Bay Watcher
    • View Profile
A Year of Military Training [Science/Case Study]
« on: April 28, 2023, 04:14:58 pm »

Hypotheses (gathered from claims made in this forum):

1. You should set the number of dwarves to something less than the squad size, to encourage them to not stop training when someone goes to sleep. Also, claims that you should set it to 4 or below, to encourage more sparring over demonstrations.
2. You should use small squad sizes (of 4, or 3, or 2), to encourage more sparring.
3. Metal armour has an enormous negative effect on XP earned for low-skill dwarves, due to the weight slowing them down.
4. Your squads should have a split of different weapon types, to discourage demonstrations and encourage sparring.



Set up:

This was done in a dedicated testing fort, with the following squads:

Squad 1: 10 dwarves, default "constant training" setup, no armour
Squad 2: as squad 1, but in full copper from the start
Squad 3: 10 dwarves, constant training but with min 4 instead of min 10, no armour
Squad 4: 10 dwarves, min 3, no armour
Squad 5: 10 dwarves, min 2, no armour
Squad 6: 10 dwarves, min 1, no armour
Squad 7: 10 dwarves, min 2, split weapons (5 sword, 5 spears)
Squad 8: 4 dwarves, min 4, no armour
Squad 9: 4 dwarves, min 4, full copper
Squad 10: 4 dwarves, min 3, no armour
Squad 11: 4 dwarves, min 2, no armour
Squad 12: 4 dwarves, min 1, no armour
Squad 13: 3 dwarves, min 3, no armour
Squad 14: 3 dwarves, min 3, full copper
Squad 15: 3 dwarves, min 2, no armour
Squad 16: 3 dwarves, min 1, no armour
Squad 17: 2 dwarves, min 2, no armour
Squad 18: 2 dwarves, min 2, full copper
Squad 19: 2 dwarves, min 1, no armour
Squad 20: 2 dwarves, min 2, different weapons (1 sword, 1 spear)
Squad 21: 4 dwarves, min 2, split weapons (2 sword, 2 spear)

In order to standardise things as much as possible, all dwarves had the following ran on them at the start (NB: unfortunately, neither assign-attributes --reset nor the assign attributes command below seems to work - both seem to randomise it across the full range, so there's a fair bit of attribute variation between dwarves):

assign-beliefs --reset
assign-facets --reset
assign-goals --reset
assign-goals --reset
assign-skills --reset
assign-attributes --attributes [ ANALYTICAL_ABILITY 0 FOCUS 0 WILLPOWER 0 CREATIVITY 0 INTUITION 0 PATIENCE 0 MEMORY 0 LINGUISTIC_ABILITY 0 SPATIAL_SENSE 0 MUSICALITY 0 SOCIAL_AWARENESS 0  STRENGTH 0 AGILITY 0 TOUGHNESS 0 ENDURANCE 0 RECUPERATION 0 DISEASE_RESISTANCE 0 ]
set-orientation --male 0 --female 0

These were also ran at the start (again, standardisation):

pref-adjust clear_all
rejevenate --all --force

These were run at the start, and after every month (to prevent tantrums and such disrupting things):

remove-stress --all
fillneeds --all

After this, all squads were simultaneously assigned to a barracks.

When things were started, a great many dwarves did not set their profession to "recruit". I don't know why. Some fixed it later, some didn't. Initial numbers below:

Squad 3:6
Squad 4:7
Squad 5:8
Squad 6:10
Squad 7:8
Squad 10:1
Squad 11:2
Squad 12:3
Squad 15:1
Squad 16:2
Squad 19:1
Squad 21:2

Monthly updates:

1 Month Update:
Spoiler (click to show/hide)

2 Month Update:
Spoiler (click to show/hide)

3 Month Update:
Spoiler (click to show/hide)

4 Month Update:
Spoiler (click to show/hide)

5 Month Update:
Spoiler (click to show/hide)

6 Month Update:
Spoiler (click to show/hide)

7 Month Update:
Spoiler (click to show/hide)


9 Month Update:
Spoiler (click to show/hide)

10 Month Update:
Spoiler (click to show/hide)

11 Month Update:
Spoiler (click to show/hide)

Final Update & Conclusions

On the Training Number: this is very obviously not a minimum, but rather a maximum. In particular, it should very much always be set to the squad size - this gave better than 400% improvements to XP earned for every single military skill, and having it set lower wiped out essentially all stat gains (NB: I unfortunately forgot to record the mental stats at the start, so all references to stat gains in this conclusion refer strictly to physical stat gains, in Strength, Agility, Toughness and Endurance - the other physical stats did not change.

On squad size: Relative to a squad size of 10, squad sizes of 2 were worse on average, by an average of 10-30% (but that might just be an artifact of less of them carrying weapons - in particular, it was vastly better for wrestler (because most of them didn't have weapons) and 66% better for discipline (which seems to correllate with wrestling quite strongly). This is probably a wash. However, squads of 4 did better on most stats, but worse on others, all with relatively small effects. In terms of overall XP earned per dwarf, squads of 4 were 4.5% below squads of 10, and squads of 2 were 6.75% higher. Looks like no effect here. However, the effect on stats were generally positive, with somewhat larger effect sizes - +24% for squads of 4 and +40% for squads of 2. So... maybe it's worth setting up small training squads? It's certainly not as big an effect as people have claimed, so for me, it's not worth the extra micro.

On wearing armour: On average, dwarves wearing no armour gained 24% more XP (29% if discounting Armour User XP) and 28% more stats than those wearing armour, so yes, it might be worth it, but again, that's not a big enough effect size for me to be happy with the extra micro and danger.

On splitting weapon types: Unfortunately, between the assumption that a training number of less than the squad size being optimal, and the equipment problems, there wasn't a lot of good data here. In particular, the only squads where both comparison points actually collected weapons were Squads 5 and 7, which gained a grand total of under 2,000 weapon XP in total between them. Those two were quite similar in terms of total XP earned, but the values are so small as to be mostly noise. This one will have to go down as "for future study".


Data
« Last Edit: April 28, 2023, 04:18:12 pm by blue sam3 »
Logged

Mobbstar

  • Bay Watcher
  • aka Mossbird
    • View Profile
    • my website
Re: A Year of Military Training [Science/Case Study]
« Reply #1 on: April 29, 2023, 04:34:44 am »

I would've liked a bit more research on why sparring is better than demonstrations and what circumstances make either better or worse.  Is there something in this regard already?  (This is why we need to cite previous research in the introduction!)

On wearing armour: On average, dwarves wearing no armour gained 24% more XP (29% if discounting Armour User XP) and 28% more stats than those wearing armour, so yes, it might be worth it, but again, that's not a big enough effect size for me to be happy with the extra micro and danger.

I assume most XP deficits from armour slowdown are because the affected dwarves arrive too late to participate in actual battles.  Practically speaking, you want to give your dwarves armour eventually, which means they need to train Armor User eventually.  Is it faster to bootstrap dwarves by delaying the armour?  I assume it would involve a lot of micro-management to hit the right point at which initial combat levels rise enough for sparring, but not too much Armor User xp has been missed out on yet.

On splitting weapon types: Unfortunately, between the assumption that a training number of less than the squad size being optimal, and the equipment problems, there wasn't a lot of good data here.

Did any swordsdwarves learn spear skills or vice versa?

blue sam3

  • Bay Watcher
    • View Profile
Re: A Year of Military Training [Science/Case Study]
« Reply #2 on: April 29, 2023, 07:16:53 am »

I would've liked a bit more research on why sparring is better than demonstrations and what circumstances make either better or worse.  Is there something in this regard already?  (This is why we need to cite previous research in the introduction!)

I'm not sure why it's better, but it definitely seems to be so: it just seems to give dramatically more XP than demonstrations do. Certainly every squad that started sparring dramatically increased training rates. My setup also isn't great for testing that specifically, and I'm not sure how to improve it.

Quote
On wearing armour: On average, dwarves wearing no armour gained 24% more XP (29% if discounting Armour User XP) and 28% more stats than those wearing armour, so yes, it might be worth it, but again, that's not a big enough effect size for me to be happy with the extra micro and danger.

I assume most XP deficits from armour slowdown are because the affected dwarves arrive too late to participate in actual battles.  Practically speaking, you want to give your dwarves armour eventually, which means they need to train Armor User eventually.  Is it faster to bootstrap dwarves by delaying the armour?  I assume it would involve a lot of micro-management to hit the right point at which initial combat levels rise enough for sparring, but not too much Armor User xp has been missed out on yet.

The claim I read was specifically that the slower actions would slow down sparring to the point of making it ridiculous. I'm re-running with the non-armour squads wearing clothes now (and a fix for the weapon problem, at last), to see where that works in.

Quote
On splitting weapon types: Unfortunately, between the assumption that a training number of less than the squad size being optimal, and the equipment problems, there wasn't a lot of good data here.

Did any swordsdwarves learn spear skills or vice versa?

Nobody had any skill in any weapon skill other than the ones that they were wielding. However, there was exactly one spear dwarf and sword dwarf in a squad, and I don't think they ever actually sparred. The new run should fix this.
Logged

Ulfarr

  • Bay Watcher
  • Going on a pilgrimage to Mars
    • View Profile
Re: A Year of Military Training [Science/Case Study]
« Reply #3 on: April 29, 2023, 07:44:43 am »


Quote
On wearing armour: On average, dwarves wearing no armour gained 24% more XP (29% if discounting Armour User XP) and 28% more stats than those wearing armour, so yes, it might be worth it, but again, that's not a big enough effect size for me to be happy with the extra micro and danger.

I assume most XP deficits from armour slowdown are because the affected dwarves arrive too late to participate in actual battles.  Practically speaking, you want to give your dwarves armour eventually, which means they need to train Armor User eventually.  Is it faster to bootstrap dwarves by delaying the armour?  I assume it would involve a lot of micro-management to hit the right point at which initial combat levels rise enough for sparring, but not too much Armor User xp has been missed out on yet.

The claim I read was specifically that the slower actions would slow down sparring to the point of making it ridiculous. I'm re-running with the non-armour squads wearing clothes now (and a fix for the weapon problem, at last), to see where that works in.


May I suggest a three way comparrison, non-armor vs lightweight armor (leather/bone) vs heavyweight armor (metal)? I think that would be needed to prove 1) if non armor has merit (outside of roleplay reasons) and 2) if the any armor is still crippling the xp gains.
Logged
Bring Kobold Kamp to LNP! graphics compatibility fix.

So the conclusion I'm getting here is that we use QSPs because dwarves can't pilot submarines.

blue sam3

  • Bay Watcher
    • View Profile
Re: A Year of Military Training [Science/Case Study]
« Reply #4 on: April 29, 2023, 10:32:00 am »

Comparisons to test:

1. Squad Size (2,3,4,10)
2. Armour vs no armour (2)
3. Split vs same weapons (2)

16 squads:

Squad 1: 10, clothes, same weapons
Squad 2: 10, clothes, split weapons
Squad 3: 10, armour, same weapons
Squad 4: 10, armour, split weapons
Sqaud 5: 4, clothes, same weapons
Squad 6: 4, clothes, split weapons
Squad 7: 4, armour, same weapons
Squad 8: 4, armour, split weapons
Squad 9: 3, clothes, same weapons
Squad 10: 3, clothes, split weapons
Squad 11: 3, armour, same weapons
Squad 12: 3, armour, split weapons
Squad 13: 2, clothes, same weapons
Squad 14: 2, clothes, split weapons
Squad 15: 2, armour, same weapons
Squad 16: 2, armour, split weapons


Much better start this time: everybody went to recruit/militia captain/militia commander immediately, and nearly everybody picked up at least some equipment: a total of four dwarves are missing weapons and shields (3 of Squad 1, 1 of Squad 2, 3 of Squad 3. However, nobody has claimed any armour. If that stays, that isn't a big deal, and we'll just test 1 and 3.

Start date: 22 Hematite

1 month report: 2 dwarves have put on pieces of armour. No idea why nobody else has. Everybody is gaining XP in their assigned weapon skills, despite some of them still not actually carrying said weapons, probably due to some demonstrations, which have started much more quickly this time. A few have already hit Novice in one or two combat skills, and one has done so in Student/Concentration.

2 month report: everybody's still doing demonstrations/individual combat drill. No obvious separations yet. Still those five dwarves not carrying weapons.

3 month report: sparring has begun in squads 2, 3, 4, and 8. Those squads have drawn significantly ahead of everybody else - Squad 8 are starting to hit proficient fighter already, while nobody else is past competent.

4 month report: Everybody except for squads 11 and 15 is now sparring. It does seem like the squads with split weapons did start sparring (and thus rapidly gaining skills) sooner. The best squads are starting to hit Talented Fighter, and Skilled in their weapon skills. Observer is up to Expert. Nobody in the two non-sparring squads is above Skilled in anything.

It turns out that literally all of the armour has vanished (it does not appear on the stocks screen). No idea what happened there. Nothing except armour (including clothes) has been affected. Maybe I accidentally gave it to a merchant? Not sure, but at least it explains why nobody is putting armour on. On closer inspection, the dwarves who are wearing a few pieces of armour aren't wearing the armour I embarked with, but armour purchased from the merchants to give them more space for masterwork adamantium goblets (in order to get this many migrants quickly enough, you need a lot of exported wealth very quickly).

5 month report: Every squad has now sparred at least once. The best dwarves are now Adept Fighter and Discipline and Proficient weapon skill. The worst (excluding those with no weapons) are Competent/Competent/Adequate in those three. The best Observer is up to Grand Master.

6 month report: The best dwarves are now at Expert/Expert/Talented in Fighter/Discipline/Weapon. The best Shield skill is up to Competent. Interestingly, it seems that everybody in the squads with people missing weapons is gaining significantly more Striker and Wrestler XP than anybody in any other squad - those squads all have between Competent and Skilled in Striker, and Adequate to Competent in Wrestler. Nobody outside of those squads has above Novice Wrestler and Dabbling Striker. The people without weapons are behind the rest of their squads, but not massively so - they're between 1 and 2 XP levels behind the best, so far.

On a side-note, I had the two non-military dwarves in the fortress (the mayor and his spouse) run around and butcher all of the non-pet animals, which took me from 100 to 150 FPS. Turns out, animals use a lot of FPS up.

7 month report: The best Dwarves are now at Accomplished/Profissional/Adept, the worst (with weapons) at Proficient/Proficient/Skilled. The best Observers are up to Legendary+5.

8 month report: Up to High Master/Great/Professional at the top end. The squads with people missing weapons have all dropped significantly behind the other squads in those three skills (but have dramatically more Striker XP to make up for it).

Final report:

Relative to squads of size 10, squads of size 4 gained a total of 21% more XP, those of size 3 gained 7.2% more, and those of size 2 gaind 43% more. However, after removing the squads with at least one member with no weapon, these change to 9.4% above, 2.7% below and 30% above, respectively. Looking at the individual skills (after subtracting out the missing-weapon squads) weapon skills were +10%, -0.23% and +41% respectively, discipline +16%, +6.4% and +42%, dodger -24, -33, and -45%, fighter +13%, -1.9% and +37%, striker -0.74%, 0.0%, and +43%, wrestler -15%, -27% and -17%, and shield user +20%, +2.6% and +33%. Dodger being lower in smaller squads makes some sense: when 5 people are sparring in a group, each one has more attacks to try to dodge. This is also the only pattern that monotonic in squad size. The size-3 squads are weird in general (apart from dodging and wrestling, they're basically identical to size-10 squads). Ignoring them, there's pretty clear improvements to weapon skills, discipline, fighter, and shield user at lower levels. These are the four main skills trained by sparring, so this might indicate more sparring among smaller squads. Perhaps odd sizes don't work well because they can't pair up for sparring as well?

Relative to squads of all swordsdwarves (and ignoring size 10 squads because of the weapon problems), the split sword/spear squads gained 13% less weapon XP, 3% more discipline, 4% more dodger, 4% less fighter, 50% more striker, 21% more wrestler, and 2% less shield user (that's 2.4% less XP total). Nobody gained any XP in a weapon skill other than their own. This is consistent with them not doing weapon demonstrations (hence lower weapon XP) and resorting to wrestling/striking more often in sparring - not sure why that would be.


Overall, I can see good reasons to use small, even-sized squads for training. I can't see any reason to split or not split weapons.


I'll have a third go at the armour comparison tomorrow, with that three-way comparison.
Logged

Panando

  • Bay Watcher
    • View Profile
Re: A Year of Military Training [Science/Case Study]
« Reply #5 on: May 06, 2023, 02:21:58 am »

While there is some interesting data here, I think it's also important to be able to look at the natural variance.

While it was roughly a decade ago, when I last did some testing on military training, one of the most insightful things was just saving the game, and then re-running the precise same scenario with no changes. The amount of variance was MASSIVE, like the exact same squad could end the year with talented weapon skill or legendary weapon skill, depending on absolutely nothing other than the way the random numbers roll. While this testing was done a long time ago, I have no reason to believe it has changed at all.

This kind of high variance is always a serious problem when doing comparisons, because to not just be measuring "who got lucky" you need a lot of trials. Probably starting at like 20 with the exact same setup. This is not that hard with DFHhack standardization of dwarves and reloading and running the game at a high speed multiplier and limiting the amount of variables examined (say, weapon skill after 1 year). In principle it probably wouldn't even be that hard for DFHack to automate the testing and logging, like once a year a script logs the skill levels, and then resets all the dwarves.

I was thinking of doing some such testing myself, but I'd probably wait until Putnam's performance enhancements are stable with DFHack.
Logged
Punch through a multi-z aquifer in under 5 minutes, video walkthrough. I post as /u/BlakeMW on reddit.

blue sam3

  • Bay Watcher
    • View Profile
Re: A Year of Military Training [Science/Case Study]
« Reply #6 on: May 06, 2023, 12:55:52 pm »

While there is some interesting data here, I think it's also important to be able to look at the natural variance.

While it was roughly a decade ago, when I last did some testing on military training, one of the most insightful things was just saving the game, and then re-running the precise same scenario with no changes. The amount of variance was MASSIVE, like the exact same squad could end the year with talented weapon skill or legendary weapon skill, depending on absolutely nothing other than the way the random numbers roll. While this testing was done a long time ago, I have no reason to believe it has changed at all.

This kind of high variance is always a serious problem when doing comparisons, because to not just be measuring "who got lucky" you need a lot of trials. Probably starting at like 20 with the exact same setup. This is not that hard with DFHhack standardization of dwarves and reloading and running the game at a high speed multiplier and limiting the amount of variables examined (say, weapon skill after 1 year). In principle it probably wouldn't even be that hard for DFHack to automate the testing and logging, like once a year a script logs the skill levels, and then resets all the dwarves.

I was thinking of doing some such testing myself, but I'd probably wait until Putnam's performance enhancements are stable with DFHack.

Yeah, I've come to the conclusion that this is going to have to wait for that (and preferably also a few more DFHack commands to speed up the setup).
Logged