Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3 4

Author Topic: The (Not-So) Horrible Truth about Military xXScienceXx  (Read 10428 times)

Bjiip

  • Bay Watcher
  • Competent Worldsmith
    • View Profile
The (Not-So) Horrible Truth about Military xXScienceXx
« on: May 30, 2018, 10:47:16 pm »

EDIT: User tussock revealed a major error in my methodology.  Leaving original post for historical reasons.

Later posts with more science:
The Effect of Equipment Material
The Effect of Weapon Choice

The Effect of Shield and Armor

Original post begins:

You know all that received wisdom about military setup?  Use blunt weapons for armored enemies, and slashing weapons for unarmored enemies?  That iron is superior to copper, and steel to iron?

Yeah, you just keep thinking that.

This is a story about how I found out that everything we know is wrong.  You want to know how to outfit your dwarves?  At the end, you'll find out.

It started when I was looking at the Military Testing page on the wiki.  I saw tests being run like "5 dwarves with adequate combat skills and iron equipment using battleaxes versus 5 similar dwarves using swords".  Hey, I thought. I can contribute to that.  So I started doing little tests.

My first tests were on the effect of skill.  One question: "What skill level does a dwarf have to be in order to be able to beat two dwarves, similarly equipped, with no skill?"  But my results were strangely inconsistent.  Sometimes the Competent (III) dwarves fared better than the Skilled (IV) dwarves.  That's strange.  The experimental design is simple; maybe my sample sizes just weren't big enough.

So I started thinking bigger.  I rewrote the arena layout to comprise a grid of 9x9 cells, separated from each other, perfect for controlled combats.

Spoiler: Large Arena Image (click to show/hide)

Populating the arena by hand became too slow.  I downloaded AutoHotKey, learned the syntax, and wrote scripts to duplicate dwarf layouts into many cells at the press of a hotkey.  I was ready to get mass statistics.  I would take all the wonderful variation possible in Dwarf Fortress's combat system, and through sheer weight of samples, beat out the noise and get the signal.

I tested the age-old debate of axe versus sword.  I populated the arena with dwarves in full iron armor; one team with axes, another with swords.  The results?  Axe won 38%, sword won 62% of fights.  Now that's a useful result!  Next time goblinite comes calling, my dwarves will carry swords.

We also know that hammers are better than axes for fighting armored enemies- right?  So I populated the arena with a team of axedwarves and a team of hammerdwarves.  This time, axe won... 42% of fights, and hammers won... 58%.  Well, hammers won, as we'd expect.  But it's strange that swords would be better against full iron armor than hammers.  After all, swords are supposed to be stopped by armor, but blunt weapons are specialized against armor.

Maybe swords are just better.  Swords versus hammers, go!  Swords win... 44%, hammers win 56%.  That's strange- hammers win against swords; but swords win harder against axes than hammers do?

Let's re-establish that the combat system makes sense.  A mirror match!  One dwarf on each side, in a no-quarter battle to the death.  With a large number of samples (>100), it should be very close to 50-50.  The results were... 45% to 55%.  That's funny.  I would have expected the results to be closer.

Well, maybe the system is just noisy- prone to random error.  What we need is more samples.  I filled the entire arena, over 1,700 cells.  In one corner, the iron-equipped axedwarves; in the other, iron-equipped hammerdwarves.  With this massive sample size, we'll certainly eliminate the noise and get the signal.  The results?  Axedwarves win 54%, hammerdwarves win 46%.

Wait a second- axedwarves win?  But last time, hammerdwarves won handily.  The sample size is pretty massive- more than most dwarves most of us will send into battle over the entirety of our Dwarf Fortress career.

Well, let's do something that's sure to show results.  There's debate over whether bronze or iron is better; but there's no debate about iron versus copper.  Let's take dwarves of comparable skill, equipped entirely in iron or entirely in copper.  The results? 97 to 96.

It's so close to 50-50, any respectable statistician would tell me that I fail at faking statistics.  Only I didn't fake them.

You want to know how to outfit your dwarves?  Outfit them in a way you think looks cool.  Nothing else matters; not weapon type, not material type.  The only effect will be on your imagination.  Dress them to die in whatever way seems best to you.

Spoiler: End Notes (click to show/hide)
« Last Edit: June 11, 2018, 01:11:29 am by Bjiip »
Logged

GoblinCookie

  • Bay Watcher
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #1 on: May 31, 2018, 06:36:25 am »

How much armour did you give your copper VS iron fighters.  Was it copper or iron?  Copper and iron are both harder than flesh, so it makes sense that it would not make a difference against unarmoured opponents. 
Logged

George_Chickens

  • Bay Watcher
  • Ghosts are stored in the balls.
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #2 on: May 31, 2018, 09:18:08 am »

These things are strongly dependeant on armour type and body. Blunt weapons absolutely WRECK HOUSE when up against moderate to moderate-large creatures, even through armour, but fall short when it comes to large creatures. Swords are great against smaller creatures but tend to fall short against larger ones, in which I much prefer an axe.
Logged
Ghosts are stored in the balls?[/quote]
also George_Chickens quit fucking my sister

Bjiip

  • Bay Watcher
  • Competent Worldsmith
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #3 on: May 31, 2018, 09:29:05 am »

How much armour did you give your copper VS iron fighters.  Was it copper or iron?  Copper and iron are both harder than flesh, so it makes sense that it would not make a difference against unarmoured opponents. 
Fighters always had a complete set of armor; details are listed in the end notes.  The "copper" fighters had copper armor and copper weapons, while their "iron" opponents had iron armor and iron weapons.  I was sure that in this test, at least, the iron fighters would dominate.  That's why I was so shocked when, with about 200 samples, it was a tie.  200 fights is probably more than the number of dwarves I've sent out to battle in my Dwarf Fortress career.

These things are strongly dependeant on armour type and body. Blunt weapons absolutely WRECK HOUSE when up against moderate to moderate-large creatures, even through armour, but fall short when it comes to large creatures. Swords are great against smaller creatures but tend to fall short against larger ones, in which I much prefer an axe.
Admittedly, I only did dwarf vs. dwarf fights.  I wanted to isolate out the effect of weapon and armor choice (or, in different tests, the effect of different skill level), so I wanted the creatures in the fight to be the same.  So I didn't test the received wisdom that (for instance) spears are especially good against very large enemies.
Logged

Ulfarr

  • Bay Watcher
  • Going on a pilgrimage to Mars
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #4 on: May 31, 2018, 09:57:34 am »


Well, maybe the system is just noisy- prone to random error. 


This is in my opinion the most important factor in DF combat. Here are a few examples of what could be the source of that noise:

1) Attack types are random:
You can give them axes but Urist McDumb might just use it like a club, hitting his targets with the flat of the blade making it useless.

2) Targets are random:
An enemy can have no armor other than just a helmet and your dwarves might still just aim for the head. Great, now half of their attacks get deflected.

3) There is no tactics in mass comabt:
In one battle your dwarves will gang up on one enemy at the time and the other they might just fight in duels. Good luck sorting these out.

4) "Critical hits" are random:
Sometimes your dwarves will cut off their enemy's limbs with one hit and other times they'll struggle cutting through a cotton sock.

 I mean there are just so many things that are out of our direct control that I doubt there is one definite answer to the "what gear should I give my dwarves" question. It's also why these fabulous stories of whole squads getting decimated by a lone dwarf armed with nothing but a sock, can happen.

With that said, material/weapon choice does have an effect as evident by both your experiment and conventional wisdom (another example is wooden bolt vs armor compared to metal bolts vs armor). It's just that right now, it's not actually restricting the player, by having an absolutely superior choice. Honestly, I think it's better that way as it gives us viable options for role playing.
Logged
Bring Kobold Kamp to LNP! graphics compatibility fix.

So the conclusion I'm getting here is that we use QSPs because dwarves can't pilot submarines.

Trif

  • Bay Watcher
  • the Not-Quite-So-Great-as-Toady One
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #5 on: May 31, 2018, 09:59:04 am »

Interesting findings!
I think this is good data, but your conclusion is flawed.

I believe it's generally well accepted that the most important fighting skill is the weapon skill.
Your setup got some good results for fights between fairly unskilled fighters (yeah, "Skilled" means not very skilled at all). And the results aren't very surprising at that skill level: fights seem to be decided at random - I'm guessing whoever gets the first lucky shot wins the battle (you also saw this in fights between Competent and Skilled dwarves).

So I attribute the randomness to the fact that both fighters can't use their weapons very effectively.

The skill level of dodger, armor user, maybe even fighter can also have an impact on the overall result.
There are many variables that you put into the background, focusing on weapon and armor material, but recognizing the effect of these variables would require much more testing.


Also (this is a separate point), it would be nice to have some error bars on your results. 100 fights feels like a lot (and it's a lot of effort, so props to you), but is it really enough to get certain results within 10%? within 5%? That would be nice to know.

Definitely thanks for your work! But I believe you were too quick with your conclusions.
Logged
Quote from: Toady One
I wonder if the game has become odd.

Werdna

  • Bay Watcher
  • Mad Overlord
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #7 on: May 31, 2018, 12:26:14 pm »

So I didn't test the received wisdom that (for instance) spears are especially good against very large enemies.

I can add one data point backing up this wisdom:

I've been monitoring my soldiers in a long-running fort (v43.05) adding 8 or so Fortress Defense races, some of which are huge (esp. War Elephants, which arrive in full plate).  My top soldier has 250+ kills with her axe, yet not one War Elephant to her name.  In the roster of close to 50 soldiers, the only ones with more than one War Elephant kill (mostly in the teens) are the Macedwarves, Hammerdwarves, and Speardwarves.  All of the Axedwarves, Swordsdwarves, and occasional Pickdwarf (which in all, constitute about 2/3 of the force) have less than a handful of WE kills among them, and I'd say ~250 WE's have attacked so far.  The squads trained with mixed weapons from the start so it's not an experience difference (they're all Legendary, currently).

Some of the comments above re: random factors should reread the original post, as the OP described vastly scaling up the number of samples which would act to average out the randomness.

What I don't understand is the copper vs iron difference.  I vividly recall combats in older versions of the game, where a dwarf exhausts and the goblins stand over him, endlessly beating on his head, and failing entirely due to a steel helmet vs whatever crap they're wielding.  The feature to remove the helmet was added specifically for this scenario.  Can copper weapons really pierce iron that easily now?  Does the weapon skill perhaps add to the 'shear' factor, allowing a copper sword wielded by a Skilled dwarf to cut through iron, and effectively rendering a copper vs iron fight meaningless?  I wonder if testing with no weapon skills will see the copper vs iron difference. 
Logged
ProvingGrounds was merely a setback.

Bumber

  • Bay Watcher
  • REMOVE KOBOLD
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #8 on: May 31, 2018, 05:08:53 pm »

The joint-twisting mechanic changed combat. You no longer need to pierce armor to do devastating damage to the dwarf underneath. It's no surprise that an axe can now seriously ruin someone's day.

Populating the arena by hand became too slow.  I downloaded AutoHotKey, learned the syntax, and wrote scripts to duplicate dwarf layouts into many cells at the press of a hotkey.
You know DF has a built-in macro feature, right?
Logged
Reading his name would trigger it. Thinking of him would trigger it. No other circumstances would trigger it- it was strictly related to the concept of Bill Clinton entering the conscious mind.

THE xTROLL FUR SOCKx RUSE WAS A........... DISTACTION        the carp HAVE the wagon

A wizard has turned you into a wagon. This was inevitable (Y/y)?

Bjiip

  • Bay Watcher
  • Competent Worldsmith
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #9 on: May 31, 2018, 10:02:45 pm »

I think this is good data, but your conclusion is flawed.
Actually, I do too.  This is really more of a call for help; I don't actually believe that nothing matters.

(yeah, "Skilled" means not very skilled at all).
This is an area where I would welcome help from the experts of the forum.  What are good levels of skills to test? 

Also (this is a separate point), it would be nice to have some error bars on your results. 100 fights feels like a lot (and it's a lot of effort, so props to you), but is it really enough to get certain results within 10%? within 5%? That would be nice to know.
This is also an area where I would welcome help.  I don't have a lot of statistics skill.

Also, admittedly, I should be keeping track of the "tristate" nature of the fights.  In my large-scale results, I only count results where one dwarf wins and the other dwarf dies; I didn't keep track of fights where both dwarves die.  For my next round, I can try to measure that.

The joint-twisting mechanic changed combat. You no longer need to pierce armor to do devastating damage to the dwarf underneath. It's no surprise that an axe can now seriously ruin someone's day.
Perhaps I should test the effect of only weapon skills versus weapon skills plus unarmed skills?

You know DF has a built-in macro feature, right?
Nifty!  I didn't know about that feature.  I still think AutoHotkey is more powerful and better for my needs, but it's great to hear about other tools.
Logged

Bumber

  • Bay Watcher
  • REMOVE KOBOLD
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #10 on: June 01, 2018, 12:25:51 am »

The joint-twisting mechanic changed combat. You no longer need to pierce armor to do devastating damage to the dwarf underneath. It's no surprise that an axe can now seriously ruin someone's day.
Perhaps I should test the effect of only weapon skills versus weapon skills plus unarmed skills?
It has nothing to do with unarmed skill. Any sufficient force can now transfer to the joints of an armored target, potentially causing mangling of body parts.
Logged
Reading his name would trigger it. Thinking of him would trigger it. No other circumstances would trigger it- it was strictly related to the concept of Bill Clinton entering the conscious mind.

THE xTROLL FUR SOCKx RUSE WAS A........... DISTACTION        the carp HAVE the wagon

A wizard has turned you into a wagon. This was inevitable (Y/y)?

Grand Sage

  • Bay Watcher
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #11 on: June 01, 2018, 06:07:23 am »

I wonder if the difference increases with the armor user skill. So that, for example, two legendary dwarves fighting in respectively steel and copper armor will lean more towards steel than "skilled" armor users.

Also, to back your claim that material type doesn't matter, you really should test Steel against copper. And, coming to think of it, throw some candy in there too, otherwise your claim only touches the "lesser" materials. And it would make sense that there isn't a big difference between them.

Another thing I came to think of was wear. I am almost certain Steel armor will last your combatants longer than copper or leather.

Also, I wonder how armor material is included in off-side combat, like raids.

So, probably safe to say that material does matter after all.
Logged
Dwarf Fortress: This feature has one or more outstanding bugs. Please visit the Bugs section for details.

And I drank the mosquito paste

GoblinCookie

  • Bay Watcher
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #12 on: June 01, 2018, 10:55:32 am »

Fighters always had a complete set of armor; details are listed in the end notes.  The "copper" fighters had copper armor and copper weapons, while their "iron" opponents had iron armor and iron weapons.  I was sure that in this test, at least, the iron fighters would dominate.  That's why I was so shocked when, with about 200 samples, it was a tie.  200 fights is probably more than the number of dwarves I've sent out to battle in my Dwarf Fortress career.

The relative casualties on the winning side have to be taken into account here.  When the iron fighters fought the copper fighters and won, did they suffer proportionally fewer casualties than the reverse case?
Logged

Bjiip

  • Bay Watcher
  • Competent Worldsmith
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #13 on: June 02, 2018, 02:23:03 pm »

I wonder if the difference increases with the armor user skill. So that, for example, two legendary dwarves fighting in respectively steel and copper armor will lean more towards steel than "skilled" armor users.
I will test this.

Another thing I came to think of was wear. I am almost certain Steel armor will last your combatants longer than copper or leather.
But we are more concerned with the durability of our soldiers than the durability of their equipment.

So, probably safe to say that material does matter after all.
That was what I thought too, until I actually tested things.
Logged

Bjiip

  • Bay Watcher
  • Competent Worldsmith
    • View Profile
Re: The Horrible Truth about Military xXScienceXx
« Reply #14 on: June 02, 2018, 03:18:27 pm »

More science!  This time, testing the importance of good unarmed skills in combat.



My reading is that at medium to high levels of skill, having unarmed skill gives a significant edge in combat.  It looks like the side with unarmed skills has close to a 2-to-1 win rate against those without unarmed skills at those levels.

Note that the Unskilled (0) bar at the left is a total mirror match, with no skill.  The fact that it's not 50/50 shows some measure of the randomness in play.

Spoiler: End Notes (click to show/hide)
Logged
Pages: [1] 2 3 4