Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Sweet-O]

Hi all. I hope you're doing well

I'd like to hear/read what you think about the following idea.

- adding an integrated benchmark into DF
means for example: 5x5 Map, 250 Dwarfs all with active(not just selected, but actually done) labor, counting time for 1 season(/1 month/ 10 days, whatever makes best sense) to elapse. Counting time from start to finish. Done by a saved game for comparable results.

Reason:
Since my introduction to the game I have been exclusively choosing my CPUs to run DF the best possible (@ my budget). I don't really care about core amount but more about IPC and frequency. No other game I've ever heard of is so CPU dependent.
While some of these values can be searched for in general benchmarks for CPUs, I think having an integrated one may improve accuracy for them for DF specifically.
I'd be interested to see whether either Intel or AMD have specific benefits in DF outside their official IPC.

Cons: Tables may need to be re-benched if an update invalidates the chosen save-game

Anybody else interested in, or bothered by this?

[Mobbstar]

A benchmark savefile would be interesting.

However, I am concerned that legendary beasts, invaders, and even crows would randomly appear and cause inconsistent results. Is it possible to make a Debug-Arena style savefile to eliminate all the aforementioned concerns?

[methylatedspirit]

I suppose if you could control all the random variables (by ensuring that Benchmark mode always runs with the same RNG seed every time, for instance), that'd go a long way for improving repeatability.

Also, in case you're not aware, Anandtech have a Dwarf Fortress 0.44.12 test, and their results are in their Bench database. For shorter and smaller worldgens, it seems that AMD wins, but bigger ones favor Intel. I still wouldn't buy an i9-11900K based on that, though. The biggest 257x257 one seems to have some very odd things at the top. A Haswell Xeon? An i5-6500? An embedded Ryzen chip?! Take it with a grain of salt, I'll say that much.

[Sweet-O]

A benchmark savefile would be interesting.
However, I am concerned that legendary beasts, invaders, and even crows would randomly appear and cause inconsistent results. Is it possible to make a Debug-Arena style savefile to eliminate all the aforementioned concerns?

Invaders and megabeasts may be avoided by world-genning without them. Interruptions should be completely avoidable. Like caravans, if you bench in the winter.
Random animals afaik never appear in large enough numbers to mess with the resource drain of a huge fortress. The bigger the load, the shorter the in-game time may be to mitigate the influence of random animals.

I suppose if you could control all the random variables (by ensuring that Benchmark mode always runs with the same RNG seed every time, for instance), that'd go a long way for improving repeatability.
Also, in case you're not aware, Anandtech have a Dwarf Fortress 0.44.12 test, and their results are in their Bench database. For shorter and smaller worldgens, it seems that AMD wins, but bigger ones favor Intel. I still wouldn't buy an i9-11900K based on that, though. The biggest 257x257 one seems to have some very odd things at the top. A Haswell Xeon? An i5-6500? An embedded Ryzen chip?! Take it with a grain of salt, I'll say that much.

No, I wasn't aware. Thank you for the info 
I've had a AMD Ryzen 5 5600X planned for my next CPU (if it's not replaced by something better by the time GPUs become affordable again). Places high in the list, although I must agree that some values do seem to be quite off (aforementioned xeon faster on bigger map )

I can't really fathom why they'd use world-gen as a benchmark (aside from the simplicity of the setup). The load from a fortress is quite different and I think would definitely lead to different tables than the world-gen. Hashing for RNG (I'd presume) and if-statements (branch prediction should not be as efficient, when the RNG is good ^^) compared to mostly path-finding algos, temperature, fluid and wear calcs, table look-ups....  (what else wrecks fortress performance?).

I might be completely wrong about these different algorithms causing different loads in a way that distinguishes CPUs from each other. But maybe it does   

[methylatedspirit]

I can't really fathom why they'd use world-gen as a benchmark (aside from the simplicity of the setup). The load from a fortress is quite different and I think would definitely lead to different tables than the world-gen. Hashing for RNG (I'd presume) and if-statements (branch prediction should not be as efficient, when the RNG is good ^^) compared to mostly path-finding algos, temperature, fluid and wear calcs, table look-ups....  (what else wrecks fortress performance?).

I might be completely wrong about these different algorithms causing different loads in a way that distinguishes CPUs from each other. But maybe it does   

Easy. Worldgen is predictable. Just input the same parameters each time, and you'll always get the same world with the same calculations each time, as you said.

The reason why it's so hard to get good results for Fortress Mode proper is that it's not predictable. The RNG is different each time, leading to different results on the same machine. I don't know if DFHack has the memory addresses for the RNG seed, but being able to control that (plus other randomizing variables) as an official feature would be nice. Even in a specific Benchmark Mode or something.

I still agree that Fort Mode does have different calculations that may lead to different sets of results (and I think it's been an on-and-off topic of discussion here), so I still hold that a Benchmark Mode would be useful. Maybe expand it to Adventure Mode as well.