Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Edward_Tohr]

I figured with DF being properly 64-bit nowadays, I'd try out a 16x16 embark.  Genned up a pocket world with all the elevation/drainage/savagery/etc. variances and max subregions set to max, let worldgen go until it stopped (year 30, as it turns out), and found a nice, six-biome area to embark in.

For the most part, it's actually running much better than expected, about 40 FPS with 10 dwarves. However, every so often, the game will lag for 10-15 minutes straight. Any idea what might be causing this, and if there's anything I can do about it?

Playing on 44.12, no mods, no dfhack.

[Immortal-D]

The game could be running at 512-bit, but that doesn't change the fact that it is very poorly optimized.  In addition to optimizing on the code size, consider that literally every interaction is saved in the world's memory; every item created, injury or death, personal history through time, etc.  A maximum size embark is only useful if you have a very specific world-building project in mind.  For a serious Fortress, no more than 5x5 tops.

[Edward_Tohr]

Aha. Seems to have been related to pathfinding and 1-2 depth water. Just needed to wait for evaporation to fix the problem for me.

[Fleeting Frames]

Vegetation and (especially evil) rain also contribute to fps loss, so for something grand glacier/no-vegetation desert/tundra/mountain would be best. 

[Immortal-D]

Aha. Seems to have been related to pathfinding and 1-2 depth water. Just needed to wait for evaporation to fix the problem for me.

Pathfinding is also a big consideration; more space means more calculations.  I'm curious to see how your FPS holds up after a couple of a years.

[Edward_Tohr]

I'm curious to see how your FPS holds up after a couple of a years.

Me too.

Jobs tasking items, especially if more than a couple are designated while the game is paused, seem to cause multi-minute lag spikes every time, even if the workshop is right next to a QSP with said materials.  News to nobody, I'm sure, but still something I'll have to work with while playing chainmailing while waiting for the program to resume.

Still, while I'm here, might as well try some ‼Science‼, right?

[Saiko Kila]

What about page file usage and general memory usage? Maybe they are culprits?

Anyway, I don't think that current PCs are fit for the game as it currently stands. I mean these big embarks on x64. If something works for x86 then it should work for x64, but if something something doesn't work for x86, then I would expect problems on x64 too.

DF x64 used to steal all my memory and torture my HDDs (probably was fixed, doesn't happen nowadays, but then I don't try big embarks anymore).

[Telgin]

If you don't have enough physical RAM to keep the game resident in memory then yes, it's absolutely going to cause potentially serious performance problems.  I'm actually kind of curious how well DF behaves with memory access locality, which impacts performance even if the whole game is resident in memory, but if the accesses are highly random across the entire address space then paging to disk will be murderous on performance as the OS has to continuously page things in and out.  If the accesses are highly localized then it's merely (potentially very) bad for performance.

Anyway, in this case it sounds like the liquid simulations and recomputing paths was the cause here.  I'm guessing the game was trying to recalculate the connectivity graphs nearly every frame or something.  A bigger map definitely impacts that.

[WJLIII3]

Doesn't DF pathfinding also check literally the entire visible area before resolving? I seem to recall that being a problem, and the reason why late-play fortresses should keep their cavern gates shut.

[Telgin]

The game organizes the map into a graph of connected areas, and it only performs pathfinding searches within the connected areas, which is why closing gates can help.  In the worst case, the entire map is connected (no doors, no walls, no floors, nada), but that doesn't happen in practice.

I'm not sure how it treats open air.  I know some people have done some testing and reported anecdotally that open air hurts pathfinding performance, which would make sense for flyers.  However, at least in older versions, flyer pathfinding was broken in ways that led me to believe that open air was completely disconnected from normal pathfinding calculations.

[Edward_Tohr]

What about page file usage and general memory usage? Maybe they are culprits?

Anyway, I don't think that current PCs are fit for the game as it currently stands. I mean these big embarks on x64. If something works for x86 then it should work for x64, but if something something doesn't work for x86, then I would expect problems on x64 too.

DF x64 used to steal all my memory and torture my HDDs (probably was fixed, doesn't happen nowadays, but then I don't try big embarks anymore).

It was only using ~3.5 gigs of RAM, and I've got 24 installed, so it shouldn't have been paging at all.

General pathfinding certainly seems to be much more efficient than the item-claiming logic for jobs, since that would lag the hell out of everything. Had a bit of lag last night that went on for about an hour before I got bored and killed the process. Even that wouldn't be so bad if I could hit escape and know the game would pause as soon as it finished processing, but...

[tussock]

That may explain some properties of who gets jobs at workshops.

The reason that might be, just guessing, is that the workshop one-tick flood fills the entire map with the walking pathfinder as a "shortcut" to find every dwarf who can path to it, which is no longer a shortcut at all when all dwarfs are within a few tiles on an extreme size map.

I'd imagine a cluster of building destroyer monsters entering the map edge would do something similar for each one as they entered, though would only fill a third of the surface or so, and sealing HFS would do it every tick thereafter for the whole volume of the circus tent. Migrants and traders will also each have to flood fill from the map edge to find their respective targets, and building a depot should flood generate a huge new map for wagon access.

If completing a depot or migrants arriving doesn't stall for similar time, there might be a bug in how often the full map pathfinder is being called there for job allocation.

[DwarfToys]

Memory bandwidth may be limited for single core apps to way below the multi-threaded limit, and with the amount of data being processed per frame / second, there's a chance the limit is being hit.  Each core can only do a certain number of memory requests per cycle.   See p. 8 of this.

The game won't make my proc scale all the way up to base speed (unless saving a world, then it gets close).  Most small programs will push it to turbo frequency single threaded.  That's usually a sign being ram r/w limited or massive pipeline stalls from expensive instructions.  There are several places in the program where it looks like optimizations should have been performed easily by the compiler but weren't.  Large sequences of redundant calculations on an index register interspersed with memory writes of a constant value that are performed out of linear order for example.    There's not much more to say on that, I don't know anything about the source code or build settings and I've had my fill of assembly for the week. 

[Fleeting Frames]

On a related note, worth noting Toady performed some optimizations in .44 cycle; Archcrystal went from 5-6 fps in 40.24 to 11 in .09 to 16 fps now.

[DwarfToys]

On a related note, worth noting Toady performed some optimizations in .44 cycle; Archcrystal went from 5-6 fps in 40.24 to 11 in .09 to 16 fps now.

Improved here too.   A clean world / fort at 100fps cap was now updating almost too fast to keep track of.  I had to drop it down to 80 so I could see what was happening better.  That's a great improvement over 43.05 based on top speed alone.