Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[peterix]

Right, but the idea is that if, even when deleted, those splatters and objects you atom-smashed still "took up" a slot in an array in an index vector (or vector-like data structure) that contains references to every object on the map, then even atom-smashing and removing splatters would still leave a "footprint" in terms of "empty slots" in the index vector that must be skipped over whenever the index vector is referenced.  That act of skipping still takes a (smaller, but still present) number of cycles, and when you have removed millions of objects or spatters from the game, then you will have millions of "empty slots" that the game will have to overlook every time.

Right. Let's get some actual info here, K?

DF has many global vectors for items (one has everything, then there's one for every item type). See: https://github.com/peterix/df-structures/blob/master/df.world.xml#L297 I don't know it off the top of my head, but I think they are ordered by item ID to allow binary search.

To speed up things like drawing, every map block (map is divided into blocks of 16x16x1 tiles, organized in a 3d array) has an ID vector of items: https://github.com/peterix/df-structures/blob/master/df.map.xml#L149
The game uses IDs stored in the blocks to search through the big all items vector (it's really a vector of pointers).

Feel free to set up a dfhack dev environment and start poking around further :-) For example, a nasty source of FPS loss could be if the ID vectors in map blocks never got emptied properly.

[Kogut]

<ops>

[Miuramir]

Get DFhack and use the commands

"clean map"
"clean units"
"clean items"
...

Right, but the idea is that if, even when deleted, those splatters and objects you atom-smashed still "took up" a slot in an array in an index vector ...

IIRC there was some strong indications in previous versions that the memory hacks used by DFhack, etc. had some serious hidden limitations in that they effectively sidestepped various optimization and/or garbage collection (in the computer sense) triggers.  It was well documented that hacking the removal of magma from a tile did *not* trigger whatever internal routines fix up temperature propagation / cleanup; and that a tile that had magama hack-removed could still burn many seasons later.  As I understand it Toady's code seems to have some fairly significant "lazy update" processing to improve performance, and using hacks can trigger weird side effects connected to not running cleanup routines. 

A useful experiment to run might be to take an existing prosperous fort with really good trade and clone off a copy.  The fort should have plenty of available haulers, several (preferably plenty of) experienced miners and plenty of picks, an atom-smashing setup ready to go but not used yet, several stonecrafting and masonry workshops with at least competent dwarves to man them, something to make reasonable bins out of (preferably ordinary wood in quantity), and be expecting a large wagon equipped dwarf caravan in the next few months. 

Fire up the copy, record averaged FPS, and then dig out a huge amount of stone with good miners, initially just having it lie there.  Record averaged FPS (hopefully significantly lower, if not keep digging), then save the fort and make several parallel copies. 

On the same system as used so far, handle all the stone differently, and record how averaged FPS changes.  Because there may be some deliberate cleanup involved in the save / load process, it's important to run the experiment from here out with the cases in a single session, record averaged FPS, then save, exit DF, and load and record averaged FPS again. 
* Control: Stone just lies where it fell from mining, before & after save/exit/load
* Hide: Stone is left as above, but hidden from view using DF's internal option, before & after save/exit/load
* Stockpile: Stone is carried by dwarves into large stone stockpiles, before & after save/exit/load
* Block: Stone is processed by masons into blocks, and stored by dwarves in bins in stockpiles, before & after save/exit/load
* Mug: Stone is processed by crafters into mugs, and stored by dwarves in bins in stockpiles, before & after save/exit/load
* Atom-smash: Stone is carried by dwarves to the atom-smasher, and smashed out of immediately obvious dwarf reality, before & after save/exit/load
* Pre-trade: Stone is set up to be traded away (probably pre-staged in stockpiles near the trading post), before & after save/exit/load
* Hack Direct: Stone is removed directly with hack tools, before & after save/exit/load
* Hack Temp: Stone is changed to something which boils away at room temperature, preferably with a minimum of Fun, before & after save/exit/load

Then once the traders arrive, run some additional copy cases:
* Trade away as much raw stone as you can, let traders leave map, before & after save/exit/load
* Trade away same number of blocks, let traders leave map, before & after save/exit/load
* Trade away same number of mugs, let traders leave map, before & after save/exit/load

There's no way I'm going to have time to run cases like this in the next few weeks, but the above set should tell us a lot about not only how DF handles storage "normally", but how it interacts with the hack tool's and some exploits fiddling around with the guts of data structures without calling methods properly.  Ideally, you'd take another set of measurements after a season change; there's some possible evidence that there is data cleanup run on the season change already. 

[peterix]

IIRC there was some strong indications in previous versions that the memory hacks used by DFhack, etc. had some serious hidden limitations in that they effectively sidestepped various optimization and/or garbage collection (in the computer sense) triggers. It was well documented that hacking the removal of magma from a tile did *not* trigger whatever internal routines fix up temperature propagation / cleanup; and that a tile that had magama hack-removed could still burn many seasons later.  As I understand it Toady's code seems to have some fairly significant "lazy update" processing to improve performance, and using hacks can trigger weird side effects connected to not running cleanup routines. 

DFHack for the most part doesn't call DF's own code (this is possible for virtual methods, but those are a tiny minority in the sea of code that is DF). Most of the time, all that is possible is to access the data. And then the burden of keeping things consistent is on DFHack and the tools built on top of it. In case of magma, the temperatures of the map tiles weren't updated, leading to heat traps. Adding sufficient amount of hackery, this was 'fixed'. The temperatures still aen't entirely consistent with what they would be if there was no magma there in the first place, but your dwarves won't melt...

In general, it's better to rely on the game's mechanics for consistency. Problems with spawned magma temperature? Place it one z-level above and let it fall. Do you want to remove magma? Don't remove it entirely and leave 1/7 magma instead - it will 'evaporate' and your temperatures will be consistent. It's all about understanding the game enough to not break things.

Removing contaminants is safe, because they are dumb objects not referenced by anything else than their container. Removing items the same way is asking for trouble. DF's code base is big, there's still plenty of stuff not understood and removing some random objects can lead to undefined behavior. Just to clarify, I'm not saying that you shouldn't try and see what's actually happening, but that if you decide to write a tool that does that, you should be pretty sure it's safe before you give it to people to use on their forts.

[Halconnen]

I wish I remembered now where I saw it, but I've seen screenshots of stonesense displaying phantom animals, snatched children and other units standing around on the mapedge. The game doesn't show them, but they are in memory somehow.

If those phantom units still cause some recurring checks that could probably be a reason for fps loss as well.

[arphen]

Removing items the same way is asking for trouble. DF's code base is big, there's still plenty of stuff not understood and removing some random objects can lead to undefined behavior.

Are you saying autodump destroy is not as good as atomsmashing?
Thanks for dfhack btw

[NW_Kohaku]

Right. Let's get some actual info here, K?

DF has many global vectors for items (one has everything, then there's one for every item type). See: https://github.com/peterix/df-structures/blob/master/df.world.xml#L297 I don't know it off the top of my head, but I think they are ordered by item ID to allow binary search.

To speed up things like drawing, every map block (map is divided into blocks of 16x16x1 tiles, organized in a 3d array) has an ID vector of items: https://github.com/peterix/df-structures/blob/master/df.map.xml#L149
The game uses IDs stored in the blocks to search through the big all items vector (it's really a vector of pointers).

I'm rather surprised.  I've never really used DF Hack, and thought it was purely about memory hex editing.  It looks like you've actually reverse-engineered the code, here.

Feel free to set up a dfhack dev environment and start poking around further :-) For example, a nasty source of FPS loss could be if the ID vectors in map blocks never got emptied properly.

Anyway, yes, that's the exact thing I'm zeroing in on - vectors never being emptied. (I'm just trying to use "normal people speak" when describing it.)

I've already set up a test fort (without the use of DF Hack, just because I've never really used it, and feel it would take more time to learn than doing a test fort).  It's going through the process of creating and destroying millions of items through reactions right now.  It's just... it takes a while to get several years of sample data

I have a "Control" set up, with those same dwarves set to infinitely repeat a "Do Nothing" task at the workshops starting at the fork point.

I can also create an atom-smasher, and try working those from the fork point, as well as possibly getting DFHack in here, and doing another test run with generating large numbers of objects, then running DF Hack to annihilate all the testrocks I'm generating occasionally.

[Kogut]

Well, if we want to specifically stress-test the specific notion that objects that no longer exist are the specific cause of lag, then what we need is a situation where dwarves are creating a very large number of objects which are then destroyed.

The easiest way to do this would be to generate custom reactions that generate and then destroy large numbers of objects at once, while at the same time turning sieges off, and setting dwarves to NO_EAT, NO_SLEEP, NO_DRINK.  "Operation FPS Bomb" would be all about doing as little as possible besides generating and then subsequently destroying large volumes of items. 

A control fortress could be set up by starting a new fortress, building the custom workshops, and setting up the reactions to start taking place.  Then, saving, copying the save, and then running a "Control" save where the dwarves basically do nothing all day, and a "Test" save, where the dwarves repeatedly create and then annihilate materials. 

Provided my educated guess is correct, this should give some corroborative evidence as to how Toady has coded the game. 

... Of course, again, Toady could simply look at it, and declare my guess inaccurate fairly easily, as well, so I'd rather take the one that's faster if I could help it.

I am trying to reproduce using your method. I disabled everything (what resulted in "Enough time to delve secure logings, ere the beasts get hungry"). I noticed two giant drops: 20000 FPS to 3000 FPS (I want this in normal game!) and 3000 FPS to 500 FPS.  It may be related to bogus 268000 FPS during pause. So it should be possible to replicate.

[NW_Kohaku]

I am trying to reproduce using your method. I disabled everything (what resulted in "Enough time to delve secure logings, ere the beasts get hungry"). I noticed two giant drops: 20000 FPS to 3000 FPS (I want this in normal game!) and 3000 FPS to 500 FPS.  It may be related to bogus 268000 FPS during pause. So it should be possible to replicate.

Yeah, I've not been able to get my FPS back down below 100 (had to crank up the max), and I forgot to eliminate caverns, and didn't bother eliminating all other creatures from the raws, so there are some oscillations that are based purely upon random critters moving around in the caverns that are going to add some noise to my data. 

I've been working with 16 dwarves (that's all I got in my first year in migrants, and I set up my fork point at the turn of the year) infinitely creating and destroying 20 stones with reactions forever. 

I'm thinking maybe I should have made that 200 stones or something.  No point in doing this sort of thing half-way, the more dramatic the results, the better, when there is noise in the data.

[Kogut]

Ops, I need to disable caverns, demons, troll, vampires etc.

[o_O[WTFace]]

Keep in mind craft like objects may have creator, quality, age and wear information, while stones are mostly inert.  I recommend using something more complex then just rocks, especially with the new economy arc stuff that has just been added. 

[NW_Kohaku]

Keep in mind craft like objects may have creator, quality, age and wear information, while stones are mostly inert.  I recommend using something more complex then just rocks, especially with the new economy arc stuff that has just been added.

... OK, I'll try that in another experimental run, then.

I'm already most of the way through my base experimental run, although I had to restart to try again with a larger number of rocks being created and destroyed, and because I ran into some unrelated bugs that were mucking up the results.

[NW_Kohaku]

Putting my results in this thread, but the short of it is that I have yet to replicate the conditions of gradual FPS decay.

[Darekun]

You could simply have a pair of pointers, one starting at the front of the list, and stopping when it hits a blank space that needs to be filled, and the other starting at the end of the list, working backwards, and stopping at and transferring any memory field that has data (and garbage collecting completely emptied arrays).

Or even just once per frame, find the first blank space, swap the last ref from the next array, and garbage-collect empty arrays. Even at ~20 FPS it should cull a 1k-pointer array from an old save once per minute or so, enough to keep up with new demands long-term? This kind of ongoing process can work wonders :J

(Or keep it head-tail doubly-linked, and instead of blanking a space in the middle, copy in the last ref of all and blank its space. Then empty arrays don't matter much, since they're necessarily next up to get filled.)

Also, if you have a decent system are you affected? my machine is getting on 3 years old, but i don't notice any fps issues.

Personally, I suspect it's more that those with decent/good systems aren't affected enough to complain. I have noticed that by the third game year or so my FPS routinely gets below [FPS_CAP:100], and in one "long-runner"(as Dare goes, eheh) it got down into the 80s even with nothing going on. That's not complaint-worthy.

[Talanic]

My computer is well over three years old and is starting to exhibit symptoms of hard disk death; I'm quite aware that many with better systems may not see any of these problems.

Also, rebooting (which I am loath to do, as it takes over a half hour to restart due to aforementioned disk death symptoms; I'm also unfortunate enough that it's running Vista) did help, which surprised me.  I didn't go back to 150 range, but was up a few notches for sure.