Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Psieye]

Anyway, Psieye, simply having more objects taking up more memory or causing lag isn't really the point - I've proven that a long time ago.

Agreed, but cross-checking results is the basic tenet of science and I never said this plant fort's experiment is over. In doing this my way, I've come across another potential lead - when people build with FPS-longevity in mind, they tightly control pathing and try to minimise items. But never have I heard anyone say they've minimised the number of stockpiles they have, or the number of workshops they have. I might decide to try out a new !!SCIENCE!! fort just to test out stockpiles and workshops, but first I'm going to try destroying these megashrooms in various ways, with and without DFhack.

So on that note, further test results:

=== Methodology ===

Install DFhack.

Take the 100k megashroom save and test various ways of destroying the shrooms and wine pots with DFhack. Also see effect of removing all the stockpile designations.

Then take the 30k megashroom save and compare manually atomsmashing all the megashrooms and wine pots to using DFhack to teleport the items into the atomsmasher.


=== Results ===

**100k save**

- Start: I decided not to wait for FPS to crawl ever so slowly up from 2 to 66 like last time. After confirming it was climbing back up, I went to the next step at 50 FPS.
memory = 55.6 MB

- After setting every megashroom for dump with no garbage zones: FPS bombed down to 2 again, but did recover all the way back to 66. Also, it is now summer not spring but this doesn't have any meaningful effect on this fort's FPS.
memory = 56.1 MB

- After also setting every wine pot for dump through the stock screen (didn't take as long to load this time, maybe because liquid stacks in pots are treated differently): FPS bombed down to 1 this time, but again crawls back up. Didn't wait until it fully recovered before proceeding to next step.
memory = 56.0 MB

> DFhack's "autodump destroy" command: FPS shoots up to 1300+ and then gradually decreases to 950 +/- 50, similar to (but faster than) how FPS slowly increased after bombing to 2 on load or mass designations before.
memory = 57.5 MB

> DFhack's "autodump" command to move everything to one location, then atomsmash: 1230 +/- 10 after the FPS settled down from the initial spike.
memory = 60.1 MB

> Remove all stockpiles, but don't destroy any items: FPS settled down and then plateaued up to 330 before the megashrooms wilted. After wilting, FPS was around 470 before they disappeared into nothingness. FPS then went to 740-ish.
memory = 57.6MB


** 30k save **

- After mass dump designations of megashrooms and all pots through stock menu: FPS = 389 +/- 2
memory = 47.7MB

> DFhack's "autodump" command, then atomsmash: FPS is settling down to 1040 +/- 10
memory = 48.3MB

> Manually dump and then atomsmash after all dumping: FPS is around 200 while the dwarves are doing the hauling. Most of the megashrooms are wilting and decaying away before the atomsmash lever is pulled, as I only want to pull this thing once instead of putting it on repeat. FPS after the lever got pulled = 1170 +/- 5
memory = 47.8MB

=== Conclusions ===

The FPS of the 100k save only rose up very, very slowly if nothing at all changed while waiting for measurements. Any change, e.g. an item set to dump or a stockpile made/removed, will ruin the 'status quo' and the FPS will bomb right back down to 2 while the game tries to find a new equilibrium.

Job queues likely only affect FPS when they're being changed. Needs a better experiment to investigate.

For FPS-longevity: manually dump & atomsmashing > autodump & atomsmashing > "autodump destroy"
However, autodump leaves more of a memory footprint than "autodump destroy".

Stockpiles take up FPS just by existing, whether they are filled or not and regardless of if there is no hauling job they generate. With lots and lots of stockpiles, this is a considerable FPS burden.


That's all I have time for today. Feel free to take my saves from the previous post and try atomsmashing as the dumping is going on.

[NW_Kohaku]

I was about to call the vaporization project to a close after only three years (because I can vaporize several million stones per year), but then I was checking results, and noticed a small increase in the amount of memory being consumed by the game, so I think I'll leave it running for another five or so years, and seeing if I can get something substantial enough to call a result.

[ratchet freak]

are you also going to check owned items (claimed scattered clothes) or are you going to wait until .06 when we a speedup there

[NW_Kohaku]

are you also going to check owned items (claimed scattered clothes) or are you going to wait until .06 when we a speedup there

Probably wait.  There's no point in checking something being fixed right now for if it is broken.

I will try testing clothing in general, however.  I don't think Toady is going to deal directly with that in his changes.

[Frogwarrior]

Do you have some sort of "summary" post in this thread, where you say in brief form the results of your experiments, i.e. "this definitely does cause FPS drop, this does not" sort of thing?

[NW_Kohaku]

Do you have some sort of "summary" post in this thread, where you say in brief form the results of your experiments, i.e. "this definitely does cause FPS drop, this does not" sort of thing?

No, although I will when I get more definitive results.

[Draxxalon]

In reference to memory usage post-smashing (via any method), you may want to also do a save + reload.

It's quite possible that the memory footprint doesnt decrease substantially while running, but on a reload, only allocates enough for the items that are presently in existance. 

It's also quite possible that there are cleanup routines built into the save and/or load process, which could affect your results.

[Elone]

Minecraft.

Do you remember those times when you could hit F3 to see mob IDs? It showed their internal ID numbers abovehead through walls and everywhere (and was removed cause it's too much a form of a wallhack).

I dont know whether these IDs were shared between all entities or just mobs in MC. I do know however that the game always spawns mobs (even if they are ineligible, then they get deleted on the next frame). The numbers often get into 6-7 digits as I keep playing. I'll take a guess that particles might be handled in the same unsorted ID manner.

I stand next to a sheep with a ~ 7 digit ID. I save and close the game (actually just unload the map and go to the main menu) and then I reload. The sheep in front of me now has a 1-2 digit ID number.

Indeed these unsorted list could be creating memory leaks, but once the game is saved and afterwards read from the save file, all the vars are just packed as they are read, they are not returned to their old addresses. The solution would be to restart the game every once in a while (both MC and DF) which resets the memory leak. Note that they could somewhat easily be resorted during gameplay, but that would cause periodic lag spikes which may be even more undesirable to the player than a memory leak that builds up over many hours before it's an issue.

What I would like to see are some results where you reload the game and see how the FPS was affected, for example "400fps before test, 200fps after the test, but 350 fps after reload" which would confirm that the drop is, actually, permanent.

What happens when you deleet stuff FROM certain dat files (like unit dats), leaving them blank?

EDIT: Scienced. First I tried deleting data from a single random (well biggest) unit dat file. The game ran without a crash. Then I deleted ALL the unit files from the folder, humming innocently all along, and surprisingly the game did not seem to mind! All the units and dorfs were just where they were before. What gives? I got a minor boost (from about 89fps to 93fps, from the same save point) but it might be worth looking into this. None of my forts has the clothing modded out so that's an undesirable factor right there. Thoughts?

[RadHazard]

I'm not sure as I haven't really played with it, but I believe the unit.dat files only store non-local historical figures.  If they're missing, when the game tries to load a historic figure (such as a migrant or trader) from one of the missing files it will give you the infamous "Nemesis Unit Load Error".

I have had one of those errors in 0.34.XX.  I accidentally opened up a second Dwarf Fortress with the same executable in the same game.  I believe the problem was that the second copy of DF cleared out the "current" folder, which stores any unit.dat files that have changed since you started that game.  The game ran fine at first, but after I fast traveled to a town that I had visited previously, it crashed with that error.  I'm assuming the same principle applies to fort mode.

[Psieye]

Let's try the reload check with the quicker destruction methods on the 45k megashroom save:

- All megashrooms (but no wine pots) designated for dump but dwarves still walled in:
FPS = 215 +/- 5
memory = 48.9 MB

> "Autodump destroy"
FPS = 960 +/- 10 (takes a long time to stabilise after the spike up, this might not be the lowest it is)
memory = 49.6 MB

> "Autodump destroy" & save/reload
FPS = 760 +/- 10 (FPS slowly increases before plateauing off - this might not be the highest it can be)
memory = 47.9 MB

> "Autodump destroy", save, QUIT, load
FPS = 960 +/- 10
memory = 46.3 MB


> "Autodump" atomsmash
FPS = 980 +/- 10
memory = 50.6 MB

> "Autodump" atomsmash, save/reload
FPS = 770 +/- 10
memory = 49.3 MB

> "Autodump" atomsmash, save/QUIT/load
FPS = 970 +/- 10
memory = 45.7 MB


> Manual dump & atomsmash, save/QUIT/load
FPS = 960 +/- 10
memory = 46.0 MB (a whole year went by while getting this dumped so extra memory being used is not surprising)

Ok, so we can conclude that there is some sort of memory cleanup on a save/reload and especially on a save/QUIT/load cycle. In this test fort though, there was no FPS gain as a consequence (stockpiles were still around though). In fact, failing to quit before loading again led to an FPS decrease that was recovered by quitting. In much larger forts, I would think that a save/quit/load cycle will help recover some FPS.

[Elone]

Thanks, Hazard! That makes sense. So there is no easy way to deal with people outside the map? What about items? Kohaku's post said that the stuff that caravans bring (but you dont buy) is deleted when the caravan clears the map, but that the items that you sell to them actually add to one of the files. Could those entries, wherever they were, be wiped without problems?

Thanks Psieye! Just what had me curious! This is a great insight into the powers of atomsmashing, autodumping, and also the arbitrarily expanding arrays! And you laid it out in a very readable manner. Atomsmash and destroy seem very comparable, meaning that both remove the items in the same way.

I'm kinda puzzled that you actually do not lose those ~200 fps of yours as long as you keep playing... If the arrays were growing as was suggested earlier, then you would be feeling a slowdown just from those fragmented arrays? Unless there's that bit where you do manage to delete everything in the array, after which the whole array goes.

Spoiler:  memory from my experience (click to show/hide)

From my understanding, apps cannot easily clean up their memory. At the very best, they can mark certain RAM areas as overwritable (by removing/invalidating the old pointers), and they are filled with new data as it becomes needed. Take Firefox for example, at the very moment I have 24 tabs on top (which is tame, i went as high as 140 tabs and they used to reload for 45 minutes due to all the processing, swapping, and transfer). They are set to load when selected, but when I start the Firefox, only one tab will be autoloaded and the RAM usage will be minimal. As I select more tabs, the RAM usage grows. So when I get an arbitrary feeling that Firefox has taken up enough, and I finished browsing with some tabs, I close Firefox and let it reload from the minimum RAM and full performance, the only downside being the 10 seconds that it took to restart.

That aside, I remember reading Toady's fairly recent market post which mentions something about city only tracking iterative changes, and the changes pile up until that area is saved to disk. Data becomes more volatile, but it saves up on disk writes a lot; so I am guessing that this could have been implemented in other cases as well. Did I dream this up or was this really said, tho...

[NW_Kohaku]

Thanks, Hazard! That makes sense. So there is no easy way to deal with people outside the map? What about items? Kohaku's post said that the stuff that caravans bring (but you dont buy) is deleted when the caravan clears the map, but that the items that you sell to them actually add to one of the files. Could those entries, wherever they were, be wiped without problems?

The "cure" to the nemesis load fail problem was that you could take the data from an older save, and copy that over into your save folder, except for the "world.sav" stuff, which was your actual fortress.  This also had the effect of any historical figures you had killed suddenly never-being-dead again.  Global retcon of sorts.

If you wanted to reset your trading status, overwriting that unit-1.txt or whatever with data from old saves should also reset your caravan status without risking the game exploding.

That isn't to say I guarantee you won't run into problems at all - it might make the next caravan believe you never traded with the mountainhomes or something.

From my understanding, apps cannot easily clean up their memory. At the very best, they can mark certain RAM areas as overwritable (by removing/invalidating the old pointers), and they are filled with new data as it becomes needed. Take Firefox for example, at the very moment I have 24 tabs on top (which is tame, i went as high as 140 tabs and they used to reload for 45 minutes due to all the processing, swapping, and transfer). They are set to load when selected, but when I start the Firefox, only one tab will be autoloaded and the RAM usage will be minimal. As I select more tabs, the RAM usage grows. So when I get an arbitrary feeling that Firefox has taken up enough, and I finished browsing with some tabs, I close Firefox and let it reload from the minimum RAM and full performance, the only downside being the 10 seconds that it took to restart.

It seems Toady is more careful about memory management than many webpages.  I don't believe this is strictly Firefox's fault, as I believe part of the problem is that many websites don't always play by firefox's rules, although I know that Chrome is supposed to specifically run each tab as a separate thread, so that closing a tab completely reclaims all memory attached to it.  (I haven't used Chrome, however, so I'm not talking about this first-hand.) 

There is an advantage in being a sole coder with the awareness that you are going to be creating and destroying a huge number of items from vectors in this regard.  Thus far, the data structures have been surprisingly robust, and handled much of what I can throw at them, with a few very notable exceptions, although these are not strictly unexpected exceptions.  (Although the "pressing t causes the game to chug for 20 minutes" thing was unexpected.) 

I suspect that the real culprits in this FPS decay test will turn out to be some of the things we already suspected just being worse than we thought - dead creatures, decaying clothes, invasions, fluid flow, general item count, and pathfinding. 

[arphen]

i'm pretty sure it runs in different processes, not threads.
keep up the good work, but tell me... do i dfhack autodestroy or melt or ... your last data shows no difference between the methods , correct?

[NW_Kohaku]

i'm pretty sure it runs in different processes, not threads.
keep up the good work, but tell me... do i dfhack autodestroy or melt or ... your last data shows no difference between the methods , correct?

I haven't tested dfhack's tools yet, so I don't know there.  It should probably be safe, though, given what I've heard from the developer of that tool. 

Atomsmashing should definitely be safe, as should custom reactions that destroy objects with no products, and I'm still running tests on vaporizing, but they should be safe if you do that, but just melting (instead of vaporizing) leaves items in memory as a lump of melted goo.

[Nil Eyeglazed]

I haven't tested dfhack's tools yet, so I don't know there.  It should probably be safe, though, given what I've heard from the developer of that tool. 

I've had DF crashes with autodump-- suggests that there are things DF is doing with dumping/destroying items that aren't happening with autodump.  Think I'd stay away from it for framerate testing purposes.