Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[NW_Kohaku]

I am testing atom-smashing and autodump destroy separately because I specifically don't know whether they have the same effect or not.

In fact, I believe I have discovered permanent FPS decay in this Atom-smashing test, as I am now permanently at 80 FPS. 

This implies that atom-smashing is not properly cleaning up what it smashes. 

This only significantly occurred after the one point where I had 30,000 goblets on the map, however, and ever since that point, even after I smashed away the glut of goblets keeping me at 60 FPS, I still have an FPS that only recovered to around 80.

Either this means the problem is that atom-smashing isn't as clean as previously thought, or that the problem is not so much individual items entering and exiting the vector, but the overall size of the vector itself never recovers. 

[bombzero]

so maybe its like uninstalling programs in that you leave lots of 'blank' data behind? insignificant in a single instance but cumulative over time?

[NonconsensualSurgery]

Conclusions so far:

1. Ignoring traders causes permanent FPS drop.

-what if they're never allowed on the map with raised drawbridges?

2. Having many many items and then atomsmashing them causes permanent FPS drop.

-what about magma disposal? I have a feeling that dropping 30,000 goblets down a volcano or flooding the chamber with magma will do something.

3. Trying to limit dwarf pathing with raised drawbridges or doors just makes it worse.

-what about constructed walls, obsidian or cave-ins blocking pathing?

The kitten crushing test should have predictable results because we already know dead units aren't cleaned up completely. We can still see them on the units screen years later. Just how many you have to kill to have a distinguishable effect is unknown; having a value of kittens atomized per FPS lost would be... interesting.

[kaijyuu]

I can confirm that destroying 95% of the items on a 60,000+ item map restores quite a bit of FPS. Went from ~50 to ~140 on one of my old forts. So, destroying items isn't pointless.

Did it with magma dumping, which may be different than atom smashing, though.

[NW_Kohaku]

Conclusions so far:

1. Ignoring traders causes permanent FPS drop.

-what if they're never allowed on the map with raised drawbridges?

2. Having many many items and then atomsmashing them causes permanent FPS drop.

-what about magma disposal? I have a feeling that dropping 30,000 goblets down a volcano or flooding the chamber with magma will do something.

3. Trying to limit dwarf pathing with raised drawbridges or doors just makes it worse.

-what about constructed walls, obsidian or cave-ins blocking pathing?

1 is not true, per se.  Blocking traders causes temporary FPS drop for as long as they are blocked. 

Being as creatures can exist off the borders of the map, and may exist there permanently, and take up FPS for being there, blocking them off the edges of the map may cause more problems than it solves.

2 is still under testing.  My FPS has climbed back up to 120 while smashing is going on, but that's because my goblet total dropped to 8000.  It may be the effects are more subtle than I at-first thought.

I have not done anything to support conclusion 3.  There is a very notable lag every time the drawbridge goes up and down.  My FPS goes up 50% whenever my lever-puller goes on break.  So, that's going from 80 to 120 from that alone. 

I have completely walled in the drawbridge, so there is nothing that can path to the drawbridge unless you are flying in the dumping pit.  It doesn't matter, it's still causing lag. 

I presume this means that every time the bridge raises or lowers, it has to recalculate the entire connectivity chart of the whole map, regardless of whether or not things like caverns have never been breached.  There have been talks before about saving connectivity charts with bridges up and down before to save processor time, and this seems to support it.

so maybe its like uninstalling programs in that you leave lots of 'blank' data behind? insignificant in a single instance but cumulative over time?

I think it might be more to do with the arrays in the vector being generated, but never sorted - merely eliminated when the whole array becomes empty, but when, if a single goblet's pointer remains in that array, the array remains behind. 

[NW_Kohaku]

I can confirm that destroying 95% of the items on a 60,000+ item map restores quite a bit of FPS. Went from ~50 to ~140 on one of my old forts. So, destroying items isn't pointless.

Did it with magma dumping, which may be different than atom smashing, though.

Magma dumping is acutally more likely to be problematic than atom-smashing.  Magma dumping leaves behind melted stone items that never quite go all the way away.

The thing is, an item that is improperly cleaned up is better than an item that is still there, but you shouldn't compare the FPS to when the object was there, you should compare the FPS to if the object had never existed at all.  If there is a full cleanup, then I should be able to create and destroy millions of items forever without any loss of FPS, but if there is a lingering footprint in memory, then even though destroying items grants a temporary memory boost, it creates a gradual decay of the data structure that leads to lag even when the objects no longer exist.

[khearn]

Have you looked at dfhack and runesmith? They do a lot of things involving the internal datastructures, including deleting things from them. You might learn more about how things are structured by looking at their code or talking to their developers. You may find that the structures you are guessing about are already well known.

[NW_Kohaku]

I am suspending my atom-smashing test for now to check on the contraction theory, instead.

I am going to try to have a million stones on my map at a single point in time before eliminating them all.

I am apparently up to roughly 100,000 stones on the map at once, but it appears as though the sheer numbers have completely overwhelmed my bookkeeper, or else that bug is back, because I'm down to lowest possible precision again. 

FPS has plummeted way down to 30 from "merely" having 100,000 stones on the map.

[blue sam3]

Right, I've done some testing on open space. All testing was done with modded (nosleep/drink/eat, speed 0) dwarves digging out areas of white sand, in full screen, with as little as possible running in the background.

Results:

initial FPS: 2000
FPS whilst digging out a small area (600 ish tiles): 50
FPS after digging out said small area: 1500
FPS whilst digging out a larger area (maybe 1500 tiles or so): 5
FPS after digging out said larger area: 1500
NOTE: at this point, several things happened:
1) First hardcoded migrant wave arrived and a ghost appeared
2) I finally remembered to turn weather etc. off
FPS after above changes: 2600
FPS whilst digging out another small area: 3
FPS after digging out said small area: 2300 (NOTE: unreliable, as I was only able to observe for a short time due to another migrant wave and a caravan arriving. After the caravan left, FPS was down at 2100)
FPS whilst digging out 7 tiles: 5
FPS whilst digging out another small area: 20
FPS after digging out said small area: 2100
FPS with an entire inaccessible z level designated: 1950
FPS after undesignating said inaccessible z level: 2050
NOTE: At this point, two migrant waves arrived, half the fortress got married, a child was born the elven caravan arrived and two ghosts appeared.
FPS after above note: 1000
NOTE: the above changed by +/- 100 points with seasonal variations, away from autosaves
FPS whilst digging out another large area: 2
FPS after digging out said large area: 900
FPS after digging out another small area and a really annoying migrant wave: 730
FPS after digging out another small area, a really annoying dwarven caravan (still present) and three really annoying births: 600
FPS after caravan left and digging out another small area: 600
FPS after digging out another small area: 600

At this point, I decided to test whether open space and floored space had the same effects, so I started channelling out all of the areas I'd previously dug out.

FPS after collapsing first area (two deaths, one birth, one lightly mangled baby, two lightly injured miners): 420
FPS after collapsing second area (one light injury, one birth): 380
FPS after collapsing ridiculously massvie area: 300
FPS after digging out another small area: 280
FPS after channelling that area out the slow way:230


Conclusions:
Empty space slows stuff up.
The accessibility of space has no noticeable effect.
Channelled out space is actually worse than floored over space.
Designating things does slow stuff down, but nothing like as much as actually having them accessible.
Weather etc. has a significant impact on FPS.
There are significant seasonal variations in FPS.
Caravans still slow things up after leaving the map.

[NW_Kohaku]

I don't think I'll be able to get to 1,000,000 objects, as the game is getting extremely laggy just trying to look at some of these workshops that have 10,000 objects in them, apiece. 

Apparently, just looking at a workshop will cause lag based upon the number of items it can be related to.

This includes the workshops I didn't even use in the experiment for lack of any worker to staff them - apparently, it's loading all the boulders in the entire fortress any time I look at the workshop.

Looking at the mason's workshop and butcher's shop causes no lag.

[NW_Kohaku]

Creating 1,000,000 simultaneous objects is harder than it seems.  Apparently, even DF has some sanity checks, and it doesn't let you create 100,000 objects in a single reaction. 

So, I'm up to creating 500,000 for the stress test, and waiting on the other reactions to finish creating 1,000,000. 

FPS has gone from 600 to 6.  It will probably be 3 by the time I hit 1,000,000 boulders.

[Girlinhat]

I think it's been confirmed that bridges/hatches/doors/etc cause an instant FPS drop, but when pathing settles down it returns to normal.  That is, there's a moment of lag when the dwarves realize "oh shit, something has changed.  Let me figure out what I'm capable of now!"  And once they realize where they can path, FPS returns to normal.

Also does "magma dumping" refer to "Into the magma sea and SMR" or "magma was placed atop it"?  SMR may trigger clean deletion, while atom smashing causes a messy deletion.

Either way, it seems like Toady could easily implement a few lines of code that will perform a loop every time you save.  Run through the array, "if defunct, remove this entry".  If it runs on every save, then seasonal autosave would keep FPS pretty damn high.

[wierd]

That might work, but could also lead to save file corruption more frequently. (The last thing you want is a race condition, or an overflow causing problems in garbage collection just before you say "ok, set everything in stone now!")

Seasonal garbage collection without saves, say when merchants are supposed to show up would be better, since the game autopauses (most of the time...) when they do, leaving the cleanup loop free to do its thing without stuff happening, and where if it crashes\smashes, it won't take the whole world with it.

[NW_Kohaku]

My contraction test has caused the entire game to go completely unresponsive.  I am unsure why.

It might be that the game just completely blows its top when you hit 1,000,000 boulders, or it may be an aspect of the custom reactions I had to made to trick the system into doing more with one reaction than it normally can (destroying 100 sets of 100 items at a time).

Performing further testing...

A quick test shows that the boulder destruction reaction causes massive lag.  The 100 step process just shut my game down for 4 straight seconds only doing it once...

Since I will need to perform this reaction over 100 times, I suppose that's the cause of the non-responsiveness.

[Kogut]

In my current test (create object, destroy object with [SPEED:0] dwarves) I see a constant FPS drop, from 4500 to 3000. I will try to reproduce it again (without changes to RAWs during game, with disabled internet and antivirus etc), maybe we will see proper bug report.