Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 5 6 [7] 8 9 10

Author Topic: Lag discussion, how to address it and be generally helpful to Toady/Threetoe  (Read 18125 times)

Rysith

  • Bay Watcher
    • View Profile

Just reporting in to report progress on some of the tests that I proposed, since I've started running them:

Starting with tests #1 and 2, embarking on a 2x2 flat embark (heavily forested freshwater swamp) got me 40 FPS in 40d9. Mining out the sand level dropped my FPS to 30, and as mining continued (as per test #2) FPS was observed to drop as low as 13. I've also only managed to get around halfway through the clearing without DF crashing, which I'll check to see if it's reproducible and file a bug report for. I *suspect* that it has something to do with passing 2^16 stones, but I'll want to check that further given that Flarechannel is using 235,000 blocks without issues.

Since there was no additional pathing load compared to the initial conditions, I'm forced to conclude that significant numbers of items have some possible effect on FPS, but the FPS drop during the sand level clearing (which generated no items) indicate that something else is going on. Perhaps some function of revealed tiles? I'll add test 2a to try to isolate that effect.

2a: Generate several thousand items using smelter reactions to isolate the sand issue.
Logged
Lanternwebs: a community fort
Try my orc mod!
The OP deserves the violent Dwarven equivalent of the Nobel Peace Prize.

darkflagrance

  • Bay Watcher
  • Carry on, carry on
    • View Profile

Just reporting in to report progress on some of the tests that I proposed, since I've started running them:

Starting with tests #1 and 2, embarking on a 2x2 flat embark (heavily forested freshwater swamp) got me 40 FPS in 40d9. Mining out the sand level dropped my FPS to 30, and as mining continued (as per test #2) FPS was observed to drop as low as 13. I've also only managed to get around halfway through the clearing without DF crashing, which I'll check to see if it's reproducible and file a bug report for. I *suspect* that it has something to do with passing 2^16 stones, but I'll want to check that further given that Flarechannel is using 235,000 blocks without issues.

Since there was no additional pathing load compared to the initial conditions, I'm forced to conclude that significant numbers of items have some possible effect on FPS, but the FPS drop during the sand level clearing (which generated no items) indicate that something else is going on. Perhaps some function of revealed tiles? I'll add test 2a to try to isolate that effect.

2a: Generate several thousand items using smelter reactions to isolate the sand issue.

I have heard from other sources that revealed tiles do cause fps drop, so I'm sure it's worth testing.
Logged
...as if nothing really matters...
   
The Legend of Tholtig Cryptbrain: 8000 dead elves and a cyclops

Tired of going decades without goblin sieges? Try The Fortress Defense Mod

o_O[WTFace]

  • Bay Watcher
    • View Profile

Isn't it the walkable space that causes lag rather then the leftover stones?  In my experience no amount of atom smashing is going to bring your FPS back to what you had on embark, even if the number of creatures and things is held constant.  It seems reasonable to conclude that open space uses CPU that unmined space doesn't, presumably because of pathfinding. 
Logged
...likes Dwarf Fortresses for their terrifying features...

Nadaka

  • Bay Watcher
    • View Profile
    • http://www.nadaka.us

walkable space does increase lag, but so do the stones. You can test this by "ramping" just under the surface instead of standard tunneling. It will produce the stone, but not the walkable area.
Logged
Take me out to the black, tell them I ain't comin' back...
I don't care cause I'm still free, you can't take the sky from me...

I turned myself into a monster, to fight against the monsters of the world.

The Architect

  • Bay Watcher
  • Breeding supercows. What I've been doing on DF.
    • View Profile

Yea, smashing 40-50 pages of loose crap tends to generate an extra 2-5 FPS depending on other factors.
Logged
Dwarf Fortress: where blunders never cease.
The sigs topic:
Oh man, this is truly sigworthy...
Oh man. This is truly sig-worthy.

Rysith

  • Bay Watcher
    • View Profile

2a: Generate several thousand items using smelter reactions to isolate the sand issue.

Test 2a completed: on an embark similar to test 1/2 (7 dwarves, 2x2 flat swamp embark) with 40 FPS at embark, creating 24,000 brass bars (roughly 12 region tiles worth of rock) dropped FPS to 20. None of the bars were tasked to be moved, so it wasn't job-queue slowness. Thus, the number of items on the map does affect FPS significantly.

Extrapolating from Tests 1 and 2, though, it seems like the effect of items scales less-than-linearly: with roughly twice as many items in test #2, the FPS loss including potential mined-out or walkable area loss wasn't twice as bad. Thus, I'd suggest something like lookups into the list of items (happening constantly, and perhaps unnecessarily) causing the item-based FPS loss.
Logged
Lanternwebs: a community fort
Try my orc mod!
The OP deserves the violent Dwarven equivalent of the Nobel Peace Prize.

darkflagrance

  • Bay Watcher
  • Carry on, carry on
    • View Profile

From the Stonesense thread, it would seem that Dwarf Fortress updates items on the map for each frame.

As in every frame, Dwarf Fortress checks every item on the map.


Thus, the amount of items has a direct effect on the amount of time DF takes to generate the next frame.
« Last Edit: January 03, 2010, 01:42:09 am by darkflagrance »
Logged
...as if nothing really matters...
   
The Legend of Tholtig Cryptbrain: 8000 dead elves and a cyclops

Tired of going decades without goblin sieges? Try The Fortress Defense Mod

G-Flex

  • Bay Watcher
    • View Profile

From the Stonesense thread, it would seem that Dwarf Fortress updates items on the map for each frame. Thus, the amount of items has a direct effect on the amount of time DF takes to generate the next frame.

There's also whenever it might have to iterate over lists to select materials or generate jobs.
Logged
There are 2 types of people in the world: Those who understand hexadecimal, and those who don't.
Visit the #Bay12Games IRC channel on NewNet
== Human Renovation: My Deus Ex mod/fan patch (v1.30, updated 5/31/2012) ==

Sizik

  • Bay Watcher
    • View Profile

I wonder what would happen if you mine out a large area of stone, then filled it back in with walls made of the stone you mined out.
Logged
Skyscrapes, the Tower-Fortress, finally complete!
Skyscrapes 2, repelling the zombie horde!

Rysith

  • Bay Watcher
    • View Profile

Test 3: 5x5 flat embark: 40 FPS on embark at the surface, 50 below or above it.
  Conclusion: Large areas do not inherently cause slowness, though they seem like they would encourage more area mined out and more items created, as well as potentially longer pathfinding distances. Walkable tiles at the current level does seem to have an effect on things, though that may be because I don't have partial print on.

Test 4: 5x5 flat embark, mined out
  Modified in anticipation of the 65k stones limit
  FPS dropped to 15 immediately on designating the first (soil) level for mining, but returned to 30 when the digging had completed. It fell as low as 20 while mining out the second (also soil). It did, however, return to 30 once the second layer was dug out, seeming to indicate that the slowdown was at least partially from the constant pathing as tiles were dug.

  Excavating the first stone layer started with FPS around 25, and ended at 18 (with all dwarves idle), a significant drop from the previous 30. That makes me suspect that revealed tiles don't hinder speed as much as either items or pathing do, since in the two soil layers we returned to a constant 30 FPS when no pathing was happening.

  This generated 56400 stones and 248 gems. Marking them all for dumping (thus, generating pathing) lowered FPS sporadically as low as 8, and by far was worse while the dwarves were hauling stones to the dump, rather than when they were getting new ones. This was with 8 dwarves (a migrant showed up as I was mining)

So, items are a significant source of "permanent" slowness, and larger areas seem not to cause more slowness than small areas in and of themselves. Pathing while hauling (though not to haul, for reasons I'm unsure of) also seems to be a significant FPS drain.
Logged
Lanternwebs: a community fort
Try my orc mod!
The OP deserves the violent Dwarven equivalent of the Nobel Peace Prize.

The Architect

  • Bay Watcher
  • Breeding supercows. What I've been doing on DF.
    • View Profile

The problems with a large area are (as far as I understand them) resultant of several factors not including the footprint size of the area. In other words: the size of a flat empty embark should not significantly affect framerate.

The only case I know of where the size itself is the direct cause of the problem is when you have a huge z-level variance. The main causes of lag on a large map are the number of creatures, plants, and other terrain items (creatures:pathing/AI, plants: interactive, terrain:reiterated each frame). Of course there is also a larger potential for liquid flow and there are longer pathing distances.
Logged
Dwarf Fortress: where blunders never cease.
The sigs topic:
Oh man, this is truly sigworthy...
Oh man. This is truly sig-worthy.

HungryHobo

  • Bay Watcher
    • View Profile

I think I may be able to add something useful to this thread.

Even up to the last page I keep seeing posts talking about throwing in threadding.
Seriously, it introduces wierdness.
lots and lots of it.

People keep talking about threading as if it's something trivial... and if your task suits it and you set out to use threads from day one it can be ... not so bad.

But you can get bugs in a threaded program that would never crop up in non threaded ones.
Deadlocks are just the start of it.

Now I'm personally only familiar with threading in java, not so much in C but here's a simple trial for anyone who feels like seeing something interesting and has javac handy.

Create a program with 2 threads, nothing fancy, you should be able to find tutorials online that show you how.
Now create a variable.

int Counter = 0;

now have both your threads do the same thing.
Have each thread do a loop where it increments Counter 10,000,000 times.
No special threading tricks. just unguarded simple ++ .

For(int i=0;i<10000000;i++)
{
Counter++;
}
At the end of the program your programmer sense would tell you that Counter should be equal to 20,000,000.

It won't be equal to that unless the JVM has been changed a lot recently.

because strange things happen then 2 threads try to mess with the same data.

one thread starts to increment the data say from 10 to 11, looses its hold on the processor, the other thread increments it 10 times up to 20 and then the first thread gets back it's hold on the processor and finishes incrementing from 10 to 11 and 10 increments have disappeared.

Or if you have a game where the players location is kept as a pair of values [X,Y] and the player loses if they step on the trap at [0,0].
Say at some point in your maze you have the player move from [1,0] to [0,1] .
but you have 2 threads dealing with it, one thread testing if the player is at the end of the maze and other environment stuff and another thread for dealing with taking in user input and altering the players state.

1:the player starts to move from [1,0] to [0,1].
2:the "player" thread  changes the variable which holds the X position from 1 to 0.
3:the "player" thread loses control of the processor.
4:the "environment" thread tests to see if the player is on the trap... well X=0 and Y=0 so the player is on the trap.
5:game over.player gets dropped into the spike pit.

these are just trivial examples but its a big problem you have to deal with whenever threading is involved.

Now imagine you're trying to split processing in DF.
one thread starts to remove a burning object as it poofs out of existance.
another thread is pathing for a dwarf looking for that item.
The thread that's removing the item deallocates the memory used for the objects description and characteristics to make room in memory.
meanwhile pathing thread finds that item in the array, starts to read it's location, yes it's closest, goes to check it's characteristics to make sure it's suitable[BANG! seg fault! crash! game broken.]

trying to turn a non threadded game into a threadded game in anything but very very limited ways would be a nightmare.



« Last Edit: January 05, 2010, 10:25:39 pm by HungryHobo »
Logged

Nadaka

  • Bay Watcher
    • View Profile
    • http://www.nadaka.us

The incrementor operator in the java language specification is not guaranteed to be atomic, and it never has been.

You could to add Object lock = new Object(); near the top and replace Counter++; with synchronized(lock){Counter++;};

or You could define a synchronize method that calls Counter++;

Its a simple issue of concurrency and synchronization. Though in something as vast as DF, there is a good chance that even someone who knew what they were doing would have a hard time ensuring correct behavior.
« Last Edit: January 05, 2010, 10:34:10 pm by Nadaka »
Logged
Take me out to the black, tell them I ain't comin' back...
I don't care cause I'm still free, you can't take the sky from me...

I turned myself into a monster, to fight against the monsters of the world.

HungryHobo

  • Bay Watcher
    • View Profile

I was trying to make the point that simple code can break in complex ways when you get into threading.
Yes there are workarounds.
I know.
I also know there's a whole world of pain when you have to start dealing with deadlocks and other fun to do with locks, semaphores and all that crap.
« Last Edit: January 05, 2010, 10:52:42 pm by HungryHobo »
Logged

Riemann

  • Bay Watcher
    • View Profile

The incrementor operator in the java language specification is not guaranteed to be atomic, and it never has been.

You could to add Object lock = new Object(); near the top and replace Counter++; with synchronized(lock){Counter++;};

or You could define a synchronize method that calls Counter++;

Its a simple issue of concurrency and synchronization. Though in something as vast as DF, there is a good chance that even someone who knew what they were doing would have a hard time ensuring correct behavior.

And of course as soon as you start throwing around locks or mutexes or other forced synchronization you not only lose the benefits of multi-threading you end up with something slower and less efficient than a single threaded loop running twice (one after the other).

Even in ideal cases multithreading has overhead. The more often you have to synch between threads the bigger the overhead and the smaller the potential gain. There is certainly a point, and one that's easy to hit, where your overhead from locks and other synching mechanisms is greater than the gain.
Logged
Pages: 1 ... 5 6 [7] 8 9 10