Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[kaijyuu]

Why the hell does item count hinder FPS? I can understand them taking up a large amount of RAM, but there's not many calculations that would be necessary for each item.

[devek]

If you were a real programmer, you could take steps to answer these questions.

Or maybe I'm someone with a real job who doesn't have time to muck about with DFhack and just comes here during compiles. Who knows...

You only say DF hack because an above poster mentioned it. The point of dfhack is to hide the inner complexity of DF. The tool of choice to figure out what is going on is IDA.

NanosecondS in the plural. As in 700-950 of them. And I've yet to hear from you an actual argument as to why exactly it takes longer than that to do a kernel mode transition and have the scheduler set a flag.

....

But all of those costs (and they are big costs) I just mentioned are paid by those additional cores, not the one running DF that took <1 microsecond to lift a wait.

Well, damn. You should write a threading library. I wouldn't mind one that could make a million asynchronous calls a second to a single thread. That would be hard core and much better than this crap I got from Intel. 

[3]

Why the hell does item count hinder FPS? I can understand them taking up a large amount of RAM, but there's not many calculations that would be necessary for each item.

It's already been said. Every time a job regarding an item is called, every item is checked.

[Makaze2048]

You only say DF hack because an above poster mentioned it. The point of dfhack is to hide the inner complexity of DF. The tool of choice to figure out what is going on is IDA.

True. If one cares enough I'm sure IDA is a fine tool for discovering the inner workings of DF, if one cares enough.

Well, damn. You should write a threading library. I wouldn't mind one that could make a million asynchronous calls a second to a single thread. That would be hard core and much better than this crap I got from Intel.

Why would I bother? The one I've got seems to do pretty well, just benchmarked it doing 10M Sets on a manual reset event in 4.060s, automatic reset took a slightly longer 4.773s. Set and a separate Reset (in case repeated sets were being ignored) on a manual reset event took an expected around double 8.483s. Those are the initial runs but repeated runs all come in within a tenth of a second.

Now, if you have any benchmarks that you believe contradict those findings or can explain to me what exactly my primary thread has to do other than set the wait handle and get back to business while another core's current process's quantum expires and schedules the now signaled thread to run I'm all ears...

[kaijyuu]

Why the hell does item count hinder FPS? I can understand them taking up a large amount of RAM, but there's not many calculations that would be necessary for each item.

It's already been said. Every time a job regarding an item is called, every item is checked.

O.O

Okay, that's some quick and easy optimization right there. Rig the pathfinding to find the closest item. The dwarf can use the result from the same operation to walk there.

I can understand it checking everything for things like looking at workshops (needs to check total number of items), but not basic stuff like item creation.

[Ratbert_CP]

Heh...  I think the multi-threading conversation just clarified a bit in my head (given that I have little knowledge of parallel processing and threading).  It appears that one camp says that threading has a lot of overhead, the other saying it has almost none.

I think you're both right, just using different metrics.  The first camp is looking at all the work being done by the OS, scheduler, and whatnot on all cores in all threads as the overhead, the other camp is looking at the overhead in just the main thread.  That simple setting of a wait handle kicks off a certain amount of processing by the rest of the system to get the other thread to run, even if that call itself returns to the main thread very quickly.

I am, of course, likely to be completely wrong in my interpretation of things.

[tps12]

It's already been said. Every time a job regarding an item is called, every item is checked.

It's not job related, it's that they're checked every frame to see if they're decomposing or bursting into flame or whatever.

[Jelle]

Hiya

Just posting in this thread because I had to abandon my first succesful fort because of FPS.

Anyway, I've been gathering bits of advice to increase the FPS in my fort. So far turning off temp and weather has done nearly nothing. Making traffic areas have helped a little bit but not much.

Then I read you guys talking about spiral ramps instead of straircases. So I did away with my current staircase (you can see it in my sig) and replaced it with a spiraling ramp.
My fps jumped from an average 10 to 20. Amazing.

So yes spiraling ramps instead of stacking stairs, it helps a whole lot.

Next up removing the hundred goblin remains along with the clothes and armor they brought. Can't have it scattered around outside, to the semi molten rock with it!

Edt: I realize I left out ten layers between ground level and the entrance, so the staircase is a lot longer then in the map vieuwer

[Vulcanius]

Heh...  I think the multi-threading conversation just clarified a bit in my head (given that I have little knowledge of parallel processing and threading).  It appears that one camp says that threading has a lot of overhead, the other saying it has almost none.

I think you're both right, just using different metrics.  The first camp is looking at all the work being done by the OS, scheduler, and whatnot on all cores in all threads as the overhead, the other camp is looking at the overhead in just the main thread.  That simple setting of a wait handle kicks off a certain amount of processing by the rest of the system to get the other thread to run, even if that call itself returns to the main thread very quickly.

I am, of course, likely to be completely wrong in my interpretation of things.

I want to clarify what I was getting at with my posts.

I'm not against using threading in the future. At some point down the road we could see some significant performance improvements by implementing it. Normally if you implement threading when you first start writing your code, it's not very difficult at all. However, when you want to take a program that's already been designed and written, it can be infinitely more difficult. I'm simplifying it for explanation, but you esssentially must go back and find every single piece of data which two threads could possibly try to access at the same time. You're almost always going to have to rethink a lot of the ways you did things before as well. Toady has basically said he doesn't have a huge amount of threading experience, which is bad when combined with the rather large amount of code he's written over roughly 5 years.

What I believe is that threading is not addressing the real issue, which in my opinion is poorly written algorithms which do not scale well at all. This is the reason the game runs fine when you first embark, but later on slows down for various reasons (in some cases to an absolute crawl). The current code needs to be improved significantly before any thought of threading is considered.

In terms of the overhead of using threads, there is very little. I can't speak for Windows, but Linux implements threads almost exactly like normal processes, which are already extremely efficient and quick. The difficulty is in determining how to divide the actual work among the threads, and how to have them communicate amongst themselves and with the parent process. None of us, except Toady, has any real idea of how that should be done because we don't have the source code, and therefore can only speculate about how the game actually works.

[Tormy]

It's already been said. Every time a job regarding an item is called, every item is checked.

It's not job related, it's that they're checked every frame to see if they're decomposing or bursting into flame or whatever.

Hopefully Toady can do something about this later on.

[Symmetry]

What I believe is that threading is not addressing the real issue, which in my opinion is poorly written algorithms which do not scale well at all. This is the reason the game runs fine when you first embark, but later on slows down for various reasons (in some cases to an absolute crawl). The current code needs to be improved significantly before any thought of threading is considered.

Given what you've just said, wouldn't a well written algorithm for pathfinding be one that scaled well with threads?  If an improved algorithm is implemented before thoughts of threading are considered won't it just have to be rewritten again in future when Toady does think about threading?

[Makaze2048]

Given what you've just said, wouldn't a well written algorithm for pathfinding be one that scaled well with threads?  If an improved algorithm is implemented before thoughts of threading are considered won't it just have to be rewritten again in future when Toady does think about threading?

Not really. The pathfinding itself still runs in a single thread either way. Whether as part of the main thread as now or in a separate thread by itself it's still a discreet unit of work that is internally only a single thread. So the core algorithm looks the same either way. The key difference is that you have to make sure nothing changes the pathing data while it's running in a separate thread. This is guaranteed automatically to happen when everything is single threaded, but care and planning must go into making sure it never happens when multithreading something.

So really what would need to change is in and around where pathfinding is called from, the actual pathfinding function would need very little modification. The scaling issue is also more about the non-linear slowdown from item count (which to some degree is going to be unavoidable, there are simply some N² operations that logically need to happen) as opposed to the linear slowdown associated with additional units pathing.

[Vulcanius]

What I believe is that threading is not addressing the real issue, which in my opinion is poorly written algorithms which do not scale well at all. This is the reason the game runs fine when you first embark, but later on slows down for various reasons (in some cases to an absolute crawl). The current code needs to be improved significantly before any thought of threading is considered.

Given what you've just said, wouldn't a well written algorithm for pathfinding be one that scaled well with threads?  If an improved algorithm is implemented before thoughts of threading are considered won't it just have to be rewritten again in future when Toady does think about threading?

No, the algorithm and the threading are separate. It also needs to be emphasized that the only people who will really gain from threading are those with multiple processors, multiple cores on their processor, or a combination of both. On the other hand, everyone gains from improvements to the algorithms.

EDIT: Now that I think about it, it would be pretty interesting to find out what the computer 'demographic' of DF is.

[Symmetry]

No, the algorithm and the threading are separate.

I guess I'm thinking of things like not writing to static/shared memory and instead using working data structures that are per task.  I realise this is difficult with a map as big as DFs, which is why I'm suggesting thinking about it in advance may prevent another lengthy rewrite.

[Urist McDepravity]

I'm simplifying it for explanation, but you esssentially must go back and find every single piece of data which two threads could possibly try to access at the same time. You're almost always going to have to rethink a lot of the ways you did things before as well.

That becomes an issue only if your threads modify that data. Worker threads could just return 'decision' for each creature on his actions for this step, which would be executed in main thread. Some stuff, like jobs queue, would still be easier with locks, but its not a problem to have global lock for it and use it for writes on the queue.

There is no need to move whole codebase in workers at once. He could start with functions which have no side effects, and do rest in main.