Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[peekama]

I'm not sure how viable it would be, but seeing as more and more computers have graphics cards/processor of some sort, wouldn't it be smart to use that power? A graphics card would be perfect for calculating flows, as GPUs proccess in parallel. This would make it possible to calculate multiple flows all at once.

I'm not saying that Toady should give the game better graphics or anything, just that it might be possible to take advantage of the idle horsepower.

[Virex]

He already does that to some extent. Some functions are calculated through openGL and I seem to recall that toady once said that these functions run over the GPU, since that's what openGL is made for.

[Kholint]

Without wanting to necro this thread, here's something that might interest you lot- Valve/steam have just released the results for their latest hardware survey:

http://store.steampowered.com/hwsurvey

The survey is going to be biased towards the mainsteam gaming community, but given how resource intensive DF is (or can be) I think the statistics fairly accurately reflect DF's demographic (especially potential demographic!):

Some interesting tidbits:
- the majority of gamers (56%) have 2 gigs of RAM. Memory is cheap! Caching and eating RAM to improve performance isn't something to be shied away from, then.

- the majority of gamers (60%) are now running multicore processors (with 50% running dual core, 10% running quad)- nudge nudge, wink wink toady (that's a very serious, heavily emphasised nudge nudge, wink wink, too). Imagine DF performance doubled (ideally) on 50% of computers (and then increasing more or less steadily with the amount of cores added every year...)!

- harddrive space isn't a problem either. Pretty much everyone has at least 10 gigs of space free, and more and more people have more than 100 - 250 gigs free.

- :p

[Granite26]

thanks for that, it was neat

[Footkerchief]

Multithreaded DF would be cool and all, but how many multithreaded game engines even exist right now?  A couple dozen?  Any of them not designed by experts, and with difficulty at that?

Letting DF use more RAM might be pretty handy for tons of stuff including path caching and lighting, though.

[Kholint]

Multithreaded DF would be cool and all, but how many multithreaded game engines even exist right now?  A couple dozen?  Any of them not designed by experts, and with difficulty at that?

The Source engine is, now. Just check out the left4dead multithreaded settings. So's the cryengine thing that crysis uses. The unreal engine is almost certainly multithreaded, and every current, actively used engine is probably in the process of being updated to make use of the extra processor power.

But that's honestly missing the point- multithreaded isn't this holy grail of super advanced techology that only the most genius of computer scientists can even begin to comprehend- I'm a 19 year old self-taught programmer (and not a good one at that!) and my pet project is multithreaded.

I will admit it takes some getting used to, but not a massive amount. It's not a one-trick pony fad like GPU-based processing (don't take me up on that in 5 years when we're all running our internet browsers and instant messengers on geforce 5 billions* :p), it's the tried and tested future of (home) computing. And it'd really suck if DF missed the multicore boat.

Worth reading articles like these: http://harkal.sylphis3d.com/2006/02/16/multithreading-game-engines-for-multicore-machines/

http://www.gamasutra.com/features/20060906/monkkonen_01.shtml


*or rather, 500s.

[Tormy]

And it'd really suck if DF missed the multicore boat.

Well, I am really wondering about the possible amount of FPS in DF v1.0 if we won't have multicore support. 

[texmith]

Games have issues with multithreading that most other apps don't. Its not easy to seperate the game environment into individual tasks that can run concurrently on their own data set as most operations are small, numerous and work directly on one integrated world rather than having an obvious input and output. Do it badly and you risk performance loss as one thread locks the other.

Most game engines don't use multithreading very well. Its generally just the interface, rendering or pehaps ai in their own threads with most of the work still handled in the main thread.

Having said all that, multithreading is going to be more and mroe important in the future. As chip manufacturers find it harder and harder to increase core speeds, the trend is going to be more cores rather than faster cores. Games will need to embrace mutithreading and it doesn't bode so well for DF which already stretches the performance of current chips.

[Andir]

Games have issues with multithreading that most other apps don't. Its not easy to seperate the game environment into individual tasks that can run concurrently on their own data set as most operations are small, numerous and work directly on one integrated world rather than having an obvious input and output. Do it badly and you risk performance loss as one thread locks the other.

Most game engines don't use multithreading very well. Its generally just the interface, rendering or pehaps ai in their own threads with most of the work still handled in the main thread.

Having said all that, multithreading is going to be more and mroe important in the future. As chip manufacturers find it harder and harder to increase core speeds, the trend is going to be more cores rather than faster cores. Games will need to embrace mutithreading and it doesn't bode so well for DF which already stretches the performance of current chips.

That used to be the case.  Developers figured people would only have one core to use and thus designed their engines to run in tight loops with strict control over when things run.  It may have been OO or Event Driven programming, but you were pretty guaranteed a single thread and one action happening at a time.  Different algorithms were made under this assumption and weren't written with horizontal scalability in mind.  The reason most game developers today will tell you that multi-core programming is hard is because of this.  They use tried and true algorithms (like A* for instance) that work great on single threaded machines and they try to adapt these techniques to work.  It's simply the wrong approach.

Dwarf Fortress on the other hand uses a lot of procedures that could easily be made parallel.  (Even down to making two or more dwarfs calculate paths simultaneously.)  The problem lies in compartmentalizing your tasks to run as a standalone unit.  Being able to decide what information is needed for the task and sequestering it to a pool of tasks or a persistent thread to be processed as needed.  Liquid calculations could benefit greatly, and more realistically, from multitasking.  If the world of DF were a server application (or a process spawn), even groups of dwarfs could be handed of to separate SMP applications that are passed their information and local maps and communicate back the result.  (Like passing a region map with 100-200 dwarfs to another process and having it update the "world".)  If it were done right, you could even split the DF world into an object that had it's own TCP/IP routines so you could play those community forts without passing files around but connecting to a community server and running through your steps or even take it to the next level and allow multiple forts in the same world, letting the server handle time without pausing.  But I'm getting off track here and this has been discussed to death on other threads.

Needless to say, threading isn't tough on today's machines when you get down to the bolts of it.  You are simply handing a process a pointer to a memory location, locking that memory location in some table, and letting the other thread work with that data.  Arrays can be split into groups of 1000 units and sanctioned to another core for processing using the same rules (functions/methods) as the other groups.  It would work great for processing a loop of calculations on a per dwarf basis for instance.  Determine how many cores (not threads... could be thousands) can be used, step your array by that number - 1, spawn that many threads and assign those chunks to each thread.  As long as you don't do any cumulative calculation, it's a winning situation.  If you can then maintain those threads for processing the same set of dwarfs the next round, you can avoid the instantiation cost of spawning threads.  If, for instance, you are counting up the number of dead dwarfs each loop, maybe you should think about why you are doing this instead of saying that that is the reason for not sectioning this loop.

Sorry for the rant.  I hate when people say multi-tasking is hard.  It's just a different though process.

[Fieari]

On such a time as Toady getting bored with DF and open sourcing it (not guaranteed to happen, ever) then you can multithread it.  But Toady, himself, personally, is entirely unfamiliar with multithreading and has stated he'd rather not take the time to learn, preferring to just keep going adding the features he wants.

[profit]

Just a side note, some more threads would do this program a world of good.
For example:
This is not only for those who have 2+ cores, multithreading can also help single core CPU's as well.   

The UI being the single biggest thing I see.  The game apparently does not differentiate between the UI thread and the Simulation thread or maybe they are even the same thread.    Hence when the simulation bogs down (For example selecting stone in the stocks menu, with 40,000 stone, or marking an area of 300 tree's to be cut)   The UI becomes totaly non responsive.  This is also shown in regular game play as the simulation speed comes under stress from too many immigrants or a slow computer the UI jerks and delays action on key press's. 

Ideally the UI and the simulation should be independent of each other, and the user interface should always be crisp and snappy.

* Having thought about it now, likely this program is a top down one, with no seperate threads other than the ones automaticly created by the IDE on compile. * it would generate one for the openGL libraries and one other one if he is using the boost libraries*

If he is doing that this program would have about 3 threads, he would have to actually not use any libraries in order for it to be less I believe so.... its probably got at least 3 threads.  Idealy it should have about 10-20 I think but thread build and tear down is so little it probably  wouldent matter if every job created a seperate thread... and it would be basically infinitely scalable if it did.... But for debugging I think he may want to stick to under 50 threads.

[Kholint]

On such a time as Toady getting bored with DF and open sourcing it (not guaranteed to happen, ever) then you can multithread it.  But Toady, himself, personally, is entirely unfamiliar with multithreading and has stated he'd rather not take the time to learn, preferring to just keep going adding the features he wants.

Ah, but that's the point: none of us have access to the source code, and I doubt many of us would even want it even if we could (I certainly wouldn't be interested in programming someone else's game!).
The best we can do is just attempt to persuade or influence toady (under the guise of DF appreciation and "helpful suggestions" ) towards our own aims.

Which is exactly why we're talking about multithreading and the multitude of advantages it would bring.

An Inconvenient Truth

Another way of looking at it is this: assume you're one of those still running a single core machine. Dwarf fortress is 100% efficient on your computer! It's powering away, grabbing as much cycles as it can to do its job as fast as possible.

But if you were to upgrade your computer to a more powerful model, it's nigh-on guaranteed to be either dual core or quad core. Your computer is much more powerful now, but lo and behold- dwarf fortress is now only 50% efficient (or, in the case of a quad core, 25% efficient!). It's only using a fraction of power that your new computer has to offer.

I'm already running a decently powerful dual core, so when I buy my next computer in a few years it'll almost certainly be a quad or eight core machine. Dwarf fortress will now only be 1/8th as efficient as it used to be on a single core machine!

Then there exists the danger of clock speeds actually slowing as time goes on. If tomorrow's processors decide to favour more cores for decreasing clock speeds (as they very well might- it's not difficult to imagine a 512 core machine with 500 Mhz per core), not only will dwarf fortress become almost incredibly inefficient with the hardware available, but it'll also be able to do less and less with that single core it *can* use.

The aforementioned clock-slowing scenario is only a possibility, but the dramatic decrease in efficient use of available processing power is a fact that can be seen already with those of us on the forums with dual or quad core computers.

This kind of rotting techno-death is basically inevitable if dwarf fortress isn't made decently scalable on modern and future computers.

[Draco18s]

If tomorrow's processors decide to favour more cores for decreasing clock speeds (as they very well might- it's not difficult to imagine a 512 core machine with 500 Mhz per core)

Already been done.
Or near enough.

I think it was 50 cores at 400 MHz or so.  IBM did it.

[texmith]

That used to be the case.  Developers figured people would only have one core to use and thus designed their engines to run in tight loops with strict control over when things run.  It may have been OO or Event Driven programming, but you were pretty guaranteed a single thread and one action happening at a time.  Different algorithms were made under this assumption and weren't written with horizontal scalability in mind.  The reason most game developers today will tell you that multi-core programming is hard is because of this.  They use tried and true algorithms (like A* for instance) that work great on single threaded machines and they try to adapt these techniques to work.  It's simply the wrong approach.

Reinventing the tried and true building blocks of game programming is hardly a trivial matter. Not that its necessarily 'hard', but its certainly a lot of work.

[Andir]

Reinventing the tried and true building blocks of game programming is hardly a trivial matter. Not that its necessarily 'hard', but its certainly a lot of work.

If you only knew how much code actually gets re-invented on a daily basis, I think you'd be amazed.  For instance, takea simple thing like Inventory management in games.  You have slots, weight, and possibly bags.  Developers have been recreating this system in just about every game that I've seen developed.  The same thing with graphics engines.  You'd think it would be more economical to use a tried and true engine published at the time... but how many people develop their own core of graphics processing in their games?  Sure, there are some that use the Source Engine, Quake Engine and Unreal, but there are far more people that grow their own.

I'd even bet that Toady developed his own pathing algorithm for DF instead of using A*.  He invented his own liquids engine.