Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Angellus]

Why is there not an external program that can trow load over to multiple porcessors?

Is it hard to get something like that working? I would recon it should see what is happening and divert it to thread 1 or 2.
Off course I have (extremely) minor knowledge of programming, but should this not be possible?

[Michael]

Why is there not an external program that can trow load over to multiple porcessors?

Is it hard to get something like that working? I would recon it should see what is happening and divert it to thread 1 or 2.
Off course I have (extremely) minor knowledge of programming, but should this not be possible?

It's not possible.  If you could do that -- analyze someone else's single threaded program, without the source code, and derive from it a new version that makes good use of multiple cores, the processor manufacturers would make you rich.

As an example of how an attempted conversion could fail for a DF-like game, suppose you tried recoding so that pathfinding would run in parallel.  One way a single-threaded program might handle enemies who can break doors is to temporarily change the map state so that all doors are open, do the pathing, and then quickly set the doors back.  In a single thread, nothing can interrupt those three steps, so this can be done safely and invisibly to the player.  But if a naive programmer just changed things to run in parallel, other monsters' pathing would occur while the map information is distorted, and thus their pathing would not be correct.

[Freaky]

Except multithreading *is* easy. Every time I read an article talking about multithreading, or read materials about multithreading in a given language, they make a point of talking about how hard it is, how many pitfalls there are, etc. But, it's not hard to write a program keeping those things in mind. It's different, so I suppose any perceived difficulty comes from being set in a single-threaded mindset, but it is by no means hard in and of itself.

Have you actually written non-trivial multithreaded applications?  The algorithms are more complex to implement and reason about, data flow gets a lot more complex with layers of locks with their own rules about when they can be acquired and message passing systems not without their own overhead and rules, even unlocked data can come with mysterious performance penalties thanks to false sharing, and one mistake can take months to show itself or even go largely unnoticed so it's very difficult to test.

Impossibly hard it's not, but it certainly is more difficult than just writing unthreaded sequential code, even if you're starting from scratch.

[Angellus]

Why is there not an external program that can trow load over to multiple porcessors?

Is it hard to get something like that working? I would recon it should see what is happening and divert it to thread 1 or 2.
Off course I have (extremely) minor knowledge of programming, but should this not be possible?

It's not possible.  If you could do that -- analyze someone else's single threaded program, without the source code, and derive from it a new version that makes good use of multiple cores, the processor manufacturers would make you rich.

As an example of how an attempted conversion could fail for a DF-like game, suppose you tried recoding so that pathfinding would run in parallel.  One way a single-threaded program might handle enemies who can break doors is to temporarily change the map state so that all doors are open, do the pathing, and then quickly set the doors back.  In a single thread, nothing can interrupt those three steps, so this can be done safely and invisibly to the player.  But if a naive programmer just changed things to run in parallel, other monsters' pathing would occur while the map information is distorted, and thus their pathing would not be correct.

Thanks a lot for the explanaition, might I also ask you why it is that a compiled program is not readable anymore? (to read the source out of DF.exe for example is impossible, not?)

[Xinael]

might I also ask you why it is that a compiled program is not readable anymore? (to read the source out of DF.exe for example is impossible, not?)

Simply put, because computers don't speak source code, and expressing even simple ideas in machine language becomes very cumbersome very quickly.

[Angellus]

But why would that not be reversable? The code is based on the source code, not? would that not be translatable back to source code?

[Michael]

might I also ask you why it is that a compiled program is not readable anymore? (to read the source out of DF.exe for example is impossible, not?)

Simply put, because computers don't speak source code, and expressing even simple ideas in machine language becomes very cumbersome very quickly.

That's not what Angellus asked.  You gave the answer to "why don't people just program in machine code and forget about C?".

The problem is that the process of translation to machine code discards a lot of information, such as what the programmer named his variables.  It becomes impossible to tell whether the programmer originally wrote "(X * 3) + 6" or "(X + 2) * 3" -- since a good compiler will generate the same code.  The impact on program understandability of losing a single such piece of data is trivial -- but "decompiling" cannot recover any of this information.

And of course the original programmer's comments, which don't effect execution at all but can be vital in understanding why things are done in specific ways, are entirely lost.

There probably are decompiling tools out there that, given an .EXE file, can produce a C program that will hopefully behave identically to the original.  But the code will be little easier to understand than an assembly-language dump.  (Assembly language is a special language designed to be easily translated both to and from machine code.  Unlike machine code, but like a normal language, it can be viewed in a text editor.  But it's much less convenient to use than C.)

(Also, there are ways to make EXE files that will utterly confuse such software.)

[Angellus]

Michael, thanks a lot for that explaination!
Xinael too, though that indeed was not exactly what I was looking for, though still information of high value to me

[Michael]

Except multithreading *is* easy. Every time I read an article talking about multithreading, or read materials about multithreading in a given language, they make a point of talking about how hard it is, how many pitfalls there are, etc. But, it's not hard to write a program keeping those things in mind. It's different, so I suppose any perceived difficulty comes from being set in a single-threaded mindset, but it is by no means hard in and of itself.

Have you actually written non-trivial multithreaded applications?  The algorithms are more complex to implement and reason about, data flow gets a lot more complex with layers of locks with their own rules about when they can be acquired and message passing systems not without their own overhead and rules, even unlocked data can come with mysterious performance penalties thanks to false sharing, and one mistake can take months to show itself or even go largely unnoticed so it's very difficult to test.

Impossibly hard it's not, but it certainly is more difficult than just writing unthreaded sequential code, even if you're starting from scratch.

Actually, you're both right, to a point.

Writing a program that appears to work and uses multiple threads is easy.

Achieving an acceptable confidence that the program has no bugs is significantly harder.

And optimization -- creating a multi-threaded program that performs a task on, say, a 4-way 2GHz machine at a speed approaching that of an 8GHz single-core, is a black art.  In fact, you could say it is impossible in general, because things that greatly benefit performance on one specific computer may poison the performance on a different rig.

[buman]

maybe someday compilers will be smart enough to do automatic parallelization on your otherwise serial programming.

[Zironic]

Some invent a time machine, and send our toady, back into the past, with the current DF version, to work alongside himself, repeatedly. Do this 10 times.
1. We will have 2+2+4+8+16+32+64+128+256+512= 1024 toadys (Each time the Toady's reach the present, they go back in time, working alongside themselves.)
2.Because they'll bring the latest version back in time, it would also become modified quickly.

[Shoku]

Why is there not an external program that can trow load over to multiple porcessors?

Is it hard to get something like that working? I would recon it should see what is happening and divert it to thread 1 or 2.
Off course I have (extremely) minor knowledge of programming, but should this not be possible?

Think of a list of chess moves as in rook to A9 or whatever. If you send every other command to a different processor it doesn't make sense anymore because now you've got two pieces moving at the same time. If you're trying to make actual sense of it you can't tell what is a response to what and with programming the response is very important- a dwarf running through his pathfound needs to respond to an orc chopping off his legs.

Now, programs like DF are basically instructions for doing the same thing over and over but with what you got last time being what you start with the next time so if it's organized a certain way (the easy to change style of programming,) it would be pretty easy to divert things like weather generation and armies marching around the map and offsite population growth to a different processor would be easy and you could even pretty easily shove temperature off onto another processor without having to think about it too terribly much but most of the time taking all of the things I've mentioned out of the equation don't speed this game up very much so this would only give Toady the option of making those things more bloated.

The things that are really eating up your processing power are all very intertwined and would take a much more complicated design to make work in multiple threads. Even if Toady had begin writing things that way from the start I think it would slow him down so much as to not be worth it. Maybe after the interface arc there will be a big influx of players and he'll be rolling in so much donation money he can hire some android interns to do a lot of that work for him in 3009.

maybe someday compilers will be smart enough to do automatic parallelization on your otherwise serial programming.

For a modern parallel that would most likely be like using Java vs C+. Java automates a lot of things for you but doing them manually in C makes for a program with more digital muscle.

[Baughn]

No, multithreading is easy, just not with the typical, imperative style of programming.

The way most programmers approach this problem is to write programs that alter some state, then do some more (different) alterations based on the altered state, rinse and repeat until you've completed a full frame.. then repeat.

Needless to say, making this style of programming work with multithreading is an iron-plated bitch. It's hard if you start off your program thinking about threading, and it's practically impossible to retrofit. That does not mean that multithreading is hard, though, just that multithreading using this style is. There are others.

For example, in (purely) functional programming, you write functions without having any state for them to alter. You pass in the old state, and they return new state, a summary of it, or whatever; it gets quite involved, and doing it well is probably harder to learn. However, this style offers several enormous benefits:

First, it becomes much easier to reason about your program, because no function can affect anything other than its own return value, and nothing needs to worry about other parts of the program altering its data (since they can't). This benefit increases with program size, offering probably better modularization than OO manages; a good thing, since the style fits very poorly with OO in general. You have to learn other ways to do what OO'd do.

Second, which is the part you care about, this makes multithreading easier. You don't have to worry about other threads altering your data since, again, they can't. Further, higher-order functions - which don't only exist in FP, but are certainly more common there - such as maps or folds, make it easier to decide what to multithread. You can always switch out a map for a parmap, which does the operations in parallel, without ever having to worry about whether this might make your program incorrect - it can't - though you still have to worry about the cost of thread creation and granularity.

Third, it's easier to reason about the programs - for computers. Automatic parallelization of non-parallel code is the holy grail, yes? Well, in functional programming, it's happening already. The research is still at an early stage, but efforts such as Data-Parallel Haskell are pushing it closer to reality every day. The compilers already do extensive single-threaded optimizations that make such languages nearly as fast as C, despite being far, far higher-level and running on processors that are, frankly, designed for C.

Fourth, it's safer. There just isn't nearly as much that can go wrong in your programs; crashes are nearly unknown, and they're fairly amenable to formal proof methods. As a result, functional programming is quickly gaining popularity in the financial sector; other areas, such as airplanes, nuclear reactors or spacecraft, are still limited by inertia and the relatively poor performance of hardened processors.

Fifth, it's cooler.  You'll never find any traditional language with anywhere near the interesting features of functional languages - hell, for you C++ programmers, Haskell allows you to overload the semicolon - and it works quite well, thank you. (Monads)

[MiamiBryce]

To answer the original question:  Dwarf Fortress uses all 4 of my cores.  I can run maps that would make other people cry, scream, and tear their hair out - without using the new OpenGL version of DF (it does nothing for me).  People seem to forget that CPU's are made to distribute single-thread application over multiple cores.  In fairness I should say that a program may not run any faster using multiple cores depending on the overhead of distributing it.

Let me assure you all, there is no need to make DF multi-threaded.  It's a simple matter of implementing more efficient algorithms for the most used functions.

</thread>

[Im_Sparks]

To answer the original question:  Dwarf Fortress uses all 4 of my cores.  I can run maps that would make other people cry, scream, and tear their hair out - without using the new OpenGL version of DF (it does nothing for me).  People seem to forget that CPU's are made to distribute single-thread application over multiple cores.  In fairness I should say that a program may not run any faster using multiple cores depending on the overhead of distributing it.

Let me assure you all, there is no need to make DF multi-threaded.  It's a simple matter of implementing more efficient algorithms for the most used functions.

</thread>

So.. You want to upload your copy of DF for me to download?