Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[The Architect]

Alright, the basic idea is as follows:

"We" make an emulator that will run Dwarf Fortress and use your whole computer's processing power. I say "we" because one thing I don't know is programming. The program could be fairly simple because it only needs to run one program, Dwarf Fortress, and DF's basic format and program structure will not change (which is what makes this useful and important).

From what I understand, right now DF's current format only enables it to run on a single processor, failing to utilize the power of the dual- or quad-core processors most of our computers currently have. A good emulator (something along the lines of DosBOX) could unlock DF's potential.

This is the gist of what I want to make happen:

You are thinking of something like DOSBox, able to act as a unique system with mutable system resources.  So while the emulator could be multithreaded across different processors, all that power would be read as one processor within the emulator, so DF would run on that alone.

And I don't know if there is one like that now, but I think that someone might make one now...

EDITED to change the title.

[Blacken]

Uhm.

Start here and see if you can figure out a way to break up a single thread of execution into arbitrarily many threads of execution.

If you figure it out, there are dozens of companies that would pay you millions of dollars to learn how.

[The Architect]

Uhm.

Start here and see if you can figure out a way to break up a single thread of execution into arbitrarily many threads of execution.

If you figure it out, there are dozens of companies that would pay you millions of dollars to learn how.

I think you're reading a little too much into this. It doesn't need to be nearly so universal, it's for running DF not splitting threads in general. We just need to trick DF. Splitting the processes DF does (if that is even necessary) might require access to code that has not previously been available to players, and we may need Toady's help. If that is so, then let's go there with it.

The point of this is to enable DF to utilize our computers' true potential without the need to rewrite the whole program. A program-specific emulator. Not a universal tool.

[Blacken]

I think you're unclear on what an emulator is, otherwise you wouldn't be using the term. I think you don't realize that I understand exactly what you're trying to ask and that what I said in my first post would be the only effective way to do it short of having the source code. I think you lack the research to realize why it's not possible. I think that you should probably do that research before you try to correct me.

Put simply: you cannot "trick" four separate instruction units into operating on one sequential string of instructions when later instructions require the results from the earlier ones. It doesn't work that way.

I suggest you go learn how to program and how modern computers work, then come back.



Toady no doubt has plans to multithread Dwarf Fortress at some point. I doubt that this "idea" is helpful, as restructuring a program for multithreading is a nontrivial task when you have the source code. It isn't like the graphics system where he could just spin it off into a separate project so people could hack on it. The perf benefits from multithreaded programming would come from core systems, not little pieces. He's said many times he doesn't want to open-source important parts, for a number of reasons.

[LegoLord]

Or possibly you know so much about it that you misunderstand what he's saying from assuming that he's even using the correct terms for what he's trying to say, due to him knowing little about it.

You're the only programmer whose responded thus far, I think it'd help to take a step back and look at this closely.  Even if it turns out he's wrong and you're right.

[Blacken]

Or possibly you know so much about it that you misunderstand what he's saying from assuming that he's even using the correct terms for what he's trying to say, due to him knowing little about it.

You're the only programmer whose responded thus far, I think it'd help to take a step back and look at this closely.  Even if it turns out he's wrong and you're right.

Sigh. Welp, let's look at what he's asking and see if it's ambiguous.

From what I understand, right now DF's current format only enables it to run on a single processor, failing to utilize the power of the dual- or quad-core processors most of our computers currently have. A good emulator (something along the lines of DosBOX) could unlock DF's potential.

This is clearly unambiguous in its intent. He doesn't know the solution (and neither does anyone else, that's the point), but his intent is clear. He is asking for a way to apply multiple instruction processors to a single (or near-single) instruction set, without touching the source code. He is essentially asking for a pet unicorn. Not even a pet pony--an out-and-out pet unicorn. There are companies spending millions of dollars on how to effectively predict code in this way. It is a decidedly nontrivial task, if it can even be done in a performant way. That's why everyone who needs it just rewrites the code.

At the risk of sounding like a jackass--it certainly would not be the first time--unless Toady wanted to give out the source code (or just do it himself) there is nothing to look at here, and it is rather silly for you to say "it'd help to take a step back" when it isn't technically feasible no matter how far you step back.

The code will eventually be rewritten to support them, I'm sure. Until then, get faster-clocked processors.

[G-Flex]

What Blacken's been saying. All of it.


We're talking about getting a single thread of instructions, and splitting it in two (or some other number) such that different instructions are being executed independent of each other.


There are certainly some things DF does that could be parallelized, but the program simply isn't designed to do it. A lot of restructuring of the code would probably be necessary.



Let me put it this way. You have the instructions for doing something. Let's say, baking something.

• Preheat the oven to 350 degrees.
• Grease a pan.
• In a large bowl, beat the eggs.
• In another large bowl, mix the sugar, flour, and baking powder.
• Add the other liquid ingredients to the eggs and mix thoroughly.
• Add the egg mixture to the flour mixture and mix them together.
• Pour this into the pan.
• Stick the pan in the oven for 40 minutes.

Now, if you think about this, you can separate this into different instruction paths that are effectively independent; you can have one person mixing the dry ingredients while another mixes the wet ingredients, while other people set the oven and grease the pan.

You can do this using intuition in real life just fine, but it's actually an extremely complicated process. You need to know which instructions rely on the results from which others, and in which order. For instance, you know not to start mixing the eggs with the milk before you've even cracked them open, or to put the pan in the oven before anything's in it, to start preheating it right before you put the pan in, or to pour the batter in the pan before it's greased, nor can you mix what's in the bowls together before the both of them have what they need individually.

To that end, you can't make a program that just figures it out for you; nothing can look at even the source code of a program and determine what you meant by it, or what the results should be, never mind a compiled executable. You know that a cake is supposed to be fluffy and delicious, with a certain taste and consistency. A computer doesn't have that going for it.


Basically, in order to multithread DF, you'd have to look at the source code, figure out which parts can run in parallel with which other parts, and how to interface them so that they interact in a reasonable fashion.

Asking a program to look at the compiled program (or even the source code) and do this for you relies on the same lack of understanding as asking someone "Why won't the computer do what I want?" - the computer never does what you want, only what you tell it to. Even the different parts of DF's code that could be run in parallel currently are not and will interface in ways that no program can magically figure out for you.

[Blacken]

G-Flex is nicer than I am.

[ctharvey]

G-Flex is nicer than I am.

Seconded.

[Footkerchief]

Okay, first off there's no such thing as a "program-specific emulator," at least that I know of.  An emulator provides a virtual architecture and system resources like memory, I/O, etc., on which any number of programs can run.  Emulators for modern systems are often known as virtual machines, e.g. VMWare or MS Virtual PC.  DF would run in a single thread inside the virtual machine, but since the virtual machine itself is multithreaded, there should be a net speed increase, right?

Well, probably not.  The only parallelization you can do on the level of the virtual processors is stuff like instruction-level parallelism, and your physical single-core already does that.  So the emulator/VM doesn't gain you anything.

Here's a paper that touches on some of the techniques being explored for parallelizing single-threaded applications.  It's still a very active area of research, so it's not out of the question that DF will someday make good use of multiple cores.  But it won't be as simple as what you proposed.

Asking a program to look at the compiled program (or even the source code) and do this for you relies on the same lack of understanding as asking someone "Why won't the computer do what I want?" - the computer never does what you want, only what you tell it to. Even the different parts of DF's code that could be run in parallel currently are not and will interface in ways that no program can magically figure out for you.

I think this is an oversimplification, if I'm remembering my Op Sys class correctly, which I'm not at all sure of.

[G-Flex]

Haha, they're calling it "Voltron". Clever.

And honestly, I just don't think the OP knows very much about emulation or why it wouldn't work. An emulator being multithreaded doesn't matter if the code it's running isn't multithreaded anyway, and if the emulator can automagically parallelize the game... well, that's not even something that you'd need an emulator for anyway, nor is it the kind of thing that seems plausible to do at run-time. It sounds like something you'd want to run on the executable itself in order to produce a new, modified, parallelized one.

[Neonivek]

Ignoring "Single Coring" the game (Which apperantly works VERY well) the rates in which computers improve may destroy the lag itself.

Though I shouldn't bet on it since computers are finally slowing down. Hense one of the major reasons for the failure of the Vista.

Note: By Single Coring, I am refering to some weird people who devoted an entire core to Dwarf Fortress

[Blacken]

Here's a paper that touches on some of the techniques being explored for parallelizing single-threaded applications.  It's still a very active area of research, so it's not out of the question that DF will someday make good use of multiple cores.  But it won't be as simple as what you proposed.

Important to keep in mind regarding that research (which is interesting, I'd never seen the full paper before--thanks for the link!):

-It doesn't use current technology. I'm sure they could build a simplified prototype, but it wouldn't be fast enough for general purpose usage.

-It wouldn't work with x86 at all, because of some quirks in the architecture. I could see it as a POWER retrofit, but it'd be one of the biggest (and thus most expensive) changes made to date in x86, and millions of computers would still lack it.

-It still requires programs to be structured in a very specific way. There isn't a "free" speed increase. Code that requires dependencies will still require previous code to be finished first, and so for something like Dwarf Fortress you gain little.


I'm sure that DF will someday make good use of multiple cores--when Toady gets around to it. This isn't how it'll happen.

Ignoring "Single Coring" the game (Which apperantly works VERY well) the rates in which computers improve may destroy the lag itself.

Though I shouldn't bet on it since computers are finally slowing down. Hense one of the major reasons for the failure of the Vista.

...No. Just...no.

Note: By Single Coring, I am refering to some weird people who devoted an entire core to Dwarf Fortress

On a properly scheduled machine, this shouldn't matter. This is an OS limitation, and one that is progressively getting better (albeit slowly).

[The Architect]

Splitting the processes DF does (if that is even necessary) might require access to code that has not previously been available to players, and we may need Toady's help. If that is so, then let's go there with it.

The point of this is to enable DF to utilize our computers' true potential without the need to rewrite the whole program. A program-specific emulator. Not a universal tool.

It seems obvious to me that I have already stated the possible need for access to the source code. You're right, I don't know a lot about what I am proposing. From what you say, access or at the very least close cooperation would be absolutely necessary. So be it, if this is in any way feasible with the source code then let's ask for it. If it's not possible either way then sure I'll forget it. But let's not dismiss it out of hand based on things that may or may not be the case (Toady/Threetoe never agreeing to such a thing).

I was actually looking for something to interpret the processes and feed DF information that it would normally need to manually cycle through. Not work the whole program on a multi-thread base. But most likely this would require non-existant output and non-existant input options. Outsourcing just the stocks database, causing DF to feed the info to a separate process and retrieve it from said process would cut the load tremendously, would it not?

DF can sequentially request info from the program rather than calculating it itself as it does now. The "emulator" would just be an interface. I do realize now that it might be the wrong term for such a program, but I dont' know a better one. The emulator would cause DF to believe xy was happening, when in fact z was performing the functions of xy separately. The same could be done with pathing, but that would be insanely problematic and likely to produce a headache just contemplating, but those are the two major drags besides job priority checks. Allowing us to set our own job priorities for dwarves and groups would probably cut lag in large forts by a ton as well. This problem could be circumvented by outsourcing the Jobs menu and feature to a separate program.

As I'm looking at it now, it definitely requires full cooperation from Toady. One thing it does not require is a rewrite of the program.

[Neonivek]

...No. Just...no

This isn't TVtropes, around here were support our ideas.

Also that was very annoying Blacken. What If I quoted you can instead of actually responding I just said "No"?