Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[sproingie]

Frankly, I'm done with this thread.  Gotten way too vitriolic.

[profit]

Frankly, I'm done with this thread.  Gotten way too vitriolic.

Ironic coming from someone who as a response to the first poster was "Mods, let's just lock the thread?"

 A statement that was obviously said in disdain for the idea, or the original poster, not because it had been brought up in the past...  Otherwise you would have suggesting locking several other threads that were duplicates of other very common suggestions that day as well.

* I do agree though my one post was borderline....  I was getting that "Boom HEADSHOT!" rush going as I kept knocking down  the ways people said it could not work.

So in closing... I have to say my first post on this thread is still very relevant now and probably we SHOULD have just left it at that because once again it is completely correct and nothing changes...

Yes, it has been considered... 

Yes, it could bring about phenomenal increases in speed...

Yes, we have also discussed using CUDA

Yes, we have also discussed FUSION

No, don't look for it to happen because Toady does not know how to do it and in order to learn would have to take time away from the actual project.

[KenboCalrissian]

Multi-threading is easily accomplished by creating two or more looms in your fort and having enough weavers to work them simultaneously.

...Why's everyone looking at me like that?

[profit]

LOL!

Epic win!

[assimilateur]

No, don't look for it to happen because Toady does not know how to do it and in order to learn would have to take time away from the actual project.

In that case, I think that people like me (and supposedly you as well) need to speak up to bring a certain detail to Toady's attention. Namely, that if improving the abysmal performance of this game isn't part of the "actual project", then I do not know what is. Piling feature upon feature is meaningless if the game all but grinds to a halt on most rigs and it takes ages to get anything done.

Of course, I'm pretty much repeating myself by now.

[Kilo24]

From my limited understanding, there is still a lot of optimization that Toady can do before he needs to rewrite the game for multithreading, like pathfinding, fluid dynamics, or graphics as the 40d# releases have shown.  He doesn't do it because he keeps adding features that might break some of those optimizations, and because he doesn't consider those to be huge, game-breaking flaws.

At this point, I still find the game quite playable, myself.  When I find that the game has become too much for my semi-recent computers to handle, then I and a lot more people will start complaining which informs Toady that optimizations are starting to be a big issue.  Pathfinding optimization is already #4 in the Eternal Suggestions thread, so I do think he's very aware of the problem.  His business model requires him to be.

[assimilateur]

I do think he's very aware of the problem.  His business model requires him to be.

Well, I do hope you're right. I also hope for some tangible evidence for that sooner rather than later, and I'm betting I'm not in a minority with that sentiment.

[The-Moon]

From my limited understanding, there is still a lot of optimization that Toady can do before he needs to rewrite the game for multithreading, like pathfinding, fluid dynamics, or graphics as the 40d# releases have shown.  He doesn't do it because he keeps adding features that might break some of those optimizations, and because he doesn't consider those to be huge, game-breaking flaws.

At this point, I still find the game quite playable, myself.  When I find that the game has become too much for my semi-recent computers to handle, then I and a lot more people will start complaining which informs Toady that optimizations are starting to be a big issue.  Pathfinding optimization is already #4 in the Eternal Suggestions thread, so I do think he's very aware of the problem.  His business model requires him to be.

The most intelligent thing anyone can do. Is prevent a problem before it becomes a problem.

Waiting around for the game to become unplayable, and then fixing it. Rather then fixing the problem to begin with.

Is just flawed logic.

After this release toady really needs to start thinking about multi processor support. Because optimizations can only go so far.

I don't care if DF is programmed completely in Assembly by the best programmers in this world. It will need multiprocessor support before it can move much further.

Also btw, nothing needs to be re-written. Toady is not going to have to completely rewrite hes code. He simply needs to redirect code to a different processor.

This can easily be done using a Client Server Type setup where you have a server storing everything from one processor and controlling the game, and then the other processors doing all the hard work.

Its mainly just a lot of cutting and pasting work for the most part. It takes a little time to do, but the sooner you get started to better. Its only going to take more time later on when toady has no choice.

Trying to optimize the game so it run faster is only delaying the inevitable.

Mainly all you need is a server app which would be the main program which has all the game data stored in, and the User interface. The server app would be the program the user controls and plays with. The main Server Program, would then load up several or more client Programs which would then control path finding or anything else the game needs to run.

This of course can be expanded so that you could run separate parts of DF on different computers and have them all connected through a lan line. So even if you don't have a muti core processor computer, but you have several computers sitting around  doing nothing. Whne you play DF, you could have DF run path finding programs or other such programs, on different computers.

This would easily only take a couple months time to setup and save a lot of time later on.

Basically toady needs know nothing about setting up threads in a single program. Windows will do that for him. He just needs to know how to use basic internet sockets and he could have everything mutil threaded within a few months if that.

He makes a server app, then a client app which does work depending on what commands the server sends to those client apps. He then only needs to connect the server and clients, and learn how to set those programs to different processors, or different computers.

Piece of cake.

[Rowanas]

A few months is a bit on the low side, given that I don't think he's done anything with client/server setups. Is it really worth the huge wait for what you're suggesting. I always run at a smooth 40 FPS, because that's my favourite speed. I don't need to limit my cat population or change my pop. cap and a lot of other people out there playing DF experience exactly the same thing that I do. Waiting around for ages for him to do that work will put off a lot of the DF community, especially when we find out that it won't even make the game any more fun, like the current extremely long update.

[Blacken]

It has nothing to do with x86, or windows xp. Short of large cluster computing and  expensive video cards in the last couple years; no CPU, operating system or programming language has been able to handle that kind of parallelization "well" for relatively trivial tasks. The vast majority of real world applications (where performance is a concern) do not and can not work like that.

Well, to be fair, though, a CUDA-equipped GPU that will blow the doors off of an x86-only implementation for a properly parallelized application is under $50.

[assimilateur]

Waiting around for ages for him to do that work will put off a lot of the DF community, especially when we find out that it won't even make the game any more fun, like the current extremely long update.

I don't think people who are experiencing acceptable framerates will be as put off by a boring update as people who are even now on the verge of quitting due to the already ridiculous slow-downs will be with an update that makes the game even slower.

[Nadaka]

Well, to be fair, though, a CUDA-equipped GPU that will blow the doors off of an x86-only implementation for a properly parallelized application is under $50.

Will? how about may. I've done some research on GPGPU and it has some serious limitations. I've never done it, never hadhardware, time or a viable problem to use it.

A: it only works with a large number of identical batch jobs
B: logic has a horrible performance cost, to preserve the performance you need to do non-conditional non-looping data transformation.
C: All tasks must be completely independent, there is not even the option to lock.
D: nothing can use the same memory for input and output, meaning you can not transform in place, possibly requiring twice as much memory for the same operation.
E: every task requires data to make a round trip through your PCI-E bus, as far as I can tell, this is always slower than copying the same data from and back to memory.

This means that it is much, much harder to do this kind of multi-processing compared to the conventional methods even if you start out intending to do so and some problems can never take advantage of it. Converting existing code is exponentially more difficult.

[profit]

Well, to be fair, though, a CUDA-equipped GPU that will blow the doors off of an x86-only implementation for a properly parallelized application is under $50.

Will? how about may. I've done some research on GPGPU and it has some serious limitations. I've never done it, never hadhardware, time or a viable problem to use it.

A: it only works with a large number of identical batch jobs
B: logic has a horrible performance cost, to preserve the performance you need to do non-conditional non-looping data transformation.
C: All tasks must be completely independent, there is not even the option to lock.
D: nothing can use the same memory for input and output, meaning you can not transform in place, possibly requiring twice as much memory for the same operation.
E: every task requires data to make a round trip through your PCI-E bus, as far as I can tell, this is always slower than copying the same data from and back to memory.

This means that it is much, much harder to do this kind of multi-processing compared to the conventional methods even if you start out intending to do so and some problems can never take advantage of it. Converting existing code is exponentially more difficult.

None of those are even close to deal breakers. 

A 200 pathfinding tasks are identical in structure.

B (This one is long I am sorry)  Here is how to do it -> http://portal.acm.org/citation.cfm?id=1413968
"ABSTRACT

In the past few years the graphics programmable processor (GPU) has evolved into an increasingly convincing computational resource for non graphics applications. The GPU is especially well suited to address problem sets expressed as data parallel computation with the same program executed on many data elements concurrently. In pursuing a scalable navigation planning approach for many thousands of agents in crowded game scenes, developers became more attracted to decomposable movement algorithms that lend to explicit parallelism. Pathfinding is one key computational intelligence action in games that is typified by intense search over sparse graph data structures. This paper describes an efficient GPU implementation of parallel global pathfinding using the CUDA programming environment, and demonstrates GPU performance scale advantage in executing an inherently irregular and divergent algorithm."
(Translation, "How to make a do CUDA pathfinding.  We will give a thread for every character and this is how. It is fast and we wrote a paper how to do it.")

C 200 pathfinding tasks do not need to touch each other in any way

D 200 pathfinding tasks will only need a byte for every step calculated at most... so... maybe 100K extra memory) bytes extra of memory to store the results temporarly... They need to store it anyhow to preserve integrity of the data in the other cuda pipelines.

E A 16x PCI Express bus has a massive amount of bandwith... around 4 GB/s to 16 GB/s Depending if you have Version 1,2, or 3.

Thats plenty to dump the entire memory of my machine 2 to 8 times in one second...  Or the entire dwarf fortress program at least 12 times a second... Now If the entire map has to be sent every time a frame comes up it would probably not work well, but sending a couple Kbytes of pathfinding data is trivial.

(For comparison the DDR 2 memory in my machine has a peak transfer rate of 6.4 GB a second. putting it inbetween V1 and V2 of the PCI express bandwith..  But way less than what PCI express V3 is capable of.)


You are correct that converting the code would be incredibly difficult if toady did not already say he wouldn't do it.

[The-Moon]

So just to Re-Explain what i posted and i am suggesting toady does....

What Toady needs to do to accomplish muti processor support is pretty simple and requires no real knowledge of threads in a single application at all. It is just learning how to use sockets.

Coding a single application, to use threads is good for certain things. But for games like Dwarf Fortress Using a server / client setup for the game is best.

If dwarf fortress is setup with the ability to use a Dual or Quad Core system that would be Fine. However if it is setup to use any amount of computers or processors, by using multipal applications. It would be able to acquire much more power then just a single computer can dish out.

A single "Server" application would be the core of dwarf fortress. The server application would be the app players interact with. It also is the main "Brain" of the game loading and storing all the information and figuring out what to do with that information.

The server would figure out how many processors the computer has. If the server is on a single processor, then it would then load up a single client application. If its a dual or quad core processor system, 1 or 3 client apps.

The client app would function as a worker unit. Taking orders from the brain / server. Then sending back the results to the server.

Depending on what functions a client app can take on is important. The client app needs to have access to the data the server has already loaded, without having to contact the server for that data. The client will need to sync with any data the server has loaded up when the client first starts up. Any time relevant data is changed which a client needs to know about, the server needs to update that client.

Clients should be able to take on all functions that the server would need it to do. Or it can be set to only perform certain functions. Taking on certain functions would minimize the extra ram usage.

Client Applications accept commands from the server application. The client then takes the command, lets just say for example, Path Find from 0,0,0 to 5,5,5. The client then calculates the path the unit needs to take, then sends back the list of data.

I Could really go on about this for a while but i assume anyone who cares and is reading up to this point has some kind of understand of what im talking about.

This system of course has alot of different things which needs to be thought out and setup so that it works well with DF. Every game is different and this system needs to be thought out and setup correctly to work correctly.


Pros:
>Ability to use any amount of Computers / Processors to run dwarf fortress.
>Being able to split up task between Computers / Processors.
>No annoying threads to worry about in one single application, and No memory errors from those threads if its not programmed correctly.
>Saves a lot of time using a client / server setup, from a normal thread setup in a single app. The single app is limited to one computer, while the client / server setup can be run on 100's of computers.
>No more multi threading, threads in the forums! Yey!
>Multiplayer of course

Cons:
>Extra Ram Usage
>May take some extra testing from us and toady to get working correctly.
>Toady has to learn sockets.... (Is that even a bad thing?)
>Toady will not be able to see annoying post in the forums about multi threading anymore.
>I'm really Trying to think up of cons honestly.....

I'm sure this all sounds really complicated and complex. But its not from the coding point of view. I haven't even skimmed the surface and there's too much to get into, i was just trying to give everyone a general idea of how this idea works and see what the responses are.

I really like this however. Not only with this idea are you able to use multi-processors on 1 computer. You can also use another computer to do work and such, since you are using sockets to talk from the server to client, all you need to do is contact another computer on the same lan line, or even over the internet. However, this method works best when all computers are connected together. Trying to use computers over the internet would defeat the purpose of this.

All the clients function the same. Depending on the computers resources and such would determine what the client does. The server should try to balance the work load to all available clients.

Don't know what much more i can say on this however right now. Anyone have any positive input for this idea?

Dwarf fortress is such a processor intensive game it only makes sense if you can connect it to other comptuers and use the resources from those computer also to run the game.

Allowing the computers to all share the work load.



With this system btw, another Pro. This would instantly make the game internet accessible, as well as pave the way for multiplayer support.

With the server client system i mention here in this thread. It could be expanded so that you could have a separate app which connects to the server app and interacts with the server.

You could have the game running at your home computer, a server app and client app running at home. Then from work, or where ever you are with your laptop, simply connect to your home computer with a separate application which runs on your laptop. You then with that program could then play the game from that laptop. The laptop wouldn't need to do much more then send and receive simple input commands from you, and sending you back visual data so you know whats going on.

The game can run from a lan network you have setup at home, while you sit and connect from the local mc donalds wireless network, and play DF without having to do a thing.

Its Great to Fantasize eh ?

Would love to see all this happen. Wouldnt take that long either and would really take DF to the next level and connecting everyone together more. People could run there own server networks and people could connect to that network from over the internet and play DF which someone else is hosting. Both being able to have there own citys / fortresses.

So many possibility's.....

[profit]

That would also pave the way for the user inteface not becoming bogged down due to the simulation bogging down... *really noticeable at FPS:0*

And would provide a neat and tidy interface for these external programs we all use *Like Dwarf TheRapist*

*not saying what you are suggesting is possible or the best way just if it was done those other things would become easy to do.

However, it will not happen =/  I am sorry but there is no other way of putting it. I do not see a single programmer able to stay mentally focused long enough to complete this task.
 There just are not enough milestones.  Toady needs mini goals to complete a task if you haven't noticed already by the development notes.

This would not have them. 
He could never complete it.