Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 2 3 [4] 5

Author Topic: Hardware and Game Performance  (Read 10461 times)

athenalras

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #45 on: December 11, 2015, 12:09:41 pm »

If you want Toady to make a multi-threaded release. Create a kickstarter in his name and raise a bunch of money. Right now DF raises between $45,000 - $60,000/year. This is pay without benefits and split between 2 people. He also has expenses to cover for his website, etc... This is far less than an entry level developer makes.

Raise $100k for Toady under the condition that he make a multi-threaded release then get back to him. Cut the guy some slack. Financially he is barely getting by doing this.

It is not as if I'm oblivious of how Toady makes a living.

What I am saying is that he will have face the problem sooner or later by either
1. Biting the bullet and reworking the code for multi-threading and multi-core processors by himself

2. Get some outside help to rework the coding for multi-threading while Toady works on more important features in single-thread programming.

This is the reality and I'm already cutting him slack by saying "Its alright no matter what you choose to do". And the reality is that he will have determine how to tackle this problem because obviously he is very reluctant to retrofit DF entirely on his own.
Logged

guessingo

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #46 on: December 11, 2015, 03:02:58 pm »

Where would toady get the money for outside help? You would need a kickstarter for him to do that. He also said he doesn't know how to code with other people. He has never had a coding job working with anyone else. He doesn't even use source control. He is 100% self taught and 100% working by himself. So he doesn't even have the 'learn from other people's code'. You also dont know what his code looks like to even know how hard it would be to thread. This is old code that he is reworking. Its likely fairly messy. He does not have any experience working with other people. You kind of need to use source control to work with other people. I also think he just prefers to work alone and with his brother. Its in his personality. Seems like a nice guy. Some people just like doing their own thing.

I think we should all understand the constraints he is under financially, no training in coding, and no experience working with other people (and seeing other people's code to see how they do things).

if people want him to multi-thread. Do a campaign to get people to pay more to his patreon account or something to show it financially. I think Toady could probably get $100,000 at least in a kickstarter for multi threadng, but its not in his personality. I base that on ADOM getting $90,000 and I don't think that is as popular. Either that or Tweet Notch. He sited Dwarf Fortress as one of his inspirations. He is loaded and apparently bored. Maybe he needs a project to invest in. Joking, but you know what I mean.

If you really want multi-threaded programming. Make a financial push and get some of the core fans to start a financial campaign. I don't think its in Toady's personality to do this himself.
Logged

Veroule

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #47 on: December 11, 2015, 04:16:42 pm »

Conceptually it is not that difficult to split path finding off into separate threads.
1. Assume the current modified A* continues to be used.
2. Each path finding calculation is done in its own thread.
3. Creatures initiate path finding, and display a thinking symbol until their path thread completes.
4. Creatures that change their mind about needing that path can do so since their thoughts and actions are still being processed on the main thread.
5. A creature that changes its mind sets an abort signal for its path finding thread.
6. The main thread will have map changing actions, some of which are of a significant enough level that all path finding should be held while the change occurs. The mechanics of this are not difficult. The main thread must know how many path finding threads are running. An atomic flag is set to tell each of the threads to pause, each thread responds to that pause with an increment to an atomic counter. When the counter is equal to the number of active path finding threads the changes can proceed, then the flag is cleared to allow the threads to continue.
7. Each of the threads on continuing increments its own internal counter for how man times the map has changed during its calculations. Each location that is added to the A* calculation would record what value that map change counter was at when it was added. When considering adding further locations from a given spot the counter would be checked. If the current change count is higher than that location the validity is checked and if the location is no longer valid all potential additions from that location are removed. This can lead to an incomplete path failing to reach its destination, and would start over from the source spot with the change counter reset to 0. If checking connectivity is fast enough then can be done with each map change or only when a starting over. This can result in some sub optimal paths when map changes occur during calculation, and so what dwarves do dumb stuff involving lots of extra walking all the time.

I think that covers all the basic conceptual components of switching path finding to multiple threads. I barely touched on implementation. The implementation would involve major changes. Among the first of which is to assure that any map changes that occur will not invalidate any pointer or array index held by the path finding calculations. The next major implementation hurdle is changing the creature behavior so that the main loop has a periodic check for did the path finding complete. The list of implementation details which can decide whether speed is gained numbers in the hundreds, if not thousands. If you look at what I said in conceptual item 6, you might notice that I am describing peeling apart a mutex structure into a combination group mutex and semi semaphore. Writing implementation details like that only comes from years of experience working with multi threading. Without that experience a programmer will tend to select from an existing mutex or semaphore and may even have the execution time that a user can perceive increase (it looks slower).

Overall, I think there is very little noticeable speed to be gained by splitting path finding off in this fashion. Threading would make a difference at specific moments when many path requests occur all at once, but at all other times the apparent gains would be minimal. Writing multi-threaded code has a steep learning curve and add whole new layers of complexity to debugging. While the path finding is a good candidate for asynchronous calculation, doing so would likely require Toady to share some portions of his existing code with a programmer that is experienced with the pitfalls of threads. Toady could then get a least enough advice to avoid some of the pitfalls. It would be a good experience for Toady and might lead to him recognizing other areas of DF that are good candidate sections for multi threading.

Toady does have a little experience working with others. BHeyler made the first SDL port, working from the Battle Champs source which Toady released specifically for that reason. That port lead to Linux and Mac versions by becoming a separate library section the source of which is included in every DF release. Baughn optomized the openGL code in that port. I recognized that the keyboard and mouse connections were a major time sink and rewrote them. Most of my code was ultimately scrapped because of style issues and availability to debug, Baughn wrote a replacement for what I had done with a similar design basis. Coordinating specific code would be difficult for Toady, but getting advice on how to do some things might not be as tough.
Logged
"Please, spare us additional torture; and just euthanise yourselves."
Delivered by Tim Curry of Clue as a parody of the lead ass from American Idol in the show Psych.

Geltor

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #48 on: December 11, 2015, 04:36:00 pm »

Where would toady get the money for outside help? You would need a kickstarter for him to do that. He also said he doesn't know how to code with other people. He has never had a coding job working with anyone else. He doesn't even use source control. He is 100% self taught and 100% working by himself. So he doesn't even have the 'learn from other people's code'. You also dont know what his code looks like to even know how hard it would be to thread. This is old code that he is reworking. Its likely fairly messy. He does not have any experience working with other people. You kind of need to use source control to work with other people. I also think he just prefers to work alone and with his brother. Its in his personality. Seems like a nice guy. Some people just like doing their own thing.

I think we should all understand the constraints he is under financially, no training in coding, and no experience working with other people (and seeing other people's code to see how they do things).

if people want him to multi-thread. Do a campaign to get people to pay more to his patreon account or something to show it financially. I think Toady could probably get $100,000 at least in a kickstarter for multi threadng, but its not in his personality. I base that on ADOM getting $90,000 and I don't think that is as popular. Either that or Tweet Notch. He sited Dwarf Fortress as one of his inspirations. He is loaded and apparently bored. Maybe he needs a project to invest in. Joking, but you know what I mean.

If you really want multi-threaded programming. Make a financial push and get some of the core fans to start a financial campaign. I don't think its in Toady's personality to do this himself.
keep in mind that when i brought up multithreading, i did not mean to retrofit everything, only to split pathfinding into threads.
Logged

Inarius

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #49 on: December 11, 2015, 04:54:02 pm »

Quote
if people want him to multi-thread. Do a campaign to get people to pay more to his patreon account or something to show it financially. I think Toady could probably get $100,000 at least in a kickstarter for multi threadng, but its not in his personality. I base that on ADOM getting $90,000 and I don't think that is as popular. Either that or Tweet Notch. He sited Dwarf Fortress as one of his inspirations. He is loaded and apparently bored. Maybe he needs a project to invest in. Joking, but you know what I mean.

If you really want multi-threaded programming. Make a financial push and get some of the core fans to start a financial campaign. I don't think its in Toady's personality to do this himself.

I think this is really true. For what I have seen in videos and all of this, Toady seems to be a good guy, but clearly he is not a "charismatic leader", able to bring load of cash. And I think that DF is as him. Genious, but not catchy. He is just someone who likes to code, and thanks to the community, he can do it everyday.

It's to us, community, to help him. I really don't think he has the capacity (it's not an intellectual issue, more related to how he is) to do a lot of communication to make a 100k campain on kickstarter. But I'm sure that, among the players, there are enough people to do it.
Do you imagine him asking Notch for money ? It can't be. It's up to us to do this.
Logged

athenalras

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #50 on: December 13, 2015, 10:28:46 pm »

Again, people don't get my point.

[Insert Quality] or not [Insert opposite quality], the point I'm making here, to be more or less blunt, is that Toady and the rest of the community should be aware of what would probably happen in the near or not near future.

-> At the rate DF development is coming along and the rate at which single-core processor technology is progressing, DF is going to hit a big, thick wall.

When this happens, it is not just finance that Toady has to deal with. It is the fact that he may eventually have to deal with having employees to make any transitions from a single core DF to multi-core DF.

Retrofitting DF to be multi-core compatible along with his x64 version may make things easier on him when the wall appears.
Logged

Button

  • Bay Watcher
  • Plants Specialist
    • View Profile
Re: Hardware and Game Performance
« Reply #51 on: December 14, 2015, 12:27:21 pm »

When this happens, it is not just finance that Toady has to deal with. It is the fact that he may eventually have to deal with having employees to make any transitions from a single core DF to multi-core DF.

Why do you think he would "have to" deal with having employees? What do you think could happen to "force" him to hire outside help?

Toady doesn't have customers, he has patrons. He's been working on a patronage model for much longer than most on that model have been, and his patrons are in it for the long term. We know how he works and what to expect. What, do you think people would stop donating as he went through a years-long slog to transition DF to multi-core? Cause there have been years-long slogs to transition DF to e.g. dynamic world, and people didn't stop donating.

Now sure, something horrible might happen that would rock the boat of Toady's patronage model. But I don't see how that rocked boat could possibly lead to hiring more coders. An emergency bad enough to require Toady to change his business model - that is, bad enough or long-term enough that his patrons wouldn't/couldn't cover it - would reduce his available cash, not increase it. That would preclude hiring other developers in itself.

I could see Toady selling off the rights to Dwarf Fortress in a time of need. But I can't imagine any circumstance in which he'd have to hire people to help him with multithreading.
Logged
I used to work on Modest Mod and Plant Fixes.

Always assume I'm not seriously back

Ghills

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #52 on: December 14, 2015, 03:08:11 pm »

When this happens, it is not just finance that Toady has to deal with. It is the fact that he may eventually have to deal with having employees to make any transitions from a single core DF to multi-core DF.

Why do you think he would "have to" deal with having employees? What do you think could happen to "force" him to hire outside help?

Toady doesn't have customers, he has patrons. He's been working on a patronage model for much longer than most on that model have been, and his patrons are in it for the long term. We know how he works and what to expect. What, do you think people would stop donating as he went through a years-long slog to transition DF to multi-core? Cause there have been years-long slogs to transition DF to e.g. dynamic world, and people didn't stop donating.

Now sure, something horrible might happen that would rock the boat of Toady's patronage model. But I don't see how that rocked boat could possibly lead to hiring more coders. An emergency bad enough to require Toady to change his business model - that is, bad enough or long-term enough that his patrons wouldn't/couldn't cover it - would reduce his available cash, not increase it. That would preclude hiring other developers in itself.

I could see Toady selling off the rights to Dwarf Fortress in a time of need. But I can't imagine any circumstance in which he'd have to hire people to help him with multithreading.

athenalras' point is that single-threaded code is going to hit a performance wall and if Toady isn't willing to multi-thread he'll have to get help.

Single-threaded performance is already a non-priority for processor research.  The reality of hardware is changing and DF is built on old assumptions.  Sooner or later Toady will face a choice between course correcting or seeing DF die off.  The longer he delays the worse it will be.
Logged
I AM POINTY DEATH INCARNATE
Ye know, being an usurper overseer gone mad with power isn't too bad. It's honestly not that different from being a normal overseer.
To summarize:
They do an epic face. If that fails, they beat said object to death with their beard.

Ghills

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #53 on: December 14, 2015, 03:08:55 pm »

I wonder how feasible it would be to run the external world stuff on a seperate thread.  The timing checks don't have to be super precise, or at least it doesnt seem to me to be so critical, so there is a certain amount of leeway that might be possible.

I do think Toady has a pretty big scaleability problem going on with his game right now.  As he keeps adding cool things his fps has been steadily dropping, which means we have to resort to mods and reducing our embark site size and/or number of dwarves.

Meanwhile modern CPUs are not giving him any performance gains (my 5 year old laptop runs the game almost as well as my top of the line desktop).  This will have to be addressed pretty soon.

This could be a good idea, depending on how the external stuff is implemented. 
Logged
I AM POINTY DEATH INCARNATE
Ye know, being an usurper overseer gone mad with power isn't too bad. It's honestly not that different from being a normal overseer.
To summarize:
They do an epic face. If that fails, they beat said object to death with their beard.

Button

  • Bay Watcher
  • Plants Specialist
    • View Profile
Re: Hardware and Game Performance
« Reply #54 on: December 14, 2015, 06:24:03 pm »

athenalras' point is that single-threaded code is going to hit a performance wall and if Toady isn't willing to multi-thread he'll have to get help.

Single-threaded performance is already a non-priority for processor research.  The reality of hardware is changing and DF is built on old assumptions.  Sooner or later Toady will face a choice between course correcting or seeing DF die off.  The longer he delays the worse it will be.

And my point was that that performance wall doesn't mean he'll have to get help - doesn't even mean he'll have to bite the bullet and multithread. Dwarf Fortress has been well behind industry standards since its inception - see, for example, the graphics. And yet, the ever-improving standards of computer graphics and graphics cards have not killed Dwarf Fortress, even as it lags so far behind them, because Dwarf Fortress isn't about graphics. As long as its graphics are good enough to be understood, it's enough.

Dwarf Fortress also isn't about performance, or real-time/twitch play. Same argument.

The Dwarf Fortress fanbase, or at least Toady's income stream, has only grown over the years, even as the technology behind DF has become more and more out-of-date. Sure, it'll probably eventually peak. But I don't think a lack of multithreading is what's going to do it. We don't play Dwarf Fortress for pretty graphics or smooth gameplay. We play to lose.

***

Also, and this is just a fun tangent, I wouldn't be surprised if automatic multi-threading becomes a thing in the next 20 years and makes the whole subgenre of multi-threaded programming something you only look into if you're trying to optimize performance - kind of like what garbage collection did to memory management.
Logged
I used to work on Modest Mod and Plant Fixes.

Always assume I'm not seriously back

Thief^

  • Bay Watcher
  • Official crazy person
    • View Profile
Re: Hardware and Game Performance
« Reply #55 on: December 15, 2015, 02:46:07 am »

There already is automatic multithreading of trivial code, e.g. accumulation loops (incidentally the same loops that can be automatically vectorized). But any nontrivial code (non-loops or loops that access other elements during the loop) will likely never be automatically multithreaded.

Edit: there's a whole subgenre of programming which is easily multi threaded, and that is functional programming. Given Toady's mathematics background (IIRC), he may even be familiar.
« Last Edit: December 15, 2015, 02:49:05 am by Thief^ »
Logged
Dwarven blood types are not A, B, AB, O but Ale, Wine, Beer, Rum, Whisky and so forth.
It's not an embark so much as seven dwarves having a simultaneous strange mood and going off to build an artifact fortress that menaces with spikes of awesome and hanging rings of death.

jaked122

  • Bay Watcher
  • [PREFSTRING:Lurker tendancies]
    • View Profile
Re: Hardware and Game Performance
« Reply #56 on: December 15, 2015, 03:06:01 am »

Eh, as it turns out, it's not the path-finding which makes the parallelization hard, deciding which dwarves use which items seems to be a task for which it is difficult to make concurrent, at least without throwing out a lot of the guarantees that exist as of now.


Pathfinding requires duplication of the entire graph in order to do across multiple threads, I don't know how weighty the map itself is, but it's nothing to sneeze at.


Also functional programming does not make hard problems in multi-threading easier, it just helps ensure that you don't screw up and get all sorts of nasty race conditions. It does make a lot of things more difficult to implement, in particular, graph transversal is a bit troublesome.


Also that kind of automation would likely require rewriting the game in a different language. That'd involve some very extensive refactoring and reworking and retooling.

Thief^

  • Bay Watcher
  • Official crazy person
    • View Profile
Re: Hardware and Game Performance
« Reply #57 on: December 15, 2015, 08:00:12 am »

In fact most of these suggestions for "easily" multithreading some aspect of DF actually turn out to require rewriting significant parts of the DF code when thought about...
Logged
Dwarven blood types are not A, B, AB, O but Ale, Wine, Beer, Rum, Whisky and so forth.
It's not an embark so much as seven dwarves having a simultaneous strange mood and going off to build an artifact fortress that menaces with spikes of awesome and hanging rings of death.

MobRules

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #58 on: December 15, 2015, 09:57:26 am »

I'm reminded of seeing Fowler speak at a conference. He said: "Concurrency. Concurrency is like a nice, fluffy bunny rabbit -- with sharp, pointy teeth and a mean streak a mile wide." (The second half of the quote was enthusiastically accompanied by the appropriate hand/body movements.)

The point being, concurrency problems often look fairly safe/simple at first glance... until you start actually trying to work on them.
« Last Edit: December 15, 2015, 10:00:18 am by MobRules »
Logged
Ninja dragons! Protect the masterwork roasts!
Is this biome reanimating? I really don't want to know what happens when "absurd numbers of megabeasts" is combined with "reanimating biomes".

Inarius

  • Bay Watcher
    • View Profile
Re: Hardware and Game Performance
« Reply #59 on: December 15, 2015, 12:46:43 pm »

The point being, concurrency problems often look fairly safe/simple at first glance... until you start actually trying to work on them.

I cannot agree more. I think that a big part of us has already coded, but for the part who doesn't, know it : multithreading can be a real nightmare.
Logged
Pages: 1 2 3 [4] 5