Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 12 13 [14] 15 16 ... 21

Author Topic: Will we ever get to a point where forts don't die FPS deaths?  (Read 65568 times)

adgriff2

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #195 on: August 09, 2010, 01:31:03 pm »

Adding support for multiple threads only takes a long time if the weaver isn't skilled.
Logged

MaDeR Levap

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #196 on: August 09, 2010, 01:33:39 pm »

Biggest problem: when main loop starts pathfinding threads and wait to complete all of them before doing other things, it can be easily ensured that memory is in consistent state. Unexpected changes in memory during access from other thread can lead to Very Bad Things, from stucked dorfs to seemingly unrelated sudden crash half of hour later and all in between. 0.31.1 bugginess would be very mild in comparison.
Agreed, that would be a problem IF you did nothing about it.
I do not think this would be optimal use of programmer time, to say at least. Most efficient and safe use would be main thread waiting for completing all pathfinding threads. Explanation below.

One method among several available being to have a separate set of pathing data that is updated on pathfinding thread startup based on data in a queue populated with any changes from the main thread.
And you claim this will be more effective and less problematic than just doing The Right Thing? "Queue populated with any changes from the main thread" sounds like waste of CPU cycles and memory space (what is average DF map nowadays, tens of megabytes?) in itself. Your pathfinding threads must be synchronized with your mysterious "set of pathing data that is updated on pathfinding thread" anyway - or you will have same problems as with taking data directly from main thread. More below.

The data the pathfinding threads are utilizing cannot be modified by the main thread and so there is no danger from data volatility.
However, data in main thread can and will change unpredictably from point of view of pathfinding thread. Armok forbid it will change during pathfinding nearby this particular tile that changed (simplest example). I have only very general concept about A*, but I imagine it will lead to Bad Things (especially with arcane optimisations). Other example, rather extreme: if memory will get reallocated for whatever reason, you are screwed completely. Crash time!

That data may be a frame or two old (at most)
You have no whasoever basis to saying that. Data that can be obsolete at all - can and will be arbitraily obsolete (many, many ticks ago). HilarityVery hard to track bugs ensues. I currently play fort with 15 FPS (yes, I am masohist, why you ask?). I hate to think how old pathfinding would be here.

Anyway, I really do not like concept where some things that happens in one ticks seeps into next ticks. This fact in itself can create big problems, unrelated to threading or pathfinding.

That situation is identical to a synchronous pathfinding call whose results are invalidated 1 frame later.
No, it is not. All things that happens in one tick are contained and predictable. If you start to doing things over two or more ticks, you open very, very big can of worms, or rather extremely nasty and esoteric bugs.

It is just not worth it.
« Last Edit: August 09, 2010, 01:40:21 pm by MaDeR Levap »
Logged

Makaze2048

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #197 on: August 09, 2010, 02:37:30 pm »

I do not think this would be optimal use of programmer time, to say at least. Most efficient and safe use would be main thread waiting for completing all pathfinding threads.
A valid argument and very possibly correct. I was only arguing that it was possible and a potential source of optimization. It's certainly got its pros and cons and the cons of potential loss of determinism, the load of ensuring non-volatile data, and most importantly increased programming effort may swamp any pros.

Quote
And you claim this will be more effective and less problematic than just doing The Right Thing?
Sorry, you're going to have to define "The Right Thing" for me here, I wasn't aware that there was a single correct choice for complex problems such as this.

Quote
"Queue populated with any changes from the main thread" sounds like waste of CPU cycles and memory space (what is average DF map nowadays, tens of megabytes?) in itself. Your pathfinding threads must be synchronized with your mysterious "set of pathing data that is updated on pathfinding thread" anyway - or you will have same problems as with taking data directly from main thread.
There is indeed overhead involved and that overhead may be greater than the gains realized. But I'm not talking about duplicating the entire world state to pathfinding. Merely a small subset or derivation of it that contains only the pathing costs. And then updating (nothing mysterious) that cost set via differentials whenever there are world changes such as constructions, mining, door locking, etc  that may affect pathing. Burrows, traffic designations, door ownership, and such may increase the size of that data set (or it may be possible to encode those flags into the upper bits of an int32 if max cost is a byte for example).

And remember I never said it's the best or most efficient method, only that it's *a* method that will technically work. There are likely superior choices if one looks hard enough.

Quote
However, data in main thread can and will change unpredictably from point of view of pathfinding thread. Armok forbid it will change during pathfinding nearby this particular tile that changed (simplest example). I have only very general concept about A*, but I imagine it will lead to Bad Things (especially with arcane optimisations). Other example, rather extreme: if memory will get reallocated for whatever reason, you are screwed completely. Crash time!
No, no it will not. That's the whole point of doing all of that complicated junk. You can argue that the method I've described is too inefficient, difficult to program, or without some additional changes causes a loss in determinism but you cannot reasonably say that it does not work or will cause a crash due to data volatility.

Quote
You have no whasoever basis to saying that. Data that can be obsolete at all - can and will be arbitraily obsolete (many, many ticks ago).
Sure I do, I can in fact simply say it by fiat by blocking the main thread if more than X frames have passed until requested paths are done. And can maintain determinism by always waiting X frames before using calced paths even if they're ready sooner. There is still a net gain in speed even when blocking the main thread like this.

Quote
I currently play fort with 15 FPS (yes, I am masohist, why you ask?). I hate to think how old pathfinding would be here.
Ironically under those circumstances (assuming you have a multicore machine) the pathfinding would likely complete even before the current frame is finished. A situation far more likely to occur when as you describe the single core the main thread is running on is swamped and other cores are almost freely available. And if setup properly where paths are requested near the beginning of the loop and utilized near the end then you'd actually use your asynchronous paths on the same frame they were requested.

Quote
Anyway, I really do not like concept where some things that happens in one ticks seeps into next ticks. This fact in itself can create big problems, unrelated to threading or pathfinding.
A reasonable position. I find myself going either way. If the game is currently 100% deterministic then yeah I'd say we need to preserve that (which we can still do with synchronous or even asynchronous pathfinding threads as detailed above). But if it's already not deterministic (and most games aren't, even when you think they are) then meh whatever works fastest and still produces reasonable results. Dwarves can be stupid enough now that I'm not sure anyone would actually notice if there were actually any problems.

Quote
That situation is identical to a synchronous pathfinding call whose results are invalidated 1 frame later.
No, it is not. All things that happens in one tick are contained and predictable. If you start to doing things over two or more ticks, you open very, very big can of worms, or rather extremely nasty and esoteric bugs.
Agreed, it is not 100% the same. It does cause determinism problems. But for most intents and purposes and specifically for the comment I was responding to which involved dwarves walking into walls that weren't there when their path was calculated then it is exactly the same thing.

Quote
It is just not worth it.
That may very well be true. I've no way of knowing without the source and even then it would probably be a judgment call.
Logged

Jayce

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #198 on: August 09, 2010, 05:29:16 pm »

Bring back magma that melts everything,even basalt is just lava thats cooled,placing basalt in lava should turn it back to lava.
Logged

MaDeR Levap

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #199 on: August 09, 2010, 05:47:46 pm »

Well, Makaze2048, it seems we will agree to disagree. I think there is no sense in even trying such thing, and you seem to think this is worth at least try. Well, that's it.
Logged

Makaze2048

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #200 on: August 09, 2010, 06:02:55 pm »

Well, Makaze2048, it seems we will agree to disagree.
Agreed.

This is just an intellectual exercise for me though. Toady has already publicly stated that threading out pathing ain't going to happen any time soon. And from a realistic perspective you're absolutely right that it would be far easier to do a single synchronous thread implementation that operates while the main thread is doing "other stuff" that can't update pathing data and that the vast majority of any gain is liable to be realized through that without resorting to crazy asynchronous voodoo.

I just like to think about what's possible even if it's not always practical :)
Logged

Vulcanius

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #201 on: August 11, 2010, 11:56:49 pm »

As a kernel dev I have to say that this thread has made me laugh, and cry. No amount of multithreading or OS optimization is going to fix a horrible algorithm. If I had it my way I'd see http://www.bay12games.com/dwarves/dev.html replaced with simply, "Refactor", for about two or three months. I love having new features and all, but giving a troll new clothes, make-up, and hair plugs doesn't change the fact that it's still an ugly f'ing troll.
Logged

Macalano

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #202 on: August 12, 2010, 01:59:03 am »

I think Dwarf fortress wins, the award, for biggest gap between system requirements and system recommended requirements.
Logged

MaDeR Levap

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #203 on: August 12, 2010, 04:54:50 am »

No amount of multithreading or OS optimization is going to fix a horrible algorithm.
I do not think pathing is that bad (obviously it could use more work, but what thing wouldn't use more work?). I think that other FPS-killing things was neglected, mainly effective management of game object (these tens of thousand rocks...).

refactor
Oh, this would be Fun. And no, I do not think two or three months would be enough for whole game. And refactoring is unrewarding job: you change code to make it easier to change or extend it in future. So after refactoring, after few months all will work same for player (except plethora of new bugs). This will not fly with this particular fanbase. I see only one solution for this for Toady: take lessons learned while making DF to heart, when (not if, when) he will start Chapter III. I predict abandoment of DF around 0.45 due to multiple issues reinforcing each other in tangled mess.

On related note, I really hope he uses his secret end-month project as opportunity to learn about things like proper multithreading. In other case (and I have very real fear that is case)... lets be blunt: he just waste time.

I think Dwarf fortress wins, the award, for biggest gap between system requirements and system recommended requirements.
Choking computers is easy. And this is niche indie game with rather insane fanbase. Toady can and will NOT care about trivialites like computing power of average computer (beyond complete neccessity like program running at all). Results are known very well.
Logged

Hammurabi

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #204 on: August 12, 2010, 09:18:34 am »

No amount of multithreading or OS optimization is going to fix a horrible algorithm.
I do not think pathing is that bad (obviously it could use more work, but what thing wouldn't use more work?). I think that other FPS-killing things was neglected, mainly effective management of game object (these tens of thousand rocks...).

Libtcod can manage the pathfinding of a thousand creatures with no apparent slowdown.  A* is very efficient, implemented correctly.  But there are many ways to screw it up, like a bad heuristic.  One hundred thousand inert rocks should have no effect on FPS.  The game has no need to examine them each frame.
Logged
Back in 1971, Nolan Bushnell of Atari said, "All the best games are easy to learn, and difficult to master," a design philosophy now treated as instinctual by nearly every designer in the industry.

tps12

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #205 on: August 12, 2010, 09:35:59 am »

One hundred thousand inert rocks should have no effect on FPS.  The game has no need to examine them each frame.

Right, but they aren't "inert"; they have material properties that cause them to respond to changes in the world.
Logged

Urist McDepravity

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #206 on: August 12, 2010, 09:40:39 am »

No amount of multithreading or OS optimization is going to fix a horrible algorithm. If I had it my way I'd see http://www.bay12games.com/dwarves/dev.html replaced with simply, "Refactor", for about two or three months. I love having new features and all, but giving a troll new clothes, make-up, and hair plugs doesn't change the fact that it's still an ugly f'ing troll.
Oh, you know, you should go to these morons who use clusters for physics/astro/bio simulations and tell them this. Wasting all those petaflops on something that could be 'refactored', eh...
Logged

Makaze2048

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #207 on: August 12, 2010, 11:01:49 am »

Libtcod can manage the pathfinding of a thousand creatures with no apparent slowdown.
That's a fairly meaningless comment without including the size and complexity of the path map and the heuristic involved. No doubt DF path finding could probably be done better but it's not quite as simple as waving the generic A* wand and pretending that it does everything that needed and runs instantly.

Quote
One hundred thousand inert rocks should have no effect on FPS.  The game has no need to examine them each frame.
Except when a mason needs a rock the game paths to each and every rock out there to determine the nearest one of each type.

But even ignoring pathing running a lot of poorly optimized N2 operations on the item list (or DwarfList*ItemList or TaskList*ItemList etc.) will swiftly bring things to a crawl as Items, Tasks, and Dwarves(who come with items and make tasks and items) increase. Pathing got a lot of discussion in this thread but it's far from the only thing slowing DF down as forts get bigger.

Right, but they aren't "inert"; they have material properties that cause them to respond to changes in the world.
Exactly, the temperature model is effecting each of these items every frame which means at very least an N crawl of the item list every frame.
« Last Edit: August 12, 2010, 11:15:32 am by Makaze2048 »
Logged

Hammurabi

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #208 on: August 12, 2010, 11:54:42 am »

Right, but they aren't "inert"; they have material properties that cause them to respond to changes in the world.
Exactly, the temperature model is effecting each of these items every frame which means at very least an N crawl of the item list every frame.

Since the only thing rocks can do is melt in magma, why not just check the squares magma is in (or next to)?  Why does the temp of every rock on z-1 need to be checked every frame when the magma is on z-50? 

Logged
Back in 1971, Nolan Bushnell of Atari said, "All the best games are easy to learn, and difficult to master," a design philosophy now treated as instinctual by nearly every designer in the industry.

Makaze2048

  • Bay Watcher
    • View Profile
Re: Will we ever get to a point where forts don't die FPS deaths?
« Reply #209 on: August 12, 2010, 12:36:11 pm »

Since the only thing rocks can do is melt in magma, why not just check the squares magma is in (or next to)?  Why does the temp of every rock on z-1 need to be checked every frame when the magma is on z-50?
Because I don't think it works quite like that. It's not looping through just rocks, it's looping through everything which includes water, mined ice, clothes, lignite cabinets, wood, everything.

Though in this case I think it's actually looping through all squares and accessing each item in a square which is actually slower if it has to then go find the item by index in an unsorted item list.
Logged
Pages: 1 ... 12 13 [14] 15 16 ... 21