Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[King Mir]

You might be able to make DF more parallel without much effort, but in the process I you'd just make it slower. Why? Firstly, you'd effectively cripple the L1 and L2 caches, because of the memory synchronization you'd need to do. Secondly, all those threads running on different cores will bottleneck on the single memory bus they share, so the execution won't really be parallel for large portions. Third, the synchronization needed just to make sure the everything happens in one tick, before the next is started would be too much of a penalty.

For efficient parallel execution you need the gains from parallelization to be greater than the costs, and with DF, that's not going to be easy.

[PatrikLundell]

@wierd:
As far as I understand from the what's exposed by DFHack, all, or nearly all, the DF data sits in a huge data structure, and since DF is written to be single threaded, my guess is that each piece of code that needs to read or write something just goes directly to the data structure to read/write it, so each of those accesses would need to be replaced by calls to access operations instead, and these accesses are likely a significant part of the code. Parts of a rewrite would probably include moving direct access to data at the points used/produced into collection of data to be passed as parameter data to the various operations, and the results being collected as a result structure at the end of each threadlet.
However, it partially depends on how far you drive the parallelism. If you take path finding, for instance, that's probably implemented as some kind of iteration over units that need path finding done during the current tick. The "input" data (i.e. the data used, such as terrain, movement costs, etc.) is probably not modified by that piece of code, while the results are probably somehow attached to each unit data representation, where the units are separate entities (sort of leaves of the big data structure). In that case it ought to be reasonably straightforward to send off the calculations to parallel threads because they wouldn't interfere (and technically you wouldn't need to protect any data access since the data several threads may access is read only). That seems to be what you propose, and, if so, we agree on that, it might be possible via patching (or a limited targeted effort by Toady). Parallelizing path finding with fluid movement, however, would probably result in varying results because of when a path happens to check a tile where fluid may slosh, and so either require some kind of protection, a sanity check that the paths are still vaild (probably a lot cheaper than calculating them in the first place) with a recalculation on a rejection, or a conscious decision that it doesn't matter.

Will parallelizing e.g. path finding help significantly though? If the average number of units that need path finding at any given tick is less than 1 the answer is probably no, and the additional administration could even lead to an overall performance loss, but on the other hand it might help larger fortresses, which are the ones worst hit by FPS losses (after all, if DF ends up waiting for the next tick to begin while running at maximum speed, it doesn't matter that you wasted a few CPU cycles).

King Mir posted while I wrote:
Parallelizing should always depend on whether you make a significant net gain. In the "simple" case of farming out the various iterations of an iterative action (e.g. the path finding above), the various threads may well use the same data from the cashes (which the current iteration would do as well). I agree, however, that DF probably is fairly memory bound, so massive parallelization may only result in additional administration and waiting for the memory bus.
In addition to determining whether there's a net gain, you also have to determine for whom there's a gain. A change that speeds things up on a bleeding edge desktop machine with cores to spare might slow an already crawling DF down even further on an old laptop. On the other hand, something that slows down small new fortresses might speed up old large ones, which are the ones in the most need of an improvement.

[wierd]

This is anectdotal, so not really data--

On my i5 and i7 systems, DF seems to be memory bound. I say this, because the running instance of DF will slow down other tasks, even though DF only saturates one core. The speed of these systems is between 2 and 3ghz.

On the flipside,  I recently picked up a weak-kneed chromebook.  It sports an integrated intel celeron processor that is crippled in just about every way imaginable. It runs at about 1.6ghz, and has 2 cores.  The speed of the ram does not seem to be the limiting factor here, as the running instance of DF does not slow down other processes.

Most user systems will likely favor the two systems mentioned first. 

Determining which obstacle the process will run into first should be pretty simple, if we learn where the cutoff is.  On weaksauce systems like this chromebook, having another worker thread would speed things up quite a lot.  On stronger systems, like those 3ghz chips my home systems are using, more cores being thrown in wont be helpful.  A quick and dirty IO vs raw CPU test cycle in the patch would allow the main process to know if it should spawn additional workers or not.

Sadly, most of the lower end chips that would benefit from parallelism are only dual core. (like the "dorito" inside my chromebook.)  One core would need to be running 2 threads at least, for there to be a performance increase, and hyperthreadding is only possible when the threads are not competing for computational resources. That would be unlikely to happen with DF.


That wasn't really the point though.  The point was that a really feverishly dedicated programmer could make it happen, not that it would be ideal.

Demonstrating areas where multiprocessing is possible, and identifying areas where the best performance increases can be squeezed out would let toady better know where he can target it in his source tree for proper support. 

Toady is a big fan of grabbing low-hanging fruit.  This kind of endeavor is more to pull a tall branch down lower, so it is more easily reached.

[King Mir]

It wouldn't make any sense to try to go part way to parallelism, when you can't be sure that going all the way would get you any performance gain. When optimizing performance like this it's important to profile to verify that the changes actually lead to an improvement, and that would be impossible if you're releasing a patches that don't in isolation have performance gain.

[FantasticDorf]

I know for a fact that the "dying > going to an afterlife > trying to make your way back" is at least a general sort of thing the game is heading towards, it's partially covered in the Cado story.

Myself I'm still hoping that it will be simple enough and agreeable to add an -o or --output-offsets=[range|list|/link/to/target/filename.format] type option.

I personally like to think of (somewhat rhetorical, not actual questions for toady though i might touch on them later) questions like this surrounding that topic.

A. What if in myth generation dwarves go to the underworld when they die and 'resurrect'/change forms and transfer their knowledge directly as new demons or some kind of intelligent undead/creature whilst persisting as themselves/mind-wiped of their old identity.

B. How many candy wearing artifact dwarves/continuous fortress efforts would it require to fight the demons of the underworld and save their souls with the power of clergy or atleast free them of their rest. (literally go on a quest to save a crucial/favourite dwarf from hell)

C. How might i stop a portal full of demons and souls of the undead trying to free themselves (or by somebody else on the outside, very diabloesque) shambling out of the underworld.

> Cue music https://www.youtube.com/watch?v=GgnVr9xUNAE

Ok actual question for toady now

> Do you have any thoughts following up the myth generator establishing 'planes of existance' as places, about applying the same treatment as caverns to the underworld? For example, defining areas so that in the case that a portal was cast (one way there or back / two way) you could identify where you were broadly going to turn up, instead of randomly appearing anywhere in the game world potentially in extreme danger

&

> In the myth creator with adjustments or definitions to how demons come to be (be it spontaneous or methodically created from game settings such as the death of a mortal/particularly 'evil' mortal) will this affect the localised populations of demons in the underworld to the point where after a long amount of time you might actually destroy them all or steady the flow of them in a manner you can account for, allowing dwarves to colonise at that depth with time and investement.

[PatrikLundell]

@FantasticDorf: Question 2: It's already possible to settle in that area with time and investment. However, it seems that area is up for some serious rework as the good/evil gets reworked/replaced by spheres, so my guess is that the answer will be "We'll see how the rework pans out. Nothing set in stone yet", or possibly that some ideas are tossed around.

[FantasticDorf]

@FantasticDorf: Question 2: It's already possible to settle in that area with time and investment. However, it seems that area is up for some serious rework as the good/evil gets reworked/replaced by spheres, so my guess is that the answer will be "We'll see how the rework pans out. Nothing set in stone yet", or possibly that some ideas are tossed around.

Im fully aware people have 'conquered hell' in game, especially by walling it off, but moreso the question was directed at literally killing everything on that layer. More to stabilise so that even with no walls that layer people could retire fortresses or go travel around in adventure mode separately with the world ticking by without being mobbed by 500 million demons in the next in-game week. It would just be empty or the demon population thinned out/clumped up in another area of the underground in the same kind of way populations decline and recover right now, you can totally clear your embark world tile of bears but there will still be bears in the woods next-door to help the world count of bears.

Demons right now can't be exterminated because they have the same kind of values as vermin for population omnipresence attached with coming in from that layer/edge of the world map or embark map in droves. I am certain that you are right patrick in that spheres and whatever magic/mythos implications may take part but from a current point of view im talking about ideas to unify that plane with the kind of systems we have on the overworld (identifying areas, population movement/density and legends logs) and use the mythlogical generator in order to establish sets of game rules that people can read and understand how planes and its dezinens work. I can't say that the latter part of your answer was really helpful but yes some ideas especially on reworks will probably be put around, i agree upcoming and prior to the new versions ahead.

[PatrikLundell]

As far as I've seen, the underworld is governed by the same rules as the layers above in that a group comes in and nothing else comes in until that group has left (including the usual camper issue) when you've dealt with the invasion force. In fact, it seems to be MORE benign that the layers above in that I've never seen two groups at the same time even though the embark is completely savage.

It's true there's a difference in that demons can't be exterminated, though, but when you've dealt with the invaders you have a controlled flow as on the other layers.

However, it will be interesting to see how the area/concept is reworked, as, frankly, it's a bit boring currently. Obviously, it's a significant challenge, which is good, but there's currently not much of a reason to go there except for bragging rights (to yourself or to others), as nothing interesting grows or lives there (as in "something you can domesticate" or otherwise get something unique out of).
Conquest/extermination/taming would be interesting if there's a reason to do it beyond it being possible to do.

[FantasticDorf]

However, it will be interesting to see how the area/concept is reworked, as, frankly, it's a bit boring currently. Obviously, it's a significant challenge, which is good, but there's currently not much of a reason to go there except for bragging rights (to yourself or to others), as nothing interesting grows or lives there (as in "something you can domesticate" or otherwise get something unique out of).
Conquest/extermination/taming would be interesting if there's a reason to do it beyond it being possible to do.

It sounds like a perfectly reasonable excuse to drop some hell-hounds at the lowest depths and define some more fauna and geological graphical features (Non roof candy veins, etc), demons in some respects (given if you can't colonise heaven and poke your myth gods or all the angels in the eye with a admantium pike) are the 'final boss' of DF at current as its not really intended to go any further and you are rewarded for your troubles with the 'bigbad' opponents, usually set to finish the game of a intermediate non-hardcore cheesing player.

Im pretty sure its possible to mod in creatures to populate the underworld, but obviously the limitation on that is that if they aren't also unnumerable they just get wiped out by demons without being overpowered. Immediately using the bottomless pits to dispose of waste trash seems like a good idea, as does literally killing all the demons and other magical incentive objectives in order to literally disrupt politics aboveground or go treasure hunting (vaults etc) on that plane in peace. It also puts a lot of distance into a inhospitable area to hide in further away from threats on the surface or to lure enemies there.

[PatrikLundell]

I have to admit I rarely think of the adventure/fortress interaction as I don't play the adventure mode, but yes, messing with politics and clearing the way for adventurers sounds like valid reasons (and with expanded off site interaction politics messing might become possible even in pure fortress mode).
Luring enemies down there you be useful only if you haven't cleared the area, though, but still have sufficient control that you can unleash some critters on invaders (and you can do that already, since the locals can breed, so you just need to capture a bunch and then have them multiply in a location you can unleash them from, optionally with an idea of how to put the cork back in the bottle.
If the 3:rd cavern isn't far away from enemies, I doubt a few additional Z levels would make much of a difference when it comes to hiding, though, unless the SMR acted like a non detection barrier of some kind.

[Taederias]

Events that aren't seen by an escaping witness don't become common knowledge until a year later, I think, and are never passed around as rumors.

Does this mean that we can't really get away with theft/murder scot-free? People will always learn of what happened in time? How do we hide something, then?

They don't end up blaming you -- it's just a hard problem to keep things completely secret from all the game's systems forever.  The historical event becomes known, but the incident report and rumor don't get converted into reputation changes.  It'd be ideal to keep track of who knows what about everything, but it's not feasible memory-wise.  So you might see supposedly unknown events bleeding into artwork and conversations after a long while.  There could be additional controls put into place, but it's all hard work for less return as you go.

I wonder, just as a rough estimate, what amount of memory would be necessary to implement a more expansive knowledge system, even just partially? Like perhaps you can have lots of more significant historical events generally known, or at most tied to whatever civilization/culture said NPC belongs, what position they hold etc., but for some smaller, more recent events, especially those related to the player (though I kind of see how making such distinctions could be difficult) the obtaining and exchange of information would be stored & simulated on a personal basis.

I imagine this could also tie in to a future release concerning laws and create cool new gameplay mechanics, where for example you could threaten or bribe a witness of a murder or theft not to tell anyone, but depending on how much they fear you, they might incriminate you anyway, or just tell some trusted friend, and this way the information might get out eventually anyway (or you could smack them on the head and hope they forget everything... lol). Hell, you could even implement spreading false information (possibly for both the player and NPCs?), which could open up a whooole lot of new possibilities (up to maneuvering entire civilizations into war based on clever lies told to important people).

At any rate, I'm asking because I imagine with the change to 64-bit architecture, there should be quite a bit more breathing room regarding memory, with many systems nowadays having 16GB+ RAM and its average amount is obviously steadily growing (and things like these could still be opt-out feature or tied to some slider controlling the meticulousness of information management). So once again, my main question: what would be your rough guess as to the requirements of such systems being implemented, and to what degree do you see it ever happening?

[crazyabe]

Color all that green if you want an answer.

[FantasticDorf]

In relevance to future thief/shady person adventurer roles/careers, will playing a role of gathering intrigue be more valuable and perhaps profitable if you say were in the position to feed juicy information to a demon master who then uses that information in world generation to forumate attacks and use you (and accountability for truthfulness/competency) as their earpiece.

It certainly makes sense that being near some gears and levers of power wouldn't be bad for somebody selling information, though it's anybody's guess how all that is going to play out.  The agents in this release will set the early stage for that kind of thing being possible though, and we're looking forward to doing more.

[Dirst]

It had been a couple days since Toady posted his worries about the Northwest windstorms, and nothing on Twitter since then.  Saw his account active on the forum today, so the obvious conclusion is that he and Threetoe managed to get a water reactor up and running.

[Taederias]

Color all that green if you want an answer.

Thanks, did just that (I'm still new on this forum).