Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[psychologicalshock]

So I was thinking - since a multicore rewrite wont happen why not have multiple instances of DF running independently where you basically have two fortresses on separate executables . The main things you can do is send items/migrants from one fortress to the next but they don't directly interact with one another. Whatever happens in fort #1 isnt going on in fort #2 (a siege for example) , but at the same time you get their combined resources and can direct them as you want. This way I think we could have  800+ dwarf fortresses where we have 4 separate fortresses with shared resources, each one running on a separate core. I am thinking that even the time wont have to be sync'd - resource transfers would only happen at some set date, once one fortress gets to that date it pauses and waits for the other fortress to catch up.

[psychologicalshock]

I am thinking it might be pretty primitive - for example there is a designated "exchange" area on the map where once the season changes everything contained within the area is transferred to the other executable.

[peskyninja]

That would actually require more programming than implementing multi-threading.

[irmo]

That would actually require more programming than implementing multi-threading.

No, it wouldn't, because the processes are all doing the same thing and communicate through a well-defined interface. The event loop just needs to listen to another source of input (commands coming over this socket), and create caravans/migrants/other stuff as directed. This is much simpler than taking some existing aspect of the game and moving it to a separate thread.

Of course once you have that set up, you can run the same thing over a TCP session and use it for multiplayer. Also, the thing on the other end isn't strictly required to be another instance of Dwarf Fortress--it could be some kind of world server (one of the items you send with the caravan could be a "Letter of Introduction" addressed to a specific fortress, which is used to route your caravan on the server side). I like this. It's interesting.

[kaenneth]

Problem is one fort would get ahead of the other; so your game FPS would be determined by the slowest fort + communication overhead. Plus cache thrashing would hurt performance as well.

[psychologicalshock]

I was thinking of the forts not being in any way synced. You can play each one separately each season and once the season is done that fort is paused until the other 3 are done. That is you can simply pause 3 and play on 1 or play all 4 at the same time - their only consideration between each other is what happens at the seasons end, data communication doesn't happen at any other time.

[psychologicalshock]

That would actually require more programming than implementing multi-threading.

No, it wouldn't, because the processes are all doing the same thing and communicate through a well-defined interface. The event loop just needs to listen to another source of input (commands coming over this socket), and create caravans/migrants/other stuff as directed. This is much simpler than taking some existing aspect of the game and moving it to a separate thread.

Of course once you have that set up, you can run the same thing over a TCP session and use it for multiplayer. Also, the thing on the other end isn't strictly required to be another instance of Dwarf Fortress--it could be some kind of world server (one of the items you send with the caravan could be a "Letter of Introduction" addressed to a specific fortress, which is used to route your caravan on the server side). I like this. It's interesting.

Cool idea!

[Kipi]

I was thinking of the forts not being in any way synced. You can play each one separately each season and once the season is done that fort is paused until the other 3 are done. That is you can simply pause 3 and play on 1 or play all 4 at the same time - their only consideration between each other is what happens at the seasons end, data communication doesn't happen at any other time.

It's not that simple.

First of all, lets say that the fortresses are using the same world. This means that several actions that may happen during the season may cause conflicts, mainly megabeast attacks and sieges. Unless the games are synced there is no way for one instance to know if that specific megabeast, for example titan, has been killed in other instance.

Secondly, if the game accepts communication between fortresses of different worlds, there is the problem of modding. Sure, if it's only you playing both fortresses it may be possible to ensure that no conflicts are caused, but especially if there is some kind of world server used, then the change of conflicts are quite high. It needs just small tweak in one of the raw files and voilá, something traded to you may corrupt your whole fortress! So, unless Toady creates some kind of fool proof check which goes through ALL raw files and makes sure those are 100% identical and exists in other instances, which would take huge amount of time and effort, there will be risk of people getting corrupted fortresses.

Thirdly, saving. If the world can be different there is no problem but what if it can't? What if you have to quit before season ends and one of your fortress is at the beginning of season while the other one is at the end of season? Separated copies of the world? That would enable the possibility of world corruption caused by edited raw files I described above.

Fourth, history aka Legends. I remember there being a thread that suggested the possibility of moving ruins from one world to another and same problem applies here as well; if a item or thing is moved from one instance to another then the history must be moved too, unless Toady dramatically changes the way the legends are stored and linked. And if parts of the history are moved then, to avoid any complications, the whole history must be moved. And since history is moved the sites must be moved, how could there be history if the sites and beings doesn't exists? Suddenly you have a world that actually contains two worlds and as the game is already lagging under the data of single world how could it handle two worlds? Or more? To make that even remotely plausible it would require both 64bit transform as well as multithreading.

There, few major issues with this idea. And I didn't even touch the subjects like whether the game should determine which fortresses are using this "shared" idea and which aren't. Or if the game should allow another fortress connecting to first one in the middle of playing (you start a fortress, play year or more, create another fortress in new instance of DF and hook it up with the first fortress)?

Not saying that the idea is terrible, I would like to see it as well, but as multithreading and 64bit are basically going to happen at some point (didn't Toady mention somewhere recently that even though he doesn't wait for that moment it will take place some point) I would like it more if Toady buts his effort to those before any kind of "Fortress to Fortress" or "Instance to Instance" communication. It's just that the suggestion made here isn't that easy to do as it may sound.

[psychologicalshock]

First of all, lets say that the fortresses are using the same world. This means that several actions that may happen during the season may cause conflicts, mainly megabeast attacks and sieges. Unless the games are synced there is no way for one instance to know if that specific megabeast, for example titan, has been killed in other instance.

Then just move it into the season calculation - for example when the season begins its determined that these goblins will attack and this beast will attack, then use a random # generator for the specific date (so that if you reload the game it happens at a different time)

[King Mir]

So, unless Toady creates some kind of fool proof check which goes through ALL raw files and makes sure those are 100% identical and exists in other instances, which would take huge amount of time and effort, there will be risk of people getting corrupted fortresses.

I agree with everything else, but this one is easy to do. Since the raws are read anyway, it's a simple matter to create a hash of the raws as you read them. Two games' raws could be compared by comparing that hash.

[irmo]

First of all, lets say that the fortresses are using the same world. This means that several actions that may happen during the season may cause conflicts, mainly megabeast attacks and sieges. Unless the games are synced there is no way for one instance to know if that specific megabeast, for example titan, has been killed in other instance.

Doesn't matter. The titan isn't going to wander from one instance into the other and create a time paradox because he was previously killed there. Communication is limited to (1) sending parties of dwarves and (2) sending items for trade. There's no reason entire worlds have to be kept in sync.

(To anyone else creating a game engine that simulates things on several different scales: Please create a clean internal interface for communication between those scales. Like, the world simulation ticks once a day, and sends the fortress a list of "things entering the fortress today", and the fortress sends back a list of "things leaving the fortress today", and other than that, they don't talk to each other. You'll be glad you did this.)

Secondly, if the game accepts communication between fortresses of different worlds, there is the problem of modding.

That's a problem of serialization, really: how much of an object's definition do you include when you package it up to send it over? King Mir's solution--refuse to talk to fortresses that aren't running the same version of the raws--is one option. Another is to just include the entire definition (basically the raw entry for that object) so that you can send over parallel-universe critters that have weird properties. This might cause some instability (like if they reference materials or syndromes that don't exist in your world), but modding already has the potential to screw up your game, so nothing new there.

[Kogut]

@OP You suggested running multiple instances of DF to create illusion of multithreading.

It is on the same level as creating second thread that executes this code:

Code: [Select]

while(true);


Spoiler (click to show/hide)

It is an infinite loop that is doing absolutely nothing

[nxcho]

One of the simplest ways of parallelizing existing code is using openmp or similar toolsets. They usually require only some restructuring of the code.
I don't have any knowledge how the inner workings of df looks like. But a qualified guess is that the best way is to parallelize on a per-creature level or possibly on sub creature level using parallel pathfinding algorithms. Possibly both. One can also suspect that the fact that creatures close to each other are far more likely to interact than creatures far away could be used to speed up the processes further.

What psychologicalshock describes is very similar to MPI that is usually used for scientific computations on large clusters. It is not totally far fetched, and it would be cool to run df on a cluster. But parallelizing with MPI and similar is usually a lot more work than when using a more standard method like openmp or even pthreads. 

[Kogut]

They usually require only some restructuring of the code.

Probably it was not intended but this sentence is an oxymoron.

[psychologicalshock]

@OP You suggested running multiple instances of DF to create illusion of multithreading.

It is on the same level as creating second thread that executes this code:

Code: [Select]

while(true);


Spoiler (click to show/hide)

It is an infinite loop that is doing absolutely nothing

Uh I am suggesting that separate parts of a selected region is run on separate DF executables with a possibility of sending resources and migrants between them, so no , not really.

What psychologicalshock describes is very similar to MPI that is usually used for scientific computations on large clusters. 

That's actually what inspired this thread