Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[khearn]

No. Once you get transistors so small that the charge from one starts causing the one next to it to fire (which is where they are now), you can't make them any smaller. If they can't get any smaller, they can't really get much faster.

That's the same logic that says we'll never be able to go from Moscow to Beijing in a day or two because horses can only gallop so fast.

No, it's the logic that says we'll never travel from Moscow to Beijing in less than 19.31 milliseconds.

(That's 5,790 km from Moscow to Beijing divided by the speed of light.)

We can go faster than horses. We can't go faster than light. There are absolute limits in this universe. We're running up against them when it comes to semiconductors. Really. Science isn't this magical hat you can always pull one more rabbit out of. Eventually you simply hit a limit.

Anyway, re: multithreading:

The short answer is that I don't know how to do it [muktithreading], it'd probably break things and take forever, but there are definitely places where it would help.

Should it be done? Probably. Is it going to happen? Not any time soon.

Exactly. He knows he needs to do it and doesn't know how, so he needs to learn. That's what I said.

The other option is that he does nothing but keeps adding features and it keeps getting slower. He's only got less than half of the features done. And already a lot of people are doing 2x2 embarks with only one cavern layer, severely lowering their pop caps, and religiously atom-smashing everything they can. The ostrich algorithm isn't going to work much longer here.

[Orange Wizard]

Well, whatever.

[Putnam]

sorry i just tried implementing a liquid system using multithreading as a proof of concept and adding a print statement to the start of the chunk processing function makes it run faster

i'm... gonna... just sorta... cry? a little bit

[Orange Wizard]

sorry i just tried implementing a liquid system using multithreading as a proof of concept and adding a print statement to the start of the chunk processing function makes it run faster

i'm... gonna... just sorta... cry? a little bit

Your code is haunted.

Anyway, Terraria does a multithreaded fluid thing, which is pretty cool. Means you can drain an ocean and only the water will lag, not the rest of the game.

[Libash_Thunderhead]

sorry i just tried implementing a liquid system using multithreading as a proof of concept and adding a print statement to the start of the chunk processing function makes it run faster

i'm... gonna... just sorta... cry? a little bit

Your code is haunted.

Anyway, Terraria does a multithreaded fluid thing, which is pretty cool. Means you can drain an ocean and only the water will lag, not the rest of the game.

Does it mean on a slower pc, it drains slower? In DF, your miner has a high chance to survive than on a fast pc?

[Putnam]

Yeah, I would probably have the main thread wait for the water code to complete. I doubt it would cause it to be slower... well, I don't doubt that at all actually, for a bad implementation, but that's a bit pessimistic to assume.

[Bumber]

No. Once you get transistors so small that the charge from one starts causing the one next to it to fire (which is where they are now), you can't make them any smaller. If they can't get any smaller, they can't really get much faster.

That's the same logic that says we'll never be able to go from Moscow to Beijing in a day or two because horses can only gallop so fast.

No, it's the logic that says we'll never travel from Moscow to Beijing in less than 19.31 milliseconds.

(That's 5,790 km from Moscow to Beijing divided by the speed of light.)

We can go faster than horses. We can't go faster than light. There are absolute limits in this universe. We're running up against them when it comes to semiconductors. Really. Science isn't this magical hat you can always pull one more rabbit out of. Eventually you simply hit a limit.

Dig a tunnel. Build a larger/smarter CPU. Lateral thinking. Improvement doesn't have to be limitless, just enough to serve our purposes.

Maybe someone will develop a library that makes multithreading less difficult than it is right now.

[Dirst]

Multithreading is not as simple as setting a compiler flag, but it's also not an indivisible design foundation that must be all-or-nothing.  Toady can start by making more formal interfaces to tasks so that they accept a state-of-the-world and return output (which may or may not be a copy of that data structure in a new state).  At first everything would still be executed sequentially, but the "encapsulation" troubleshooting won't be commingled with timing troubleshooting.  Once there are a handful of severable tasks, then he can worry about spawning threads and weaving the results back in.

Of course, it makes sense to have a notion of how multithreading works before the start to identify the relatively low-hanging fruit.

The reason I said that multithreading "may or may not help" is because the updates that cause lag are already stuttered across time (growth checks every ten ticks, flow is not calculated for every tile every tick, temperature is only checked when certain events occur, etc.).  Players may not see a performance boost from a multithreaded simulation, but the simulation itself ought to be smoother.  Fortunately, this means that if some update cycles that take several ticks per run, it won't alter the fundamental nature of the game.  Though it will probably affect edge cases like carving fortifications into warm stone.

[khearn]

Multithreading is not as simple as setting a compiler flag, but it's also not an indivisible design foundation that must be all-or-nothing.  Toady can start by making more formal interfaces to tasks so that they accept a state-of-the-world and return output (which may or may not be a copy of that data structure in a new state).  At first everything would still be executed sequentially, but the "encapsulation" troubleshooting won't be commingled with timing troubleshooting.  Once there are a handful of severable tasks, then he can worry about spawning threads and weaving the results back in.

I agree 100%. This is why he needs to start working on it now, not just put it off until "the end." If he decides to redo the way mechanisms work, he should go ahead and redo it in a way that makes each operation that takes place in a tick be independent of the other operations in the same tick. So that some time in the future it will be possible to multithread it without another major rewrite.

If you say "I'm not going to optimize this for multithreading now, because I might rewrite it some time in the future" what you're really saying is "I will have to rewrite this in the future when I get around to multithreading."

Of course, it makes sense to have a notion of how multithreading works before the start to identify the relatively low-hanging fruit.

Yeah, this is the tough part. It's a complicated thing, and trying to learn while implementing it on a major project like DF is likely to result in plenty of "learning experiences." That's why I think he should try a side project specifically so he can get experience at multithreading before trying to implement it in DF. A mentor who can coach him through the learning process would also be wonderful, but hard to come by. I'm not volunteering because I don't have enough experience in the subject to be able to help.

If I got stuck managing a software project similar to DF, something singlethreaded that really needed to be multithreaded, but had no one on the team with multithreading experience, I wouldn't enroll my programmers in multithreading courses. I'd try to hire someone with experience who can join the team and act as a guide and architect. But DF isn't being developed by a team, and that's not likely to change (and I'm not criticizing that, I respect Tarn for it). That does make it a lot harder to add new skills when they are required, though. But it doesn't mean those new skills shouldn't be acquired.

And as a programmer who is getting a little bit along in years, I can say from experience that it's easier to learn major new paradigms earlier than later. When I was in my 20s & 30s, picking up new languages and systems was easy, now that I'm in my 50s I have to work a lot harder at it. He can learn multithreading now, or he can wait 20 years and have to work a lot harder.

Or I suppose he can just hope he dies before he gets to the point where he has to deal with it, but I sure hope he's not taking that strategy.

[Shonai_Dweller]

Toady often mentions that he takes breaks from coding DF to work on 'side projects' but doesn't like to mention what they are. Since one of these resulted in a 64-bit DF, it could be that he's already been experimenting and learning about multi-threading (and more effective pathing and other such stuff) for a while. Just because he hasn't mentioned it, doesn't mean he isn't aware of the issue or is hoping it will go away as a lot of these type of threads seem to imply.

[Bumber]

The reason I said that multithreading "may or may not help" is because the updates that cause lag are already stuttered across time (growth checks every ten ticks, flow is not calculated for every tile every tick, temperature is only checked when certain events occur, etc.).  Players may not see a performance boost from a multithreaded simulation, but the simulation itself ought to be smoother.  Fortunately, this means that if some update cycles that take several ticks per run, it won't alter the fundamental nature of the game. Though it will probably affect edge cases like carving fortifications into warm stone.

And there might be other things that could be checked less often, but aren't.

In my fort I had jobs for clothing queued up in the manager. When my cloth ran out, my announcements were filled with constant spam that dropped my FPS from 100 down to 20. Checks like this need to happen less frequently because they are not high precision and aren't likely to change state quickly. This would also apply to unreachable (e.g., in a tree) items. If you can't reach it, stop trying for a while. It's not going to change unless the map changes. Unnecessary pathfinding is expensive.

[Libash_Thunderhead]

In my fort I had jobs for clothing queued up in the manager. When my cloth ran out, my announcements were filled with constant spam that dropped my FPS from 100 down to 20.

Maybe it is because the game log. The messages are being written in gamelog.txt.

[Putnam]

game...log... written to hard drive... as you play.

the game runs faster... with an ssd...

because...

SHIET

[Miuramir]

game...log... written to hard drive... as you play.

the game runs faster... with an ssd...

because...

SHIET

One thing I've noticed with both DF and KSP (Kerbal Space Program) is that the logging works OK under normal circumstances, but isn't well handled when something abnormal occurs.  In KSP that's usually because of a buggy mod (or interactions between two mods that don't play well with each other), but DF is more likely to generate weirdness due to odd combinations of player actions, and the default level of logging is higher. 

How practical would it be for someone to experiment with a setup that puts the files DF writes to regularly on a RAM drive, which then only copies to the actual hard drive in the background every so often (per minute?)?  A bit fiddly, but in certain cases it could probably provide a speed boost. 

In some senses, as an early Alpha piece of software, logging directly to disk as you go despite any performance issues is the "right" answer, in case the program crashes or locks up.  But from an "operational" standpoint, some sort of cached log is more efficient. 

[Dozebôm Lolumzalìs]

Honestly, the practical solution would to have saving sync the announcement file in RAM with the announcement file on the hard drive, because if the program crashes, you lose all progress since the last save.

So if it's:

1 Granite: Save

2 Granite: Do stuff

3 Granite: Crash

My way would have the log include everything up until Granite 1st, and thus include the actual history of the world after the crash.

The way it is now, the log would say:

1 Granite: Stuff

2 Granite: Stuff (that never happened)

3 Granite: Stuff (that never happened)

2 Granite: Stuff that happened

Which is very confusing.