Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 7 8 [9] 10

Author Topic: What language is Dwarf Fortress made in?  (Read 47974 times)

devek

  • Bay Watcher
  • [KILL_EVERYTHING]
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #120 on: September 13, 2010, 06:24:06 pm »

Dude, i'm too drunk to get the [] right. My mind is spinning with possibilities.

_java_enum Day
{
    SUNDAY, MONDAY, TUESDAY, WEDNESDAY,
    THURSDAY, FRIDAY, SATURDAY;
}

std::string day = "MONDAY";

switch (Day.valueOf(day))
{
    case SATURDAY:             
    case SUNDAY:
        // weekend processing ...
        break;
    default:
        // weekday processing ...
}
« Last Edit: September 13, 2010, 06:26:27 pm by devek »
Logged
"Why do people rebuild things that they know are going to be destroyed? Why do people cling to life when they know they can't live forever?"

ZCM

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #121 on: September 13, 2010, 06:56:54 pm »

Sweet. So based off your reading, I can make extensions to GCC that allows it to compile java source files. (someone did!)

Now I can be an asshole and confuse every discussion by claiming that my program is really a C program, since it can be compiled by a compliant implementation.

C for all!
Doesn't work - Java has some reserved words that are legal identifiers in a strictly conforming C program, trying to make a compiler that accepted both would be hopelessly confused. ("class Object { ..." etc.). I think there's some other restrictions too, like certain things are required to generate errors.

I was even thinking, I could add intrinsic java functions to C since inline java might not be optimal

So you could be like...

char []hello = "Hello World!";

_java_System_out_println(hello);

Gcc could be all smart and convert that char to a java.string, reorder it, the possibilities are limitless and it would still totally be c and not c code with java in it.
I think that would work, actually. At least from a standards compliance perspective.
« Last Edit: September 13, 2010, 06:59:10 pm by ZCM »
Logged
Badger badgers badger badger badgers badgers badger.

devek

  • Bay Watcher
  • [KILL_EVERYTHING]
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #122 on: September 13, 2010, 07:02:58 pm »

Dude, I totally get what you're saying.

Between #pragma, inlines, and intrinsics there are tons of ways to get that java in there.

But still, it would be pretty stupid to say that the result was a c program with no java in it...
Logged
"Why do people rebuild things that they know are going to be destroyed? Why do people cling to life when they know they can't live forever?"

ZCM

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #123 on: September 13, 2010, 07:09:12 pm »

So I was playing with a simple DF-like simulator I'm building in C#, and I switched one part from doing a periodic sweep over the entire map to instead keeping a priority queue of what needed updating when, and the CPU usage for that part dropped by a factor of 1000.
Logged
Badger badgers badger badger badgers badgers badger.

devek

  • Bay Watcher
  • [KILL_EVERYTHING]
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #124 on: September 13, 2010, 07:37:12 pm »

Without talking about how I modded GCC to automatically detect by structure the difference between C and C# and compile with 100% accuracy, rendering your C# program to C, I must say.. color me surprised :P

There are tons of little things that can be done like that too.

Create a mesh graph for rooms and use some sort of a r-tree for items, and the example of a dwarf running out to grab something would be solved. It would be beautiful.
Logged
"Why do people rebuild things that they know are going to be destroyed? Why do people cling to life when they know they can't live forever?"

Veroule

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #125 on: September 13, 2010, 10:09:12 pm »

So I was playing with a simple DF-like simulator I'm building in C#, and I switched one part from doing a periodic sweep over the entire map to instead keeping a priority queue of what needed updating when, and the CPU usage for that part dropped by a factor of 1000.
Yes, there are a lot of algorithm changes that are possible, and many could make significant differences.  Changing the way the objects are allocated and hld data is generally easier and would make a large difference.  That is what that pdf was all about.

Something like 1-2 months ago a topic cropped up where someone setup a save with generally stable init.txt options, and asked people to to post FPS and system specs.  While not the most thorough examination, it did give some results that supported the concept behind that optomization.  That was 2 nearly identical systems, one had a CPU with a lower clock speed and larger L2 cache than the other.  The one with the larger cache got about double the FPS of the "faster" system.  I think a few other comparisons can be drawn from the results posted, and they all point to a good gain being achieved by reducing the memory bottleneck.

You might want to check the underlying implementation of that priority queue.  The right way to do it is a binary heap.  Items in a binary heap get partially sorted into position when added, but are not fully sorted.  When the top item is removed then additional sorting occurs to determine the new top item, again this doesn't fully sort the heap.  Only the top item of the heap is guaranteed to be sorted into the right spot, and that is the only one you need to properly prioritize a queue.
Logged
"Please, spare us additional torture; and just euthanise yourselves."
Delivered by Tim Curry of Clue as a parody of the lead ass from American Idol in the show Psych.

KrunkSplein

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #126 on: September 14, 2010, 12:22:33 pm »

One tiny tidbit of information:

DF is not feature-complete.  Optimization is something you generally do once you are sure you're not adding anything new.  Attempting to optimize while coding will literally drive you insane.

Starting off with something reasonably optimized is not dumb, but spending a large amount of time optimizing code that may not make it to release is.

Heap optimization and the like will be hugely important... eventually.
Logged

qwertyuiopas

  • Bay Watcher
  • Photoshop is for elves who cannot use MSPaint.
    • View Profile
    • uristqwerty.ca, my current (barren) site.
Re: What language is Dwarf Fortress made in?
« Reply #127 on: September 14, 2010, 08:33:21 pm »

When pathfinding to a task(construction, mining, etc.), rather than path to each of the 8 directions, why not path to the tile itself(temporarily marking it as passable, if required) and then stop one tile short?

Also, pathfinding will remain a huge issue for the exact same reason that you propose it isn't: Exactly how many times will it need to retrieve data about the terrain? Reducing the excess sprawl of the pathfinding would reduce cache misses thus, if I understand correctly, leaving more cache for other things.
AFAICT it doesn't path to each of the 8 directions; it picks one of them arbitrarily, and paths to that one. That's why your dwarves sometimes decide to mine a block right next to them but run hundreds of tiles around to get to the other side. Of course, that wastes pathfinding time.... It would probably be faster to use a reversed A* search to path to all eight possible squares simultaneously.

Dwarves have a priority over which side they will mine out from. Unless there is something in the way, miners will try to walk around unmined areas to reach these preferred sides, even if that path is very long. In order from most preferred to least preferred, dwarves prefer to stand on the tile to the:
West > East > North > South > NW > SW > NE > SE

That looks suspiciously like an area that could be optimized from what looks like 8 pathfinds quitting after the first success, since if the first success requires pathing through a maze that occupies half the map, then you just had a cache miss on just about half the map, and, again, if I understand how the cache works(and I probably don't), a lot of important things have been displaced that were about to be used once that dwarf found his way...
Logged
Eh?
Eh!

devek

  • Bay Watcher
  • [KILL_EVERYTHING]
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #128 on: September 14, 2010, 08:49:30 pm »

Well, you're kind of on the right track. It is the path fails that hurt, not the path successes.

Now try visualizing the world in 3d. Put a cube around the tile you want to reach and path toward that. You'll find the shortest path every time and the dwarf won't make a fool of himself :P

If you want performance... the first change to make is obvious. Change mapblocks from 16x16 to 48x48. It would take 5 min to code and the speed difference will be huge. With reverse engineering I can make DF do that, it would just take a bit of time to find each and every part of the code that accessed a tile... it might be easy, how many other places would you be calling for something / 16 and then something % 16? Change the 16s to 48s to enjoy a possible 50% more fps...



Logged
"Why do people rebuild things that they know are going to be destroyed? Why do people cling to life when they know they can't live forever?"

qwertyuiopas

  • Bay Watcher
  • Photoshop is for elves who cannot use MSPaint.
    • View Profile
    • uristqwerty.ca, my current (barren) site.
Re: What language is Dwarf Fortress made in?
« Reply #129 on: September 14, 2010, 09:56:27 pm »

Well, failures due to the destination being solid rock are nice, but a pathing failure that could have been solved by quitting on a tile that it probaably passed over....
Logged
Eh?
Eh!

vlademir1

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #130 on: September 15, 2010, 09:25:21 pm »

Considering the optimization discussion going here , here's one that struck me.  As far as I can tell, without reverse engineering the code from the binary, DF polls every mob on the map every frame for pathing and moving.  Wouldn't it be an easy speed optimization to move every instance of a mob in a cage from the main list (that calls them to path and move) to a separate list that's part of the cages data structure and only move them back when they leave the cage?  It's the kind of optimization that doesn't seem likely to be broken by changing features (a chance of escaping the cages being tho only one I can think of) and doesn't need a lot of those speed limiting memory lookups ;D, and at the same time cuts dozens of path and movement calls every frame.  Implementing that list as an array would then also add verisimilitude as cages wouldn't be able to hold an infinite number of creatures.  It would of course be less efficient populating the Animals and Units lists since you'd have to call multiple memory areas, but those are comparably infrequent calls anyway.
Logged

G-Flex

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #131 on: September 15, 2010, 09:32:41 pm »

One tiny tidbit of information:

DF is not feature-complete.  Optimization is something you generally do once you are sure you're not adding anything new.  Attempting to optimize while coding will literally drive you insane.

Starting off with something reasonably optimized is not dumb, but spending a large amount of time optimizing code that may not make it to release is.

Heap optimization and the like will be hugely important... eventually.

The problem is that it's not just "optimization" that's important here. It's also the concept of programming with a strong foundation/structure using good practices from the very beginning.
Logged
There are 2 types of people in the world: Those who understand hexadecimal, and those who don't.
Visit the #Bay12Games IRC channel on NewNet
== Human Renovation: My Deus Ex mod/fan patch (v1.30, updated 5/31/2012) ==

Veroule

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #132 on: September 15, 2010, 11:45:55 pm »

Pathing really isn't the huge slow down that everyone thinks it.  The proof is very simple, stop everything in your fort and set all your dwarves to do refuse hauling.   When the entire job list is clear note the fps, then set a number of whatever to be dumped.  The number should be closely equal to your current idlers.  After just a little bit you should notice a small freeze, that is all of the pathing requests.  Not exactly optimal, but continue to watch the fps as the dwarves go to work.  It should return to the same number it was when they were all idle, there are no more pathing calculations done unless a traffic jam ensues.

As ZCM noted, using a prioritized queue to determine which objects need updating is massively faster then checking all items.

Quote from: KrunkSplein
babble
My rule for when to optomize is always 2 levels.  If a class has 2 child classes, or any function has 2 levels of calls above it; then it should be considered a lower level item and should be optomized.  I will bend that while writing the particulars, but once a specific algorithm is determined and that lower level item is clearly defined I make sure it is pretty close to optimal before moving on.

For something the size of DF waiting until the other 69% of the features are written to do optomizations would be idiotic.  It would also make the game nearly unplayable as well.  Consider if you will, that in 8 years 31% of the game has been written.  Toady is writing it faster then when he started, but it will likely be at least another 5 years until 100%.  If some good easy optomizations are found, tried, and learned now; then Toady is only undoing the mistakes in the first 31% of the game, instead of having to fix the same mistake 3 times as often.

My experience says it is always easier and less expensive to do something right the first time, than it is to do it, take it apart, and do it again.  This is one of the few things that seems to be true for all fields of human endeavor and study.
Logged
"Please, spare us additional torture; and just euthanise yourselves."
Delivered by Tim Curry of Clue as a parody of the lead ass from American Idol in the show Psych.

vlademir1

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #133 on: September 16, 2010, 05:05:15 am »

I'm assuming this is in relation to my post, if not just ignore me :D
Pathing really isn't the huge slow down that everyone thinks it.  The proof is very simple, stop everything in your fort and set all your dwarves to do refuse hauling.   When the entire job list is clear note the fps, then set a number of whatever to be dumped.  The number should be closely equal to your current idlers.  After just a little bit you should notice a small freeze, that is all of the pathing requests.  Not exactly optimal, but continue to watch the fps as the dwarves go to work.  It should return to the same number it was when they were all idle, there are no more pathing calculations done unless a traffic jam ensues.

It's not the pathing call itself, but the associated loading of each data member into processor memory to check if you actually need to call pathing I'm talking about bypassing.  I'm assuming each frame every creature from all your dwarves, to any visitors (hostile or no), and all critters on your map are loaded up, some data checked for if they need to path, move, etc and then unloaded.  If something is caged it reasonably shouldn't need to even be loaded up since it's not able to do anything and loading that takes time. 
A good test case for this is to gen a world, copy the save, and load up an embark with just seven dwarves then do the same in the copy at the exact same location (so no terrain difference to affect the numbers) with just seven dwarves and thirty dogs.  The second is slower on my system by a measurable margin, -10 fps on my system,  just because it takes extra time per frame to check each of those creatures and tossing them immediately into a cage only gives me back 3 fps.
Playing with the gfps tells me something similar is done by the display code too, since dropping the gfps to 1 gives me huge fps increases at the cost of easy visual tracking of what's going on. I keep gfps at 13 explicitly to have a reasonably smooth visual while still having better fps than the default gfps of 50 allows me.

As ZCM noted, using a prioritized queue to determine which objects need updating is massively faster then checking all items.
I fully agree, but that is one of many fairly minor to implement tweaks that would help as were many of the various ideas thrown around in this thread (devek's 48x48 instead of 16x16 as an example).

Quote from: KrunkSplein
babble
My rule for when to optomize is always 2 levels.  If a class has 2 child classes, or any function has 2 levels of calls above it; then it should be considered a lower level item and should be optomized.  I will bend that while writing the particulars, but once a specific algorithm is determined and that lower level item is clearly defined I make sure it is pretty close to optimal before moving on.

For something the size of DF waiting until the other 69% of the features are written to do optomizations would be idiotic.  It would also make the game nearly unplayable as well.  Consider if you will, that in 8 years 31% of the game has been written.  Toady is writing it faster then when he started, but it will likely be at least another 5 years until 100%.  If some good easy optomizations are found, tried, and learned now; then Toady is only undoing the mistakes in the first 31% of the game, instead of having to fix the same mistake 3 times as often.

My experience says it is always easier and less expensive to do something right the first time, than it is to do it, take it apart, and do it again.  This is one of the few things that seems to be true for all fields of human endeavor and study.
That right there is a big issue, tying closely into G-Flex' immediately preceding post.  It was made obvious by the September 2008 to April 2010 dev cycle from 0.28.* to 0.31.* with the stuff that was rewritten and now works in a very different way (largely combat related) that Toady wasn't really working from a perspective of working out how something should work then implementing that from the start so much as "kludge now and fix it when I have to".  The later, while a valid method, isn't a good way to work on a project of this game's massive scope because you then have to later change all sorts of very early code that may or may not any longer have valid comments.  That said, that is a methodology common to many until they get to doing a really big, complex project that ultimately shows them it's very small working limits.  The real elephant here, much as it may upset the end user to contemplate, is that the longer the project goes without optimization, the more and more newly implemented systems rely on the old code and the harder optimization is without breaking several other things that rely on the old implementation's workings later.
Logged

Veroule

  • Bay Watcher
    • View Profile
Re: What language is Dwarf Fortress made in?
« Reply #134 on: September 16, 2010, 09:03:03 pm »

Regarding the map blocks, I don't think 48x48 would be that good of an idea.  The specific number would really depend on a lot of different pieces, but my first instinct is say keep the dimension on a power of 2.  Going to 32x32 is 4 times the abstraction which should be enough to make things a little faster.

The main reason I like to keep some things on powers of 2 is for divisions.  Both the floating point and integer division operations have a variable number of clock cycles depending on the divisor.  If will also need a modulus then it becomes even more important, because I have yet to see a compiler make use of the fact that the division operations return both the mod and dividend at the same time.  By staying on a power of 2 you can do the division with a right shift '>>' operation, the modulus can be gotten with a bitwise and '&'; each of which is only 1 clock cycle.  Also if you have to put the value back together you can use << and |, instead of * and +, for a few more clock cycles.

Another reason 32x32 might make more sense is because it totals 1024.  Assuming the needed data for different movement types can be organized into 3 bytes per tile, you can do a structure with {int32 x1,y1,z1,x2,y2,z2;int8 a1[1024],a2[1024],a3[1024];} and it works out to just enough under 4Kb.  4Kb is the magic number on Windows for the smallest allocatable block, of which some bytes are reserved for OS data.  Then your language memory manager splits that up and reserves a few more bytes with each chunk for its data.  Adding up the structure above it is exactly 1000 bytes shy of 4Kb; allowing a very generous 50 bytes for both OS and MM, there is 950 left for connection information.

The details about pathing are pretty thoroughly discussed in topic in the suggestions forum.
Logged
"Please, spare us additional torture; and just euthanise yourselves."
Delivered by Tim Curry of Clue as a parody of the lead ass from American Idol in the show Psych.
Pages: 1 ... 7 8 [9] 10