Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Erasmus Darwin]

I've got a couple ideas for utilities that I wanted to write that require at least partially figuring out DF's save format.

For example, I wanted to cook up a utility to automatically add custom names to dwarves based on the names of various Something Awful goons.  Sankis is doing the same thing manually in his Castlerooter thread ( http://forums.somethingawful.com/showthread.php?s=&threadid=2378686 ), and the results are pretty entertaining.

Another idea I had was to provide a list of dwarves along with their skills, allowing the player could more easily configure labor and workshop profiles.

Anyway, I tried to start decoding the save file (focusing on the region#.sav and ignoring all those unit files in subdirectories), and I really didn't get far at all.  It's obviously compressed, and the strings in the dwarfort.exe imply zlib was used.  However, I scanned the entire .sav attempting to use zlib's inflate on anything that looked like it was a valid zlib header, and I failed to uncompress anything.

I tried my tool for finding zlib-compressed data on Kobold Quest, and it worked just fine -- just skipping the first 4 bytes of data was enough to find data that zlib was happy to decompress.  I'm guessing that somewhere between then and now the save format got a little trickier.

Anyone have any hints?  Right now, my choices seem to be either trying to debug dwarfort while it's saving to figure out what it's doing (difficult!) or do all my data-related operations in memory while dwarfort is running and the data is uncompressed (slightly less difficult?).  Neither one seems particularly appealing.

[flap]

I have much experience into the field of altering memory while games are running. And did a bit of research already about that. If you need I could help in that part. (for exemple write some very simple bits of code).

But of course for this technic, the adress used will have to be updated for every new version of DF

[Erasmus Darwin]

It's looking more and more like I'm going to have to go the live memory route.  I'm a little intimidated, since doing a memory dump on a running, minimal fortress clocked in at around 70 megs.  But it'll be a fun learning experience.

I really appreciate the offer for help.  I suspect I'll have quite a few questions, as I don't have experience fiddling with live memory.

I guess one of the first steps would be trying to figure out how to find the dwarf structure in memory.  I'm guessing it's either a linked list (in which case I need to locate the head pointer), or an array of pointers (in which case I need to locate the array).

[flap]

You already know a bit about that technic. That's a good start.

I am home this week end, and will try to send you information I have. I know that I already had located some address. But I don't remember if I had the pointer...

I have a rough tool I wrote for me a while ago. It does not replace a regular gametrainer to spot the address, but is usefull to explore arrays and linked lists.

I also have some bits of code. I will update them to what I know already for DF, and you'll be able to read the source. The only langage I know is C. (and am using LCC-win32 as a compiler. But it should not be important)

[Jaqie Fox]

I have always been a hardware girl, not software, and the only programming language I ever learned was MS Qbasic in high school, but I do know of an (old) program that may be able to help with this.

google for "Tsearch"...

just trying to help.

[Erasmus Darwin]

quote:Originally posted by flap:
<STRONG>I also have some bits of code. I will update them to what I know already for DF, and you'll be able to read the source. The only langage I know is C. (and am using LCC-win32 as a compiler. But it should not be important)</STRONG>

That's great, thanks.  It looks like my stuff'll wind up being in C, too.  I tried going the Perl route (since I've been playing around with Perl under Win32 a bunch lately), but it seems there isn't support for the debugging API, yet.

quote:Originally posted by Jaqie:
<STRONG>google for "Tsearch"...</STRONG>

I played with Tsearch a bit, and it's been pretty helpful.  I was able to locate a custom dwarf name and then backtrack it into what seems to be the main dwarf structure.  From there, I managed to locate a couple things, but I haven't yet figured out how big the structure is, how the structures are related to each other, or how to find the structures starting from a fixed point in memory.

[qwip]

Hmmm...I seem to remember reading about someone (on these forums? on the something awful forums?) who had written a utility that would read DF and output your dwarves' labor preferences.  Perhaps you could search for that utility and build off of it?

[Erasmus Darwin]

quote:Originally posted by qwip:
<STRONG>I seem to remember reading about someone (on these forums? on the something awful forums?) who had written a utility that would read DF and output your dwarves' labor preferences.</STRONG>

I did a lot of looking but couldn't find anything.  Is there anything you remember that could help me narrow down the search?

Also, I managed to do some decoding of the dwarf structure so far:
Dwarf structure (all values 4 bytes)
------------------------------------
 4 - First name ptr
 8 - First name len
20 - Nickname ptr
24 - Nickname len
32 - Surname word 1 (some sort of index)
36 - Pluralize (0 - no, 1 - yes; different for verbs?)
40 - Surname word 2
44 - Pluralize
92 - ?? (0x01010001)
   - Setting third byte (94) to 0x00 blanks name
100 - Custom profession ptr
104 - Custom profession len
120 - Race code (0x00000048 = dwarf)
128 - X pos
132 - Y pos
140 - Wander X
144 - Wander Y
160 - Work X
164 - Work Y

I haven't figured out how big the structure is, and I haven't had much luck trying to decode other structures that're pointed to by this structure, but it's a start.

Oh, and I've made some progress in being able to reliably locate the dwarves in memory, but I don't think I'm done yet.  Once I've got that down, I think I've got enough information to make utilities to teleport stuck dwarves out of the fort and to nickname dwarves with goon names.  Still, the latter would work better once I can figure out skills and preferences so that people can pick what sorts of dwarves are named after them.

[ March 25, 2007: Message edited by: Erasmus Darwin ]

[flap]

Nice, I see that you really have a bit of knowledge into that field. That's cool !

Here is the application I wrote. It will give you the name surname of all dwarfs on the map, their nickname, customized profession.

You can also change their nickname if they have one.

It should help you to progress.
http://www.flap.fr/grenier/MemoryDF_V02.zip
http://www.flap.fr/grenier/missearch_v2-2.zip.

ArtMoney might be more powerfull than Tsearch (but slightly messy)

About profession update, I have spotted were it is located (much later in the structure you describe, like +1120, or so). For each profession it is a BYTE : 00 work is off, 01 labor is on. The order in this list in not the same as the one on the screen. And the list is longer than what is actually displayed

[ March 29, 2007: Message edited by: flap ]

[Erasmus Darwin]

quote:Originally posted by flap:
<STRONG>Nice, I see that you really have a bit of knowledge into that field. That's cool !</STRONG>

Thanks.  I'm still learning, but it's been fun so far.  Unfortunately, I've been having trouble figuring out how to reliably locate the dwarves in memory (right now I just backtrack from a custom name that I set).  But looking at your utilities should fix that.

 

quote:<STRONG>
http://www.flap.fr/grenier/memorydf.zip
http://www.flap.fr/grenier/missearch_v2-2.zip.
</STRONG>

I tried downloading these, but the zip files keep coming out as corrupt.  They're also each only 1572 bytes long.

 

quote:<STRONG>
ArtMoney might be more powerfull than Tsearch (but slightly messy)
</STRONG>

I'll have to check it out.

 

quote:<STRONG>
About profession update, I have spotted were it is located (much later in the structure you describe, like +1120, or so).
</STRONG>

Interesting.  I had assumed that that was stored in a completely different spot in memory.  I had found a four byte value at position 248 in the structure that was way too low to be a pointer.  Changing it at all would make the dwarf's profession, skills, and labor (but not attributes) disappear.  So I had assumed it was an index to somewhere else in memory where that stuff was defined.

Also, another fun thing I found: 00789C1C and 00789C20 are two 4 byte integers that seem to define where the pointer is when you hit 'v', 't', 'k' or similar functions.  I haven't tested it a lot to make sure that's always the correct location, but it seems to be right.

I should be able to use that to set up a nice "teleport stuck dwarves" utility.  Just position the cursor on top of the stuck dwarves, read the X and Y value, compare it to the X and Y for each dwarf, and move any that match.  A number of people on SA have mentioned problems with dwarves getting stuck on magma smelters (and then falling into the magma if the smelter is unbuilt), so that should help out a bit.

[ March 26, 2007: Message edited by: Erasmus Darwin ]

[flap]

Ouuups. Sorry.
I'll post memoryDF tomorrow (I just erased it from my USB key)
I recreated the link to MisSearch, and added that new download (with the source, but not commented this time... Sorry). It is a tool to deal with dwarfs labor preference.
http://www.flap.fr/grenier/LabourDF_V02.zip

[ March 29, 2007: Message edited by: flap ]

[flap]

You do really look for the good things. That's nice. What I might be looking for (But I don't have that much time), is the place in which in memory wich tells what menu you are browsing. This could be usefull for you too (with your pointer thing)

About locating the unit structure (with dwarfs, enemies, animals...) :
At 0x00AC2DB4 you have the pointer to an array of pointers. Each pointer there is a different unit. And you already a lot about these. How could you have found it ? If you have a rough idea of where is the pointeur of your unit (dwarf), with ArtMoney do a search in a range of values not too far from that one. You will find that one pointeur, close to what you already know from that structre is refered too wuite often in memory. Ok, it might the good one.

Do the same for an other dwarf. Now for both dwarfs, you have a pointer (not 100% sure but you can check that the features you know are at the same distance from it). and series of places where you find those pointeres in memory. Look at what is hapening where you find the pointer of both dwarfs not too far one from the other. You might find a array full of pointers. Here MisSearch or a memory dump can be usefull to check what the othere pointers refers too, and where this array would start. Once you've found where this array starts, simply search for that address. You hvae found the static address of the pointer to that array !

By experience, address starting with 0x00... are often stable address. The ones dynamically allocated would often be found further.

[Xeirxes]

I feel kind of bad about this, reverse engineering Toady's code  

[Erasmus Darwin]

quote:Originally posted by flap:
<STRONG>At 0x00AC2DB4 you have the pointer to an array of pointers.</STRONG>

This doesn't seem to be working for me.  At 0x00AC2DB4, all I'm seeing is lots and lots of 00s.  Similarly, when I try and do anything with the labour program, it crashes since it's trying to dereference an invalid pointer.  I had tried following the trail of pointers backwards before, but I could never find something that was always stable.

quote:Originally posted by Xeirxes:
<STRONG>I feel kind of bad about this, reverse engineering Toady's code   </STRONG>

Well, it's not like we're doing it to make a game that competes with Dwarf Fortress or anything like that.

Besides, the stuff I'm planning on doing (auto-naming dwarves to Goon names and teleporting stuck dwarves) seems like it would either be outside the scope of what Toady's working on (and he already has a lot on his plate right now) or a temporary workaround for a bug that's going to get fixed sooner or later anyway.

[Erasmus Darwin]

Ok, now I'm really confused.  I'm getting different results in ArtMoney versus Tsearch.  I located the same dwarf structure in both programs, but I get two different addresses.  They're both in hexadecimal, so I know it isn't a hex/dec conversion thing.  And I know it's the same set of memory because when I change it in one, the change shows up in the other.  But the memory addresses are completely different.