Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[kutulu]

Status Report:

I am still making progress on this!  I can now load everything except the skill levels out of the running copy of dwarfort.  Next up I'm going to confirm that I can write changes back to the process without crashing it.

However, since d12 is now out, and d13 seemingly on the way, I'm going to try writing a Linux variation on the "memory layout amalyzer" that the Win32 version has, so it doesn't take me another 4 weeks to find everything for the new version

--Mike

[peterix]

Vectors look like:

Code: [Select]

  [address - 4] Length
  [address    ] Start
  [address + 4] End


If we're speaking about std::vector here, it looks like this:

Code: [Select]

  [address    ] Start
  [address + 4] End
  [address + 8] End of allocated space


[0x517A5D]

If we're speaking about std::vector here, it looks like this:

Code: [Select]

  [address    ] Start
  [address + 4] End
  [address + 8] End of allocated space


Based on the layout of main_creature_vector in L.40d11, I believe this to be correct.

The main_creature_vector has a 16-bit sized variable at location -4, so it can't be the allocated length.

However I don't think it matters very much, as we should only care about the .Start and .End fields.

Incidentally, MSVC++ adds 3 extra fields of cruft to a std::vector, at least with the compile options used to build DF.

[kutulu]

If we're speaking about std::vector here, it looks like this:

Code: [Select]

  [address    ] Start
  [address + 4] End
  [address + 8] End of allocated space


Based on the layout of main_creature_vector in L.40d11, I believe this to be correct.

The main_creature_vector has a 16-bit sized variable at location -4, so it can't be the allocated length.

However I don't think it matters very much, as we should only care about the .Start and .End fields.

Incidentally, MSVC++ adds 3 extra fields of cruft to a std::vector, at least with the compile options used to build DF.

I was basing my layout on the race vector, which always seems to have the vector length in the word just before the first entry, but you're completely correct: it doesn't matter, since start and end are all we need.  (It also might not matter if they switch to using icc to compile DF and we need to use that compiler's layout...)

Some pretty good news: I can now read an entire dorf from memory on both 40d11 and 40d12, including the skills and labors.  Integrating this back into Dwarf Manager shouldn't be too problematic, as I made sure to use the same MemoryAccess interface.

I'm also testing the write-back process with some safeguards to make sure I don't write past the end of the existing character buffer.  There are a couple of minimally-intrusive ways to integrate that check back into Dwarf Manager but I'm not sure I like any of them (the problem being that DM right now doesn't have any mechanism to limit the length of custom professions/nicknames.)

Also, for anyone who's interested, the locations of the key vectors can be found by scanning the .text segment for these byte sequences (which someone smarter than me can probably even decipher ):

Code: [Select]

Legend: XX = Dwarven Race Index
        VV = Vector Start
        EE = Vector End

Race Vector and Dwarven Race Index:
0f bf 15 XX XX XX XX 66
85 d2 78 18 8b 0d VV VV VV VV a1 EE EE EE EE 29 c8 c1

Creature Vector:
66 90 31 ed 8b 0d VV VV VV VV a1 EE EE EE EE 29 c8 c1

Language Vector:
20 8b 14 88 8b 0d VV VV VV VV a1 EE EE EE EE 89 4c 24

Translation Vector:
57 74 08 90 8b 0d VV VV VV VV a1 EE EE EE EE 89 4c 24

And the values so far:

Code: [Select]

40d11:
DwarvenRaceIndex   : 0x092a7b84
RaceVector         : 0x093016b0
CreatureVector     : 0x092bee50
LanguageVector     : 0x09301770
TranslationVector  : 0x09301788

40d12:
DwarvenRaceIndex   : 0x08f3df84
RaceVector         : 0x08f97acc
CreatureVector     : 0x08f55250
LanguageVector     : 0x08f97b8c
TranslationVector  : 0x08f97ba4


[0x517A5D]

Code: [Select]

Race Vector and Dwarven Race Index:
0f bf 15 XX XX XX XX 66
85 d2 78 18 8b 0d VV VV VV VV a1 EE EE EE EE 29 c8 c1

Individually, these are not good patterns.  Concatenated, they are okay but not great.  There are two occurrences, both of which do point to the race vector and the dwarven race index.  I am uncomfortable because it hardcodes both register allocation and a jump distance.  If the compiler happens to do different optimizations to this function, the pattern will not match.  Moreover, if the compiler happens to do different optimizations to an unrelated function, this pattern might get a false positive.

Code: [Select]

Creature Vector:
66 90 31 ed 8b 0d VV VV VV VV a1 EE EE EE EE 29 c8 c1

This one makes me very uncomfortable because it hardcodes a NOP used for alignment, an unrelated register used as a loop counter, and the registers used to reference the vector.

Code: [Select]

Language Vector:
20 8b 14 88 8b 0d VV VV VV VV a1 EE EE EE EE 89 4c 24

Again, I think this is a poor pattern.  This one hardcodes the location of a stack variable, an unrelated multiplication, and the registers used to reference the vector.

Code: [Select]

Translation Vector:
57 74 08 90 8b 0d VV VV VV VV a1 EE EE EE EE 89 4c 24


This one hardcodes the location of a table in the code, a NOP used for alignment, and a stack variable.  I think this one is a very poor pattern.

Overall, I recommend against putting any of these patterns into the DM detection code.  They should be used under human supervision.

If you want, I will try to figure out some detection methods that are likely to be more stable.  This will probably involve more than simple memory matching.

[kutulu]

Overall, I recommend against putting any of these patterns into the DM detection code.  They should be used under human supervision.

If you want, I will try to figure out some detection methods that are likely to be more stable.  This will probably involve more than simple memory matching.

Thanks for the highly informative response! 

And no, I had no intention of trying to update the existing DM detection routines to work on Linux, at the very least not until they decide if they're switching from gcc to icc for the compiler :\  I am using those patterns only to grep through my memory dumps.  Realistically, the existing DM detection code method is much much slower but probably WAY safer -- scan the memory dump for known strings and then work backwards.  Now that I have the memory layouts for strings, vectors, etc. I could probably update the Win32 code to work, but not really top of my priority list

[kutulu]

Awesome news on this front: I can now load dorfs from a running copy of d11, d12, or d13 into the DM user interface, running under Mono, and get everything right!  I am working now on integrating my safety checks for strings during the write-back, and writing back the labor flag updates.  In the mean time, if anyone wants to see this work read-only, or (better yet) help test to make sure I got it right, I can:

* Send you the compiled binary and updated .xml files that will run under Mono, or
* Send you a patch to apply to the latest svn source code for DM, to compile using MonoDevelop

Anyone interested just let me know.

--K

[CautionToTheWind]

I'll gladly help you test this. Please send me the binary to the email hugo1000 at the usual gmail.com

What packages does it depend on?

Cheers and Thanks.

[Nabobalis]

I'm been following this thread and I have to admit your doing excellent work

I would like to help test but I'm not sure I would know what to do to get it working 

Although if you send me either to nfreij@hotmail.com I'll be willing to have a go at it. Too much free time right now.

[chmod]

Hey guys. I just saw this thread, and thought I might offer some help if I can. I just posted a new tool that does just about everything dwarf manager does, but in C++ using Qt. Thread here: http://www.bay12games.com/forum/index.php?topic=39229.0

My hope was to branch out and make the software available for linux and OSX as well. If anyone is interested I'd be happy to start discussing support for more platforms.

[kutulu]

The compiled binary can be downloaded from http://home.roadrunner.com/~kutulu/dm-linux.tar.bz2 (at least for now).  Please keep in mind that it's read-only for now.  In fact, if you attempt to write back to df after making changes I can't guarantee you will live to tell about it.

--K

EDIT

To run this you'll need to first install a mono runtime environment.  I am using mono v2.4.2, but I believe any 2.x version of the runtime should suffice.  Once you have mono installed, just unpack the above tarball somewhere and "mono DwarfManager.exe", then proceed just like the Windows version.

A few additional issues to be aware of:

* ABSOLUTELY DO NOT shut down dwarf fortress while the program is running.  I am currently exploring a bug where attempting to hook into the process Exit event (which works fine on Windows) freezes DM on Linux, so currently, the app has no idea that df has shut down.

* If something goes horribly wrong and DM crashes, you may start to receive very confusing "File not found" errors trying to re-connect.  This is a confusing error message that really means that ptrace() access to the dwarfort.exe process is locked and DM can't get connected.  You'll have to kill df and restart it to get going again.  This hasn't happened to me outside of IDE debugging yet, but it's confusing enough that I thought I'd mention it.

[denspb]

Could you look up addresses for d13 version?

[kutulu]

Could you look up addresses for d13 version?

They're already included in the data file in the tarball I posted; if you're just looking for the Windows numbers they can be found here:

http://www.bay12games.com/forum/index.php?topic=25585.msg648755#msg648755

[Shades]

*replying to keep track of thread*

[Jeff Carr]

I'm definitely interested in how this works out.  And I'm more than willing to help test.