Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Clément]

Also I have a weird bug with the classic version only where game.external_flag is always reset to 0 when a fortress is loaded (I can set it in the main menu but not in game).

Code: [Select]

[DFHack]# :lua df.global.game.external_flag=1
[DFHack]# :lua !df.global.game.external_flag
0

[Clément]

Also I have a weird bug with the classic version only where game.external_flag is always reset to 0 when a fortress is loaded (I can set it in the main menu but not in game).

Code: [Select]

[DFHack]# :lua df.global.game.external_flag=1
[DFHack]# :lua !df.global.game.external_flag
0

Commenting the "Steam-only stuff" in df.ui-menus.xml solves this (at least it solves Dwarf Therapist, I did not recompile DFHack). The layout for gamest seems to be different between steam and classic still.

[ab9rf]

I checked during initial analysis to verify that gamest in classic was _not smaller_ than it was in steam. I neglected to check whether it was _larger_, in part because it didn't make sense for it to be.

Sadly, it is. We know that the external flag is the last element in gamest, and so the length of gamest is the difference between the address of game and the address of external flag (plus four bytes). In Steam, this length is 0x141e2554 - 0x141e1adc0 + 4 = 0xa7a8; in Classic this length is 0x141e193d4 - 0x141e0ebc0 + 4 = 0xa818. For some reason that is inexplicable to me, gamest is now actually _longer_ on Classic than on Steam.

I don't have an analyzed 50.05 Classic image handy, but in 50.04-Classic this was 0x141dbf3c - 0x141db4fc0 +4 = 0xa400, while in 50.04-Steam this was 0x141dcb52c - 0x141dc10b0 + 4 = 0xa480.

I honestly do not know what Bay12 is doing here.

[Clément]

I had to remove fields to make external_flag work on classic, so I made gamest smaller. If it is actually bigger, it means there are extra fields after external_flag.

[lethosor]

I had to remove fields to make external_flag work on classic, so I made gamest smaller. If it is actually bigger, it means there are extra fields after external_flag.

Well, if you had to remove fields to make external_flag align, it means there is something larger before external_flag on classic (vs Steam). It could be extra fields, or the padding Putnam added in an attempt to align gamest.

[Putnam]

I honestly do not know what Bay12 is doing here.

finding it difficult to measure what the changes i'm making are actually doing, mostly

[Clément]

50.07-classic, external_flag offset is 0xa724

50.07-steam, external_flag offset is 0xa7a4


The difference is in mod_manager. classic is smaller than steam, at least until external_flag.

[ab9rf]

I checked during initial analysis to verify that gamest in classic was _not smaller_ than it was in steam. I neglected to check whether it was _larger_, in part because it didn't make sense for it to be.

Sadly, it is. We know that the external flag is the last element in gamest, and so the length of gamest is the difference between the address of game and the address of external flag (plus four bytes). In Steam, this length is 0x141e2554 - 0x141e1adc0 + 4 = 0xa7a8; in Classic this length is 0x141e193d4 - 0x141e0ebc0 + 4 = 0xa818. For some reason that is inexplicable to me, gamest is now actually _longer_ on Classic than on Steam.

I don't have an analyzed 50.05 Classic image handy, but in 50.04-Classic this was 0x141dbf3c - 0x141db4fc0 +4 = 0xa400, while in 50.04-Steam this was 0x141dcb52c - 0x141dc10b0 + 4 = 0xa480.

I honestly do not know what Bay12 is doing here.

I figured out my error above: I transposed two digits while manually doing these calculations. The address of game in .07-Classic is 0x141e0ecb0, which is not what I used above.

Clément's analysis is correct, and thus it would appear that the decision to implement padding (as previously mentioned) has not gone forward. That was my mistake; I misinterpreted our initial reverse-engineering results. I apologize for any inconvenience this may have created for DFHack users.

For DFHack, we'll be moving forward with a Plan B approach that doesn't depend on Bay12 keeping the game structure aligned between versions.

Code: [Select]

PS E:\Kelly\Projects\df_misc> & ruby -I../metasm dump_df_globals.rb --raw 'E:\kelly\df\df_50_07_steam\Dwarf Fortress.exe' | where-object {$_.contains("game")}
Global table starts at 141318080h
<global-address name='game.external_flag' value='0x141e25564'/>
<global-address name='game' value='0x141e1adc0'/>
<global-address name='gamemode_cansave' value='0x141876a15'/>
<global-address name='gamemode' value='0x141319840'/>
<global-address name='gametype' value='0x14131983c'/>
PS E:\Kelly\Projects\df_misc> & ruby -I../metasm dump_df_globals.rb --raw 'E:\kelly\df\df_50_07_classic\Dwarf Fortress.exe' | where-object {$_.contains("game")}
Global table starts at 14130c050h
<global-address name='game.external_flag' value='0x141e193d4'/>
<global-address name='game' value='0x141e0ecb0'/>
<global-address name='gamemode_cansave' value='0x14187e18c'/>
<global-address name='gamemode' value='0x14130d804'/>
<global-address name='gametype' value='0x14130d7dc'/>

steam: game.external_flag - game = 0x141e25564 - 0x141e1adc0 = 0xa7a4
classic: game.external_flag - game = 0x141e193d4 - 0x141e0ecb0 = 0xa724L


[PatrikLundell]

Despite that, I can still get some data from the Classic version using DFHack, with many things seemingly working (gui/gm-editor, some basic scripts to look at some data of interest, such as units, for instance). I've run a script over the DF data structures to try to find things that I might be able to fix, and after blacklisting parts of it (crashing or running out of memory on "broken" structures) I've been able to go through it, although without finding anything that's within my capability to fix so far, unfortunately.

Still, it's usable as a probe (with extra caution) with Classic, despite there apparently being a big wart on it.

[ab9rf]

Despite that, I can still get some data from the Classic version using DFHack, with many things seemingly working (gui/gm-editor, some basic scripts to look at some data of interest, such as units, for instance). I've run a script over the DF data structures to try to find things that I might be able to fix, and after blacklisting parts of it (crashing or running out of memory on "broken" structures) I've been able to go through it, although without finding anything that's within my capability to fix so far, unfortunately.

Still, it's usable as a probe (with extra caution) with Classic, despite there apparently being a big wart on it.

To the best of my knowledge, the only variable that is mislocated due to this issue that is actually used in DFHack is, ironically, the external flag, which is only used at present by autolabor. With this single exception, none of the variables in the "tail" of gamest (the members that appear after mod_manager) that is misaligned due to this issue are currently being used by any tool or script that is being distributed with DFHack.

[0x517A5D]

I've found start_dwarf_count for 50.07 Classic.

Unfortunately, it's not going to be that easy.  Setting it above 9 causes a crash.

.text:0000000140CB5976                         loc_140CB5976:                          ; CODE XREF: sub_140CB3AD0+1E35↑j
.text:0000000140CB5976 BF 07 00 00 00                          mov     edi, 7          ; START_DWARF_COUNT
.text:0000000140CB597B 48 8B 74 24 60                          mov     rsi, [rsp+2C0h+unit]
.text:0000000140CB5980 4C 8B 75 B8                             mov     r14, [rbp+1C0h+var_208]
.text:0000000140CB5984 44 8D 67 F8                             lea     r12d, [rdi-8]   ; OKAY!  This parameter needs to be set to -1 !
.text:0000000140CB5984                                                                 ; That chooses caste at random.
.text:0000000140CB5984                                                                 ; If it's >= 0, it selects that caste.
.text:0000000140CB5988
.text:0000000140CB5988                         loc_140CB5988:                          ; CODE XREF: sub_140CB3AD0+1FEB↓j
.text:0000000140CB5988 FF CF                                   dec     edi
.text:0000000140CB598A E8 A1 49 3B FF                          call    new_unit
.text:0000000140CB598F 48 8B D8                                mov     rbx, rax
.text:0000000140CB5992 48 89 44 24 60                          mov     [rsp+2C0h+unit], rax
.text:0000000140CB5997 45 8B C4                                mov     r8d, r12d
.text:0000000140CB599A 45 33 C9                                xor     r9d, r9d
.text:0000000140CB599D 0F B7 15 28 26 15 01                    movzx   edx, cs:word_141E07FCC ; example value: 23Ch, 572, DWARF.  Unit type.
.text:0000000140CB59A4 48 8B C8                                mov     rcx, rax
.text:0000000140CB59A7
.text:0000000140CB59A7                         this call crashes when startdwarf is 10 or higher.
.text:0000000140CB59A7                         turns out that's because of the caste flag in r12d
.text:0000000140CB59A7                         (passed in r8d, per above code).
.text:0000000140CB59A7                         This parameter should be -1.  When startdwarf is forced to 10,
.text:0000000140CB59A7                         r12d is set to (10-8) = 2, which is a nonexistant caste.
.text:0000000140CB59A7 E8 94 4D 27 00                          call    sub_140F2A740   ; parameters: rcx=*unit,
.text:0000000140CB59A7                                                                 ; dx = creature type index (572=DWARF),
.text:0000000140CB59A7                                                                 ; r8w = caste (-1 for random),
.text:0000000140CB59A7                                                                 ; r9b flag: set unit 6-byte field @ 0D70h

This excessively-optimized code depends on knowing that EDI is set to 7 as a space-saving way to set R12D to -1, using the calculation (7-8).

R12 is then used to select the caste of the unit being generated.  -1 means random; 0 means FEMALE; 1 means MALE; 2 or higher lead to a crash.

We need to enter the call to sub_140F2A740 with R8 == -1.  I don't see a way to squeeze that into the current code.

Anyone have thoughts?

[lethosor]

Oh, that is interesting. We have deliberately been leaving start_dwarf_count out of symbols.xml (thereby blocking startdwarf) because of this crash, but hadn't tracked down the specific reason why > 9 crashed more reliably. Good to know why that happens!

We've been considering asking Bay12 to make start_dwarf_count a proper global, or something similarly useful to us. Obviously that's not a high priority for them, and it's not near the top of our list either, so that hasn't happened yet. I'm no x86 expert so I don't have a great idea of how to squeeze in a patch in the meantime.

[ab9rf]

I've found start_dwarf_count for 50.07 Classic.

Unfortunately, it's not going to be that easy.  Setting it above 9 causes a crash.

.text:0000000140CB5976                         loc_140CB5976:                          ; CODE XREF: sub_140CB3AD0+1E35↑j
.text:0000000140CB5976 BF 07 00 00 00                          mov     edi, 7          ; START_DWARF_COUNT
.text:0000000140CB597B 48 8B 74 24 60                          mov     rsi, [rsp+2C0h+unit]
.text:0000000140CB5980 4C 8B 75 B8                             mov     r14, [rbp+1C0h+var_208]
.text:0000000140CB5984 44 8D 67 F8                             lea     r12d, [rdi-8]   ; OKAY!  This parameter needs to be set to -1 !
.text:0000000140CB5984                                                                 ; That chooses caste at random.
.text:0000000140CB5984                                                                 ; If it's >= 0, it selects that caste.
.text:0000000140CB5988
.text:0000000140CB5988                         loc_140CB5988:                          ; CODE XREF: sub_140CB3AD0+1FEB↓j
.text:0000000140CB5988 FF CF                                   dec     edi
.text:0000000140CB598A E8 A1 49 3B FF                          call    new_unit
.text:0000000140CB598F 48 8B D8                                mov     rbx, rax
.text:0000000140CB5992 48 89 44 24 60                          mov     [rsp+2C0h+unit], rax
.text:0000000140CB5997 45 8B C4                                mov     r8d, r12d
.text:0000000140CB599A 45 33 C9                                xor     r9d, r9d
.text:0000000140CB599D 0F B7 15 28 26 15 01                    movzx   edx, cs:word_141E07FCC ; example value: 23Ch, 572, DWARF.  Unit type.
.text:0000000140CB59A4 48 8B C8                                mov     rcx, rax
.text:0000000140CB59A7
.text:0000000140CB59A7                         this call crashes when startdwarf is 10 or higher.
.text:0000000140CB59A7                         turns out that's because of the caste flag in r12d
.text:0000000140CB59A7                         (passed in r8d, per above code).
.text:0000000140CB59A7                         This parameter should be -1.  When startdwarf is forced to 10,
.text:0000000140CB59A7                         r12d is set to (10-8) = 2, which is a nonexistant caste.
.text:0000000140CB59A7 E8 94 4D 27 00                          call    sub_140F2A740   ; parameters: rcx=*unit,
.text:0000000140CB59A7                                                                 ; dx = creature type index (572=DWARF),
.text:0000000140CB59A7                                                                 ; r8w = caste (-1 for random),
.text:0000000140CB59A7                                                                 ; r9b flag: set unit 6-byte field @ 0D70h

This excessively-optimized code depends on knowing that EDI is set to 7 as a space-saving way to set R12D to -1, using the calculation (7-8).

R12 is then used to select the caste of the unit being generated.  -1 means random; 0 means FEMALE; 1 means MALE; 2 or higher lead to a crash.

We need to enter the call to sub_140F2A740 with R8 == -1.  I don't see a way to squeeze that into the current code.

Anyone have thoughts?

We found the startdwarf location relatively early on in the development cycle (we have a script for it that worked, so it was trivial), but we decided (as lethosor notes) not to publish it because our testing showed that it failed for counts greater than 9. I never investigated deeply to figure out why, since we wanted to get a release out and it wasn't worth blocking for this, and honestly I've never worked back to the problem.

Good find. I'm not inclined to try to try to come up with a patch that will make this work "as desired". If someone else wants to, more power to them. If someone comes up with a patch that works for start dwarf counts less than 7 (such as, oh I don't know, one), that I might be more interested in...

Poking at it a bit, the obvious solution is to patch n into the startdwarf location and (255-n) in the last byte of the LEA instruction, exactly 16 bytes later.

[0x517A5D]

If someone comes up with a patch that works for start dwarf counts less than 7 (such as, oh I don't know, one), that I might be more interested in...

But... you can.  I just did it in the debugger.  The only trick is, you need to embark with at least 6 animals.  (4 purchased animals and the 2 wagon animals.)

Poking at it a bit, the obvious solution is to patch n into the startdwarf location and (255-n) in the last byte of the LEA instruction, exactly 16 bytes later.

On inspection, that would work.  The limit would be 126 127 dwarves before the LEA starts adding instead of subtracting.

Fragile as anything, though.  There's no telling what the optimizer will do to the code in future releases.  I sure wouldn't want to hardcode a 16 byte offset in hopes that it magically will always work.

Thank you for looking at this.

[Clément]

Where is the "init.d" directory mentioned in the documentation supposed to be created? It says "main DF directory" but it does not work for me. The only occurence of init.d I find in the code is for "raw/init.d".

Or what is the best way to run a lua command on startup? Should I rather create a "dfhack-config/init/dfhack-....init" file containing ":lua ..."?

I'm asking again. Sorry for spamming, I think it may have been forgotten at the end of the previous page.