Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Dutchling]

Im DL'ing the program right now, I assume it is the working test file? This app sounds awesome!

[thewonderidiot]

4) DF finishes the job and tries to free its memory. "Undefined" behavior by the C standard, but harmless with MSVC 8. Crash under MSVC2010.

Would you mind going into more detail about that? When you say you relocated the manager vector... you updated all of the references to that vector in DF to point to your allocated vector? What is DF trying to free, exactly?

[devek]

4) DF finishes the job and tries to free its memory. "Undefined" behavior by the C standard, but harmless with MSVC 8. Crash under MSVC2010.

Would you mind going into more detail about that? When you say you relocated the manager vector... you updated all of the references to that vector in DF to point to your allocated vector? What is DF trying to free, exactly?

The actual job is an entry in the vector. They are 64 bytes.

[agatharchides]

I'd used the test app and it spits out a lot of materials and items at me, so I guess it works?  I'm no programmer and I'm not exactly sure what I am looking for.  The idea sounss very useful and I look forward to it coming out though. 

[thewonderidiot]

The actual job is an entry in the vector. They are 64 bytes.

Well, I meant, why is it crashing, more precisely? Is it trying to remove a job from where the vector was before you moved it, or is it trying to remove a nonexistent job from your moved vector, or what?

[devek]

It isn't removing a job where the vector was before I moved it. It isn't removing a nonexistent job.

It is removing a valid job I made myself, and when it goes to free the memory it crashes since its heap manager didn't allocate that memory. MSVC8 was "graceful" enough to realize the address was outside of any valid bucket and just did nothing, which was what I relied on

[thewonderidiot]

It isn't removing a job where the vector was before I moved it. It isn't removing a nonexistent job.

It is removing a valid job I made myself, and when it goes to free the memory it crashes since its heap manager didn't allocate that memory. MSVC8 was "graceful" enough to realize the address was outside of any valid bucket and just did nothing, which was what I relied on

Ahhh, right. Are you opposed to just obliterating the call to free() with nops?

EDIT: Actually that's probably a really bad idea. Let me think about this some more.

[devek]

Well, it might have a deconstructor I could obliterate I am pretty sure it is a struct though and not a class. I could find the code that frees it and nop it out, it wouldn't be that big of a deal leaking 64 bytes of memory each time a job finished compared to the hundreds of megabytes DF already uses.

[Reverb]

Hey devek, glad to see ur back and working on this. I have some code that I wrote that does this stuff for .25 for work orders and everything seems to be working. I can add things to the queue and remove. Also I have code to produce all the possible work orders and turn them to strings. Anyways, just saying that I have code but just was kind stuck on the item linking to work order as I didn't have an elegant solution as you did, I don't think.

Update 1:
I'm using thewonderidiots remote function hook. Also I'm willing to release my code... I think... but it's not pretty and that scares me. Second, it should build and run with mingw (and QT creator) but I haven't tested for this in a while, as I'm messing with it in MSVC++ 10 atm.

[devek]

You are the yin to my yang, sending you a pm

[0x517A5D]

With the old MSVC, it was forgiving if you tried to free a memory address you didn't allocate.. with the new MSVC compiler he has been using since .13, it crashes every freaking time.

1) Foreman connects to DF.
2) Foreman allocates memory inside of DF and relocates the manager vector to that memory.

You're using VirtualAllocEx() or a close relative to reserve your memory?

3) Foreman creates a job in the allocated memory space and links it into the vector.
4) DF finishes the job and tries to free its memory. "Undefined" behavior by the C standard, but harmless with MSVC 8. Crash under MSVC2010.

It may be undefined in C, but it's explicitly illegal in C++.  I'm not going to try to dig out the actual spec, but here's a bit from cplusplus.com:

void operator delete[] (void* ptr) throw ();
Parameters
ptr Either a null pointer, or a pointer to a memory block to be released, type-casted to a void*.
This pointer value should have been returned by a previous call to operator new[].

Where it says should have, I would substitute must have.

Do you, by any chance, know the addresses of the relevant DF code that is doing the allocation or freeing?  I mean not the malloc(), free(), new(), or delete() functions, but the relevant call to them, or the constructor or destructor.  In any DF version.  I'd like to trace through them.

I would probably attack this problem by:

• inserting a 4K memory block via VirtualAllocEx(),
• giving the block execute privileges, using VirtualProtectEx()
• injecting a bit of assembly code into that block that
  • calls realloc() or new(),
  • stashes the return values of the call at the end of that block of memory,
  • and then calls ExitThread().
• create a new thread pointing at that injected code using CreateRemoteThreadEx() or similar.
• WaitForSingleObject() on that thread's handle,
• grab the stashed return values and do whatever memory massaging you need to.
• optionally call VirtualFreeEx() on our injected memory block, just to be tidy.  If the allocation needs to happen more than once, just leave it in place forever.

But that's just me.

Fake Edit: Since DF already imports all of malloc(), free() realloc(), new() and delete(), you can just grab the addresses of them from the import table.  You don't need to use LoadLibrary() / GetModuleHandle() / GetProcAddress().

[devek]

Haha... so I fixed the crash problem. I redirected the msvc2010 delete operator to a piece of code that checks to see if the memory belongs to me or dwarf fortress. If it belongs to me, it just returns but if it belongs to df it sends it to the real delete operator. This is actually perfect because it will result in 0 memory leaks AND get rid of the pesky undefined behavior of what happens when a program frees something it didn't allocate.

This might be useful to somebody someday.. or not. I'll be busy this weekend with my daughter, but I should be able to get a release out next week for sure now.

Code: [Select]

    // one last EVIL dead to do. put the free operator at +FB0 and load our own goodie in there.
    uint32_t realfree = readDWord(dfbase+0x67E2C0);
    actionLog("Hooking into the free operator at " % QString::number(realfree,16));
    uint32_t mystuff = readDWord(queuePointer);
    writeDWord(mystuff+0xFB0, realfree);
    realfree = mystuff+0xFB0;

    uint8_t in[45];
    in[0]  = 0x55;                                                              // push ebp
    in[1]  = 0x89; in[2]  = 0xE5;                                               // mov ebp, esp
    in[3]  = 0x83; in[4]  = 0xEC; in[5]  = 0x18;                                // sub esp, 18
    in[6]  = 0xA1; memcpy(in+7, &mystuff, 4);                                   // mov eax our memory
    in[11] = 0x39; in[12] = 0x45; in[13] = 0x08;                                // cmp [ebp+08],eax
    in[14] = 0x72; in[15] = 0x0F;                                               // jump +15 if it isn't our memory
    in[16] = 0xA1; memcpy(in+17, &mystuff, 4);                                  // mov eax our memory
    in[21] = 0x05; in[22] = 0x00; in[23] = 0x20; in[24] = 0x00; in[25] = 0x00;  // add eax 0x2000
    in[26] = 0x3B; in[27] = 0x45; in[28] = 0x08;                                // cmp eax, [ebp+08]
    in[29] = 0x73; in[30] = 0x0C;                                               // jae 12 it is our memory
    in[31] = 0x8B; in[32] = 0x45; in[33] = 0x08;                                // mov eax, [ebp+08]
    in[34] = 0x89; in[35] = 0x04; in[36] = 0x24;                                // mov [esp], eax
    in[37] = 0xFF; in[38] = 0x15; memcpy(in+39, &realfree, 4);                  // use the real free operator
    in[43] = 0xC9;                                                              // leave
    in[44] = 0xC3;                                                              // ret

    WriteProcessMemory(hDF, (void *)(mystuff+0xFC0), (void *) in, 45, 0);
    writeDWord(dfbase+0x67E2C0, mystuff+0xFC0);

[devek]

Do you, by any chance, know the addresses of the relevant DF code that is doing the allocation or freeing?  I mean not the malloc(), free(), new(), or delete() functions, but the relevant call to them, or the constructor or destructor.  In any DF version.  I'd like to trace through them.

First, thanks for the response.

Second dfbase + 0x29126C (.text:0069126C in IDA) is where it calls delete on a manager job. you can trigger it by going in the job->manager menu and adding and then deleting a job.

Second, while your method would be perfect. I have a (probably too large) single chunk or memory I am doing all my work from. As long as DF simply ignores anything in my chunk, all is good. What do you think?

Fake Edit: Since DF already imports all of malloc(), free() realloc(), new() and delete(), you can just grab the addresses of them from the import table.  You don't need to use LoadLibrary() / GetModuleHandle() / GetProcAddress().

Correct, I nabbed it from the .idata segment

[thewonderidiot]

Haha... so I fixed the crash problem. I redirected the msvc2010 delete operator to a piece of code that checks to see if the memory belongs to me or dwarf fortress. If it belongs to me, it just returns but if it belongs to df it sends it to the real delete operator. This is actually perfect because it will result in 0 memory leaks AND get rid of the pesky undefined behavior of what happens when a program frees something it didn't allocate.

This might be useful to somebody someday.. or not. I'll be busy this weekend with my daughter, but I should be able to get a release out next week for sure now.

Code: [Select]

le codes

Ah, well done. I approve of this.

[devek]

Ok, I kind of dumped out a little prerelease for you guys. It hasn't really been that tested well and its .25 only. What it does do, it should do well and with few if any bugs.
http://dffd.wimbli.com/file.php?id=2806

This is kind of where I am heading:

0.7 = support more complicated jobs and actually be tested
0.8 = add most jobs and put auto tree cutting back in, i hate elves
0.9 = support all jobs put the ability to mill specific things back in(something you can't do in DF even ) changes to beta
1.0 = fix all bugs found in 0.9, throw the thing up on google code and start working on my next project.

Feed back is appreciated!