Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: 1 ... 29 30 [31] 32 33 ... 244

Author Topic: DFHack 50.14-r1.1  (Read 889203 times)

lethosor

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #450 on: November 27, 2017, 05:00:22 pm »

Where do you need hands at?  I'm not the greatest at things (more brute force than anything), but I don't mind trying.  I also have a couple of programmer friends on call.  I guess what I'm asking is what needs to be done to get a test DFHack version up and going?
We mostly need to verify that structure layouts are right, and fix ones that have changed. It's hard for me to explain exactly what to do there, but if you can compile a bleeding-edge version of DFHack (with up-to-date structures, which might require an extra "git pull" in library/xml), you could poke around at things in the Lua interpreter and point out anything that looks wrong or crashes.

I'd be looking at it myself but naturally can't, but what is the new stuff in the xml export exactly?

Assuming you mean the DF XML export, I'm not that familiar with it, but here's the output of "strings dwarfort.exe | grep '^<.*>'", which grabs all strings starting with a <:
Spoiler (click to show/hide)
Besides things like <SWAP> and <no colors>, those all seem to be XML export-related, and some look new.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

Moonshine Fox

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #451 on: November 28, 2017, 02:14:51 am »

What is the best method for someone who is not a developer (my coding abilities extend to semi-complex python scripting and some basic Java coding) to find out the progress towards having DFHack up to date with DF? I don't want to bother the devs or spam threads every time there's a new DF version. Instead, is there a way for me to find this out directly, on my own? If so, could some kind soul point me in that direction?
Logged

Max™

  • Bay Watcher
  • [CULL:SQUARE]
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #452 on: November 28, 2017, 03:27:58 am »

I meant the: "Various additional data in the XML export" from the release notes, is there not anything relevant to getting dfhack working added this time?
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #453 on: November 28, 2017, 04:48:29 am »

I would have thought you already had a set of basic sanity checking tools for the structures that are mapped?
I'm thinking of something that crawls through df.global depth first and verifies that fields with known bounds are within those, and probably crashing (with a log indication of where) when assumed pointers don't point to legal locations as it attempts to follow the pointers.
- The most trivial sanity check would probably be to verify that enum values either have a legal value or -1.
- Another check would be to verify that a value that's a reference into a vector is within that vector's range, or that an identifier (such as a hist fig id) lies within the range of the value of the first entry and the last one. That would require meta info on the field to specify what it refers to, though.
- A third check would be to verify that arrays that should be the same length are (such as the coordinate vectors where X and Y are stored in separate arrays). Again, meta info would be required.
- A fourth check would be to check that arrays have the correct length, such as arrays that should be the size of the world width/height. Meta info required, of course.

Edit: Some rather brief poking around using some of my scripts as probes only found a single issue: the DFHack Lua scriptery that displays which key a command is bound to always prints a question mark for me (as seen in gui/gm-editor's help screen, where all the commands are presented as "?: ...". The bound keys seem to work when pressed, however. I guess this is really not a structure issue, though, but a script one, for the next stage of the update.
« Last Edit: November 28, 2017, 08:08:16 am by PatrikLundell »
Logged

Clément

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #454 on: November 28, 2017, 09:48:53 am »

From devel/dt-export-ini.lua:
Code: [Select]
--WARNING below value should be: "general_ref::vtable","1","0x8","0x4","vmethod","getType","general_ref_type",""
value('ref_type',0x8)

I think this one should be 0x10 on 64 bits platforms (getType is the third method in general_ref vtable). Does the comment mean there is a proper way to get the value or should I add a test based on the architecture?
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #455 on: November 28, 2017, 09:59:48 am »

Minor result from poking around:
Found a dorf with a misc_unit_trait.id = 63 (nil)
                                               .value =45
I suspect this means new misc_trait_type values have been added, since I can't find any value out of bounds on half a dozen 0.43.05 dorfs.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #456 on: November 28, 2017, 11:22:14 am »

What is the best method for someone who is not a developer (my coding abilities extend to semi-complex python scripting and some basic Java coding) to find out the progress towards having DFHack up to date with DF? I don't want to bother the devs or spam threads every time there's a new DF version. Instead, is there a way for me to find this out directly, on my own? If so, could some kind soul point me in that direction?
In the past, we have pointed people to "v0.xx.yy.lst" in https://github.com/dfhack/df-structures. There isn't one yet for 0.44.x, though, because the Linux build was only just released. Here is 0.43.05's; "UNCHECKED" is (potentially) bad, "ALIGNED" is good, "VERIFIED" is human-checked, so hopefully entirely right.

There are usually other things to be done too. You can look for the "r1" milestone under https://github.com/dfhack/dfhack/issues, although it doesn't currently have anything assigned.

I meant the: "Various additional data in the XML export" from the release notes, is there not anything relevant to getting dfhack working added this time?
Toady added global addresses in 0.44.01, which are in a global table in the DF executable, nothing to do with the XML export.

I would have thought you already had a set of basic sanity checking tools for the structures that are mapped?
Yeah, we have some tools (cl-linux-debug, although it wasn't as useful without a Linux build). Mifki is actually working on another one right now, a Lua script.
Quote
I'm thinking of something that crawls through df.global depth first and verifies that fields with known bounds are within those,
Not sure what exactly this part means - it's nearly impossible to check primitive field sizes (integers)
Quote
and probably crashing (with a log indication of where) when assumed pointers don't point to legal locations as it attempts to follow the pointers.
We can check pointer validity, but it's a little slow. Checking vector validity isn't too hard either, but other things (like strings) can be more complicated.
Quote
- The most trivial sanity check would probably be to verify that enum values either have a legal value or -1.
Mifki's script does this, I think. (-1 is only legal for some.)
Quote
- Another check would be to verify that a value that's a reference into a vector is within that vector's range, or that an identifier (such as a hist fig id) lies within the range of the value of the first entry and the last one. That would require meta info on the field to specify what it refers to, though.
The problem there is that "ref-target" attributes from the XML files aren't available to Lua, so Lua scripts can't tell which vector a field like "unit.histfig_id" is for. cl-linux-debug can do this, I think.
Quote
- A third check would be to verify that arrays that should be the same length are (such as the coordinate vectors where X and Y are stored in separate arrays). Again, meta info would be required.
- A fourth check would be to check that arrays have the correct length, such as arrays that should be the size of the world width/height. Meta info required, of course.
Assuming you mean vectors; there's no way to check this reliably for C arrays. There are relatively few cases where we could use this metadata, and fewer still where we'd run into valid vectors that should have the same dimensions but don't, so it's probably easier to just check things like that manually.

Quote
Edit: Some rather brief poking around using some of my scripts as probes only found a single issue: the DFHack Lua scriptery that displays which key a command is bound to always prints a question mark for me (as seen in gui/gm-editor's help screen, where all the commands are presented as "?: ...". The bound keys seem to work when pressed, however. I guess this is really not a structure issue, though, but a script one, for the next stage of the update.
That's because the "keydisplay" offset is missing on Win64 (I'm assuming you're using that, since that's the only platform that has any sort of 0.44.02 support but not that offset, but specifying is helpful). That points to a map of interface_key's to strings, which DF uses to display keys, and DFHack relies on as well (in Screen::getKeyDisplay()). You'll see the same issue in things like the manipulator plugin, so it's not Lua- or script-specific.

From devel/dt-export-ini.lua:
Code: [Select]
--WARNING below value should be: "general_ref::vtable","1","0x8","0x4","vmethod","getType","general_ref_type",""
value('ref_type',0x8)

I think this one should be 0x10 on 64 bits platforms (getType is the third method in general_ref vtable). Does the comment mean there is a proper way to get the value or should I add a test based on the architecture?
Sounds like you want 2*sizeof(void*), aka 2*dfhack.getArchitecture()/8.
The comment is referring to a line from CSV files exported by some df-structures-related tool that I've never really used much. I believe the old Perl script to generate DT layouts uses those CSV files, so maybe that's why the comment is there.

Minor result from poking around:
Found a dorf with a misc_unit_trait.id = 63 (nil)
                                               .value =45
I suspect this means new misc_trait_type values have been added, since I can't find any value out of bounds on half a dozen 0.43.05 dorfs.
Yeah, that's probably the case, thanks. That's a "new" dwarf, right? (One generated in 0.44)
« Last Edit: November 28, 2017, 01:17:21 pm by lethosor »
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #457 on: November 28, 2017, 11:41:21 am »

:
I'm thinking of something that crawls through df.global depth first and verifies that fields with known bounds are within those,
:
Minor result from poking around:
Found a dorf with a misc_unit_trait.id = 63 (nil)
                                               .value =45
I suspect this means new misc_trait_type values have been added, since I can't find any value out of bounds on half a dozen 0.43.05 dorfs.
Yeah, that's probably the case, thanks. That's a "new" dwarf, right? (One generated in 0.44)
The bounds comment intended to refer to the contents of the field, such as an X coordinate being 0 - X_Dim - 1 or -30000. The intended purpose of that kind of checks (as well as those of vectors) would be to flag that the field might no longer be at that location.
 
My poking is done on a Win64 installation (the LNP is the base), correct, and I didn't find it useful to poke in a ported save, so it's a 0.44.02 world, but it's a relevant question, of course. (Note to self: remember to provide all the relevant info).
Logged

Clément

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #458 on: November 28, 2017, 11:55:11 am »

Sounds like you want 2*sizeof(void*), aka 2*dfhack.getArchitecture().
The comment is referring to a line from CSV files exported by some df-structures-related tool that I've never really used much. I believe the old Perl script to generate DT layouts uses those CSV files, so maybe that's why the comment is there.
"2*(dfhack.getArchitecture()/8)" more exactly. So, there is no way in the current script API to know where "getType" is in the vtable.
Logged

Moonshine Fox

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #459 on: November 28, 2017, 12:51:53 pm »

What is the best method for someone who is not a developer (my coding abilities extend to semi-complex python scripting and some basic Java coding) to find out the progress towards having DFHack up to date with DF? I don't want to bother the devs or spam threads every time there's a new DF version. Instead, is there a way for me to find this out directly, on my own? If so, could some kind soul point me in that direction?
In the past, we have pointed people to "v0.xx.yy.lst" in https://github.com/dfhack/df-structures. There isn't one yet for 0.44.x, though, because the Linux build was only just released. Here is 0.43.05's; "UNCHECKED" is (potentially) bad, "ALIGNED" is good, "VERIFIED" is human-checked, so hopefully entirely right.

There are usually other things to be done too. You can look for the "r1" milestone under https://github.com/dfhack/dfhack/issues, although it doesn't currently have anything assigned.


Thanks! That's exactly what I was looking for.
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #460 on: November 28, 2017, 01:23:40 pm »

"2*(dfhack.getArchitecture()/8)" more exactly. So, there is no way in the current script API to know where "getType" is in the vtable.
Whoops, fixed. You could also use something like df.reinterpret_cast('pointer',1):sizeof(). Apparently df.sizeof('pointer') isn't supported, though.

It doesn't look like it, sorry. I poked around in debug.getmetatable(df.general_ref) and debug.getmetatable(df.general_ref:new()), although the closest thing I could find, "_index_table" in the latter, just has data fields, not methods (and is empty in this case).

Thanks! That's exactly what I was looking for.
If you're talking about the .lst file, it's not always kept up-to-date, unfortunately, but we'll try.

Someone (Japa?) made an SVG chart back in the early 0.40 days that updated periodically - I could perhaps look into doing that again.
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.

Max™

  • Bay Watcher
  • [CULL:SQUARE]
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #461 on: November 28, 2017, 05:26:51 pm »

Ohhhh, so the global addresses may be easier to get translated into a symbols table then, it sounded like he was going to do one of the "pass -a flag or some shit and it spits out a table of addresses" suggestions a while ago but I never saw where he went with it.
Logged

PatrikLundell

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #462 on: November 29, 2017, 07:12:19 am »

Another DF structure issue:
There's something off with unit.status.current_soul.performance_skills.

The same save sometimes have the field containing nil, sometimes a value that looks like a legal pointer (and points to data that seems reasonable, with empty vectors), and sometimes contains a suspect pointer value that crashes DF when you try to follow it.
Note that the same dorf can have a pointer in one load of the save, but not the next one (with the save not being updated in the mean time).
Current example: unit personality pointer: 000001B715BA8638 and performance_skills pointer being 00D2090000170612, and, as expected DF went belly up as I tried to follow the pointer.

My save is a win64 0.44.02 generated world (from PSV data) with the raw changes being lowered ATTACK_TRIGGER values for (semi)megabeasts as well as changed siege triggers for the civs. Goblins set to start on glaciers only.

Edit: Concerning the misc_unit_trait mentioned earlier:
- Immediately after embark all 7 had WantsDrink, except for the expedition leader who also had a RoomComplaint.
- Releasing the pause for a few second addeded 3 values to all of them: ClaimClothingCooldown, CleanSelfCooldown, and the reported 63. The expedition leader also added id=21, value=545, so it seems one of the previously unused values has been used (and my guess would be that they probably all have been used, as 63 is a fair bit higher than the previous last value, indicating a need for a fair number of new values).

Edit2: Probably a binding issue of some kind rather than a layout issue: a unit.meeting.state value of 0 is considered to be nil by Lua (gm-editor as well as a home brew script). As far as I see from the XML that should be "SelectNoble". It can be noted that the enum is defined inside the struct element rather than as its own type.

I also remembered that I still failed to describe the world gen conditions, as I'd forgotten that I'd tried to force goblins to start on glaciers (corrected above).

Edit3:
- df.global.world.units.all[].job.mood_skill: Enum out of whack with 4-5 digit values.
- df.global.world.units.all[].body.body_plan.interaction [].interaction.material_breath: Several values of 20, with current max = 19. The one I looked at was a cat and other data indicates it's a licking interaction, and also a head-bump interaction with the same value for the same cat.
- df.global.world.entities.all[].armies[].controller: Following this pointer crashed both gui/gm-editor and my script. The pointer address looked reasonable.
« Last Edit: December 01, 2017, 05:36:48 am by PatrikLundell »
Logged

Clément

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #463 on: December 02, 2017, 05:01:41 am »

Is there any blocker for updating df-structures' symbols.xml with linux versions? Is help needed?
Logged

lethosor

  • Bay Watcher
    • View Profile
Re: DFHack 0.43.05-r3.1
« Reply #464 on: December 02, 2017, 12:26:03 pm »

There weren't Linux builds of DF for a while, so that's behind. You could run devel/dump-offsets and provide the output of that, but I'm not sure what else needs to be done.

Edit: looks like that was done already.
« Last Edit: December 02, 2017, 10:13:29 pm by lethosor »
Logged
DFHack - Dwarf Manipulator (Lua) - DF Wiki talk

There was a typo in the siegers' campfire code. When the fires went out, so did the game.
Pages: 1 ... 29 30 [31] 32 33 ... 244