Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[clinodev]

Thank you for all your work on this project!

[RedDwarfStepper]

A new DFHack version has been released. This also means all changes made to this plugin since the last release are released as well.
I'll try to remember what has been done over this period...

Splendid!

Being nitpicky I know, but there was also the bug triggered by pocket worlds:

Thanks for the crash report.

It took a bit of time to find out what was wrong, but it is indeed a bug.
The line should be

Code: [Select]

            tile->region_type[i][k] = world_data->regions[tile->biome_index[mlt->at(i).at(k).biome_offset]]->type;
i.e. "biome" -> "biome_index".
The reason it crashes only on pocket worlds is that the number of regions in those worlds tend to be fewer than the number of biome types, i.e. the index passed in was the biome type, but was interpreted as the index of the region.
The bug should screw up the reasonably (I think) uncommon searches for region types when it doesn't cause a crash.

Also thank you for your fast answer concerning incursions.
I'll have to do some more code reading and stepping through the code during survey and matching phase in a small world to get a better grasp of the details.
What I can say is this: The index structure actually would allow for storing data of an incursion from a neighbor before the receiving MLT was being surveyed, all that is needed is the correct key, that can be calculated - thanks to you - but it might be rather complicated to get rid of those if they are an ocean/mountain and the receiving MLT is e.g. a desert.
This also results in a question: Does the following sentence only mean that the biome won't be transferred or does it also include all other features (aquifer...).

There are also certain rules forcing ocean/lake biomes to lose edges to mountains, and both of them to anything else, no matter what the original array value is.

If this includes the biome and all the features I would have to retain all this info for later, so it could be undone.
If this only refers to the biome it just could be removed if the receiving MLT has a "stronger" biome type.

Also, you can't just merge things together as you go, as that would cause incursion info into a tile to be propagated into further tiles as part of the "real" data for that tile.

Currently my design keeps the real source data (MLTs) separate from the index data (pure sink) during the survey phase - so any further propagation should not happen, but I'm wondering, how could further propagation happen if always 2 tiles (with the exception of corners) are being linked as you said here

every "receiver" is matched by a "giver" in the corresponding neighboring tile (corners has one "giver" and 3 "receivers"), and there is no "neutral" choice: there is an intrusion in one direction or the other.

Sorry if I should have understood/interpreted that wrong, which is quite likely. That would be just more evidence I need to see those rules and the data in the context of program execution.
Ah - I think I'm starting to understand: There are up to 8 incursion giver candidates (the neighbors of the MLT as corners and edges) for every MLT (exceptions world-corners and -edges...) but only 1 of them actually will be a giver, but the other 7 could be the giver to another MLT, which includes the winning candidate, which would lead to further propagation. Yes?

Again thanks for your patience!

[PatrikLundell]

Yes, the bug you reported was fixed as well, although I missed reporting it.

Incursions are packet deals, to the mountain etc. incursion inversion applies to everything. Instead of a mountain incursion into a steppe, we'd get a steppe incursion into the mountain, with all the associated data.

Hypothetical incursion contamination process:
- Survey World Tile A, and apply incursion information to border World Tile B MLTs that haven't been surveyed yet. As a result, some of these border MLTs may have info about containing biome A, B, and C (because it happened to receiving incursions both from the corners and the edge).
- Process World Tile B, where the edge tile above is identified to have a native biome D, so its biome set is now (A, B, C, D).
- Process incursions in World  Tile B, at which time we note that the problem MLT above makes incursions into two neighboring tiles, resulting in those tiles getting tagged with biome A, B, C, and D in addition to whatever biome it has natively, but the incursion actually contains only D.
The above also goes for other properties, such as e.g. aquifers.

You can't keep all the data from all MLTs in the world throughout the survey process, or you'd use up too much memory: you have to discard the data when the processing of it is done as you go. I've retained the border MLT info for incursion processing but discard the interior data when the processing of that world tile is finished. You'd have to do something similar, but can be more intelligent about it and discard the border data as it has been used for all relevant incursions.

It seems like you misunderstand the giver-receiver relationships. For a given (interior) MLT there are 4 edge incursions either going inwards or outwards so the tile can receive 0-4 incursions and perform 0-4 incursions, with the sum being exactly 4.
For each corner there is exactly one incursion provider and exactly 3 incursion receivers (all receiving from the same provider).
Thus, if you know the situation for one side of the arrangement, you also know the situation for all the other sides of that arrangement, which is why DF stores the info in a single place for each of these interface points. If the East side of an MLT received, that also means the West side of the MLT to the East gave.
There is no relation between different interface points (i.e. edges or corners) of an MLT. Each one of them is related only to the counterpart(s) for that interface point (which reside(s) in (a) different MLT(s)). Having an incursion in one place says absolutely nothing about the state of other edges/corners of the same MLT.

[RedDwarfStepper]

Thank you for those explanations!

I did some more reading and are currently in the process of stepping through the code that handles the incursions - as part of that I created the following annotated call tree:

Code: [Select]

# matcher::embark_match
// tldr; does the actual matching for the current MLT and delegates the incursion processing for the outer edges of the embark rectangle as there is no need to process the inner part of the embark rectangle
// starting ~ line 773 the both north corners (NW+NE) and the north edge of every northern embark MLT of the embark area, then the south corners (SW+SE) and the south edge of every southern embark MLT are being processed
// starting ~ line 888 both west corners (NW+SW) and the west edge of every western embark MLT and then both east corners (NE+SE) and the east edge of every eastern embark MLT of the embark area are being processed
// the second loop respects all already processed corners, so no duplicate processing
    # survey::translate_corner/survey::translate_ns_edge/survey::translate_ew_edge
    // those 3 methods apply the rules described in world_region_details for edges.biome_corner/edges.biome_x/edges.biome_y to find the origin direction of the incursion to the current MLT corner/edge
    # matcher::process_embark_incursion_mid_level_tile
    // locates the correct incursion origin MLT + related RTD/world tile with the passed origin direction for further processing       
        # matcher::process_embark_incursion
        // actually processes the matching with the incursion data with the correct MLT and RTD/world tile

Is that basically correct?

And I'm wondering: Could it be, that in embark_assist::survey::translate_ew_edge instead of

Code: [Select]

biome_x

Code: [Select]

biome_y should be used at the following lines:
https://github.com/DFHack/dfhack/blob/82f082d7cbd6b823ab4c54d0dfbbf578f7fb1c4a/plugins/embark-assistant/survey.cpp#L1912
https://github.com/DFHack/dfhack/blob/82f082d7cbd6b823ab4c54d0dfbbf578f7fb1c4a/plugins/embark-assistant/survey.cpp#L1917
as the edges between east/west tiles run vertically/in y-direction?
Or am I getting something wrong here again?

[PatrikLundell]

It's a bit hard to follow your description as the line numbers are too approximate to allow me to be certain about what you refer to. However, I assume it's the section following "//  Take incursions into account." at line 731 in my copy of the code.

Yes. You don't have to process incursions in the interior of the embark rectangle, because those incursions will contain things that are already accounted for as part of the info for the MLTs they originate from.

Yes. The first loop deals with the top and bottom rows of the embark, failing to perform incursion handling on interface points whose neighbors haven't been surveyed yet (because they're in a different world tile that hasn't been surveyed).

Yes. The second loop does the same for the east and west edges.

Yes. The translate... figure out incursion directions with respect to the subject (the magic direction number parameter), taking complications (world borders, oceans, etc.) into account.

Don't understand what Return To Desktop means in this context...
Yes. process_embark_incursion_mid_level_tile just locates the correct incursion MLT and feeds it to process_embark_incursion, which performs the actual processing. It would have been possible to use a variable to store the pointer to the MLT and make a call at the end rather than perform the call directly in each MLT selection case, which would have resulted in fewer lines.

Yes, again. It's a bug, as you assumed. It should indeed use "y", not "x".

[vjek]

Looking for a feature enhancement to this tool, here's the use case, briefly, with more details after:
- Dynamic population of a list of REGIONAL_ effects that can be selected for filters
- Application of these filters shown on the world map, same as the filters for flux, sand, biome, metal, and so on.

The goal of the change would be to allow players to find which REGIONAL_ interactions exist on what tiles.
Currently, effects such as blood rain, thralling, and re-animation are covered.  This would be an additional feature, but not quite so specific or static.

In advanced worldgen, when "Number of Regional Interaction Types" is set to a value (value of 10 shown in the spoiler), this number of REGIONAL_ values is created in: df.global.world.raws.interactions

Spoiler: region interactions list (click to show/hide)

Code: [Select]

[lua]# for x=0,#df.global.world.raws.interactions-1 do print(df.global.world.raws.interactions[x].name) end
CLEANING
BP_BUMP
PET_ANIMAL
MATERIAL_EMISSION
MATERIAL_EMISSION_WITH_HIDE_EFFECT
UNDERGROUND_SPECIAL
REGIONAL_1
REGIONAL_2
REGIONAL_3
REGIONAL_4
REGIONAL_5
REGIONAL_6
REGIONAL_7
REGIONAL_8
REGIONAL_9
REGIONAL_10

If this list of REGIONAL_ values could be enumerated in a select-able list for embark filtering, this would allow players considerable flexibility in selecting which type of unique Regional Interaction they wanted to embark on.

It doesn't particularly matter where the list is enumerated in the UI.  It could go somewhere between Blood Rain and Reanimation in the list, perhaps with the heading/title: Interactions or Regional Interactions.
As a bonus, if, when traversing the list (similar to biome manipulator) it also showed the features of each Regional Interaction, then an informed decision could be made by the player regarding which one they wanted.  These REGIONAL_ values vary randomly by world, and are procedurally generated.

I think all the lists, arrays, etc and locations of them are known, and all the relevant code already exists in either survey.cpp or biomemanipulator.lua, currently.
Any questions for clarification, happy to oblige.

[PatrikLundell]

Unfortunately, that's tricky for a number of reasons:
- There's no telling what values a player may select for the number of regional interactions. It may well be in the thousands (the world I look at now has 11601 interactions, although most aren't regional, of course, but I wouldn't rule out hundreds or a few thousand).
- Second level selections (i.e. have an selection criterion expand) is not trivial. I've sort of worked around it for a future update by adding the dynamic entries at the end, but that trick can only be performed once. It's probably not impossible to implement a two level UI, but it has to be readable as well when contracted. However, the criterion "regional interaction has to be X" where the list is populated dynamically isn't that hard to implement (the set of minerals, for instance, is read from the raws section to allow for modded ones). However, finding out that you want REGIONAL_798 is better done using some other tool (such as the Biome Manipulator or, better, a new tool someone writes to mess with interactions).
- Interactions is a painful lot to deal with due to how messy they are (but there are people who like to get their hands dirty!), and it can be noted that the Biome Manipulator displays only a subset of the info (and there's a reason there's no functionality to change interactions themselves: it's possible to do, but would require a lot of UI, including understanding of additional parts).
- Also note that when you select a particular interaction you implicitly select a particular region (I think interactions can be shared, but if they can, it probably doesn't happen often), and that means you've got very little room for additional constraints on your embark. Thus, I think changing the world to match the desires is probably better than searching for a lucky hit in most cases.
- When it comes to the mapping of interactions, there's some new stuff that I don't think has been identified, but I suspect it doesn't affect regional interactions, at least not vanilla ones. Also note that I'm not sure how much the spreading evil thingie affects the logic. I think the logic is still the same, but there might be subtle changes that will cause the current logic to make incorrect evaluations.

[vjek]

Ok, so, say you've isolated that REGIONAL_4 is the regional interaction that you're interested in finding, to embark where it exists.
Manually (or otherwise), what structures would you need to iterate through to see if it's present in a given embark tile?
Presumably something in df.global.world.world_data ?  Or is it not present there, in a searchable form?

[PatrikLundell]

You actually have to go via the interaction, and an interaction can only be tied to one region. Here's the relevant part from the Biome Manipulator:

Code: [Select]

    for i, interaction in ipairs (df.global.world.interaction_instances.all) do
      if interaction.region_index == region [Surface] then
        Weather_Page.Interaction_Index = interaction.interaction_id
        break
      end
    end

So, once you've found the region, you'll have to iterate through the list of world coordinates in the df.global.world.world_data.regions [interaction.interaction_index].

Going from an embark tile, you'd find the df.global.world.world_data.region_map [x]:_displace (y).region (or possibly region_id, I don't have the XML in front of me) and then iterate through the interaction_instances to find a match (or not, of course) for that id.


[PatrikLundell]

Updates have been accepted into the DFHack develop branch. Since we haven't yet reached the stage where DFHack is released as such, but nightly builds are made available, they're announced now (but if you don't build DFHack yourself you may have to wait another night for a build...).

Version 0.11 provides some new/changed functionality:
- Aquifer searches and display now distinguishes between Light and Heavy aquifers. The embark display can e.g. show "No Lt Hv" if the embark has some parts without aquifer, some with Light and some with Heavy.
- Min/Max tree search criteria have been introduced. I still don't understand why people want it, but now it's there...
- The largest addition is the ability to search for embark with particular neighbors (made possible through the identification of a previously misunderstood flag). This includes the ability to search for min/max necro neighbors. The neighbors are listed among the embark info, primarily because the Kobolds that the vanilla display suppresses are displayed.

[lethosor]

Since we haven't yet reached the stage where DFHack is released as such, but nightly builds are made available, they're announced now (but if you don't build DFHack yourself you may have to wait another night for a build...).

"nightly" doesn't actually mean that - builds are usually finished within an hour, unless there are a lot of builds running.

[RedDwarfStepper]

Ok, so here we go, after working on the indexing for the last couple of months I have a meaningful update:
I found an easy way to speed up (~10%, which translates into around 1:25 minutes for a 257x257 region) the search and also one to reduce the consumed memory significantly (~ 29%-36% of total used memory or ~ 77%-79% of the additional memory footprint of the plugin which translated to ~460 - 490 MB for a 257x257 region):
Have a look at
https://github.com/bseiller/dfhack/commit/9f53558267ef9f3769b310641d4c6fbc7e9576de
for the speep up - basically there were two maps being filled but never read (dead stores), removing them does the trick.
and here
https://github.com/bseiller/dfhack/commit/c07a0f1ff4cca85cd752ff26f358be49f8ec7cfe
for the changes to save memory - using a smaller struct with only those attributes that are relevant for incursions

I have incorporated those changes in my development - currently I use about 140MB RAM (some attributes are still missing) for the indices so I've still around 320MB to spend
@PatrikLundell:
Let me know what you think.

PS: I also added some output to the console to make measuring the time a search takes more convenient:
https://github.com/bseiller/dfhack/commit/532cccde8a290c2f9e02d945c963c7fa3f65b063

[PatrikLundell]

- Good find with the speedup. This comes from replicating the code from Prospector, but that tool actually uses the info to reduce the amount of minerals found when features are present, but that part isn't relevant to this code.

- Splitting the data into a part relevant to all and a part for the non incursion related stuff is a smart move.

- Adding a timer during the work is definitely useful. It can be commented out when the work is done (and uncommented when more work is needed: There's a reason for commenting out the "out" variables rather than removing them, beyond pure laziness).

[RedDwarfStepper]

I was just going through the incursion processing code and stumbled upon this:
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/survey.cpp#L2427
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/survey.cpp#L2484
vs.
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/matcher.cpp#L1083
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/matcher.cpp#L1152
Shouldn't this be
embark_assist::survey::translate_ew_edge
in survey.cpp#L2427 and survey.cpp#L2484, too?

[PatrikLundell]

Yes, it should.