Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[RedDwarfStepper]

Ok, good to know - that means it should be as well used here
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/survey.cpp#L2427
and here
https://github.com/DFHack/dfhack/blob/6bdbf5b0ddaa045a8fc6ff91e91dc30cb3d21e3f/plugins/embark-assistant/survey.cpp#L2484
in embark_assist::survey::survey_embark

Never mind - having multiple branches of the sources in parallel is not easy on my memory

Do you want a pull request for that?

[PatrikLundell]

I'll make one, thank you. I just haven't been in a hurry given that it's likely there won't be a new DFHack version for some time as the rate of change has slowed down.

[RedDwarfStepper]

Ok, got it.

I think I might have found another bug, for real this time
Actually one could count it as two, but with the same pattern and the same fix.
Have a look at
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1922
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1996
If k is 15 (in the case of translate_ns_edge) or i is 15 (in the case of translate_ew_edge), which means the incursion would cross world-tile boundaries to the south/east a wrong/illegal value for biome_x/biome_y is being used (out-of-bound read access for i/k==16). Whatever memory is being read is being interpreted as int8_t, but might be something completely different.
There is no access violation or any other error thrown, but there is a chance that the incursion results for these corner cases are bogus - also depending on the regions of the two embark/mid_level_tiles
involved in the comparison.
Do you agree with my assessment?

If I'm right, the fix is two-fold:
1. store world_data->region_details[0]->edges.biome_x[0-15][0]/biome_y[0][0-15] in the region_tile_datum, similar to
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/defs.h#L116
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/defs.h#L117
retaining the the northern row of biome_x and the western column of biome_y.

2. add distinct cases when accessing biome_x/biome_y similar to
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1684
to
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1692
including adjust_coordinates as in
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1660

To allow for a 1/1.5 pass incursion processing I'll need the data anyway - so I can provide the necessary changes once I've made them and they work as intended.

Ah, and one actual question concerning the incursion processing at the edges of the world.
Of course there can't be any processing of edges from/to the outside of the world.
But what about the corners? Let's say we're at the southern edge of the world (y == world_data->world_height - 1 && k == 15).
Would it be possible to process the southern (-west/-east) corners to see if there is an incursion from/to the neighboring tile (i+/-1) within the same world-tile, ignoring the 2 non-existing corners/tiles further to the south?
Or would the needed data live in the non-existing southern world-tile neighbor (y+1)?
What about the other edges of the world (west/north)? Can those corner (north-west + south-west / north-west + north-east) be processed with the available data?
Long question short:
Is it possible to process corner incursion at the edge of the world with only two candidates and if so for which world-edges is it possible?

[PatrikLundell]

Nope, not this time. For some reason Toady decided to make the structures 16*17 and 17*16, with actual data in the extra row/column. That's fine as it saves you from going to the next world tile to get the data, except that the corner, 16/16, is missing, so for that one you still need the that world tile.

I do agree that we really need the data for the other world tile(s) anyway to process the data properly, so that extra row/column doesn't help all that much.

For the western/northern edges the incursion info exists within the world, but there's special handling for the cases where the incursions are specified to come from the non existing side, which really affects corners only (for edges it's just a flip). Unfortunately, DF generates data indicating usage of data from the non existent tiles.
For the southern/eastern edges the extra row/colum provides the incursion info for everything but the corners (although you have to ignore it when going in the wrong direction for edges). While it should be possible to use the data for (0, 16) of the next world tile in the X direction (and correspondingly in the Y one, that's not how DF has implemented it. The corners along those edges all have the same pattern, i.e. the NW tile being the source (you can see that pattern if you use the showbiomes script on embarks along those edges).

[RedDwarfStepper]

Nope, not this time. For some reason Toady decided to make the structures 16*17 and 17*16, with actual data in the extra row/column. That's fine as it saves you from going to the next world tile to get the data, except that the corner, 16/16, is missing, so for that one you still need the that world tile.

Hm, I can see that split_x and split_y are [16][17]/[17][16]
https://github.com/DFHack/df-structures/blob/cbeed8b52482ce749fc2ff13a9db9733931e7da0/df.world-data.xml#L418
https://github.com/DFHack/df-structures/blob/cbeed8b52482ce749fc2ff13a9db9733931e7da0/df.world-data.xml#L422
and neither split_x or split_y is used/referenced in the plugin code.
biome_x and biome_y seem both to be [16][16]
https://github.com/DFHack/df-structures/blob/cbeed8b52482ce749fc2ff13a9db9733931e7da0/df.world-data.xml#L437
https://github.com/DFHack/df-structures/blob/cbeed8b52482ce749fc2ff13a9db9733931e7da0/df.world-data.xml#L440
or as generated code

Code: [Select]

      int8_t biome_x[16][16]; /*!< 0=Reference is N, 1=Reference is current tile (adopted by S edge to the N) */
      int8_t biome_y[16][16]; /*!< 0=Reference is W, 1=Reference is current tile (Adopted by E edge to the W) */Am I missing something?

[PatrikLundell]

Nope, the one who was missing something was me. I should have gone to bed rather than try to write an answer at that time.

The code works for the "own" case, but, as you point out, it doesn't for the "other" one, so yes, there is a need to cache the edge case data, and yes, the same kind of translation logic is the one you refer to is needed.

[RedDwarfStepper]

Ok, so I finally found the time to adapt the code for incursion processing, which resulted in "uncovering" of the missing special cases (runtime errors) for the world-borders (northern world-bound [y=0, k=0], eastern world-bound [x=world_width -1, i=15], southern ... western ...) But I'm unsure how to handle those cases.

For the western/northern edges the incursion info exists within the world, but there's special handling for the cases where the incursions are specified to come from the non existing side, which really affects corners only (for edges it's just a flip).

By "flip" do you mean wrapping around to the other/opposite end of the world so e.g. southern tiles (y=world-height - 1, k=15) check for incursions against y=0, k=0?
Or just return "4" for no incursion here?

[PatrikLundell]

A "flip" is just reverse the incursion direction, i.e. returning 4.

Edit: I'm unsure how to interpret the comment about the "uncovered" cases. Does it mean your handling has trouble, or does it mean there's an issue in the released version? I don't get any errors when I try it along the world edges.

[RedDwarfStepper]

A "flip" is just reverse the incursion direction, i.e. returning 4.

Ok, understood the handling of edges at the world borders.
I'll come back once it works and had another look at the incursion processing of corners.

Edit: I'm unsure how to interpret the comment about the "uncovered" cases. Does it mean your handling has trouble, or does it mean there's an issue in the released version? I don't get any errors when I try it along the world edges.

"manifested" would have been the better word.
Compare
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1916
till
https://github.com/DFHack/dfhack/blob/f20446534bb7f39425e102bd70daec46e328004f/plugins/embark-assistant/survey.cpp#L1925
with

Code: [Select]

if (own_edge) {
// the edge belongs to the currently processed tile thus its counterpart is the north tile (k - 1)
effective_edge = world_data->region_details[0]->edges.biome_x[i][k];
// region_type_of actually properly handles the case that we need information from the world tile north (y - 1) and returns df::world_region_type::Lake in case of y < 0, which prevents incursion processing
south_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k);
north_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k - 1);
}
else {
// the edge belongs to the tile south of the currently processed tile thus its counterpart is the south tile (k + 1)
// here the case that we need information from the next world tile south is being handled properly
if (k < 15) {
effective_edge = world_data->region_details[0]->edges.biome_x[i][k + 1];
}
else {
// outside of the world - so no incursion
if (y + 1 == world_data->world_height) {
return 4;
}
                // just the next world tile to the south
effective_edge = survey_results->at(x).at(y + 1).northern_row_biome_x[i];
}

north_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k);
south_region_type = embark_assist::survey::region_type_of(survey_results, x, y, i, k + 1);
}
The above line (reading cached data)

Code: [Select]

effective_edge = survey_results->at(x).at(y + 1).northern_row_biome_x[i];resulted in an illegal access error without handling the case

Code: [Select]

y + 1 == world_data->world_height were before it was just

Code: [Select]

effective_edge = world_data->region_details[0]->edges.biome_x[i][k + 1];which read invalid data but did not crash

[PatrikLundell]

Ah, I see. Unfortunately, DFHack is FUBARed currently, so I can't do any kind of work with it.

Edit:
I've update the code with your discovery, although I've used the names north_row_biome_x and west_column_biome_y for consistency with the other names used.

[Telear]

[ARGH! Ignore this! I forgot I'd asked this before, and that I didn't notice that Patrik has implemented something that means I can write my searcher as a plain old shell script. Onwards! And thanks so much for the change]

I'm not entirely sure if this is the right place to ask, or if I'd be better off starting a new thread, but I'm busily porting a fragile Mac only Keyboard Maestro script that generates lots of worlds and then runs embark-assistant over them in search of particularly rare embarks (in my case volcano + big waterfall + Iron + Coal, but in principle, anything that can be searched for), and it would be really useful if there were some information exposed to lua about search state and how many matching tiles have been found so that I could just fire off the search, and wait until either there was at least one matching tile or the search and completed. The Keyboard Maestro script was doing some horrendous screen scraping in order to detect when the search was over, and I really don't want to have to reimplement a similar mechanism in lua, because... ick.

I'd do it myself, but I have very little idea of what I'm doing in C++ land.

If that's not doable, is the search time linear with the area of the world being searched? In other words, could I fire off the search and then do something like

dfhack.timeout(world_area * a_constant, 'ticks', check_for_success)

and be confident that they search will have finished?

[PatrikLundell]

(Despite the Ignore this! statement)

It's possible to use a callback based Lua script to detect when world gen is done. If you make a callback at every frame (which is approximately equal to once per year once generation gets going, with some jumps throwing exact year matches off) you can check for various progress criteria (such as e.g. civ placement, good/evil placement, and, of course, reaching the end of the generation). Using that, you can have a script perform manipulations of the world as it's generated (I use it to tailor my PSV worlds to have all plants and creatures legal to regions being present, all glaciers being evil (that's where my glacier starting biome only gobbos start with their Blizzard Men slaves), etc. My slab_civ script uses the same harness logic to detect civs that ought to be dead and push them over the edge at the end of world gen.
I think only the "frames" callback works during world gen, as there is no time related concept that makes sense at that scale.

[Telear]

Right... I saw your script that did that in your repo and it's cool, but I like the idea of making seeds that will reliably generate the interesting embark in vanilla DF, if only because it's a damn sight easier to distribute a simple text file than it is to send out a save file. I can sort of see the properties I'd need to be watching for worldgen, but I can't quite work out where I need to be looking to watch what embark-assistant is up to. Of course, now I'm wondering whether you could interleave embark-assistant with worldgen. Pause at year 0 and search for the physical geography features that are locked in at that point and throw a rejection if they're missing. Then, once you have matches, just keep looking at those as history is run to see if the 'dwarfish geography' of the potential site is right too. If there's sufficiently few interesting sites, searching that way might even be more efficient.

But that assumes the datastructures that embark-assistant needs are available during worldgen.

Once I've published a first cut at a unicorn finder script, I might see about repurposing the lua version of embark-assistant for a proof of concept – if it works, then I hope that once DFHack is back on its feet, it wouldn't be hard to expose the necessary bits of the plugin to Lua for shenanigans.

[PatrikLundell]

If embark-assistant can be run during world gen, you'd have to be very careful with the search criteria. For instance, if you look for an evil embark before good/evil has been placed, you won't find any. You would probably be able to run it at the start of history (you can try it yourself, by invoking it from the DFHack window), but it will still reflect evilness/savagery as it is at that time, not at the end of history, so even then you'd have to be careful.

However, when it comes to early aborts, I'd use a a Lua script for that and accept that a few of the candidates that are approved at that stage will fail at the final evaluation, in particular if the criteria don't require loading of the MLT info.
I've certainly used the strategy of aborting world gen early if Eye Ball Mk I assessment finds the world lacking.
Iterating over all the volcanoes, for instance, is more or less trivial using Lua. Just iterate over all the peaks (and there won't be that many of them), and look for the ones that have the volcano flag set.

By the way, DFHack is back: the change that threw things into disarray has been reverted until a solution to the problem has been found.

[Telear]

Right. I think you're getting caught up in the specifics of my example search target. I might want Volcano/Waterfall on a 3x3 with lots of neighbours, but someone else might want a big frozen river, magma in cavern one and joyous wilds in a 500 year old world. It seems (in theory at least) that I might be able to pause worldgen at the start of recorded history and run a limited embark-assistant that's only looking for the things that may have been placed at that point in time (Volcano/Waterfall/3x3, or Big Frozen River and Cavern 1 Magma) so worlds without those features can get rejected before we've sat around twiddling our thumbs waiting for 500 years to go by.

That said, in my original, fragile version of this, I just searched for the physical stuff with a 5 year history, then set the History, Name and Creature seeds to RANDOM, increased END_YEAR and added extra rows to my embark_assistant_profile.txt. It works, but it's fiddly and not great when I'm aiming at a DFHack script that anyone can run by setting up a search profile, then doing something like make-dream-embark PRESET at the DFHack prompt.