Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[A_Curious_Cat]

(I wonder if this is the problem A_Curious_Cat had but failed to reproduce?).

I think it did have something to do with sand.  I, also, just realized that I may have made a mistake when I tried to reproduce the problem.  Back when I first applied for an account on DFFD, I made sure that I could reproduce the problem.

The correct sequence to reproduce the problem (with the save that I was planning on putting on DFFD) was (IIRC)

1.  Load the world by choosing "Start Playing" and "Dwarf Fortress" mode and use embark-assistant to search using the search parameters that I planned to provide in a screenshot (which did not require that sand be present) and then search again (according to what I've heard about needing to always search twice with the same parameters).  This should have resulted in a number of potential embarks being found.

2.  Abort the game and return to the main menu.

3. repeat the actions in step 1 with the exception of changing the search parameters to require that sand be present.  This should have resulted in no embarks being found.

4. Without repeating actions in step 2, change the search parameters back to the way they were in step 1 and perform search again (remembering to search twice with the same parameters).  This should have resulted in no embarks being found (conflicting with the results from step 1 even though both searches used the same parameters).

I think my mistake was that I forgot to do step 2.

Anyways, that world has since been deleted after the fort I made on it imploded.

[PatrikLundell]

I'm not a fan of your definition of All as "present in all tiles", rather than being the reverse of Absent.

It's true that you can achieve the original (i.e. no intrusion processing) functionality of All with Present for the desired criterion and Absent for all of the others, but it's not a user friendly and intuitive way of setting the criteria, which is why All was introduced.

With your definition of "All", players that selected Good:All and Savagery:Low:All could end up getting an embark with all tiles being Evil and Savage natively , with your criteria being fulfilled by some incursions that are Good and some of which have Low Savagery for every tile, as long as the embark isn't larger than 2 in at least one dimension (this extreme outcome would be very uncommon, of course, but would also be the cause of bug reports).

If you think there would be a demand for your versions of "All" I would satisfy that by two new criteria called something like "Present_As_All_Tiles" and "Present_In_All_Tiles", where the former would demand that the tiles themselves meet the criterion, but allow incursions to differ, and the latter would be satisfied with all tiles meeting the criterion either in the tile itself or through an incursion. I'm not convinced there's a demand for those criteria, though.
  I can't come up with a scenario where I'd ever somehow would want a feature present in all tiles but being satisfied with it being provided by a tiny incursion somewhere in it: If a small incursion would be sufficient, it would probably be enough to have it anywhere in the embark, rather than demand it in every tile.
  I can make a case for demanding every tile to meet the criterion natively while allowing for incursions to differ as it might result in a smaller set of matches when you want the dominant part of your embark to match the criterion, but incursions vary a lot in size, so that's more an illusion of control that actual control.

In my view Absent and All are symmetric: Any presence invalidates Absent, and any absence invalidates All.

On the issue of dealing with incursion propagation to the world tile summary, I've thought about it and currently believe it can be handled in the current version by:
- At the end of the survey of each world tile, post process the tiles along the edges in the previously processed neighboring tiles to process incursions into them for the purpose of propagating the info to the world tile (but not affect the search results). The processing would need to defer processing of the lower right corner until all four of the touching world tiles have been processed, and it would need to process world edge tiles as if the fictive world tile on the other side has been processed already.

With your caching, I'd probably aim for a three step process:
- Process all MLTs in all world tiles.
- Process all incursions into all MLTs. However, it might be that it may be better to keep incursion processing for the interior MLTs in the first stage, as is done currently.
- Perform matching against all the fully processed MLTs. This step would be the only one needed for matching attempts after the first one.

Edit: My approach to deal with the incorrect top level data won't work: You need access to all the surrounding MLTs to process incursions, and attempts to process incursions as they become available would require the storage of both the native MLT results and the partial summary. Back to the drawing board...

[RedDwarfStepper]

@A_Curious_Cat:
I just reread your previous posts and your last one and yes, that seems to be the problem I've found as well.
But leaving/aborting to the main menu is not necessary to provoke this behavior, at least that's my observation.
It is sufficient to have a world save that contains a "specific" configuration (which is actually quite common, or so I think) of characteristics (like my save here) and to search twice, with the same or at least similar parameters in a row. Those characteristics include aquifer, clay, sand, flux and coal others as well. Three of those are eligible for incursion processing: aquifer, clay and sand, which is relevant.
What happens is as follows:
- after starting embark-assistant (via console or automatically via init somehow) an initial high level world survey processes all world tiles exactly once. It identifies potentially existing characteristics using counters to keep track of the number of occurrences a characteristic mighthave within a world tile.* These counters are being used to decide if a world tile is a preliminary match and thus will be processed during matching
- after starting the search the number of preliminary matches is being evaluated, based on the counters containing potential numbers  => more preliminary matches than later on
- during the very first search run a more detailed survey takes place that evaluates all 256 embark tiles in every world tile. The counters are being reset to contain the actual number of occurrences of every characteristic, which might be lower than the previous value, even 0.
- where possible cross world tile incursions are being taken into account which in some cases result in incoming characteristics# (aquifer, clay or sand!) that actually don't natively exist inside the world tile but potentially could have as discovered during the initial high level world survey (<= this is the kicker! These two circumstances * + # aligning in some world tiles produces matches that won't reappear with the second search)
- the matches of the first search run are being displayed
- the second search is started, with unchanged or slightly modified search criteria, still containing at least one of the above mentioned characteristics.
- during evaluation of the preliminary matches all world tiles that have a relevant counter of zero are no longer considered a preliminary match and won't be processed during regular matching (no cross world tile incursions!) and thus won't produce a match even if they should => less matches than before.

And to make it even more interesting: That some previous matches are missing might be disguised by the fact that the second and all subsequent searches often result in more total matches as all world tiles are fully surveyed which makes the complete processing of cross world tiles incursions possible.

[RedDwarfStepper]

@PatrikLundell: I stand corrected.

I'm not a fan of your definition of All as "present in all tiles", rather than being the reverse of Absent.

... but it's not a user friendly and intuitive way of setting the criteria, which is why All was introduced.

... (...but would also be the cause of bug reports).
...
  I can't come up with a scenario where I'd ever somehow would want a feature present in all tiles but being satisfied with it being provided by a tiny incursion somewhere in it: If a small incursion would be sufficient, it would probably be enough to have it anywhere in the embark, rather than demand it in every tile.
  I can make a case for demanding every tile to meet the criterion natively while allowing for incursions to differ as it might result in a smaller set of matches when you want the dominant part of your embark to match the criterion, but incursions vary a lot in size, so that's more an illusion of control that actual control.

In my view Absent and All are symmetric: Any presence invalidates Absent, and any absence invalidates All.

Just today I have told a colleague of mine one should not change the user experience without a really good reason - we discussed the changes he made to a filter in a list of possible options - of all the things
I really should heed my own advice and listen to your very strong arguments: strict "all" it is.
My gut feeling/intuition was that a less strict "all" might suit the users better.
Also it would not have required any changes...
Yes, I'm lazy I know

With your caching, I'd probably aim for a three step process:
- Process all MLTs in all world tiles.
- Process all incursions into all MLTs. However, it might be that it may be better to keep incursion processing for the interior MLTs in the first stage, as is done currently.
- Perform matching against all the fully processed MLTs. This step would be the only one needed for matching attempts after the first one.

Hehe, that's pretty much exactly what I've implemented, have a look here
If you want I can go into details - but right now I need to do some chores and get some sleep - I know I know sleep is overrated

PS:

Edit: My approach to deal with the incorrect top level data won't work: You need access to all the surrounding MLTs to process incursions, and attempts to process incursions as they become available would require the storage of both the native MLT results and the partial summary. Back to the drawing board...

Yeah, I had some fun with that situation as well - it added quite some complexity to the code. Also I store all 64 outer/edge mid_level_tile/mid_level_tile_basic (<= contains only data relevant for incursions) of a world tile inside region_tile_datum => north & south row +west & east column. The internal/intra world tile incursions are being processed parallel to the native data, the cross world tile incursions delayed until all neighboring world tiles have being processed - using counters and such to track the progress...

[PatrikLundell]

I've just committed a PR for a fix of the bug (the world showing it was handy).

The fix isn't elegant. On the first pass the incursion criteria aren't checked at all (not that it should matter, as every tile is surveyed).
Once all tiles have been surveyed, all world tiles are checked for potential incursion resources from neighboring tiles. The keyword here is potential: at that location in the code the incursion direction info isn't available, so the code just checks all the neighboring MLTs saved for incursion checking and marks the resource as supplied by neighbors. When determining whether the world tile matches the high level criteria on the subsequent scans they're considered matched (for further checking) if the resource is available natively or might be provided by incursions as per the data collected above.
It would be possible to do a correct incursion checking by adding fields to the MLT info and populate them through a lighter incursion checking process, but it has the issues of a fair bit of additional messy code.

[RedDwarfStepper]

Good that the provided world was helpful!

I just looked at the changes in the PR hoping I could "backport" them easily to my version but alas the changes regarding the aquifers (the new separation in light and heavy) are too extensive to allow for an easy integration/merge.
The logical next step would be to migrate/update to the current develop branch (which I'll have to do eventually anyways) - but that would mean I'd have to adapt a lot of existing code, some of which isn't final/done yet. I worked on some larger projects were such an migration was done during the initial development phase - it never was a fun experience.
The main problem is bugs and being able to track if they have their origin in the newly developed code or in the merged/update base code. Without a stable feature baseline that can be used as a reference this can result in some extensive hide-and-seek-egg-hunt and it is just not Easter yet.
I'll have to ponder my options here a little more.

Also I'm wondering: Don't the other incursionable (?) characteristics (savagery, evilness, elevation/flatness, soil depth and biome) need the same special treatment as aquifer, sand and clay?
A quick test searching for biome1:"grassland_temperate" with the same world ("pocket_world_scarce") and a 4x4 area got me additional hits in the lower right corner (eastern part of the southern row) of the world tile at x:5,y:3 for example, also at x:6,y:3 (complete southern row) and at x:7,y:5 (western column, upper corner) with the index search.
If necessary I can provide exact coordinates or other additional data.

[PatrikLundell]

You're correct in that there's something broken with the temperate grassland search. However, it's not caused by missing incursion info as the whole 4*4 embark area in the lower right corner of the (5, 3) tile has temperate grassland as the native biome in each of the tiles, and even a single one ought to propagate it to the world tile level.

That does not mean that the rest of the incursion info shouldn't be made available, as I've got a sinking feeling that it should.

On to bug hunting...

Changing the baseline part way through is a pain, and I would only do it if my current work would largely have to be modified again to align with the new baseline. If I decided to shift the baseline, I'd probably port my changes from the modified older baseline piece by piece where each piece could be tested before moving the next one. If that wasn't possible because nothing works until everything's done I'd prefer to stick to the older baseline until it works (I might well move before the last bug was dealt with, but at least the general functionality should be there). If there was a risk that the work might turn out to have to be abandoned due to problems that couldn't be dealt with I'd stick with the older baseline simply to waste less effort before that unpleasant result was revealed.

[RedDwarfStepper]

However, it's not caused by missing incursion info as the whole 4*4 embark area in the lower right corner of the (5, 3) tile has temperate grassland as the native biome in each of the tiles, and even a single one ought to propagate it to the world tile level.

Hm, curious, yet all the additional matches seem to be located at one of the edges of the world tile, either x == 0 or y == 12 (or more general y == 16 - y_dimension)
Could it be an "one-off"?

[PatrikLundell]

This got ugly: https://github.com/DFHack/dfhack/issues/1708

In addition to that, I do believe I've seen the number of matches shrink between the first and second pass, so yes, I think the rest of the incursion factors need to be taken into world tile level consideration as well.

[RedDwarfStepper]

Oh my, reading the ticket sounds like quite an adventure down the rabbit hole.
One question of understanding though: Is the issue described in the ticket the reason for the biome1:"grassland_temperate" mismatch or was it an additional finding while debugging and affects only ocean biomes?
I'm wondering, because I'm currently running on dfhack 0.44.12-r2 and the new search code I added is indirectly using survey::high_level_world_survey and thus also DFHack::Maps::GetBiomeTypeWithRef and it correctly finds those matches, while the previously existing match code does not. So GetBiomeTypeWithRef can't be broken completely at least not for "grassland_temperate".

[PatrikLundell]

GetBiomeTypeWithRef definitely works most of the time. It doesn't with the your original temperate grassland patch, and the tropical ocean boundary has changed, but I can't say how much has changed, beyond that it doesn't seem to be a major one. I spent a fair bit of time looking at temperate grassland, shrubland, and savanna without finding any obvious mismatch.
However, I came to the conclusion that it would be wasted time to investigate further, as that might give additional indications as to what has changed, but would do nothing to actually help solving the issue.

My guess is that there are a small number of minor adjustments that affect niche cases, plus a push to get the tropical ocean to reach further from the equator. A very possible change might be to have ocean turn tropical if it's in the "maybe" zone in addition to the "definite" one, and grassland might have had a parameter adjustment. There may well be other adjustments to things I haven't looked at.

I did compare the results with the Biome Manipulator that still uses a Lua implementation of the logic, and, as expected, it gives the same result as GetBiomeTypeWithRef, so it's not a matter if it being changed since or bungled up when introduced (which I have no reason to think would have happened, but it's still something that could be compared, resulting in the expected outcome).

In conclusion: The logic is not completely broken, but only suffering from minor dents and scratches, although you have to be wary about oceans (which I don't think would be a common search target anyway).

I believe I've seen cases of "found in the first search but not the second one" for grassland, so my next task will be to propagate the remaining incursion resources to the top level.

[BoogieMan]

This is awesome and extremely helpful, thanks for making it!

[RedDwarfStepper]

@PatrikLundell:
I think I found another issue while verifying the result of the min/max river-search:
If a world tile has at least 2 rivers with different sizes this will result in false positives/matches when searching for the occurring larger river size.
E.g.: A world tile has 1 medium and 1 minor river. Search for min-river:medium will return matches for all embark tiles that contain a river, even if it is only the minor river.
The reason for that is to be found here: matcher.cpp#L688 and here matcher.cpp#L1591. The embark tiles themselves only know that they "have" a river, but not its size (river_present).
As I had to calculate the river size from the max-min values (rivers_horizontal.y_max - rivers_horizontal.y_min / rivers_vertical.x_max - details->rivers_vertical.x_min) I could provide code.
Also the large world (region4) I've uploaded way back

I uploaded the save (http://dffd.bay12games.com/file.php?id=14584) and profiles I use for getting an idea how much memory it would take to "index" everything for a large world.

has at least one example (major river + stream) of this at x:166, y:61; starting from i:0, k:13 going to i:4, k:13.

[PatrikLundell]

The game data only "knows" of at most one river per world tile. Other rivers may have their end point in a tile containing a "registered" river, thus joining it, but there's no info back to the top level about it, so there's no way to retrieve that info from the world tile. As you say, to check this properly you'd have to manually measure the river size at the MLT level (and, while doing that, you'll find that rivers change size gradually and thus can change from one size to the next one along its course, in the middle of a world tile).

Thus, it's an issue of an (over) simplified implementation of the check, but you're right in that it ought to be checked properly.

[RedDwarfStepper]

argh - stop being so fast 
I'll never be able catch up if you keep this up