Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[PatrikLundell]

I've taken care of the sleeping for you, so you should be ready to move on

[PatrikLundell]

@Telear: I've experimented with the code to get the plugin to work in a semi-non-interactive way, but can't get all the way. Trying to get the plugin to shut down itself crashes DF, probably because you can't dismantle it from deep in a call chain: the shutdown probably has to come from an "outside" thread, such as the key input one (and emulating the "q" input doesn't result in this thread shift).
What I've done is:
- Allow for a single optional case sensitive parameter "fileresult" to trigger a special behavior.
- When the parameter is provided, the file "./data/init/embark_assistant_fileresult.txt" is deleted if it exists.
- A search is triggered immediately, first loading the contents of the saved profile (I haven't tested without any profile available, but suspect it will just result in the starting profile, i.e. X*X embarks with nothing else specified).
- Once the search is finished, a second search is started off immediately (committed and accepted changes that are not yet released in a new DFHack release provides detection of "incursions" into mid level tiles (bits of the biome of neighboring tiles jutting into the mid level tile examined), and that results in mutual analysis dependencies along world tile borders that cannot be resolved in a single pass).
- Once the second pass is through "./data/init/embark_assistant_fileresult.txt" is created, the number of matching world tiles followed by a newline is written into the file, and the files is closed.
- That's it. From then on the plugin should work as per normal (although I haven't actually tried, apart from using the "q"uit command manually).

Thus, using external scripting, I would expect that you could:
- Start DF and generate a world.
- Use the world to enter the pre embark stage.
- Optionally delete "./data/init/embark_assistant_fileresult.txt" if that makes trigger detection easier/safer.
- Enter "embark-assistant fileresult" into the DFHack window.
- Detect the appearance of the "./data/init/embark_assistant_fileresult.txt" file.
- Read the results from the file.
- Feed "q" into DF (or "kill" into the DFHack console if you don't care about doing things nicely).
- Repeat (probably deleting the world first if it had no matches).

It can be noted that even if I would be able to get the plugin to auto shutdown, it cannot be made to return a tile matching number, because plugins already have a return code (a success or failure code the framework recognizes).

[RedDwarfStepper]

@PatrikLundell
Haha, very thoughtful of you to take care of my sleep - still feel kind of tired though.
Actually I really need some holidays - so my mucking around in the code will have to wait a little longer - but I like the idea of first modifying the movement (diagonal + 10x) pattern to get to know the code better. Which already has me thinking about some details I will have to ask you about eventually.
Also I'm wondering: Is this here the right place for me to ask detailed questions or is there a more suitable place, github perhaps? Just don't want to cause bordome to the other members with my learning the ins and outs of DF plugin development and the specialties of embark-assistant.

One more thing though - might using marcos help Telear? (https://dwarffortresswiki.org/index.php/DF2014:Macros_and_keymaps)
And is there a way to get the output of the dfhack console directly into a log file?
That would allow you to just output the number of found matches into the console without the need for embark_assistant_fileresult.
But of course then the log-file would have to be monitored for new entries to shut down the game....

Cheers!

[PatrikLundell]

@RedDwarfStepper:
After making my previous post I continued on to look at modifying the movement logic, and it turned out to be a lot simpler than I thought, as well as uncovering that I'd missed to cater for a need to scan everything in the first scan to get the incursion logic working correctly when the original implementation only scanned on a "need to examine" basis. This results in the first scan taking a long time (basically the same as currently), while subsequent ones are a lot quicker if the number of preliminary matching world tiles is reasonably low.

I think this thread is probably the best place to discuss issues regarding the plugin: I don't think there's a huge crowd of people that would find lots of time being lost reading posts irrelevant to them, and there is at least some chance something posted might be of use to someone else. If there are protests about that assessment I'd switch to a separate thread in this forum.
Note that I'm definitely NOT an expert on plugin development: This is my first plugin attempt, and I wouldn't have been able to get it to work without help from others, in particular Lethosor. I'm also not a friend of git: it keeps stabbing me in the back, apart from using backwards logic and "help" and documentation that simultaneously drown you in words about a seemingly infinite number of variations, while at the same time managing to just tell you half of the ones you need to use leaving the rest in the dark.

I think macros would be of limited use to Telear, as the process requires things outside of the DF context, detection of triggers, and entering of "key presses" only when trigger conditions are fulfilled. A macro essentially compresses several key presses into one, but since the key presses are entered through a kind of script anyway, a few lines of key presses does the same as a macro does, with the sequence being repeated every time that sequence of the script is processed.
I don't know how to get stuff off the DFHack console, but I doubt it would improve the situation in that case, as the script then would have to process everything else written to the console, like the number of preliminary matches

[clinodev]

I think this thread is probably the best place to discuss issues regarding the plugin: I don't think there's a huge crowd of people that would find lots of time being lost reading posts irrelevant to them, and there is at least some chance something posted might be of use to someone else. If there are protests about that assessment I'd switch to a separate thread in this forum.

I can't speak for anyone else who's subscribed to this thread, but I personally find these sorts of questions and speculations on what future versions of the plugin may or may not be able to do endlessly entertaining and informative, and sincerely encourage them.

[PatrikLundell]

It certainly wouldn't be a matter of shifting requests and discussions about those elsewhere, but it's rather a matter of whether discussions about this section in that file looks the way it does (or what it actually does), i.e. things that may be of little interest to those interested in the sausage, not how it's made, would be better off elsewhere. Currently I don't foresee such detailed discussions to take up so much of the thread that it would be hard to get the interesting bits, though.

[ibanix]

Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.

[PatrikLundell]

Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.

No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
This, in turn, would require ensuring all the DF structures relied on being available in the old version (or added to it, after verifying they indeed were the same back then), etc.

I have no objections to you trying to perform that feat, but to get a release you'd likely have to do all the groundwork as well as persuading the DFHack maintainers to release your updated old DFHack version (and I'd secure a promise before starting to ensure the work won't be in vain). Of course, you can do the work "privately" by downloading a base version of DFHack to your computer and do the work there for your private use.

I'd recommend 0.40.24 if you're looking for stability, but I started with DF at 0.40.0X (6, 7?), so I've never seen the old version.

[Pvt. Pirate]

Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.

No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
This, in turn, would require ensuring all the DF structures relied on being available in the old version (or added to it, after verifying they indeed were the same back then), etc.

I have no objections to you trying to perform that feat, but to get a release you'd likely have to do all the groundwork as well as persuading the DFHack maintainers to release your updated old DFHack version (and I'd secure a promise before starting to ensure the work won't be in vain). Of course, you can do the work "privately" by downloading a base version of DFHack to your computer and do the work there for your private use.

I'd recommend 0.40.24 if you're looking for stability, but I started with DF at 0.40.0X (6, 7?), so I've never seen the old version.

for me 44.09 is the most stable version. i was able to play 3 ingame years without a crash.

[ibanix]

Did this ever get backported to 0.34.11? Still running on that version due to all the crashes in the latest version.

No, and it won't be. Doing that would require digging up a corresponding version of DFHack, modify that, and release such a version.
This, in turn, would require ensuring all the DF structures relied on being available in the old version (or added to it, after verifying they indeed were the same back then), etc.

I have no objections to you trying to perform that feat, but to get a release you'd likely have to do all the groundwork as well as persuading the DFHack maintainers to release your updated old DFHack version (and I'd secure a promise before starting to ensure the work won't be in vain). Of course, you can do the work "privately" by downloading a base version of DFHack to your computer and do the work there for your private use.

I'd recommend 0.40.24 if you're looking for stability, but I started with DF at 0.40.0X (6, 7?), so I've never seen the old version.

I don't have the skill, so I'll just wait until a stable version of 0.44. Thanks for answering!

[PatrikLundell]

I don't think there will be any stable version of 0.44: You'll have to wait for the Villains release to stabilize, and hope Toady will start to adopt a new development strategy whereby he fixes critical bugs in the latest released version in parallel with the development of the next one (he'll need to adopt such a strategy for the Premium release onward,, but I've got no idea if he'll wait until the last moment or tries to ease himself into it during the Premium arc).

Back porting DF mappings a large number of versions is no simple task, and I don't think there are many who could do it (I don't think I could, beyond the reasonably trivial "yes, it seems to be exactly the same" cases). Adapting the plugin code to work with what it managed to port back (most probably some things would not be found) would be the "easy" part.
In addition to the difficulty of pack porting DF mappings, it's probably going to be a fairly substantial amount of work, even if you specifically aim for just the things used by this plugin.

[Fleeting Frames]

On that backporting note: It can be very much not enough to merely edit the plugin calls from named fields to anons; some of the functions plugin asks from dfhack could be outright missing (not familiar enough with embark-assistant code to say for sure); this means you'd have to do some updates for relevant parts of dfhack as well.

Relevant parts can be determined by failure to compile, then looking at dfhack's github history of relevant files, then adding things in. Also, relevant files may be entirely missing in 34.11.

For as great a difference as this, I'd expect it to take weeks.

[RedDwarfStepper]

@PatrikLundell:
I'm back from my vacation (quite a while actually) and I've been busy.
First with getting to know the code and then with doing preliminary tests with a nice little data structure called RoaringBitmap.
Basically it's a compressed bitset, that can hold a huge number of entries while needing way less memory than regular implementations like vector<bool>.
This makes it possible to index/cache all values that can be expressed as bool, including inorganics (tested). It also should work well for those numerical attributes that only have a few valid values like savagery, with each value being represented by a separate index. It probably won't work for values that have a wider range like elevation, also since that retrieving would get inefficient.
My first test held 6 bools (sand, coal, ...) and all of the inorganics for all mid_level_tile entries of a 257x257 world in ~ 200MB RAM.
I'm trying to get my head around the logic of incursion-processing (got your last version from github), which might by possibly moved into the index as well.
That might take a while, also I'll have questions
As soon as there is something to show I'll put it up on github.

[PatrikLundell]

Compressing the data into single bits per bit should indeed make it possible to reduce the memory needed a fair bit, at the cost of speed (which is important).
However, I'm unable to see how you can compress what you mentioned into 200 MB, as my quick calculation results in:
(264 non procedurally generated inorganics + 6 booleans) / 8 = 33 bytes *
257 *257 * 256 mid level tiles (= 16908544) => 557981952 bytes, i.e. about 560 MB.

Nevertheless, this might be a useful approach, although the speed impact will have to be investigated, and I'd definitely check the overhead of this RoaringBitmap thingie compared to doing it "manually". Note, though, that there are currently 3 vectors over the inorganics: one for all of them, one for metals, and one for economics. While the data itself can be stored in a single vector, it comes at the expense of then having to probe the appropriate indices for metals and economics, which may or may not turn out to be an issue.

Incursion processing is indeed complicated, made more so by the way Toady ended up representing the information so that it almost, but not quite, provided all the information needed to evaluate a world tile in isolation along the eastern/southern edges. As you can see, the incursion processing ended up with partial storage of the base information (notably excluding the inorganics) along the world tile edges, and I've considered whether it would be reasonable to expand that partial information storage to cover all mid level tiles, but haven't investigated that.

[Fleeting Frames]

It might be possible to compress the 264 inorganics by finding groups of frequently-appearing together inorganics, so if a geo_biome has 100111111 for layer materials, could mark the 1111111 as seven₁₀(=111_binary) 1s instead. It's probable that two adjacent embark tiles in same biome share lot of base geology, so there's no need to use memory for that on both of them.

But that trades even more memory for performance.

Also, caching search result should probably be a auto-off toggle. I imgine the common use case is quick enough or just single search, not needing the cache, where cache might be prohibitive on older systems attempting it on words on the boundary of what their RAM can support.