Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[myk]

Since you copied the files from a package manager-managed directory, are there any symlinks involved? or does the launcher script set the current working directory? We saw a similar issue in https://github.com/DFHack/dfhack/issues/1671. Are the comments in that bug helpful?

If you put a .csv blueprint file in the blueprints/ directory, does it show up in quickfort list? How about if you add a file to your system-installed blueprints/ dir?

[bassmannate]

I don't see any symlinks anywhere. Just for grins, I created a ~/.dwarffortress folder and copied dfhack-config and all its contents over. Still nothing when I run order import automation. Everything is showing up just fine and it's finding the blueprints but it's just not finding this json for some reason.

[myk]

If you manually create a few manager orders and run

orders export testytest

Where does testytest.json show up?

[lethosor]

Similarly, does

Code: [Select]

:lua ~dfhack.filesystem.getcwd()
print

Code: [Select]

~/Applications/linux-dwarf-pack/df_47_05_linux/
or something else? If something else (other than expanding "~"), then you should be looking in whatever folder it printed for the "dfhack-config" folder to edit. Alternatively, your pack might be setting the working directory incorrectly.

Another thing I might have missed: did the "dfhack-config" folder already exist? It should have, so if it didn't, the one you're modifying was likely created in the wrong place - see paragraph above.

[bassmannate]

Aha! It went to ~/.local/share/linux-dwarf-pack/df_47_05_linux/dfhack-config/orders which kinda makes sense since this is a system install. I'm a little new to Linux Mint and even before switching, Ubuntu was in the process of moving configs all over the place. This looks like the "normal" place for Mint to put configs judging by what else is in the ~/.local/share folder.

I just copied my automation.json over to the ~/.local location and it appears to have worked fine since I didn't get an error. It should also be noted that this is my first real attempt at using dfhack so quite a bit of this is all new to me and I'm not entirely sure what all is possible with everything which is why I didn't even know that export command exitsted.

Also, I don't know if it's worth mentioning but the dfhack-config folder was present but the orders folder was not under my installation folder. I'll stop now since this is getting more into issues with the pack I'm using than anything. Thanks for the help, guys!

[myk]

I've been silent for a while about development, so I should write an update. I've been focusing on tests for the core quickfort algorithms. Some 2000 lines of unit tests later, I've fixed a good number of bugs that now won't get to annoy you ; )

I still need to write the higher-level functional and integration tests, but I think I'll take a break from testing for a while. Instead, I'm going to take a look at the blueprint plugin and start thinking about what it will take to make it more useful.

[clinodev]

Your labors are much appreciated!

[myk]

Thanks!  : ) I love working with this community!

Ok, here's a rough roadmap for blueprint. All this is open for discussion/suggestions, but the goals I have are:

• Make it easier to generate blueprints from a map
• Capture as much map information as possible in the generated blueprints
• Improve integration with DFHack quickfort

Blueprint usability
This essentially comes down to adding an interactive visual interface. I'm thinking: use the cursor to select the boundaries of the blueprint area (complete with flashing blocks indicating the selected area) and then select the "start" position of the blueprint (where the cursor should be when replaying the blueprint). Other on-screen elements could allow other options to be set before the blueprint is generated.

Capture more map state
There are lots of opportunities for improvement here. Here's a few items that I plan to cover:

• all building types (there are a bunch missing)
• multi-category stockpiles (e.g. stockpiles that have both corpses ('y') and refuse ('r') enabled)
• non-rectangular stockpiles and buildings (like farm plots)
• zones
• stockpile/building/zone names
• room configurations (size, justice settings, door settings, etc.)
• stockpile links (give/take) if within same blueprint
• stockpile top-level settings (i.e. "links only" setting and container counts)

I'd like to be able to exactly reproduce stockpile configuration with regards to which item types are enabled, but it's difficult to do. Perhaps I can integrate with the stockpiles plugin to export the configuration, but I'd need to build functionality into quickfort to be able to automatically import it again. I'll think about this more later, once blueprint is in better shape.

Quickfort integration
Quickfort supports a lot of blueprint metadata that the blueprint plugin doesn't currently provide, like descriptive comments, labels, and blueprint chains. These could be added automatically or customized with blueprint plugin commandline/gui settings.

Finally, I'd like to add a section to the quickfort user guide on how to create blueprints with the blueprint plugin and the bits of syntax you need to know to make finishing touches. A guide for creating blueprints with absolutely minimal effort.

Big questions to answer first

• How important is it to keep compatibility with the original Python quickfort? Should I implement a "--legacy_mode" that generates compatible blueprints? For example, DFHack quickfort supports multi-category stockpiles, but Python quickfort does not. That feature would be disabled if --legacy_mode were specified. It would be a lot of work to consider legacy quickfort for every new blueprint feature, though, and a lot of "if" statements would be sprinkled throughout the code that make it harder to maintain. I wouldn't want to do this unless I were sure there is a good size group of people who would benefit from it.
• Should I re-implement blueprint in Lua? (beware: technical ramblings ahead) It's not that I *hate* C++, but it is a *lot* easier to add functionality in Lua than it is in C++. I knew nothing about Lua when I started quickfort, but the language quickly gained my respect for how it enabled simple and compact code. Blueprint is currently written as a C++ plugin, but there's nothing about the functionality that requires it to be a plugin instead of a native Lua script. All new GUI work would be done in Lua anyway over a C++ / Lua bridge layer. The existing logic in blueprint is minimal, so there would not be much to port. Moreover, most of the blueprint code is the database of mappings from map information to keystrokes. Quickfort already has this database, albeit keyed in reverse. I plan to factor out the quickfort database and make it usable by both quickfort and blueprint. I could still use the shared database from C++ over the aforementioned C++ / Lua bridge layer, but it would introduce a plugin -> script dependency, which is architecturally questionable (it's possible, but it would be the first dependency in that direction, and it could lead to confusion/strange bugs later since it's an unexpected dependency chain).

[ldog]

1. Sounds like a lot of work for little/no gain. I don't see any reason to use the original anymore. Yours is fuller featured. I guess if DFHack stopped being a thing the standalone Quickfort would still work, but it doesn't seem a likely scenario.
2. It looks like you made your case for Lua here. Isn't most of DFHack Lua scripts anyway?

[lethosor]

Blueprint usability
This essentially comes down to adding an interactive visual interface. I'm thinking: use the cursor to select the boundaries of the blueprint area (complete with flashing blocks indicating the selected area) and then select the "start" position of the blueprint (where the cursor should be when replaying the blueprint). Other on-screen elements could allow other options to be set before the blueprint is generated.

Awesome. I was thinking pretty much along the same lines. gui/mass-remove might be useful to reference for some of the UI flow, and there are probably others as well. I hadn't even thought of allowing the start position to be chosen, but that makes sense to me (maybe a default of the top left corner and/or a way to choose presets like the corner/center of a blueprint could be useful?).

Finally, I'd like to add a section to the quickfort user guide on how to create blueprints with the blueprint plugin and the bits of syntax you need to know to make finishing touches. A guide for creating blueprints with absolutely minimal effort.

Thanks, that'd be great!

Should I re-implement blueprint in Lua? ... Blueprint is currently written as a C++ plugin, but there's nothing about the functionality that requires it to be a plugin instead of a native Lua script.

"Convential wisdom" is that map-based operations over a large area tend to be noticeably slow in Lua, with an extreme example being "reveal". I think most reasonably-sized blueprints wouldn't be too slow to create from Lua, but you might want to benchmark something dreamfort-sized and be open to implementing bottlenecks in Lua edit: C++. (I don't have a great understanding of exactly what slowed down Lua "reveal", but DF field accesses can add up, so caching those as locals could potentially help in some cases.)

[myk]

"Convential wisdom" is that map-based operations over a large area tend to be noticeably slow in Lua

ahh, ok. thanks! let me run some benchmarks to see what's feasible

[myk]

Ok, I implemented blueprint in lua for dig. There is a performance difference of about 50x. For a large number of tiles, this is very significant. For example, scanning a 190x190x100 block (that's 3,610,000 tiles) takes only 200ms in C++ but a whopping 10s in lua. 10 seconds is way too long to make users wait for something that's supposed to be interactive.

However, we're expecting blueprints to be over much smaller areas. The entirety of Dreamfort, for example, is less than 60x60x15 (54,000 tiles). On those scales, even the lua implementation only takes 150ms.

So, for the expected common use case (manually run by players, at most once a minute as they make edits to a map and export their blueprints), performance of C++ and Lua is essentially equivalent.

C++ would make a difference if blueprint generation is automated and/or run over a huge area. For example, a script that takes periodic blueprint snapshots of the entire map in the background while a player plays. Do we expect a use case like this to ever happen?

[lethosor]

Looks like I made a typo in my earlier post - I meant to suggest implementing slow parts of the logic in C++ and the rest in Lua. I haven't looked too closely at the blueprint source recently, but do you have an idea of which operations are slow in Lua (if there are any in particular)?

A multiple-second slowdown isn't great, but also not likely for users to run into IMO. That said, I don't know for sure if we would want to support larger blueprints/snapshots down the line, so I'm hesitant to migrate pieces to Lua if they would only be migrated back to C++ later.

[myk]

Looks like I made a typo in my earlier post - I meant to suggest implementing slow parts of the logic in C++ and the rest in Lua.

no worries, I understood what you meant : )

do you have an idea of which operations are slow in Lua (if there are any in particular)?

I implemented blueprint dig in a number of ways, trying to determine the performance of different parts. The breakdown was:

- 500ms for lua loop logic
- 1s for local pos = xyz2pos(x, y, z)
- 4.5s for local tt = dfhack.maps.getTileType(pos)
- 3.5s for local shape = df.tiletype.attrs[ tt ].shape

and on the C++ side:
- 2s for calling lua from C++ for every tile (i.e. Lua::SafeCall())

so doing just about anything in lua is slow at this scale (even xyz2pos, which just puts the vars in a table. I guess there's memory allocation involved), and even just switching between lua and C++ for every tile would take unacceptably long (2s).

edit2: incidentally, caching the dfhack.maps.getTileType function in a local variable to avoid the table lookups shaves a full 700ms off the runtime, so local caching is actually worth it for performance-critical code in lua.

edit3: interestingly, similarly caching df.tiletype.attrs only saves 100ms.

A multiple-second slowdown isn't great, but also not likely for users to run into IMO. That said, I don't know for sure if we would want to support larger blueprints/snapshots down the line, so I'm hesitant to migrate pieces to Lua if they would only be migrated back to C++ later.

yeah, that's fair. Let me think a bit about what bits of logic would really be useful to have in Lua and come up with an architecture.

edit: more detailed performance breakdown

[myk]

so, given that nothing that involving iteration across tiles can be in Lua, I'm thinking the core algorithm will need to be all in C++, with just the gui, commandline parsing, and other pre/post processing tasks being done in lua.