Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[lethosor]

Yeah, the combination of iteration and SafeCall() being slow makes sense to me. Cutting down on the number of SafeCall() invocations would essentially require moving the iteration into C++ as well as the logic (or keeping it in C++, rather), but that should eliminate both of the major slowdowns. Thanks for looking into this!

I don't have a good explanation for why looking up dfhack.maps.getTileType would be slower than df.tiletype.attrs... they are both stored in Lua tables, as far as I can tell, and the "df" table is considerably larger than any of the others involved.

[myk]

Ok, I spent some more time thinking and writing things down in a design doc. I apologize for the formality, but it helps me get my thoughts in order. Let me copy in the user-facing sections here, and I'd love to hear everyone's opinions on which features you're looking forward to and which features you don't really care about. The group of people I'm really trying to reach with these changes to blueprint, though, is the newbies who are generating blueprints and using quickfort for the first time. I'm not expecting them to know about this thread, so I'm afraid ultimately I'll have to move forward with some guesses about what is best.

To recap, the point of this overhaul is to make blueprint a more productive member of the quickfort ecosystem. The blueprint plugin is the easiest way to create useful, valid quickfort blueprints, especially for people new to quickfort. However, there are a lot of opportunities to make blueprints easier to create, edit, and play back. There are also many ways to improve the quality of the generated blueprints via capturing more map information.

The next sections are kind of long, so I'll put them in spoiler tags.

Spoiler: Use Cases (click to show/hide)

There are two major groups of users that this work is targeting:

• Users who want to generate a blueprint and use it with quickfort without having to understand anything about the format
• Users who want to use generated blueprints as a starting point for creating more advanced custom blueprints

I am catering the default settings to members of group 1, with most of the non-default options intended for members of group 2.

For group 1, the expected workflow is:

• run blueprint gui or gui/blueprint to generate a minimal .csv blueprint file
• run quickfort gui or quickfort run at some later time to apply it

I want everything to be as simple as possible for this group. Quickfort has the capability to display messages after a blueprint is run, so we can autogenerate a walkthrough for replaying the blueprints later.

Group 2 will use a greater variety of options and spend time editing the blueprints before replaying them with quickfort. The features that make blueprints easier to read, understand, and edit are for this group. Group 2's expected workflow is:

• run blueprint gui or gui/blueprint, fiddle with options, and generate pretty-printed .csv or .xlsx files
• edit the files in a text editor or spreadsheet program
• save/export the modified blueprints
• run them through quickfort

I expect members of group 2 to make iterative changes to the modified blueprints, either via direct editing or by using the blueprint command to generate blueprints for specific map areas, which they will then use to overwrite portions of their modified blueprints.

Spoiler: Features and commandline API (click to show/hide)

This section is written in the format of DFHack plugin documentation since we'd have to write that later anyway : )

note: I haven't checked it for formatting bugs yet. Also, this text assumes all features are already implemented. In reality, I plan to add bits of help text incrementally as the related features are completed.

blueprint
=========
The ``blueprint`` command records the structure of a portion of your fortress in
a blueprint file that you (or anyone else) can later play back with `quickfort`.

Blueprints are ``.csv`` or ``.xlsx`` files created in the ``blueprints``
subdirectory of your DF folder. The map area to turn into a blueprint is either
selected interactively with the `blueprint gui` command or, if the gui is not
used, starts at the active cursor location and extends right and down for the
requested width and height.

Usage:

    blueprint <width> <height> [<depth>] [<name> [<phases>]] [<options>]
    blueprint gui [<name> [<phases>]] [<options>]

Examples:

blueprint gui
    Runs `gui/blueprint`, the interactive blueprint frontend, where all
    configuration for a ``blueprint`` command can be set visually and
    interactively.

blueprint gui bedrooms --format pretty
    Starts `gui/blueprint` with some configuration options already set to custom
    values.

blueprint 30 40 bedrooms
    Generates blueprints for an area 30 tiles wide by 40 tiles tall, starting
    from the active cursor on the current z-level. Output is written to the
    ``bedrooms.csv`` file in the ``blueprints`` directory.

blueprint 59 63 -15 --splitby level --format xlsx --playback_start "30;32;central stairs"
    Generates blueprints for an area 59 tiles wide by 63 tiles tall for
    15 z-levels, starting from the active cursor and going down the z-levels.
    Blueprints are written to ``blueprints/blueprint_level1.xlsx`` through
    ``blueprints/blueprint_level15.xlsx``. When the blueprints are run with
    quickfort, the cursor should be placed 29 tiles in and 31 tiles down from
    where the cursor was when the blueprints were created (that is, in row 30
    and column 32 of the blueprint area. the comment indicates that the
    cursor should be placed where you want the central stairs of the fortress to be).

Positional Parameters:

:width:   Width of the area (in tiles) to translate.
:height:  Height of the area (in tiles) to translate.
:depth:   Number of z-levels to translate. Positive numbers go *up* from the
          cursor and negative numbers go *down*. Defaults to 1 if not specified,
          indicating that the blueprint should only include the current z-level.
:name:    Base name for blueprint files created in the ``blueprints`` directory.
          If no name is specified, "blueprint" is used by default. The string
          must contain some characters other than numbers so the name won't be
          confused with the optional ``depth`` parameter.

Phases:

If you want to generate blueprints only for specific phases, add their names to
the commandline, anywhere after the blueprint base name. You can list multiple
phases; just separate them with a space.

:dig:    Generate quickfort ``#dig`` blueprints.
:construct:  Generate quickfort ``#build`` blueprints specifically for
         constructions, like walls and floors. Constructions and buildings are
         written to separate blueprints so walls can be built before doors and
         floors can be built before the furniture that sits on top of them.
:build:  Generate quickfort ``#build`` blueprints for building non-construction
         buildings.
:place:  Generate quickfort ``#place`` blueprints for placing stockpiles.
:zone:   Generate quickfort ``#zone`` blueprints for designating zones.
:config: Generate quickfort ``#query`` blueprints for non-room configuration.
         Query commands that can be run immediately after the previous phases
         without waiting for anything to be constructed (like stockpile
         configuration) go in this phase.
:rooms:  Generate quickfort ``#query`` blueprints for configuring rooms and
         other query commands that can't be run until after furniture is placed.

If no phases are specified, blueprint will autodetect which blueprints should
be generated. For example, if there are no zones defined in the area, no
blueprint will be created for the zone phase. If a phase was explicitly requested,
a blueprint for that phase will be generated, even if there is nothing in it.

Options:

:-c, --cursor <x>,<y>,<z>:
    Use the specified map coordinates instead of the current cursor position for
    the boundary of the blueprint range. If this option is specified, then an
    active cursor is not necessary.
:-f, --format <format>:
    Select the output format of the generated files. See the ``Output Formats``
    section below for options. If not specified, the output format defaults to
    "minimal", which will produce a small ``.csv`` file.
:-m, --nometa:
    `Meta blueprints <quickfort-meta>` let you apply all blueprints that can be
    replayed at the same time (without unpausing the game) with a single
    command. This usually reduces the number of quickfort commands you need to
    run to rebuild your fort from about 6 to 2 or 3. If you would rather just have the
    low-level blueprints, this flag will prevent meta blueprints from being generated
    and any low-level blueprints from being `hidden <quickfort-hidden>` from the
    ``quickfort list`` command.
:-s, --playback_start <col number>;<row number>;<descriptive comment>:
    Specify the column and row (relative to the upper-left corner of the blueprint)
    where the player should put the cursor when the blueprint is played back
    with quickfort, in `quickfort start marker <quickfort-start>` format. E.g.:
    ``"10;10;central stairs"``. Note that if there is a space in the comment, you
    need to surround the entire string in double quotes.
:-t, --splitby <strategy>:
    Split blueprints into multiple files. See the ``Splitting output into
    multiple files`` section below for details. If not specified, defaults to
    "none", which will create a standard quickfort
    `multi-blueprint <quickfort-packaging>` file.

Output formats:

Here are the values that can be passed to the ``--format`` flag:

:minimal:
    Creates ``.csv`` files with minimal file size. This is the default.
:pretty:
    Makes the blueprints in the ``.csv`` files easier to read and edit with a text
    editor by adding extra spacing and alignment markers.
:xlsx:
    Creates a binary ``.xlsx`` file with each blueprint on a separate worksheet.
    This option is best for viewing and editing the blueprints in a spreadsheet
    application like Microsoft Excel or `Google Sheets <https://sheets.new>`__.

Splitting output into multiple files:

Here are the values that can be passed to the ``--splitby`` flag. The "ext" in
the filename formats below represents either "csv" or "xlsx", depending on the
requested output format.

:none:
    Writes all blueprints into a single file. This is the standard format for
    quickfort fortress blueprint bundles and is the default.
:level:
    Creates one file of blueprints per z-level. Filenames will be in the format:
    basename_levelN.ext, where N is the level number, starting from 1 as the
    top-most level.
:group:
    Creates one file per group of blueprints that can be played back at the same
    time (without have to unpause the game and let dwarves fulfill jobs between
    blueprint runs). Filenames will be in the format: basename_phases.ext, where
    phases is a string like "dig" or "build_place_config", depending on which
    phases are included in the group.
:phase:
    Creates a separate file for each individual phase. Implies ``--nometa``
    since meta blueprints can't combine blueprints that are in separate files.

Spoiler: GUI design (click to show/hide)

Right panel is overwritten with the blueprint interface and all keyboard input
is intercepted. Panel has 28 columns for text. Format:

Blueprint

Create quickfort blueprints
from a live game map.

Select the first corner.


n: name  blueprint

a: Phases  Autodetect

f: format  Minimal text .csv
  File output format

m: meta  On
  Combine blueprints that
  can be replayed together.

s: playback start tile  Unset
  Set where the cursor
  should be positioned when
  replaying the blueprints.

t: split files:  No
  Split blueprints into
  multiple files.

ESC: Back

The game cursor is active and can be moved with the cursor keys. When Enter is
hit, the "Select the first corner." line changes to "Select the second corner."
and the ESC line changes to:

ESC: Cancel selection

After the first corner is selected, a green flashing X appears where the cursor
is. As the cursor is moved, the area from the start position to the current
cursor position will flash with green X characters. The selected dimensions and
area will appear on the gui panel, for example:

Select the second corner.
  10x15x1 (150 tiles)

When the second corner is selected, the gui sends the configuration to the main
blueprint generation routine (as if the blueprint command were invoked directly
with those options).

- "n" replaces the current name string with an editable field, pre-populated
with the current name (or a blank if the current name was the default "blueprint" string).
Enter commits the new name. ESC dismisses the edit widget
without changing the name. If files with the given basename are detected in the
output directory, a warning is displayed, e.g.:

n: name  blueprint
  Warning: may overwrite
  existing files.

- "a" toggles between Autodetect and Custom, with the following lines shown
if Custom is set. Each phase toggles between On and Off.

a: Phases  Custom
d: dig        On
B: construct  On
b: build      On
p: place      On
z: zone       On
Q: config     On
q: rooms      On

- "f" cycles across "Minimal text .csv", "Pretty text .csv", and "Binary .xlsx"
- "m" toggles between On and Off
- "s" starts a workflow where the player can select a map tile to use as the
  playback start position, and is then prompted to enter a string to use as the comment.
  When set, it will look like this:

s: playback start tile  Set
Comment: user-entered description, truncated to fit in panel width
  Set where the cursor...

Hitting s when the start tile has been set will toggle it back to Unset (though the comment will be saved for when the user selects s again to select a different tile).

- "t" cycles across No, Level, Group, and Phase

Minimal UI workflow
Although setting of the blueprint name is technically optional, I expect the minimal flow to be:

- hit n, enter a name, hit Enter
- use cursor or mouse to select starting tile
- use cursor or mouse to select opposite corner of target area
- As soon as the second corner is selected, appropriate blueprints will be generated.

If you're interested in the architectural or development milestone planning, or just want to see an easier-to-read version of what's in the spoiler tags above, please check out the full design doc -- but please leave comments here so we can have all the discussion in one place.

[myk]

Ok, done with the interactive gui (the gui/blueprint script) and the commandline parsing changes to the blueprint plugin. I have some questions on details of the implementation, though, so it might take a bit to submit all the pull requests and do the reviews.

This forum doesn't allow embedded images, but here's a screenshot of the gui in action: https://drive.google.com/file/d/1YT9wMtTbghaRMxTBmmQkTsyv33sQQe-G/view?usp=sharing
Just imagine the green X's flashing to indicate the selected area.

[Abadrausar]

First of all let me tell you that you are doing a very nice and much needed job with DFHACK QuickFort, for my part I have wished for years for something to appear capable of replacing the good old quickfort within the DFHACK environment, you are responsible for making this dream of mine and many other people in our community come true.

This was surely no easy feast, as the original quickfort is packed with functionality and beautifully designed as attests to the extraordinary longevity of Joel's utilities.

For all that you have already offered to this community, I thank you for the effort you have dedicated, selfishly and especially for the promise of what is yet to come, I would like you to find as much satisfaction developing and polishing DFHACK quickfort as users of this utility will feel using it in the years to come.

Thanks again!

... done with the interactive gui (the gui/blueprint script) and the commandline parsing changes to the blueprint plugin. I have some questions on details of the implementation...

Normally I prefer to play with nano-embarks of 1x1 with 48x48 tiles, that means usually very vertical fortress that use all the area disponible over 6-10 levels for the active or live part of the  fortresss.

In those limite conditions, Would DFHACK QuickFort be able to blueprint all the 48x48xDeep of the fortress?

[myk]

Thank you very much for your kind words! I love working with this community and with the core DFHack developers!

Would DFHACK QuickFort be able to blueprint all the 48x48xDeep of the fortress?

I've run blueprint and quickfort over hundreds of z-levels and millions of tiles. 48x48x10 should be no trouble for it at all. In fact, dreamfort is just about that size since I originally designed it to be usable on a 1x1 embark (though now that I look through the blueprints, it looks like I let the guildhall level get too big to fit. I'll have to fix that. Recently I've been testing on a 4x4 embark for the easier access to trees so I didn't notice that I'd gone beyond 1x1 boundaries.)

You can do all the z-levels at once or you can split them up level by level. Designating all the digging at once makes sense, but you might want to separate out the other phases for the other levels. It would slow down your fort development a lot if you had to wait for the digging to be done for the entire fort before you could start doing any building. I do suggest using the dreamfort blueprints as a model for creating whole-fort blueprint sets. If you haven't already seen it, I wrote a guide for how to build these kinds of large-scale blueprint sets while still making them as easy as possible to reapply later.

[A_Curious_Cat]

How well do they work on 16x16 embarks?

[myk]

I'll have run some tests and get back to you : ) blueprint has to touch every tile, which is slow, but is written in C++, which is fast. quickfort only has to touch the tiles that need modification, which is fast, but is written in lua, which is slow. It will depend a lot on exactly what we're doing with them. What do you think a "typical" 16x16 blueprinted fortress would look like?

[A_Curious_Cat]

Not really sure what a typical one would look like.  Maybe like and 8x8 one with more space to expand laterally?  I’m still trying to survive beyond a few years.  My forts so far seem to have the same basic layout for the main levels.  First I dig ramps down to two levels below the highest rock layer and then dig a long 3-tile-wide tunnel leading to my 1x1 central staircase (which is in a 3x3 room) and then dig out an 11x11 room for my main stockpile behind that.  Then I dig out a 5x5 room in front of the staircase and to the side of the tunnel, with a 3xtile-wide entrance for the trade depot.

Then I dig 4 levels down making 3x3 rooms for each level of the staircase.  On the lowest level so far, I dig out a 33x11 room on one side of the stairwell to act as a meeting hall/dining room with chairs and tables against the walls (which I eventually convert into a tavern).  My tavern always has an engraving of the fortress symbol directly opposite the entrance. Behind that I dig out another 33x11 room for prepared food, drink, and mug stockpiles.  Then, behind that, I dig out a series of 5x5 workshops for my kitchen(s), still(s) (I only recently learned it was better to put my stills down there instead of up with the farms), fishery, butcher’s shop, tannery, and leather works.  Behind that, I have another 33x11 room for raw food stockpiles.  In then dig out three 3-tile-wide hallways from the central stairwell forming a “T” shape.  I then dig out 3x3 offices for my manager and clerk on the inner corners of the “T” near the entrance to my meeting hall.  Next to these (and across from the tavern), I dig out two 3x3 rooms (one on each side of the “T”) with stairs going up to the next level (which is where my visitor housing is).  On each side of the main hallway, I start digging out 11x11 rooms.  The first of these is invariably used to temporarily house my carpenter’s and mason’s workshops and feed stockpiles until I can dig out the industry level.  Later it becomes my display room.  The second one becomes a non-denominational temple.  Other ones become my well room (with 5 wells in a quincunx pattern drawing from a 9x9 reservoir with it’s surface on the level below).  Next to this goes my hospital (with a door directly connecting it to the well room).  Another room is used for my library.  And other rooms are used for dedicated temples (which I only set up when asked to).  I also, start digging out additional 11x11 rooms behind these and make additional 3-tile-wide hallways to access these (thus turning the “T” into an “m”).  These are used for my guildhalls (which I, also, don’t set up until asked to).  Guildhalls seem to go the same way every time.  Start by digging out an 11x11 room and smoothing it.  Place four tables and four chairs.  Add Statues as necessary.  Guildhalls can be converted to grand guildhalls by simply engraving them and adding more statues as necessary.  Finally, on the outer sides of these auxiliary hallways, I put my noble suites.  Currently, these include three connected 5x5 rooms: an office, a dining room, and a bedroom.  The suites are tessellated to reduce the amount of space taken up.  I’m currently considering adding a fourth room with an upright spike in the center linked to an adjacent lever…

I then dig down another level and make my industry level.  This features a central 3x3 stairwell room with 3-tile-wide hallways shooting off of it in all four directions.  I tile workshop/stockpile areas on this level as I am able.  Each area consists of four 5x5 workshops in a row sandwiched between two 24x11 stockpile rooms (except that some workshops -such as the jeweler’s- get their own 5x5 stockpile rooms), one for feeder stockpiles, and one for product stockpiles.  These are usually dedicated to a particular category of industry (woodworking, stone working, metalworking, ceramics, glassworking, etc.) and are completely surrounded by 3-tile-wide hallways.  Of course, I can’t place these “areas” everywhere.  I have to contend with the reservoir.

I then dig down eight levels and dig out an 11x11 room to use as a dormitory.

On the level below the dormitory I start digging out housing blocks.  Each housing block consists of 10 3x3 bedrooms in a 2x5 matrix.  Each housing block is surrounded by 3-tile-wide hallways.

The visitor housing on the level above the tavern level follows a similar pattern.

I also, extend the main staircase up into the soil layers where I build my farms and farming related workshops (as well as a 5x5 seed stockpile room).

Above the farming level, I dig out a large underground pasture and a set of 11x11 rooms for my poultry.

(In one fortress, I even managed to dig out a multi-level tree farm, but didn’t last long enough to harvest anything from it).

I also dig out 5x5 sand and clay gathering areas on appropriate levels.

The reason that I have the entrance tunnel sandwiched between two unused z-levels above and two below, is because I eventually want to use the space for a “welcoming center” for unwanted guests (complete with a trauma conga line).

Anyways, the above might not be dug/built in the order listed, and I might have missed something, but hopefully, that should give you and idea of how I’m building my forts.

And all of this, so far, has been on 8x8 embarks.

[myk]

From what you describe of your fort, I think both blueprint and quickfort should complete in less than a second. If you edit the blueprint files and do something really complex, maybe quickfort will take two or three seconds (and yes, there are ways to make pathologically complex blueprints that can take longer to apply), but the relatively simple blueprints generated by the blueprint plugin should get applied by quickfort as fast as you can type the commands. You probably would want to split the blueprints up per level so your dorfs can concentrate on getting one thing done at a time, though.

For each level (or group of related adjacent levels), this is how it could work:

• in your existing, built fort, run gui/blueprint and select the (possibly multi-level) area to generate blueprints for
• in a new, blank fort, run gui/quickfort to apply the dig mode blueprint (or use the non-gui quickfort command according to your preference)
• once the area is dug out, use quickfort to apply the place and build blueprints
• once furniture is placed, use quickfort to apply the query blueprint, if there are any rooms to configure

There will still be some manual steps, like digging the ramps down through a variable number of soil layers, but blueprint and quckfort should be able to make the repetitive parts go by much more quickly.

Note that stockpile configurations are not yet recorded in blueprints. My current idea is that if you name the stockpile something obvious, like "plants" or "pots", blueprint can figure out what quickfort alias to use to do the configuration. Recording arbitrary stockpile configuration might be too difficult, though. I'll know more once I start looking into this near the end of the planned overhaul.

[Garfunkel]

How well do they work on 16x16 embarks?

What sort of supercomputer do you have to that you can do 16x16 embarks! 

[A_Curious_Cat]

How well do they work on 16x16 embarks?

What sort of supercomputer do you have to that you can do 16x16 embarks! 

Iirc, it has an older Core i8 Extreme Edition processor (Ivy Bridge, I think) and 16 GB of memory.  I think it can be upgraded to 32 GB.

[A_Curious_Cat]

Now that I come to think of it, I think it might be a sandy bridge processor.

Anyways, that machine isn’t working right now so I’m using a notebook with a skylake processor.  It’s only got 8GB of memory, only 4 cores, and runs at a lower clock rate, but for some reason it feels like it’s genning worlds faster.

[myk]

Normally I prefer to play with nano-embarks of 1x1 with 48x48 tiles

I shrunk the dreamfort guildhall level down to eight 7x7 rooms instead of sixteen so it fits more comfortably in a 1x1 embark.  If a fort needs more hall/temple/library space, you can always run

Code: [Select]

quickfort run library/dreamfort.csv -n /guildhall1 and lay down another guildhall level.

old guildhall level


new guildhall level


The tables, chairs, altars, etc. are not part of the blueprint, of course. They were just what I needed for that particular fort.

[myk]

I needed to make an insta-dig tool so I could test the blueprint-quickfort-blueprint loop. Since it will be a standard DFHack plugin visible to players, I tried to make it useful to and usable by players in addition to serving my purposes for testing. So without further ado, I bring you dig-now:

Code: [Select]

dig-now
=======

Instantly completes dig designations, modifying map tiles and creating boulders,
ores, and gems as if a miner were doing the mining or engraving. By default, all
dig designations on the map are completed and boulder generation follows
standard game rules, but the behavior is configurable.

Note that no units will get mining or engraving experience for the dug/engraved
tiles.

Currently, trees are not chopped down and any designated trees or roots will be
ignored. Engravings are also not automatically generated by this plugin as they
depend on the skill and creative choices of individual engravers.

Usage::

    dig-now [<pos> <pos>] [<options>]

Where the optional ``<pos>`` pair can be used to specify the coordinate bounds
within which ``dig-now`` will operate. If they are not specified, ``dig-now``
will scan the entire map.

Any ``<pos>`` parameters can either be an ``<x>,<y>,<z>`` triple (e.g.
``35,12,150``) or the string ``here``, which means the position of the active
game cursor should be used.

Examples:

``dig-now``
    Dig designated tiles according to standard game rules.

``dig-now --clean``
    Dig designated tiles, but don't generate any boulders, ores, or gems.

``dig-now --dump here``
    Dig tiles and dump all generated boulders, ores, and gems at the tile under
    the game cursor.

Options:

:``-c``, ``--clean``:
    Don't generate any boulders, ores, or gems. Equivalent to
    ``--percentages 0,0,0,0``.
:``-d``, ``--dump <pos>``:
    Dump any generated items at the specified coordinates. If the tile at those
    coordinates is open space or is a wall, items will be generated on the
    closest walkable tile below.
:``-e``, ``--everywhere``:
    Generate a boulder, ore, or gem for every tile that can produce one.
    Equivalent to ``--percentages 100,100,100,100``.
:``-h``, ``--help``:
    Show quick usage help text.
:``-p``, ``--percentages <layer>,<vein>,<small cluster>,<deep>``:
    Set item generation percentages for each of the tile categories. The
    ``vein`` category includes both the large oval clusters and the long stringy
    mineral veins. Default is ``25,33,100,100``.
:``-z``, ``--cur-zlevel``:
    Restricts the bounds to the currently visible z-level.

This is certainly a "cheat" plugin, but I expect it to be useful for players who want to rapidly test quickfort blueprints.

[Nameless Archon]

This is certainly a "cheat" plugin, but I expect it to be useful for players who want to rapidly test quickfort blueprints.

Excellent. Just in time for me to begin poking at this for real, instead of in my head.