Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[Rose]

Making a thread to move the discussion out of the DFHack thread.

So the idea is to export an entire DF world to an external program, as a one-time operation (As opposed to Armok Vision, which is realtime)

How would be the best way to do this?

One idea was Unreal Engine, but I don't know how well it handles 10,000 square KM of terrain out of the box.

I'm partial to Unity, mainly because I know it, but it, too would need trickery to support loading that much terrain.

[Algorithman]

UE4 handles level streaming quite well and there are ways to even do it without (world shifting for example) and it comes with some nice goodies like RuntimeMeshComponent (which supports collision checks out of the box). (here's a nice example what's possible with RMC: https://www.youtube.com/watch?v=G84M1NJiERE)

I worked with unity before (when i first created my dungeon generator) but i switched to ue4 mostly because of the speed improvements on calculations. pure c++ is just faster.

btw, good name

[Algorithman]

As for the data needed for each tile, is there a 'compendium' how tile data is organized in DF?

[Rose]

There's DFHack, and all the source files it uses.

But I think it really comes down to how accurate you want things to be.

[Rose]

Worldgen buildings can be done using available info

[Algorithman]

If i can be accurate, then i want to be accurate.

Starting of course with the pure landscape, and step by step adding the other features. I guess i need to do some reading this evening.

[Rose]

An exact tile per tile heightmap would be 38957 megapixels, and would be 38gb uncompressed, if you discard the bottom of the ocean, and don't store color.

[Roses]

An exact tile per tile heightmap would be 38957 megapixels, and would be 38gb uncompressed, if you discard the bottom of the ocean, and don't store color.

For what world size?

[Rose]

257 region tiles wide, or 4112 embark tiles, or 197,376 fortress tiles, or about 400km

[Algorithman]

Actually i'm not scared about 40GB, only how much data i need to load for displaying the part where the spectator is. it needs to be efficiently organized, to have low loading times. (i'm so glad i bought that nice samsung 960 evo m2 ssd, 3.2GB read)

japa, how many bytes per tile do you think are needed? for the heightmap alone i only need one byte, which would represent the center of the tile. the four corner vertices then can be calculated.

[PatrikLundell]

One byte isn't sufficient for Elevation, as it can be higher than 256 even without taking what's underground into consideration.
The number of bytes you need per tile depends on the number of pieces of info you want to store for the tile, so instead of asking Japa to pull a guess out of a hat, state what you want to have available and determine the size of that.
Basic stuff:
- Elevation
- Rainfall
- Vegetation
- Drainage
- Salinity
- Savagery
- Evilness
- Volcanism
Most of those can affect the biome. At a guess, though, you only want Elevation.
Stuff on the tile:
- Tree trunk/branches, etc. Stored in a sparse enum that requires 2 bytes
- Tile "floor" material
- Vegetation, if any (tree trunk just says that, but not that it's an apple tree)
- Flowers and other "contaminants" both up in the trees and on the ground
- Tile material (soil/stone, air, water, magma, etc.)
- Critters occupying tile (and I'm not sure there's a limit to that list)
- Items stored on the tile (and a QSP can contain thousands)
- Smoke/mist, etc. Not sure where that is stored, but I would guess not as part of the above.
- "Ordinary" contaminats, such as blood (including critter producing it, as e.g. troll blood has a different color from insect ichor), vomit, booze, etc. I don't know if this list has a limit either.

The above is from the top of my head, so there should be stuff missing.
I'd start looking at the data provided by DFHack "probe" command for tile stuff and look at the various DF structures that may be of interest.

[Algorithman]

Thx patrik for the nice list.

some questions/remarks:

Elevation; i concluded 1byte size from the legends export height map, since it's greyscale. seems DF is compressing that already in legends mode
Rainfall, Vegetation, Drainage, Salinity, Savagery, Evilness and Volcanism: I need those in a much lower granularity (if i need them)
Trees: i'll store these in a seperate list, needs to be seperated from the heightmap anyway for efficient display, and probably i wont care about the upper levels of the trees, except maybe the final height of the tree
Tile material: A must, but maybe it can be compressed with a rle compression, need to look at actual data first
Critters, items, smoke and contaminants: as i understand mifki, the whole worldmap is only accessible on embark screen, so i don't know if they are available at that point, will try out when i come home from work

Another thing i need to check is if DF can be read fast enough to get a (somewhat cached/read ahead) 'live' readout while in embark.
I don't aim (yet?) for a ingame mode like armok's vision, a still map is quite enough for me

@mifki (if you're reading this): Do you remember how long it took to read the data out of DF for your bigmap?

[mifki]

Critters, items, smoke and contaminants: as i understand mifki, the whole worldmap is only accessible on embark screen, so i don't know if they are available at that point, will try out when i come home from work

@mifki (if you're reading this): Do you remember how long it took to read the data out of DF for your bigmap?

No, the reading is actually done in adventure mode by forcing the game to load adjacent areas like it does when you go near an edge with the adventurer. So the delay is the same as when you travel in adventure mode.

[PatrikLundell]

Elaborating on what mifki said, the embark map can only provide you with region level data: the site level doesn't exist at that time.

"Natural" coarse levels for parameters are region tile level and world tile level. Note that biome boundaries are visible at the embark level (try embarking at a site with a desert with zero rainfall and a jungle, and you'll see a 2 m tile level jagged boundary). However, the actual values are the same for the whole area that refers to the same world tile (biomes can refer to any of the 8 surrounding world tiles, plus the embark tile), so it's possible to store the data as DF does, i.e. with the data at the world tile level with embark tiles referring to the controlling world tile (and with site level tiles referring to region level tiles). Elevation is the exception to the above, as it varies both at the region and site level, and elevation variation does not follow biome boundaries.

[ragundo]

You can only get the data for the world in the following ways

• In the embark screen as you move the cursor. DF fills world_region_details array and generate the tiles needed
• Exporting it in legends mode, of course, not all data
• Mifki's way

I did a plugin sometime ago precisely for this, but it needed to call DF subroutines in assembly in order to fill world_region_details_array vector with the data corresponding for each world coordinate. Hence, it needed to get updated in each DF release, and also, I did it when DF was 32 bits, and needed to fight with different calling conventions to subroutines, etc.
It worked, but I would need to update it to 64 bit DF now that the calling conventions are simplified in 64 bit programming.
https://github.com/ragundo/exportmaps

Anyway, I would wait for the new DF release, which, I think, will have a world map while in game. I'm dying to begin to disassemble that stuff.