Bay 12 Games Forum

Please login or register.

Login with username, password and session length
Advanced search  
Pages: [1] 2 3

Author Topic: Python Utilities for DF now with Modding with Excel! (v2.3)  (Read 8862 times)

Roses

  • Bay Watcher
    • View Profile
Python Utilities for DF now with Modding with Excel! (v2.3)
« on: August 19, 2013, 06:30:46 pm »

HAS NOT BEEN UPDATED TO .40x! WILL BE UPDATED SHORTLY.

So I have written a couple different python utilities for analyzing and modifying dwarf fortress raws. I figured I would start uploading them as I make them public worthy.

Modding with Excel!
 
Modding with Excel is a new set of python scripts I have been working on that allows you to move data between .txt and .csv (comma separated value) files. This allows you to add entries via Excel (or your favorite CSV reader) easily and cleanly, allowing for sorting and quick analysis of entries. Now working for items and inorganics. There are included csv files so you can see how to set them up.
Screen shot for MWE:
Spoiler (click to show/hide)

PLEASE NOTE: In the columns with more than one entry the python program is expecting enters between the entries (i.e. \n). If people feel this is too limiting I can have it acknowledge the entries in a different way.

MWE_ITEMS: This utility will handle all item types (Weapons, Ammo, Armor, Gloves, Helm, Pants, Shoes, Food, Instruments, Tools, Toys, Siege Ammo, and Trap comps) by making separate files for each. To use, simply place the python script in your raws folder and run. It will prompt you if you wish to copy from Raw to CSV or CSV to Raw, choose by entering 1 or -1.

MWE_INORGANICS: This is a preliminary utility. It has all the function you need to create inorganics in Excel, but it doesn't break up the inorganic tokens by what they are commonly used for (i.e. there is a column for MEAT_NAME for all them). I am planning on adding some more logic to the program so that the csv files aren't so large, but currently it works fine (also supports syndromes in your inorganics). Functions the same as MWE_ITEMS.

MWE_PLANTS: Another preliminary utility that suffers from the same drawbacks as MWE_INORGANICS (although it does successfully parse the plants into trees, grasses, and crops) which makes the csv files more manageable.

Both MWE_INORGANICS and MWE_PLANTS works better to generate raws from then as a means for sorting data. More work needs to go into making the csv files generated useful.

MWE_CREATURES: This utility differs from the others in that it only works in one direction (namely to go from CSV files to RAW files). Because of the complexity of creatures I have also introduced a new feature, Templates! (I will be adding this feature into the other MWE scripts as well). Templates allow you to group your creatures with similar bodies, attacks, castes, and traits, while still giving you the ability to modify them. A sample attack template is provided below;

Code: [Select]
[TEMPLATE:BITE]
[ATTACK:BITE:BODYPART:BY_CATEGORY:#ARG1]
[ATTACK_SKILL:BITE]
[ATTACK_VERB:bite:bites]
[ATTACK_CONTACT_PERC:#ARG2]
[ATTACK_PENETRATION_PERC:#ARG3]
[ATTACK_FLAG_EDGE]
[ATTACK_PRIORITY:#ARG4]
[ATTACK_FLAG_CANLATCH]

you can reference this template by providing a line in the CSV file under the attack column BITE:BEAK:100:100:MAIN. Templates are able to handle an infinite number of arguments, but for ease of use it is best to keep your arguments limited in number. An example CSV file (creatures.csv) is included to show case the use of this script. In addition four template files are included (templates_ATTACK.txt, templates_BODY.txt, templates_CASTE.txt, templates_TYPE.txt) each with example templates that are referenced in the CSV file. This script allows fast an easy creation of new creatures. (NOTE: Currently only the ATTACK and CASTE templates support variable arguments, the BODY template uses specific arguments for materials and tissues, and the TYPE template has no argument support. This will be fixed in a later version)

Misc Scripts

ADD_CASTES: This utility adds caste information to creatures raws. It only modifies files specified and adds castes that are user specified to the creatures in those files (UPDATE: There is now an external file named creature_iist.txt that lets the user specify which creatures are actually modified). This allows adding things like giant variations directly to the creature instead of having them as a separate creature. This script requires input from the user. To use open the script with some text editor (I personally use VIM) and modify the required fields (there is short comments that explain what to do). Because of the extra level of complication with this script files in the objects folder will not be overwritten, instead a new file will be created in the raws folder which can then be checked to make sure it did what you wanted, and then put into the objects folder.
Screen shot for ADD_CASTES:
Spoiler (click to show/hide)

ADD_SIZE: This utility reads all of the creature raws and puts the max size of the creature (as listed in the creature raws) in the creatures description. This utility currently has several limitations. It does not add a size to any creature that doesn't have a size listed in the raws (most creatures that use the creature variation templates, like GIANT and ANIMAL_PERSON, don't have their own size listed), and it only lists the largest size in the raws. Also, subsequent runs will not change the values in the description, even if the values in the raws are changed. To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files.
Screen shot for ADD_SIZE:
Spoiler (click to show/hide)

ORGANIZE_ENTITY: This utility reads the entity files and separates each entity into their own files. In addition it organizes the file into different components (e.g. Equipment, Items, Behavior, Ethics, Flavor, etc...) This allows for easier navigation of the entity files. To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files.
Screen shot for ORGANIZE_ENTITY:
Spoiler (click to show/hide)

SPLIT_CVARIATIONS: This utility reads through all of the creatures raws and seperates out the ANIMAL_PERSON and GIANT creature variations into their own files. This allows for easy deletion of these creatures for those that don't like having them in the game, or for easy manipulation.  To run, simply put the python file in the raws folder and run it, it will modify all creature files in the objects folder. As always, it is a good idea to back up your files before running any utility that directly modifies files. (UPDATE: Fixed a bug that was brought to my attention by CryptoCactus)
Screen shot for SPLIT_CVARIATIONS:
Spoiler (click to show/hide)

SPLIT_REACTIONS: This utility reads through all of the reaction raws, and separates them into individual reaction files. I use this for taking apart mods where I just want to use one specific building. To run it all you need to do is place the python file in the raws folder and it will spit out an individual file for each building mentioned in the raws. I have personally used it with Masterwork and Genesis Reborn with no issues. Please note that some mods make use of hundreds of buildings, so it will generate hundreds of text files. Only use if you are wanting a specific buildings reactions separated out.
Screen shot for SPLIT_REACTIONS:
Spoiler (click to show/hide)

SPLIT_VERMIN: Same as SPLIT_CVARIATIONS except it pulls out vermin.

More utilities will be added as I make them fit for public use.
http://dffd.wimbli.com/file.php?id=7916

Python 3+ is required for use. Please let me know if you run into an trouble with any of the utilities
« Last Edit: July 17, 2014, 04:58:57 pm by Roses »
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Python Utilities for DF (v1.0)
« Reply #1 on: August 20, 2013, 02:08:31 pm »

I think I do like that screenshot quite a bit. With some extra work I could seperate different features of the mod and offer them as a stand-alone download for vanilla DF.
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.0)
« Reply #2 on: August 20, 2013, 02:20:59 pm »

I think I do like that screenshot quite a bit. With some extra work I could seperate different features of the mod and offer them as a stand-alone download for vanilla DF.

The python code itself is fairly simple and can easily be extended to other raw files (I've experimented with splitting out Mega and Semi-Mega beasts from creature files, economic vs war metals, and  above ground vs under ground plants). Pretty much anything that has a distinct raw token (or tokens) can be extracted and split into separate files.
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Python Utilities for DF (v1.0)
« Reply #3 on: August 20, 2013, 02:56:01 pm »

Thats nice. Would help me sort stuff for my UI. But I dont think that I can export complete features, because too many things interlock.

But yeah, sorting only aboveground vs underground plants/creatures, and only megabeasts, only creatures with interactions... thats all interesting. :)
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.0)
« Reply #4 on: August 20, 2013, 03:10:53 pm »

Thats nice. Would help me sort stuff for my UI. But I dont think that I can export complete features, because too many things interlock.

But yeah, sorting only aboveground vs underground plants/creatures, and only megabeasts, only creatures with interactions... thats all interesting. :)

I am actually working on a more robust version that can look through all the raws and grab all of the dependencies. For example, say I want to use your runesmith reactions, it would go through the raws, give me the buildings required, any inorganic tokens, items, materials, and other things needed. I am just having trouble with how to approach reaction classes, and how to make sure everything needed gets included (first run through might tell me x is needed, but x might need y and y might need z etc...)
Logged

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.1)
« Reply #5 on: August 20, 2013, 03:28:17 pm »

A new utility has been added, also a bug in the STRIP utility has been fixed.
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Python Utilities for DF (v1.1)
« Reply #6 on: August 20, 2013, 03:39:58 pm »

A while ago I tried to make a flowchart for the library... but it was impossible.

Ok, you can write on paper, vellum or papyrus. Thats 3 sources from 3 buildings, from plants, leather, or wood. So do I include woodcutting into the chart? How about treefarms as a source of wood? Or caravans? The Merchant Stalls I added can also sell you wood. Or you raid elves, they have wooden items to steal. Papyrus comes from plants... so herbalism/gathering? And farm-plots. And the Herbalists Lab can also make plants. Where do the dependencies end? ^^

Anyway, good luck :) I am still curious about your "soul system / kill dragon learn dragonfire - private mod" ;)
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.1)
« Reply #7 on: August 20, 2013, 03:48:42 pm »

Anyway, good luck :) I am still curious about your "soul system / kill dragon learn dragonfire - private mod" ;)

Yeah, it can get really really complicated, right now I am thinking of just doing the absolute minimum (i.e. all tokens are present, and DF won't give you any errors), which would mean as long as they aren't special leather, plants, or wood, they wouldn't get included in the extraction of the library.

I have a couple python scripts for adding souls to all creatures and such that I am prettying up and getting rid of errors so that I can post them. Trying to come up with a reasonable automation system so that you don't have to go through all the creatures and assign their soul level individually. It is pretty straight forward for some creatures (you can just say all Megabeasts have a level 5 soul, Semi-Mega Beasts have level 4) but other creatures aren't quite so easy. I suppose I should post my own raws at some point, but I keep making changes and additions, and never seem to be quite happy with them.
Logged

Meph

  • Bay Watcher
    • View Profile
    • worldbicyclist
Re: Python Utilities for DF (v1.1)
« Reply #8 on: August 20, 2013, 03:59:01 pm »

If you figure out a nice way to use corpses, let me know... its driving me mad atm.

I want to use corpses of invaders in specific reactions, but I cant target them... I have to give them itemcorpses, which works. Then I wanted to use an itemcorpse of a civ-member, but it still counts as corpse, not as item. I even transformed a dwarf into a completely different creature using TrueTransform, then kill it, then use its itemcorpse... but no, its still a corpse that gets buried. So that idea I have to scrap completely.

Same thing goes for using DISPLAY_TILE:TILE on civ-members. It seems that the profession graphics overwrite the added display tile from syndromes. So instead a dwarf miner will look like a dwarf miner, even if he has DISPLAY_TILE:TILE:64, which is the tile humans use. Damnation and curses on DF for doing stuff like that. Have to scrap three different ideas I had because of that.  ::)
Logged
::: ☼Meph Tileset☼☼Map Tileset☼- 32x graphic sets with TWBT :::
::: ☼MASTERWORK DF☼ - A comprehensive mod pack now on Patreon - 250.000+ downloads and counting :::
::: WorldBicyclist.com - Follow my bike tours around the world - 148 countries visited :::

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.2)
« Reply #9 on: September 11, 2013, 11:49:42 am »

New scripts added
Logged

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF (v1.3)
« Reply #10 on: November 04, 2013, 12:54:46 am »

New scripts added and a fix to a previous script brought to my attention by CryptoCactus.

Please let me know if you have any ideas for new scripts or any trouble with these.
Logged

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF now with Modding with Excel! (v2.0)
« Reply #11 on: December 01, 2013, 05:45:13 pm »

New script added, the first of many in the Modding with Excel set! Read the first post for more information.
Logged

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF now with Modding with Excel! (v2.1)
« Reply #12 on: December 02, 2013, 08:45:53 pm »

Modding with Excel now handles inorganics as well as items. Sample csv files have been included so that you can see how to better utilize these tools.

Up next are plants!
« Last Edit: December 02, 2013, 08:49:27 pm by Roses »
Logged

Roses

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF now with Modding with Excel! (v2.2)
« Reply #13 on: December 03, 2013, 10:09:35 pm »

Plants and inorganics are now both preliminarily handled. (See first post for more information about the drawbacks and difficulties)

Next up are the much dreaded creatures. They will work slightly differently by supplying templates (e.g. specifying BIRD will put in all the raws that are in the BIRD template). Will make assigning creatures and their materials much easier, but will not allow RAW -> CSV conversion (i.e. will be CSV -> RAW only)
Logged

Maklak

  • Bay Watcher
    • View Profile
Re: Python Utilities for DF now with Modding with Excel! (v2.2)
« Reply #14 on: December 05, 2013, 11:17:30 am »

I'm working on an off on a set of similar RAW analysers in Perl. They only print tables of data, though; I have no ambitions to have them aid modding directly. CSV is one of my output formats. http://www.bay12forums.com/smf/index.php?topic=126953.0

My question is, how do you handle a list of something in a CSV file? For example an item has a set of properties (ID, name, size, whatever) that I can separate by commas. What do I do to get a column that has multiple things, like Attacks?  What did you do to display them so nicely in a spreadhseet?
Logged
Quote from: Omnicega
Since you seem to criticize most things harsher than concentrated acid, I'll take that as a compliment.
On mining Organics
Military guide for FoE mod.
Research: Crossbow with axe and shield.
Dropbox referral
Pages: [1] 2 3