Print

Please login or register.
1 Hour 1 Day 1 Week 1 Month Forever
Login with username, password and session length

[voliol]

The below suggestion (and description of the current state of modding/mod installing) is in part based on the discussion held between primarily Mr_Crabman and I on this thread on the modding board. In case you decide to read it, the thread uses some different terminology, though this suggestion should stand on its own. When it comes to the mod manager mentioned, I have written a mockup which can hopefully help visualize the idea better, though it does not work exactly as written in this suggestion, or use identical syntax.

So the mod structure will soon be getting reworked. Let's talk about that.

There are three kinds of goals to achieve with this new reworked structure:

• First off, mods should become more available. Players should be able to use mods without having knowledge of how they work, and be less alienated by the way mods are installed.
• Second, everything that is possible using the current methods of creating, finding, and installing mods should also be possible in the new system. This might seem trivial, but it turns out to have some conflicts with making it more available, and moving away from folders and manual mod-merging.
• Third, various goals related to improving the raw syntax. These are more of stretch goals, as none are needed as a core part of making and using mods, but many of them are still very useful, and/or have been on the to-do list for a long time, so doing some of them along with other mod structure changes might not be a bad idea.

There is also the non-functional requirement of having support for Steam Workshop, as that has been promised. At the same time, the system should work for those who play Classic, or Premium via an itch.io purchase. For that reason, it should not be too dependent on Steam Workshop.
Finally, there are goals related to the graphics' need to move out of the save files. I will not talk about them much in this initial post, but I encourage the discussion in the thread. This is obviously an important part of the rewrite, as it is half of what prompted it (the other being Steam Workshop)(as far as I know). The reason I won't is that I don't have a good grasp on graphics, so I simply don't have the suggestions.

In this post, I will first map out the way mods are currently installed, and then move on to suggesting how they could/should be installed (by using an in-game mod manager). This way we will see what is needed, and what can currently done (goals of the first and second kind) before arriving at the suggestion. Third, there is the technical aspect of how to merge mods, and finally the syntax improvement suggestions, as they tie into that technical aspect (and are also less important) they can't come before it.

How mods are currently installed:

• The player first finds a mod online, likely on the Modding board here on the forum, or directly from DFFD (though these are my own guesses from where I know there are mods). Sometimes there are multiple versions of a mod available, the most common being modular (raw files only) vs. pre-installed (comes with a full install of Dwarf Fortress, and the mod already installed in it).
  Mods may have installers of their own, like Masterwork, but these mods are by far in minority (and may indeed only be Masterwork),
  so I will not be going over them.
• Then they download it.
• The downloaded mod is usually found in a compressed folder of some kind, but the structure here may vary, even if we include the pre-installed. Some mods contain only one set of raw objects, some contain multiple subfolders with versions or parts of the mod to be chosen by the player. For this reason, most mods have an installation instruction somewhere, and these are not all identical.
• In all cases but with pre-installed mods (and these are already installed, so they won't be mentioned from now), the player then drags some files into the appropriate folder (but which one this is may vary!) of their DF install, with the goal of getting the mod's raw files into the /raw/objects folder (and sometimes some other files into other folders, like graphics and speech files). The mod's raw files either overwrite existing files, or don't.
  All raw files contain so-called raw objects describing the objects in the game, each with a supposedly unique ID (like CREATURE:DWARF for the dwarf creature), and a definition built up by "tokens" (like [DESCRIPTION:A sturdy creature fond of alcohol and industry.]). When mods overwrite existing files, this is usually to make changes to existing objects, by providing a version of the file where one or several objects have been manually edited by the modder. The modder may also add new objects to an existing file (adding a sphinx to creature_standard.txt), but this is considered bad practice for reasons that will become apparent. New objects are instead better placed in their own files, these are the files that do not overwrite.
• When the mod has been installed, any new worlds created will use the mod. It is also possible to mod the raws of existing saves, though this comes with some limitations as many objects (such as creatures and entities) will only appear in-game if they were in the raws at the time of world generation.

Installing multiple mods at once is complicated, due to two kinds of conflicts.
The first is raw duplication, when two objects (possibly in different files) have the same object ID it causes some kind of offset error, with grievous (if funny) results. Raw duplication is usually easy to avoid by using prefixes/suffixes in object IDs; CREATURE:MOD1_SPHINX is a different id from CREATURE:SPHINX_MOD2, but this is a convention new modders often do not know about. Raw duplication may also arise even with unique names, for example when the same objects are used as a base in multiple mods by the same creator, and two or more are installed together.

The other kind of conflict is a file conflict, when two mods use files of the same name they overwrite another, and thus the content in all but last installed of those files is left out. Perhaps the most common example is two mods adding reactions to ENTITY:MOUNTAIN in entity_default.txt, as that is needed for dwarven civilizations (and thus players) to use the reactions in-game. As a solution for that case, some mods provide lists of reactions to manually add to the entities when installing the mod. While functional, this is very hefty.

Is is not possible to uninstall a mod that changes Vanilla files any other way than replacing your modded /raw folder with a clean one, nor is it possible to switch between mod loadouts quickly; the fastest way to do so is to open a separate installation of DF with the other set of mods already installed.


Suggestion:

What I suggest, is to have an in-game mod manager. The mod manager would depend on a folder of "all mods". In this folder, the player would place mods as separate sub-folders (or compressed .zip files, if that works better with Steam Workshop) in some unified format, unlike the current downloads. For those using Steam Workshop, the "subscribe" button would place a mod in the folder, and "unsubscribing" would remove it, while those not using Steam would have to find mods elsewhere and place them manually there. This is the extent that the player would have to dig in the files (not at all with Steam Workshop), with the mod manager handling the use cases of installing and uninstalling mods.

I suggest reusing many of the same GUI elements from the the trade screen for the mod manager screen, both to instill understanding through familiarity, and to save time developing (if it does, this is an assumption on my part).

• When the player enters the mod manager screen for the first time, after having placed a few mods in the "all mods" folder, all mods are listed on the left side of the screen.
• By selecting them they may see info on each of the mods, such as mod name, creator, version (of the mod), and what version of Dwarf Fortress it is meant for. It should also show a description (written by the creator), and dependencies with other mods, including what mods (the creator knows) the mod is incompatible with.
• The player may also move mods to the right side of the screen, where they are listed as mods to be installed, and conversely mods may be moved from the right list to the left to be "uninstalled". When the player exits the mod manager this configuration is saved, so the to-be-installed mods are the same as last time when they come back.

There is also a "mod" which is on the right "to-be-installed" side by default: the vanilla raws. These may or may not be kept in the "all mods"
folder - if they are kept separately there is a smaller risk of players accidentally erasing them, and it becomes easier to have a "vanilla toggle", turning off all mods without losing your configuration for the future. Regardless I argue they should be in the mod manager screen, to show vanilla raws are not hard-coded, and also that they should be possible to uninstall, for the sake of total conversion mods.

• I'm not sure when the installing is best done, and where the mod manager is best placed. Perhaps on the main screen, or as a step when creating a new world, as new worlds are the prime targets.
  In either case, at the time of "installing" mods (which could be as soon as the mod manager screen is exited), all the mods in the right list have their folders read, and their combined raws are what is used by a save. Here comes the difficult technical part, how do you merge mods? How do you deal with the aforementioned conflicts? The easier way is probably to do it with "patch-style edits" (rather than to use git-like merges of full files) where some parts of a mod relate to a base file (or base raw object) and have instructions on how to add/remove/change(convert) certain tokens, or remove the whole object.

There is one more integral part for the mod manager to cover the use cases we currently have (before getting to patches): mod options and/or mod bundles. The point of both of them is to allow variants of a mod and mods subdivided into optional parts, without separate downloads.
With mod options, the idea is to have options for each mod (presumably set using similar methods to the "main" options, as those will have some in-game solution as well). These can toggle parts of the mod on/off, and set values within the raws. From a technical standpoint I imagine they would work similar to the arguments used in creature variations (this explanation/suggestion might be expanded as well), but for the whole mod.

A mod bundle is something like a bin in the trade screen, it contains several "mods" and you can choose to only include some of them to be installed. In the "all mods" folder, a mod bundle is a folder containing multiple mod folders. Mod bundles cover the same use cases as mod options (the bundle containing all variations of the mod as separate mods, the same thing with optional parts), though they also have a technical function related to the merging of mods, if one of the "extra" goals below is met. Mod bundles could also be used for mod packs, though I think the collections on Steam Workshop work pretty well to fulfill that niche.

[voliol]

Patch-style edits:

The point of a patch-style edit (or a patch, or an edit) is to describe only the changes made to a base object, if tokens are added/removed/changed(converted) or if the whole object is removed.

When applying a patch, the base object must first be read by whatever installs the mods, so some sort of order is implied. Patches should also be able to negate one another. For example: we have Mod A, Mod B and Mod C.

• Mod A includes the base version of the object (let's call it TOAD)
• Mod B includes a patch to remove some tokens in TOAD, but also other patches/additions
• Mod C is meant to revert the changes Mod B made to TOAD, by including a patch which re-adds the removed tokens

Here, it is important that the base Mod A is first, and the patches in B and C must be applied in order, otherwise C will add the tokens only for B to remove them along with the original tokens. Solving this order for every set of mods is difficult, I think this is a case where the task is best offloaded onto the player. The mods in the right list are ordered, and the player is able to change the order, guided by instructions in the mod descriptions, and the dependencies (as these imply which mods are before).
Whether this order should be top-down or bottom-up, with the base mods on the top or bottom of the list, is arbitrary. There is intuition and examples in other games for both.

But before that, here's aside on how to solve duplication errors, have "later" objects (in the installation order) overwrite "earlier" ones. This can be a useful tool, redefining an object from scratch when the corresponding patch would be too substantial, but mostly it solves duplication errors. Even when misused, the worst case scenario is that objects that should not be overwritten are, and parts of mods are left out like with the current file conflicts.

For the patches to work, I suggest EDIT "objects", to exist alongside the normal objects in a mod's raws.
EDIT objects do not describe a new object, instead they describe a patch to apply on an already existing object.
EDIT objects should have the powers that the current creature variations (though they shouldn't replace them) have, they should be able to add, remove, and convert tokens. This can be done with tokens like ADD_TAG, REMOVE_TAG, or CONVERT_TAG (the raws themselves call tokens "tag"), but it should be noted that ADD_TAG can as well be left out. If it is, then EDIT objects adding tokens will look mostly like the objects they edit, making it easy to copy-and-paste between them during the modding process.Not having this is currently an issue with the creature  variations and CV_ADD_TAG.
An EDIT object might look something like this:

Code: [Select]

[EDIT:ENTITY:MOUNTAIN]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]It adds two PERMITTED_REACTION tokens to the end of ENTITY:MOUNTAIN, presumably allowing the dwarves to create mithril (the reactions and materials for this still have to be defined elsewhere).

In addition, EDIT objects should be able to remove objects altogether. This is probably best done by adding a token like REMOVE_OBJECT, which slates an object for removal, but doesn't remove it there and then in the installation process. This way mods can easily add back objects removed by earlier mods.
For example: Mod A removes an object (CREATURE:TOAD) using the following EDIT object:

Code: [Select]

[EDIT:CREATURE:TOAD]
[REMOVE_OBJECT]which adds the REMOVE_OBJECT token to the end of CREATURE:TOAD. Then all Mod B has to do to add it back is using:

Code: [Select]

[EDIT:CREATURE:TOAD]
[REMOVE_TAG:REMOVE_OBJECT]The alternative would be for B to include a full definition of CREATURE:TOAD, to add it back. This latter solution has the weakness of B having to keep up with the original definition of CREATURE:TOAD, should it be changed. Any edits of CREATURE:TOAD before A (say by Mod C) would also be ignored by this new definition.

However, creature variations (and COPY_TAGS_FROM) complicate EDIT objects. I will assume a familiarity with the former here, otherwise I refer to their Wiki article. Say you want to be kind to slug men, and allow them to use APPLY_CREATURE_VARIATION:ANIMAL_PERSON instead of APPLY_CREATURE_VARIATION:ANIMAL_PERSON_LEGLESS. This sounds reasonable, how would we do it with an EDIT object?

Code: [Select]

[EDIT:CREATURE:SLUG_MAN]
[CONVERT_TAG]
[CT_MASTER:APPLY_CREATURE_VARIATION]
[CT_TARGET:ANIMAL_PERSON_LEGLESS]
[CT_REPLACEMENT:ANIMAL_PERSON]Now, we are also upset that slug men get the SAVAGE token from CREATURE_VARIATION:ANIMAL_PERSON; we want our legged slug men to appear in all biomes. We only want that for slug men though, not other animal people. How would that look?

Code: [Select]

[EDIT:CREATURE:SLUG_MAN]
[REMOVE_TAG:SAVAGE]Finally, we want to re-add the PREFSTRING which was removed by CREATURE_VARIATION:ANIMAL_PERSON.

Code: [Select]

[EDIT:CREATURE:SLUG_MAN]
[PREFSTRING:slimy trails]Of course, these three can be written together, like

Code: [Select]

[EDIT:CREATURE:SLUG_MAN]
[CONVERT_TAG]
[CT_MASTER:APPLY_CREATURE_VARIATION]
[CT_TARGET:ANIMAL_PERSON_LEGLESS]
[CT_REPLACEMENT:ANIMAL_PERSON]
[REMOVE_TAG:SAVAGE]
[PREFSTRING:slimy trails]These are reasonable, and can be achieved using manual edits, but here they won't work together; the first example can only be done if EDIT objects are applied before APPLY_CREATURE_VARIATION (because creature variations are irreversible),
while the latter two require that they are applied after it (SAVAGE must be removed after it is added, and the creature variation removes any instances of PREFSTRING when applied). I said this can be achieved using manual edits, but how?
The former is very simple, replace [APPLY_CREATURE_VARIATION:ANIMAL_PERSON_LEGLESS] with [APPLY_CREATURE_VARIATION:ANIMAL_PERSON].
To do the latter you use creature variations, you add the following to the slug man raws after (below) [APPLY_CREATURE_VARIATION:ANIMAL_PERSON]:

Code: [Select]

[CV_REMOVE_TAG:SAVAGE]
[APPLY_CURRENT_CREATURE_VARIATION]
[PREFSTRING:slimy trails]Note that we would also need to apply an EDIT before if it removed or added any APPLY_CREATURE_VARIATION. What this means is essentially that we need to add/remove/convert any instances of APPLY_CREATURE_VARIATION in a step before they are applied, and add/remove/convert normal tokens in a step after.

EDIT objects also need to be able to edit creature variations, e.g. in case you want to change all animal people, or giant animals, or [insert custom modded creature variation here]. This is of course also possible to do currently, manually.

With these use cases explained, I think it is possible to take multiple paths. The one followed in the thread leading up to writing this suggestion (and when writing the mockup) was treating the add/remove/convert tokens within EDIT objects like they were creature variation tokens, and applying them as such. I am unsure if that is the best path, it lead to certain complications and was conceptually confusing, so I won't go into details here. That said, it did lead to many insights about creature variations, that have been baked into the
"goals/suggestions of the third kind" below. The best path probably depends on the structure of the code as well, something we can't know.
Toady, if you choose to go this far with the mod structure rework, I believe any solution will be a good one  


Goals/suggestions of the third kind:

• Generalize creature variations to be useable by all raw objects. They are general enough that it feels strange they only exist for creatures. As an asset, that kind of very general templates is really useful, though it has been underutilized in the modding community. A proposed name is "object templates", skipping "variation" as that presumably had to do with the animal people and giant animals being variations on the base animal, and with that origin being so removed from the use it makes little sense to keep it.
  
  A wish from those making a DF markup language (Mr_Crabman being one of them) is that if many kinds of objects (entities, creatures, tools, etc.) use the same kind of template, they should be distinguished somehow to clarify the context for the parser.
• Make creature variations/object templates nestable. This is an old goal from when creature variations were first revealed. It is also a good idea, and could be done "while you're at it", though that is probably a dangerous mindset.
• Merge body detail plans with creature variations/object templates. Likewise, an old goal.
• Condense the syntax of X_CONVERT_TAG tokens. Why have

Code: [Select]

[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
[CVCT_REPLACEMENT:HUMANOID]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED_NECK]
[CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CV_CONVERT_TAG]
[CVCT_MASTER:BODY]
[CVCT_TARGET:QUADRUPED_HOOF]
[CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]when it can be condensed into the following?

Code: [Select]

[CV_CONVERT_TAG:BODY]
[CVCT_TARGET:QUADRUPED]
    [CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_FRONT_GRASP]
    [CVCT_REPLACEMENT:HUMANOID]
[CVCT_TARGET:QUADRUPED_NECK]
    [CVCT_REPLACEMENT:HUMANOID_NECK:3FINGERS]
[CVCT_TARGET:QUADRUPED_HOOF]
[CVCT_REPLACEMENT:HUMANOID_HOOF:3FINGERS]This applies to whatever conversion tokens are used in the EDIT objects as well.

• Split CVCT_TARGET (and corresponding EDIT token) into CVCT_TARGET and CVCT_TARGET_STRING. The way CVCT_TARGET currently works by looking for strings is quite uncumbersome, as seen with the body conversions which have to be in a certain order so [CVCT_TARGET:QUADRUPED] won't affect [BODY:QUADRUPED_HOOF]. The suggestion is to let CVCT_TARGET look for whole token arguments, as separated by colons, and give the string-searching functionality (which is still needed for e.g. giant bushmasters) to CVCT_TARGET_STRING.
• Give all objects "classes" like the CREATURE_CLASS and SYNDROME_CLASS that already exist. Primarily for the mass-editing purposes described below, but also because these kinds of arbitrary tokens are good to build off of and use in the future. They give freedom, and modders love that.
• Allow EDIT objects to select not only one, but multiple objects at once, by following some criteria. Suggested criteria are specific tokens, object classes (see above) and of course object IDs. This is both so you can shorten some expressions, instead of having

Code: [Select]

[EDIT:ENTITY:MOUNTAIN]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]
[EDIT:ENTITY:MOUNTAIN_DARK]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]
[EDIT:ENTITY:MOUNTAIN_DEEP]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]to add the mithril reactions to the vanilla dwarven entity and their two modded cousins (made up on the spot as examples) it could be shortened to

Code: [Select]

[EDIT:ENTITY:MOUNTAIN]
 [PLUS_SELECT:ENTITY:MOUNTAIN_DARK]
 [PLUS_SELECT:ENTITY:MOUNTAIN_DEEP]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]or even to

Code: [Select]

[EDIT:ENTITY:SEL_BY_CLASS:DWARVEN]
[PERMITTED_REACTION:MITHRIL_MAKING]
[PERMITTED_REACTION:MITHRIL_MAKING2]if we imagine all these dwarven civ to have a shared "DWARVEN" class. This class-based selection may also select other "DWARVEN" civs, even ones the person creating the mithril mod had no knowledge of (for instance because they were made after it). Another example would be

Code: [Select]

[EDIT:ENTITY:ALL]
[ALL_MAIN_POPS_CONTROLLABLE]

to make all entities available for adventurers to come from. This kind of mass-editing is thus a boon for the connectivity between mods, as long as the community can come up with some standard for classes. Though as I have already seen propositions for the limited creature classes, and other modding communities like the Minecraft one has been able, I have no doubts we would.

The examples given here are all inclusive (like a logic "or"), but it should also be able to be restrictive (like a logic "and"), e.g. only select the plants that are both giant mushrooms and found underground. Or a combination thereof.

This is also where the "technical function" of mod bundles lie. Take the mithril mod we've used as an example. It consists in part of the mithril raws themselves (which we haven't seen) and the reactions, and in part of the EDIT object giving the mithril reactions to all dwarven entities. We also assume it contains those dark dwarves and deep dwarves implied by the object names. Now imagine another mod, let's call it the orichalcum mod. Much like the mithril mod, the orichalcum mod adds a new material, and a few dwarven civs (forge dwarves and moss dwarves), and it too intends for the new metal to be usable for all dwarven civs, by using something like the SEL_BY_CLASS construction seen above.

How do you order these mods?
If you put the mithril mod before the orichalcum mod, then the dark and deep dwarves will get both mithril and orichalcum, but the forge and moss dwarves will only get the latter. If you reverse the situation and put the orichalcum mod the reverse will be true, with dark and deep dwarves only getting mithril but no orichalcum.
The solution is to split the mods into two parts each, one for the definitions and one for the EDIT-based assignments. The assignment parts. As you don't want separate downloads/subscriptions for these parts, and want them grouped together in the left list for readability, mod bundles are perfect for this.

(As a final note (for now, and obviously I want people to reply here, I just didn't know where to put this), Steam Workshop mods may be auto-updated, so if that feature is used then presumably the mods in the "all mods" folder are periodically overwritten with newer versions. This could be an issue if mods aren't stored somehow in conjunction with each save file, as changing the raws of existing saves doesn't always work well.)

[Mr Crabman]

Good stuff. Here are my thoughts on some of these issues, and further suggestions:

Goals/Suggestions of the Third Kind

A couple of things Voilol forgot to mention (or perhaps chose not to), or which he did mention but I wish to re-emphasize briefly:

1. Very important, the syntax changes to creature variations and body detail plans (especially CV_CONVERT) could still be compatible with current saves with the use of a couple of regexes; there is no need to shy away from making these changes out of fear of breaking players existing saves with the Steam release, because it's certainly possible to easily convert the saves (though maybe the changes to graphics will break saves anyway?). I really recommend having a look at the mockup linked in the top post, which does such a conversion.

2. Scrap [APPLY_CURRENT_CREATURE_VARIATION] (note the CURRENT), and simply apply REMOVE_TAG and CONVERT_TAG and APPLY_CREATURE_VARIATION line-by-line, in order. This would normally break CV_CONVERT_TAG because of how the string replacement works (this is why conversions are applied in reverse!), but with Voilol's suggestion of splitting CVCT_TARGET into CVCT_TARGET and CVCT_TARGET_STRING, this wouldn't be a problem anymore.

3. I very strongly recommend abandoning CV_ADD_TAG in all situations, in favor of just using the token itself to be added directly. It makes copy-pasting rather awkward. Voilol did mention this, but I just wanted to re-emphasize that. Considering it wasn't done this way from the start this may be infeasible, but hopefully it's still doable. CV_ADD_CTAG is slightly different and has a use, so that could be kept (though I posted another suggestion about this a couple of posts down which could make copy-pasting easy even there).

4. There should be a way to target arguments (for CONVERT_TAG) based on their index, not their value. For example, say you want to change [BODY_DETAIL_PLAN:VERTEBRATE_TISSUE_LAYERS:SKIN:FAT:MUSCLE:BONE:CARTILAGE] in a dwarf, to use SCALE instead of skin? The way to do that right now would be to use:

Code: [Select]

[CONVERT_TAG]
   [CT_MASTER:BODY_DETAIL_PLAN:VERTEBRATE_TISSUE_LAYERS]
   [CT_TARGET:SKIN]
   [CT_REPLACEMENT:SCALE]
But the problem is, what if another mod already replaced it with CHITIN? Now this won't work. You could just use REMOVE_TAG and then re-add the plan with your desired arguments, but if you want to leave other modders to have freedom to edit the other arguments, this won't work. So what I suggest instead, is to be able to target the argument number, not the value, so something like this:

Code: [Select]

[CONVERT_TAG]
   [CT_MASTER:BODY_DETAIL_PLAN:VERTEBRATE_TISSUE_LAYERS]
   [CT_TARGET_ARGNUM:2]
   [CT_REPLACEMENT:SCALE]
Now it doesn't matter what is there; SKIN, CHITIN, BRONZE, anything will be made into scales. Note that here I'm using the current syntax for CONVERT_TAG, not the improved one that Voilol mentions above (which merges CONVERT_TAG and CT_MASTER, and lets multiple CT_TARGET and CT_REPLACE tokens share the same CONVERT_TAG block).

Also note that I said ARGNUM:2, because VERTEBRATE_TISSUE_LAYERS is technically argument 1 of BODY_DETAIL_PLAN.

Note that you could also do something like:

Code: [Select]

[CT_TARGET_ARGNUM:2:5]
   [CT_REPLACEMENT:SCALE:IRON]
To replace both the SKIN and BONE materials, regardless of their actual value; you could do as many ARGNUM's as there are arguments in that token; if that specific argument doesn't exist for that token but could (BODY could have dozens of arguments, or only 1), then using a number higher than what exists... I'm not sure if it should just add it (and should it automatically create a bunch of "empty" arguments in between or not?), or just ignore it; maybe better/more experienced modders than me could say what they think would be most useful/safe?

5. The likes of REMOVE_TAG and CONVERT_TAG have an unfortunate weakness; some tokens are identical, but used multiple times in different places, and it's impossible to target one without the other; for example:

Code: [Select]

[ATTACK:BITE:BODYPART:BY_CATEGORY:BEAK]
   [ATTACK_SKILL:BITE]
   [ATTACK_VERB:bite:bites]
   [ATTACK_CONTACT_PERC:100]
   [ATTACK_PENETRATION_PERC:100]
   [ATTACK_FLAG_EDGE]
   [ATTACK_PREPARE_AND_RECOVER:3:3]
   [ATTACK_PRIORITY:MAIN]
   [ATTACK_FLAG_CANLATCH]
[ATTACK:SCRATCH:CHILD_TISSUE_LAYER_GROUP:BY_TYPE:STANCE:BY_CATEGORY:ALL:TALON]
   [ATTACK_SKILL:STANCE_STRIKE]
   [ATTACK_VERB:snatch at:snatches at]
   [ATTACK_CONTACT_PERC:100]
   [ATTACK_PENETRATION_PERC:100]
   [ATTACK_FLAG_EDGE]
   [ATTACK_PREPARE_AND_RECOVER:3:3]
   [ATTACK_PRIORITY:MAIN]
   [ATTACK_FLAG_WITH]
   [ATTACK_FLAG_BAD_MULTIATTACK]
How are you to do something like change the ATTACK_PREPARE_AND_RECOVER of the talon attack, without also changing the bite with the beak? Or perhaps you want to remove the scratch attack entirely, deleting all its tokens? You can't, this is impossible, because most of the tokens are shared with the bite attack, and you will end up affecting it while editing.

Same goes for the likes of interactions, or really any object with "nesting" tokens (where 1 token, like ATTACK, has other tokens that are "nested" under it), as they are liable to be repeated exactly; one annoying case I myself have encountered while modding is APP_MOD_NOUN being shared by both TISSUE_LAYER_APPEARANCE_MODIFIER and BP_APPEARANCE_MODIFIER, making it impossible to blank out the tissue appearance stuff without breaking body appearance stuff too.

There should be a way to deal with this; one option could be to specify the index of the token; like in the talon attack above, you could specify that you want to target the second instance of ATTACK_PREPARE_AND_RECOVER in the file; this has the problem of being a bit clunky and not easily copyable between objects, since different things are done in different orders.

Another way would be to specify that you want to target the next instance of ATTACK_PREPARE_AND_RECOVER that occurs after ATTACK:SCRATCH, the weakness there though is that if someone else removes that token first in their mod, then that would lead the token to jump ahead to a third attack and remove the token from that.

If there was some way the raw parsing engine could be aware of which tokens were nested under which others, that could be ideal, because then you may be able to target the talons ATTACK_PREPARE_AND_RECOVER on the basis of it being nested under the "talon" attack directly.

Steam workshop automatically updating files

The first part is simple; as Voilol says, store a copy of the mods in each save, with only the players active mods for that save there; this makes sense, because players will want different saves with different mods anyway.

The second idea would be to make is so that downloaded Steam workshop files should not go directly in the mods folder, but instead in a separate folder just for storing downloaded Steam workshop files (on Linux this would be `/home/user/.local/share/Steam/steamapps/workshop/content/gameidhere/`, but I'm not sure what the path would be on other OS's).

The first time installing a mod it would copy it into the mods folder, but after that, the mod manager UI could have a button detecting whether the "workshop files" have been changed, and therefore prompt the player to click an "update/refresh mods" button (the "refresh" is in case a raw-savy player decided to edit a downloaded mod themselves), which would update the "all mods" folder. Ideally each mod could have an individual button for this in case you only wanted to update specific mods.

Of course, having an toggle to auto-update "all mods" whenever Steam does an update so you don't have to think about it would be good too, and with saves storing copies of mods, there would be no worries about mod updates breaking things.

Individual saves internal mod folders may also perhaps have separate "update/refresh mods" buttons which could pull in changes from the "all mods" version of the mod (this could be useful even for non-Steam versions), but I'm not sure how feasible updating raws on an active save is, or how feasible it's intended to be.

For example, I know ENTITY changes basically never work and always require a new save, and AFAIK new creatures will not be added to the world, but changes to current creatures often work. For sure there would have to at least be a pop-up warning that "updating mods on a current save may not apply certain changes, and could possibly break things, and you should make a backup with this helpful button!", if this was to be done at all.

[TheLifeOfRyanB]

I've been thinking about mods in terms of source control, e.g. git. Essentially the same issues apply as for applying two or more patches (changes) from separate branches (mods) back to the develop/master branch (vanilla), each of which may overlap in several areas of code (raws). Even git is only so good, too much overlap and it gives up and returns the merge conflict back to the user to fix. This is a problem for which a complete solution didn't exist yet.

So yeah, loading mods in order isn't quite sufficient, although maybe having some kind of priority for resolving conflicts would be helpful, and having several stages for applying parts of mods would be helpful in standardising what changes can be applied, e.g. add new objects, delete existing objects, apply blanket changes to sets of objects...

[Mr Crabman]

Something else I just thought of; if you do go for making creature variations/object templates nestable, the "inner templates" should be able to be applied only conditionally, much like REMOVE_CTAG, ADD_CTAG (even if ADD_TAG is removed or made optional, ADD_CTAG should be kept), and CONVERT_CTAG, and there should be a way to pass in the chosen arguments. Imagine something like this for example (syntax is just to illustrate my point):

Code: [Select]

[OBJECT_TEMPLATE:UPPER_TEMPLATE]
    [USE_OBJECT_CTEMPLATE:1:CONDITIONAL_ARG:VERTEBRATE_TISSUE_LAYERS:!ARG2:!ARG3:!ARG4:!ARG5]

This would mean UPPER_TEMPLATE could be used and you may trigger the addition of vertebrate tissues only if the first argument passed in is CONDITIONAL_ARG.


This one is also a "stretch goal", but please consider this syntax improvement for both EDIT objects and creature variations/object templates:

Code: [Select]

[BULK_REMOVE_TAG] // naturally there is a CTAG version of this
    [NAME]
    [GENERAL_CHILD_NAME]
    [GENERAL_BABY_NAME]
    [CASTE_NAME]
    [CHILDNAME]
    [BABYNAME]
    [POPULATION_NUMBER]
    [CLUSTER_NUMBER]
    [BODY:HUMANOID_SIMPLE] // only removed if it has this leading argument; you can add extra arguments as with the normal REMOVE_TAG
[END_BULK_REMOVE]

The old version ([REMOVE_TAG:NAME]) could be useful as well for when removing only 1-2 tokens, but this would be useful when removing any more than that, because it:

- Reduces the amount of typing (each REMOVE_TAG is 10 characters extra, and it only gets worse with REMOVE_CTAG)
- Makes it more readable (no more wall of REMOVE_TAG!)
- Allows for more condensed text; here I put each one on a separate line, but you could easily put multiple per line and still have them all be readable on a narrow coding window, which you couldn't do now because of the extra length added by each REMOVE_TAG.

Of course, while ADD_TAG itself is hopefully going to be abandoned for just writing the token directly (for ease of copy-pasting between templates and objects), ADD_CTAG could use the same sort of thing, for when you want to add a block of tokens only under a specific condition (without the need for massively repeating that conditional check over and over).


EDIT: One other thing that would be useful is being able to, creature variations/object templates, conditionally do something only if there is no argument given; for example:

Code: [Select]

[CV_ADD_CTAG:5::MUNDANE]
That code there doesn't add the token under any circumstances, but it would be ideal if it would instead add it most of the time, but not add it if argument 5 has been given any value. In other words, [APPLY_CREATURE_VARIATION:TEST:A:B:C:D] would allow MUNDANE to be added, but [APPLY_CREATURE_VARIATION:whatever:A:B:C:D:E] would not.

I'm not sure how the syntax would work, but it could also be useful to execute a token change only when there a specific argument has not been given; this is more specific than above; you would do something like this (but with probably better syntax/better phrasing):

Code: [Select]

[CV_ADD_NOTAG:1:FANTASY:MUNDANE]
And it would add MUNDANE all the time, unless the first argument given is FANTASY.

[Mr Crabman]

A couple more suggestions.

There should be a way to make specific changes (in a creature variation/object template) conditionally, but not based on an argument passed in (like we already have), but a token the creature has.

This sort of almost exists with CONVERT_TAG, but that only allows you to change an argument of a token (or add an extra argument), or remove an argument, or as I just tested, remove/convert the token itself (was easy to test; added a block to convert FLIER to AQUATIC and saw whether crow men would drown on land).

But you definitely can't add extra tokens; you can't for example, make any creature with FLIER also able to breathe under water, you could only replace it; you could theoretically just add the AMPHIBIOUS token, but then that would be applied to all creatures this variation/template is used on, not just the ones who had FLIER.

Also, even for the conversions, I'm pretty sure this is only actually safe to use on tokens with no arguments (like FLIER an AQUATIC, or PET and PET_EXOTIC).


EDIT: Speaking of conversions, a generalized way to modify integer values when CONVERT_TAG-ing would be handy; right now a select few tags have specific ways to CHANGE_BODY_SIZE_PERC, or CHANGE_FREQUENCY_PERC, and GRAVITATE_BODY_SIZE and such, but a more generalized way to change an arbitrary integer value in an argument (selected with TARGET_ARGNUM of course) by a certain percentage, or to add or subtract specific amounts, could be rather useful.


One other useful thing I believe would be being able to sort raw files into folders without turning them into mod bundles, purely for organizational purposes by the modder (in cases where there are lots of files in one mod, sorting them into folders could be helpful).

[Mr Crabman]

In FotF:

8. We use identifiers for dependencies.  I'm not sure how to avoid trouble if people use the same identifier, since we don't ourselves control a central hub with links or anything like that - for instance, I don't know how much Workshop helps with that, or enables chains of downloads etc., if the modder provides some kind of workshop id/page, but our own system of dependency tracking has to work independently from Workshop.  This makes it less friendly, since we don't have our own mod server.  It shouldn't be too long before I finally get a chance to look at Workshop integration and we'll see what more there is to say, and if our own internal system changes to something that can capture both that and elsewise.  Integration may mean mods have to have two identifiers for all I know, which'll be fun.

My advice here is that for "our own system of dependency tracking" that needs to be independent from Workshop, do something like this (I don't know what the format of the info.txt files is, and it's probably not this mockery of json, but it gets the point across):

Code: [Select]

dependency {
  mod_name: Example Mod
  version: {<=1.4.2, >=2.0}
  author: Fake Toady One
  workshop: https://steamcommunity.com/sharedfiles/filedetails/?id=85000600250
  dffd: https://dffd.bay12games.com/file.php?id=95781
  source: https://github.com/FakeToadyOneGithubAccount/example-mod/
  details: You need to enable "Extra Amphibians".
}

Some of this information (mostly the author name and source code link) may be technically unnecessary to specify, but it's useful. Obviously not all of it would be visible at all times, it would only become visible when hovering over (or maybe even selecting) the dependency.

The links would hopefully allow for a clickable link in the mod menu somewhere to take you to the download page for the desired dependencies automatically, and they would be optional (no sense forcing everyone to have a dffd version of the mod for example, or to specify a "source code" page).

The 'workshop' and 'dffd' things should definitely enforce using actual links to the workshop and dffd somehow, to prevent confusion or potentially sending people to dodgy places. I don't know if it would be perhaps better to instead have the workshop and dffd id's instead of just direct links, and then just generate a correct link (for the end user to click on) using the id.

If there is some way using the workshop API to check if a mod is currently installed/subscribed to (so the mod manager gives a nice friendly green checkbox next to the right dependencies), you could possibly also have an automatic version check (using the '<=' and '>=' thingies and the mods version numbers, to see if you're using the correct version).

The details thing is mostly for cases where perhaps mods have configurable options (if/when Tarn adds those), and just having the dependency installed isn't good enough, you need to turn on a specific setting (maybe this mod adds giant salamanders, but to do that you need to install "Example Mod" as a dependency and enable the optional "Extra Amphibians").

[voliol]

Steam workshop has support for dependencies (example, see the box to the right). It shows a green check too if you have them installed. I don’t know if it’s possible to link them up to mod info files so they could be auto-generated, or if having them means twice the configuration for the uploader. There is also afaik no way (in workshop) of showing anti-dependencies, mods that don’t work together, other than putting it in the description.

As for concerns that mods would use the same identifier, I don’t think that is a too likely occurence. Modders want unique names for their mods, and mods should still be mostly concentrated on DFFD/the mod release forum and Steam workshop, so they’ll be able to search and see if a name is ”taken” before. Community members can also work it out by asking ”hey, our mods have similar names, do you want to talk about it?” or ”I found two versions of this, which should I use for example mod?” when confusion arises.

Author name is important! Not only because most authors like crediting themselves, but it’s also something for users to look up similar-quality/style mods with, and because a mod can have different authors and uploaders if they are modpacks, community mods, continuations (like OldGenesis), or simply because the author doesn’t have a Steam/forum account but is okay with someone else mirroring them there.

[Mr Crabman]

There is also afaik no way (in workshop) of showing anti-dependencies, mods that don’t work together, other than putting it in the description.

You're right, there's no anti-dependencies on the workshop, but you could do something like that for the info.txt anyway (kind of like listing dependencies, with at least some of the same attached data).

As for concerns that mods would use the same identifier, I don’t think that is a too likely occurence. Modders want unique names for their mods, and mods should still be mostly concentrated on DFFD/the mod release forum and Steam workshop, so they’ll be able to search and see if a name is ”taken” before. Community members can also work it out by asking ”hey, our mods have similar names, do you want to talk about it?” or ”I found two versions of this, which should I use for example mod?” when confusion arises.

Thanks for mentioning the mod forum, that would be another place that would be good to have ingame links to (perhaps instead of or as well as the DFFD links).

I suppose it's not going to be super common to have conflicts, and just the author name itself would be enough to ensure never getting conflicts basically, but while Workshop has its own dependency system, it would be good to have one for the game itself in case of non-workshop mods, and for this situation, linking to the correct mod pages would be big usability features.

Author name is important! Not only because most authors like crediting themselves, but it’s also something for users to look up similar-quality/style mods with, and because a mod can have different authors and uploaders if they are modpacks, community mods, continuations (like OldGenesis), or simply because the author doesn’t have a Steam/forum account but is okay with someone else mirroring them there.

You make a convincing case, you're absolutely right.

[voliol]

Here's a reflection post-the-release of 0.50.01, which implemented a new mod structure, which the above was meant to guide. The new system is largely a vast improvement, and I really do like it. I hope I don't come off as a downer, just because pointing out cons is easier. 
That said, in focusing on "goals of the first kind" (availability and ease of use), it has some weaknesses in terms of "the second kind of goals", meant to ensure the functionality of the old system of manually editing raws (and of "the third kind", since these were stretch goals). It's worth noting the old system can still be emulated as raws are loaded somewhere, but as this is both trickier and messier, the new system should have all functionality needed to make people not want to do that.

Here are (imo) the most important points:

• SELECT_ and CUT_ does not work on all objects. Not only does this mean a lack of functionality, but it is annoying to remember/have to look up which ones work and not.
• Creating a new object with an already used ID does not overwrite the one already using it, meaning duplication errors still exist. I have neither heard or know a reason why this can't be done, now that the CUT_ functionality does exist. I believe that in conjunction with the above, it also means e.g. graphics objects can't be replaced without replacing the whole mod said graphics object comes in.
• Removing and changing tokens is possible by adding APPLY_CREATURE_VARIATION or APPLY_CURRENT_CREATURE_VARIATION to a SELECT_CREATURE. This is great, but as the name implies it only works on creatures. Above, the suggestion of generalizing creature variations was listed as a suggestion "of the third kind", but that assumed there would be some other method of removing/converting tokens.
  Since nothing else has seemingly changed here, the other suggestions for "variations"/"templates" still apply, but they are second to having the functionality for all objects.

[Salmeuk]

reading this thread felt like reviewing a friend's thesis. . a lot of work went into the structure. thanks.

 DF's future is in it's modding community. vanilla will never hold a candle to the creative output of 10,000 obsessive nerds. so here's agreement that the application of mods should become a standardized and fully functional process, without spending hours getting up to speed on the minute definitions of a hundred different tokens

[voliol]

Haha, thanks ^^. This is definitely the tidier part of the work, compared to the initial discussion between Mr Crabman and I, and the demo.

With Dwarf Fortress being as complicated a game as it is, token knowledge is something I don't think can be avoided, so the best we can do is shape a system around them making them easy to use. And this thread is an attempt in taking part in that.