Mr_Crabman, I replied to some of your previous comment in the comment above, but the other point were complicated enough that I felt I needed better tools to get my head around them, and it lead to this. Those replies are at the bottom of this comment, using the model described. I'm sorry for dropping this bomb on you, but at the same time a more complete description like this can be used to base an eventual suggestions-board post on, so I hope you bear with it...
Below is a model for a mod loader for Dwarf Fortress, either one integrated into the game, or a third-party tool. The core parts are the same for either, so in most cases the distinction won't be made. I will assume a familiarity with DF modding.
The player is presented with a list of select-able mods. This list is extracted from a mods folder, the mods presumably coming from Steam Workshop or DFFD. Unselected mods may also be grouped together as "mod packs", or just "mods with multiple components". This is so the player can easily recognize which mods are related, if they have downloaded many of them. Players may select mods, moving them into a separate list, and they may also freely change the order of mods in this "selected" list. The order is guided by the description of each mod, which may include dependencies, other mods that should be higher up in the list.
The mods are loaded top-down. If a list of selected mod looks like this:
- Mod A
- Mod C
- Mod B
then they are loaded as
1. Mod A
2. Mod C
3. Mod B
Within each mod, files are parsed in order according to their first line. This is how it currently works in vanilla, see
https://dwarffortresswiki.org/index.php/DF2014:Raw_file#Parsing. The only difference is that the new "object variations" have priority, so files with "o_variation" are parsed first in each mod. This has less importance than in current DF, but it is still worth mentioning.
If the mod list (expanded so we can see the files in each mod) looks like this:
- Mod A
- creature_a
- entity_a
- Mod B
- creature_b
- inorganic_b
- o_variation_b
- Mod C
- entity_c
Then the parsing order is:
- Mod A
1. creature_a
2. entity_a
- Mod B
3. o_variation_b
4. inorganic_b
5. creature_b
- Mod C
6. entity_c
Each file contains one or multiple objects, and each object contains tokens. The order of objects withing files, and tokens withing objects, is the natural order within a text file.
An object/token written above is parsed before one written below. The parsing/loading of mods is done in three steps.
- First, going by parsing order, each object is read.
If the object is an object variation, it is just saved to memory. That is also the case if the object is anything but an "EDIT" object, so creatures, materials, entities etc. are just saved to memory for now.
If the object is an EDIT object, then it is not saved to memory. Instead, the EDIT object selects one or more of the already encountered objects, and makes changes to them. Here, it is relevant to divide the tokens into 4 categories: "EDIT-level" tokens, "special" tokens, "object variation" tokens ("ov" tokens for short), and "normal" tokens.
EDIT-level tokens do something in this first step, and are only found in EDIT objects. PLUS_SELECT and UNSELECT allow you to further specify which objects should be edited. The rest of the EDIT-level tokens (ADD_SPEC_TOKEN, REMOVE_SPEC_TOKEN, CONVERT_SPEC_TOKEN, CST_MASTER, CST_TARGET, and CST_REPLACEMENT) add/remove/convert special tokensand ov tokens from/to/in the selected object(s).
Special tokens are the three GO_TO tokens GO_TO_END, GO_TO_START, GO_TO_TAG, as well as COPY_FROM_OBJECT, REMOVE_OBJECT (which is a new token), and APPLY_OBJECT_VARIATION and APPLY_CURRENT_OBJECT_VARIATION.
When these are in EDIT objects, they are copied to the end of the selected object(s).
Object variation tokens all start with "OV"; OV_ADD_TAG (and the synonymous OV_NEW_TAG), OV_REMOVE_TAG, OV_CONVERT_TAG, OVCT_MASTER, OVCT_TARGET, OVCT_REPLACEMENT, as well as the conditional "_CTAG" counterparts to the first three listed.
When these are in EDIT objects, they are copied to the end of the selected object(s).
Normal tokens are the rest, like NAME and BRAG_ON_KILL, but also ones that may have syntax meaning simply not relevant to the mod loading process, like SELECT_CASTE and SYNDROME.
When these are in EDIT objects, they are not simply copied over, instead they are added using an OV_ADD_TAG with said token as the argument. This is because actually copying over normal tokens complicates things, so OV_ADD_TAGs are disguised so you don't have to type them out. The "disguise" also makes it practical to move tokens from normal objects to EDIT objects.
When all tokens in the EDIT object have been read, if it added any ov tokens, APPLY_CURRENT_OBJECT_VARIATION is added to the end of the selected object(s).
Also, if an object has the same object ID as one that has already been saved to memory, then the later object overwrites the former.
- In step two, all the objects saved to memory are read once more, except the object variations. For each in-memory object, an "output object" is prepared. Then, each token of the object is read, in order. If the token is a special token, then it executes a special command.
The three GO_TO tokens change the insertion index for new tokens in the output object, moving it to the start, the end, or a specified token. The GO_TO tokens only affect the output object, not the in-memory object currently being parsed.
COPY_FROM_OBJECT copies tokens from another output object. If that output object has not been created yet, the corresponding in-memory object is given priority, and parsed before copying over the tokens. A loop of COPY_FROM_OBJECT causes an error (message).
REMOVE_OBJECT prevents the eventual writing down of the output object. This is so COPY_FROM_OBJECT can still copy from an object slated for removal. That technicality aside, what REMOVE_OBJECT does is removing the current object.
APPLY_OBJECT_VARIATION applies an in-memory object variation. Again, these work like current DF creature variations, but generalized to work not only with creatures. See https://dwarffortresswiki.org/index.php/DF2014:Creature_variation_token
APPLY_OBJECT_VARIATION also takes arbitrary arguments which can be used with the "_CTAG" tokens.
APPLY_CURRENT_OBJECT_VARIATION applies the ov tokens that have directly preceded it in the current object.
If the token is an ov token, it is saved up (with others like it) and applied at the next APPLY_CURRENT_OBJECT_VARIATION.
If the token is a normal token, it is inserted into the output object's list of tokens. Initially, this means they are added to the end, but the GO_TO tokens can change this.
- Step three is taking all the output objects, except the ones slated for removal with REMOVE_OBJECT, and writing them into "compiled" text files (as compared to the original list of mods, they are not compiled per-say. If we imagine the mod loader to be integrated into Dwarf Fortress, these compiled files are likely what a save uses, so it won't have to parse the more involved list of mods each time it is opened. If we imagine the mod loader to be third-party, these compiled files are what are dragged into the game's raws folder, as one would install a total conversion mod in current Dwarf Fortress.
Some further thoughts:
Body detail plans could be made into OBJECT_VARIATIONs. As far as I can tell, they already work the same as diminished creature variations only able to add tokens. Body details plan are older than the latter, I believe, but the overlap still seems strange.
All tokens related to creature variations could be kept as synonyms for their corresponding object variation-related token, for the sake of backwards compability. At the same time, this could be problematic if targeted by REMOVE_SPEC_TOKEN/CONVERT_SPEC_TOKEN, if a search for e.g. OV_ADD_TAG doesn't find instances of CV_ADD_TAG. There is a similar problem with OV_ADD_TAG and OV_NEW_TAG, and to some extent the "_CTAG" tokens. Perhaps some repetition will be needed in the EDIT objects to cover all cases.
As APPLY_CURRENT_OBJECT_VARIATION is only added automatically at the end of the EDIT object, this may lead to some misleading EDIT objects. E.g.
[EDIT:CREATURE:BY_ID:TOAD]
[GO_TO_START]
[ADOPTS_OWNER]
[GO_TO_END]
gives
[CREATURE:TOAD]
...
[GO_TO_START]
[OV_ADD_TAG:ADOPTS_OWNER]
[GO_TO_END]
[APPLY_CURRENT_OBJECT_VARIATION]
which adds ADOPTS_OWNER at the end of TOAD, not at the start. Maybe APPLY_CURRENT_OBJECT_VARIATION should be added not only at the end of the block made using EDIT, but
before any GO_TO tokens it adds.
I'm thinking of ways that [IF]-blocks or other conditionals could be added. I think the way might be something similar to the current creature variation arguments, where all occurrences of a text string are replaced by a conditional value, at the very start of loading a mod. Lets call these "mod arguments". Then, the definitions for the mod arguments is found in a text found of their own. Something like:
...
[CONDITIONAL:CAN_DWARVES_TAME_DRAGONS]
[DESCRIPTION:Allow dwarves to tame dragons?]
[COND_TYPE:BOOLEAN]
[CURRENT_VALUE:TRUE]
...
or maybe
...
Allow dwarves to tame dragons?
[CAN_DWARVES_TAME_DRAGONS:BOOLEAN:TRUE]
...
Either way, I hope that some of the GUI work here can be repurposed from reading init or advanced world generation settings files,
in an integrated version. They should have similar requirements, and having to know whether a gui element is supposed to be a boolean, an integer, a multi-choice option, or an arbitrary string (like the world name in advanced world gen).
And also there should be some token that checks whether a mod argument has a certain value, and allows/restricts other tokens based on that. I'm not sure when this would be done, or how it would work with the other parts of the mod loader described, there's a lot to figure out here.
Now, to finally making those replies using this model:
Also, this is a bit of a pipe dream/unlikely hope, but if they're kept separate/specialized, one could conceivably scrap the CV_NEW_TAG syntax and just directly write in the tags, so instead of this:
[CV_NEW_TAG:CAN_DO_INTERACTION:PET_ANIMAL]
[CV_NEW_TAG:CDI:ADV_NAME:Pet animal]
[CV_NEW_TAG:CDI:USAGE_HINT:GREETING]
[CV_NEW_TAG:CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CV_NEW_TAG:CDI:VERB:pet:pets:pets]
[CV_NEW_TAG:CDI:TARGET:A:SELF_ONLY]
[CV_NEW_TAG:CDI:TARGET:B:TOUCHABLE]
[CV_NEW_TAG:CDI:TARGET_RANGE:B:1]
[CV_NEW_TAG:CDI:MAX_TARGET_NUMBER:B:1]
[CV_NEW_TAG:CDI:WAIT_PERIOD:20]
You'd have this:
[CAN_DO_INTERACTION:PET_ANIMAL]
[CDI:ADV_NAME:Pet animal]
[CDI:USAGE_HINT:GREETING]
[CDI:BP_REQUIRED:BY_TYPE:GRASP]
[CDI:VERB:pet:pets:pets]
[CDI:TARGET:A:SELF_ONLY]
[CDI:TARGET:B:TOUCHABLE]
[CDI:TARGET_RANGE:B:1]
[CDI:MAX_TARGET_NUMBER:B:1]
[CDI:WAIT_PERIOD:20]
Which would just be so much nice and neater, allowing the likes of easily copy/pasting things into an object variation from an actual object (instead of having to add in CV_NEW_TAG one by one).
If we allow EDIT objects to include disguised OV_ADD_TAGs, it could also be done with the object variations; when reading an ov object in step 1,
replace all non-ov tokens with OV_ADD_TAG:whatever-token-that-was.
They don't need to be separate/specialized for this though; ov objects will never include any non-ov tokens so its obvious that any token that appears that way must
be a disguised OV_ADD_TAG (or OV_NEW_TAG, the synonymous tokens still weird me out tbh).
No, I mean, what's the difference between the pairs of 1&2, and 3&4. I might be missing something really obvious here, in which case I'm sorry.
1&2 is for CREATURE_VARIATION objects, and 3&4 is for CREATURE objects (because every single CREATURE_VARIATION needs to be edited by all mods, before being applied to even vanilla creatures).
This is not needed in the above model, because APPLY_OBJECT_VARIATION is a special token processed in step 2. All edits to object variations and objects such as creatures alike
are done in step 1.
My worry is that the mods need to be applied in order; a top-level token needs to be able to reliably re-add a token that was removed, without knowing necessarily that it was removed (rather than just not present to begin with). I'm worried kind of that something like GO_TO_TAG could be used in ways that would place a token like say, PET_VALUE to somewhere high up in the object, and another mod places REMOVE:PET_VALUE at the end, such that no matter what load order the mods are placed in, you would get the tag deleted, which is bad....
.... And now mentioning that has brought to mind/gotten me worried about how REMOVE_SPEC_TAG and adding in tags again would work with the 3 `GO_TO` tokens, since they're not meant to be processed until step 6.
This is why the EDIT-level tokens can only make changes to special tokens, and why the edit objects add OV_ADD_TAGs instead of directly adding tokens.
During step 1 (i.e. when edits are being made), all edits done to normal tokens remain as unprocessed ov tokens in the in-memory objects, and can thus be reverted.
As the GO_TO tokens are special-only and have no EDIT-level counterparts, edits can't place ov tokens anywhere but the end of the selected object(s),
and so they will be processed in step 2 in the same order as the edits were parsed, top-down in the list of selected mods.
Say we have the raws of the TOAD, and a mod A:
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
and mod B:
[CREATURE:EDIT:BY_TOKEN:TOAD]
[GO_TO_START]
[PET_VALUE:50]
loaded in that order. After step 1, we would have:
[CREATURE:TOAD]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
[PET_VALUE:10]
...
[GO_TO_END]
[OV_REMOVE_TAG:PET_VALUE]
[APPLY_CURRENT_OBJECT_VARIATION]
[GO_TO_START]
[OV_ADD_TAG:PET_VALUE:50]
[APPLY_CURRENT_OBJECT_VARIATION]
. In step 2, this would be processed line-by-line, and as we can see REMOVE_TOKEN would be processed/applied first. As REMOVE_TOKEN does not add a token,
it is unaffected by the change of insertion index made by GO_TO_END. GO_TO_START still affects where PET_VALUE is inserted. Ultimately, we would get the desired result:
[CREATURE:TOAD]
[PET_VALUE:50]
...
[CREATURE_TILE:249]
[COLOR:2:0:0]
...